Carter scored 19 points and Mack had 14 for the Scarlet Knights (9-7, 1-2 Big East), who had knocked off then-No. 10 Florida in double overtime on December 29.
Rutgers then fell to South Florida and West Virginia to open its conference schedule, but rebounded Saturday to defeated the defending national champion for the first time in school history.
The Scarlet Knights also snapped a 10-game losing streak against Connecticut.
"They've got to play hard no matter what the circumstance is," said Rutgers head coach Mike Rice. "There is no question that these guys like the big moment and they're not afraid of the big moment."
Shabazz Napier netted 23 points for the Huskies (12-3, 2-2), who were coming off Tuesday's setback at Seton Hall and lost in Jim Calhoun's return to the bench. The head coach was serving a three-game suspension levied against him last year by the NCAA for failing to maintain "an atmosphere of compliance" within the program.
UConn held a 30-28 lead after a close first half, but the Huskies didn't score for the first three-plus minutes coming out of the break. That allowed Rutgers to go on an eight-point run and grab the lead. Dane Miller converted a three- point play, Carter drained a three-pointer and Gilvydas Biruta sank a jumper for a 36-30 advantage.
Napier followed with back-to-back threes to tie the game, and the teams traded the lead several times until the Scarlet Knights made a decisive run.
Down by three with 12 minutes to go, they got a jumper from Mike Poole, who drilled a three-pointer a few minutes later, after both teams missed a series of shots. Jerome Seagears added a jumper, and Mack's three capped the run with 7 1/2 minutes to play, making it a 52-45 contest.
"Responding in the second half to a very talented team -- whether it was zone or man-to-man, whatever we did, even offensively, our guys continued to execute," Rice said. "It was a tremendous team performance tonight."
The Huskies countered with the next four points, but Biruta answered with a critical dunk, and UConn couldn't muster enough offense down the stretch to make a comeback.
Following a Jeremy Lamb layup with six minutes left, the Huskies didn't score another basket until Napier made a three-pointer with 2 1/2 minutes to play. By that time, the Scarlet Knights had built a 59-50 advantage, and the hosts made enough free throws down the stretch to secure the win.
Rutgers had been 0-11 when facing the defending national champion. The Scarlet Knights last faced a reigning titlist on December 28, 2009, when they dropped an 81-67 decision against North Carolina...UConn fell to 17-2 against Rutgers in Big East regular season games...Rutgers' last win against UConn came on January 30, 2002, a 61-53 decision over the No. 17 Huskies at the RAC.