Withers redshirted his first season at Louisville to focus on gaining strength and weight.[6] On November 29, 2020, he recorded a freshman season-high 20 points and nine rebounds in an 86–64 win over Prairie View A&M.[7] As a freshman, Withers averaged 10.1 points and 7.7 rebounds per game, earning Atlantic Coast Conference (ACC) All-Freshman Team honors.[8]
He started 15 out of 29 games his sophomore season, averaging 5.8 points per game and was second on the team in rebounding with 4.6 per game. In his junior year, he started 29 out of 32 games on the season and averaged 8.9 points and 5.3 rebounds per game. He led the Cardinals with a 41.7% three-point percentage.[8]
Following the season, he entered his name into the NCAA's transfer portal.[9] Withers would ultimately choose to return to his home state and decided to transfer to North Carolina.[10]
Withers was born when his father, Curtis, was 16 years old and a high school sophomore. Curtis played college basketball at Charlotte, where he was an all-conference player, before embarking on a professional career in the NBA Development League and overseas.[11]