Castleton played in 19 games as a true freshman and averaged 1.1 points and 1.1 rebounds per game.[4] As a sophomore, he averaged 3.1 points on 54% shooting and 2.4 rebounds over 25 games played, all off the bench.[5] Following the end of the season, Castleton entered the transfer portal and ultimately transferred to the University of Florida, which had offered him a scholarship coming out of high school.[6][7]
Castleton was granted a waiver to make him eligible to play for the Florida Gators immediately rather than have to sit out one season per NCAA transfer rules.[8] During the season, he became the seventh Gator (following Vernon Maxwell, Dan Cross, Joakim Noah, Scottie Wilbekin, Michael Frazier II (3) and Jalen Hudson) to ever earn Southeastern Conference (SEC) player of the week honors at least twice in the same season.[9] He was named second-team All-SEC after averaging 12.4 points and 6.4 rebounds with a conference-high 2.3 blocks per game during the regular season.[10][11] Following the season, Castleton declared for the 2021 NBA draft while maintaining his college eligibility.[12] He ultimately opted to withdraw from the draft and return to Florida.[13]
On November 14, 2021, Castleton recorded 15 points, a career-high 16 rebounds, and six blocks in a 71–55 win against Florida State.[14] That effort contributed to his third SEC Player of the Week honor.[15] He was named to the Second Team All-SEC as a senior.[16]
On January 16, 2023, Castleton was recognized with his fourth career SEC Player of the Week award, making him the second Gator to achieve four, one behind Vernon Maxwell in Gator history. The effort partly recognizes the rare stat line of 16 points, 13 rebounds, six assists, three steals, and two blocked shots against Missouri on January 14.[17][18] On February 15, 2023, Castleton broke his hand in a 79–64 win against Ole Miss and was lost for the season. Before the injury, he was averaging 16.5 points (third in the SEC), 7.9 rebounds (sixth) and 3.0 blocks (first) per game. He had been on a hot streak prior to the injury, with averages of 24.8 points and 9.5 rebounds per game over his last four games.[19] The coaches recognized him as a first-team All-SEC selectee.[20]