Fagg attended Davidson, where he was captain of the football and wrestling teams. He later earned a master's degree from Boston University. Fagg began his coaching career in 1962 as head wrestling coach and assistant football coach at Garinger High School in Charlotte, North Carolina. In 1964, he moved on to The Citadel in Charleston, South Carolina to serve in the same two roles for a year. Fagg returned to Davidson in 1965 when he was hired as an assistant on the staff of Homer Smith, who was newly hired as head football coach.[3]