The fort was built in 1861 by Confederate army troops under General Simon Bolivar Buckner at the peak of Vinegar Hill and was named Fort Vinegar. It was renamed for William Haines Lytle after his 1863 death leading a cavalry charge in the Battle of Chickamauga. The hill was later called College Hill.