Hormel married four times. He married his first wife, movie actress Leslie Caron, on September 23, 1951. The couple divorced on April 26, 1955. His second marriage was to Kim Wadsworth on March 23, 1957. The couple had two children, and the marriage ended in divorce in 1961. In 1968, Hormel married Nancy B. Friedman, with whom he had one child; they divorced in 1973. Hormel married his fourth and final wife, Jamie Renee Vincent, on June 2, 1992. The couple had two children, and were married until his death on February 12, 2006.[6]
His daughter Geri was featured on the MTV show My Super Sweet 16 in December 2007.
Hormel has two brothers, American composer and environmental philanthropist, Thomas Hormel and Ambassador James Hormel.