Peltier's mother was the former May Ayo. He had a sister, Bernice P. Harang, and three brothers, Donald Louis Peltier, Richard Benton Peltier, and Dr. James R. Peltier Sr., a Thibodaux oral surgeon, a founder and president of the Louisiana Society of Oral Surgeons, and member of the "good government" groups the Public Affairs Research Council and the Council for a Better Louisiana.[8] Peltier's brother-in-law, Warren Harang Jr., was a former president of the Thibodaux Chamber of Commerce and the American Sugar Cane League, a member of the Lafourche Parish School Board, and the mayor of Thibodaux from 1968 to 1978, 1986–1990, and 1994–1998.[9]
In 1945, Peltier married Irma Mary Geheeb (1924-2014), the third daughter of Albert John and Cleo Belou Geheeb. Known as "Mickey", she graduated from the former Ursuline College, now Ursuline Academy, in her native New Orleans. The Peltiers lost an infant son in 1952 and have three surviving children, Patricia P. Crum, Harvey "Drew" Peltier III, and wife Linda, and Mary Ellen Peltier.[10] Peltier's son-in-law, John Mitchell Crum (1945-2012), was a district attorney of the 40th Judicial District of St. John the Baptist Parish.[11]