Eli Porter Ashmun was the eldest child of Justus and Kezia Ashmun. He was born in the vicinity of Fort Edward, New York, from whence the family fled in 1777 during the advance of British troops of John Burgoyne in the Saratoga campaign of the American Revolutionary War. They settled in Blandford, Massachusetts, where Ashmun's father operated the tavern. Ashmun's education was quite rudimentary, but he was taken under wing by Theodore Sedgwick, who gave him legal training. He was eventually admitted to the bar, and opened the first legal practice in Blandford. He married Lucy Hooker, daughter of John Hooker of Northampton, with whom he had five children.
In 1807, Ashmun was awarded an honorary degree by Middlebury College, and moved to Northampton where he continued his law practice.
Ashmun was the father of George Ashmun (1804–1870), who served in the U.S. House of Representatives, and of John Hooker Ashmun. The latter served as a partner in his father's law practice, and operated the Northampton Law School for several years. He then became a prominent legal instructor at Harvard Law School.
