After returning to Massachusetts, Foster held various positions in the government. He served as justice of the peace for Worcester County from 1781 to 1823, as special justice of the court of common pleas in 1792, and as sheriff of Worcester County in 1792.[3] In 1791, he was elected as a Federalist candidate to the Massachusetts House of Representatives.
He was elected as a United States House of Representatives to the 3rd United States Congress in 1793, and was re-elected to the three succeeding Congresses, defeating Levi Lincoln, Sr. each time. He served in Congress from March 4, 1793, until his resignation on June 6, 1800.[4] While in Congress, he was Chairman of the United States Senate Committee on Claims.
In 1799, he was a delegate to the State constitutional convention[5] and on June 6, 1800, he was elected to the United States Senate to fill the vacancy caused by Samuel Dexter's resignation.[6] He served in the Senate until his resignation on March 3, 1803. He was a member of the State House from 1808 to 1809 and a member of the Massachusetts Governor's Council in 1818.[7]
In 1813 he was elected a member of the American Antiquarian Society.[8]
Foster died in Brookfield on April 29, 1823, aged 65.[9]