It was built originally as a Georgian in 1771 by Hans Smith, one of the early German settlers of the town. Twenty years later, Miller, a prosperous local businessman, acquired it and rebuilt it in the Federal style. In 1835, his descendants added some Greek Revival elements.