His younger brother Thomas De Witt Talmage was also a clergyman, and his family, within the Reformed tradition, migrated to North America from the Netherlands. His father's family had emigrated from England, and were the founders of the towns of South Hampton, and East Hampton in New York.