He inherited the title Baron Willoughby de Broke and Baron Latimer on the death of his father in 1728, his elder brother Thomas having died of smallpox in 1710. Two other elder brothers died young.[2]
He married Penelope, daughter of Clifton Packe of Prestwold Hall, Leicestershire (died 1730). He married secondly Elizabeth Williams, daughter of Nathaniel Williams.[2]
His only son died in infancy. Upon his death, on 11 August 1752, the title passed to his nephew John Peyto-Verney, who was the son of his younger brother Sir John Verney.[1]