He was the son of Charles Agar, an Englishman who acquired lands in County Kilkenny, including Gowran Castle. His mother was Ellis Blanchville, daughter of Peter Blanchville of Kilkenny.
He married firstly Susannah Alexander, and secondly Mary Wemyss, daughter of Sir Henry Wemyss of Danesfort. By his second wife, he had at least four children, including Henry Agar, James Agar, and Ellis (Elizabeth) Bermingham, Countess of Brandon in her own right. Both Henry and James, after Henry's death, sat in the Irish House of Commons for Gowran. James fought a long and bitter battle to retain control of the borough of Callan, which eventually led to his death in a duel.
Agar's daughter Elizabeth or Ellis, Countess of Brandon in her own right
