In 1734, James married Grace Lockhart (1706–1738), granddaughter of the 9th Earl of Eglington and widow of 3rd Earl of Aboyne. Before her death in 1738, they had two children, Francis, (1737–1810), who succeeded as Earl of Moray, and Euphemia (1738–1771). He married again in 1740, this time to Margaret Wemyss, eldest daughter of the Earl of Wemyss; they had two sons, Lt-Colonel James Stuart (1741–1809), [a] and Lieutenant (RN) David Stuart (1745–1784).[b][1]
In 1755 he purchased Balmerino House in Leith from the Crown who had confiscated the house due to Lord Balmerino's active support of the Jacobite Rebellion.[2]