Pryor was the only son of Marlborough Robert Pryor, DL, JP, Weston Park, Hitchin, Hertfordshire and his wife. He was educated at Eton and Trinity College, Cambridge where he gained a BA (1901), and an MA (1905).[3] In 1910 he married the writer Ethne Philippa, only daughter of Sir Norman Moore, 1st Baronet. They had three sons; John Marlborough Pryor (1911-1984), Mark Gillachrist Marlborough Pryor (1919-1970) and Lieutenant Colonel Robert Matthew Marlborough Pryor MBE (known as Matthew; 1917-2005). His grandson by Matthew is the archaeologist Francis Pryor; by Mark his grandchildren include the author William Pryor and the artist Lucy Raverat.