He married Chiyohime (1597–1649) daughter of Ogasawara Hidemasa and adopted daughter of the second Tokugawa shōgun[definition needed], Hidetada. His childhood name was Mitsuchiyo (光千代).
Having studied the Yagyū Shinkage-ryū under Ujii Yashiro, Tadatoshi wanted his guest, Musashi, to fight against the sword master of his fief, and see which style was the strongest. But Ujii, despite his full license in Yagyu Shinkage style, could not strike a single blow against him after numerous bouts. Lord Tadatoshi took over, but he too was powerless against Musashi. He said then about Musashi: "I never imagined there could be such a difference in levels of accomplishment!"