Under Minamoto no Yoritomo, Doi Sanehira was awarded for his achievements in the Genpei War against the Heike clan, and was appointed governor of the three provinces of Bizen, Bitchu and Bingo. Sanehira's son Enpei took the name of "Kobayakawa", which was the place name of his residence, but was subsequently awarded the Nuta shōen in Aki Province for his distinguished service in the Jōkyū War and relocated to Aki. His grandson, Kobayakawa Shigehira completed Takayama Castle in 1206 as the clan's stronghold. Takayama Castle was built on Mount Tsumataka, a 150 meter height overlooking the Nuta River, about 10 kilometers west of current center of Mihara city. On the opposite of the river is Mount Niitaka, with an elevation of 197.6 meters, and these two mountains stand like the pillars of gate. In 1276, Kobayakawa Masahei built a secondary fortification on the site of the current Niitakayama. In 1552, Kobayakawa Takakage relocated his seat from Takayama Castle to Niitakayama Castle and rebuilt the fort into a castle.[2][3] The new castle extended for 400 meters from west-to-west and was protected by the Nuta River to the north and east. It consisted of several enclosures protected by stone walls and dry moats. The site was the original location of the Kobayakawa clan bodaiji, Kyoshi-ji, which dated from the Kamakura period. The temple was removed to make room for the castle.
In 1561, his father Mōri Motonari visited Niitakayama and stayed for several days.[4] Over the next 20 years, the Kobayakawa clan helped the Mōri clan defeat the Ouchi clan and the Amago clan to secure supremacy over the Chugoku region of Japan. The Mōri entrusted the Kobayakawa with the San'yo region, whereas Kobayakawa Takakage's elder brother Kikkawa Motoharu was entrusted with the San'in region. The Mōri and Kobayakawa eventually came into conflict with Oda Nobunaga, and subsequently with Toyotomi Hideyoshi. On Nobunaga's assassination in 1582, The Kobayakawa reached an accord with Hideyoshi, and Kobayakawa eventually became a vassal of Hideyoshi, who needed the Kobayakawa navy in the Seto Inland Sea. Kobayakawa Takakage was awarded Iyo Province in Shikoku and later the post of magistrate of Hakata in Kyushu. Together with Kuroda Yoshitaka (Kanbe), he was one of Hideyoshi's most trusted military advisors.
Niitakayama Castle remained the seat of the Kobayakawa clan until 1596, when Mihara Castle was completed. During the construction of Mihara Castle, the stone walls and structures of Niitakayama Castle were gradually dismantled as building materials for the new castle. At present, all that remains of the castle are some remnants of stone walls. The castle site is a 15-minute walk from JR WestHongō Station.