Under Minamoto no Yoritomo, Doi Sanehira was awarded for his achievements in the Genpei War against the Heike clan, and was appointed governor of the three provinces of Bizen, Bitchu and Bingo. Sanehira's son Enpei took the name of "Kobayakawa", which was the place name of his residence, but was subsequently awarded the Numata shōen in Aki Province for his distinguished service in the Jōkyū War and relocated to Aki. His grandson, Kobayakawa Shigehira completed Takayama Castle in 1206 as the clan's stronghold. The Nuta Kobayakawa clan was the main line and ruled from Takayama Castle, and the Takehara Kobayakawa clan was a cadet branch.[2]
From the middle of the 16th century, the Ouchi clan, which ruled Suo and Nagato Provinces became the strongest warlord in the western half of Japan, supported by its monopoly of foreign trade. The Takahara-Kobayakawa gained in power through an alliance with the Ouchi, whereas the Nuta-Kobayakawa was rent by internal conflicts. Takayama Castle was on the front line of the conflict between the Ouchi clan and their aggressive neighbors to the north, the Amago clan. In defending against the Amago, the Nuta Kobayakawa clan lost three consecutive leaders in youth by illness or battle. In 1540, when the Amago again invaded Aki, the Takehara-Kobayakawa switched sides, but the Amago were defeated. The Ouchi clan seized Takayama Castle and much of the Kobayakawa territory. Around this time, the inlet to the Seto Inland Sea became silted up, adding to the clan's financial woes.
In 1541, the head of the Takehara-Kobayakawa branch (Kobayakawa Okikage) died without an heir, so the clan adopted the third son of Mōri Motonari as Kobayakawa Takakage in 1544. On the other hand, the head of the Nuta-Kobayakawa clan (Kobayakawa Shigehira) was young and in such poor health that he became blind, so he was forced into retirement, and Kobayakawa Takakage united the two branches in 1550. In 1552, he relocated his stronghold to a new castle, Niitakayama Castle on the opposite side of the Nuta River, and the Takayama Castle was abandoned.[2]
Takayama Castle was built on Mount Tsumataka, a 150-meter height overlooking the Nuta River, about 10 kilometers west of current center of Mihara city. On the opposite of the river is Mount Niitaka of similar height, and these two mountains stand like the pillars of gate. Originally there was an inlet with a length of over 10 kilometers from current river mouth on the Seto Inland Sea to just the below of the castle. This was an important port for the clan, which was also a maritime power on the Seto Inland Sea. Mount Tsumataka has two lines of ridges extending north and south. The northern ridge is the core part of the castle, which consisted of several enclosures protected by stone walls and dry moats. The total size of the castle was about 600 meter long and 400 meter wide.
At present, no structures of the castle survive. The site is a 20-minute walk from JR WestHongō Station.[2]