The place selected for Chihaya Castle was very defensible: a narrow ridge with an elevation of 150 meters with steep cliffs, protected on both sides by rivers. The only access was by a climb of 500 steep and narrow stairs and the surrounding topography was not conductive to siege operations. Moveable bridges were among the fortress' chief defensive measures, alongside its wooden walls, earthwork defenses, and strategic location on Mt. Kongō. The fortress was surrounded with felled trees and boulders, which could be rolled down the mountain on an approaching army, and screens of brush were used to help protect from arrows. The castle itself consisted of five enclosures spread 200 meters along the ridge, protected by dry moats. The main enclosure measured around 100 by 20 meters and may have had a yagura tower in one corner. Although the castle was small, it could house several hundred men.[2]
In 1333, Kusunoki Masashige emerged from hiding, attacking shogunate positions in Kii province, Osaka and then the Rokuhara tandai offices in Kyoto, The Kamakura shogunate sent a large army against him, which took Kami-Akasaka Castle after a siege of two weeks and then attacked Chihaya Castle. Per the medieval chronicle Taiheiki, the greatly outnumbered defenders put up a strong defense, forcing the attackers to abandon direct assault and to resort to a siege. However, the castle had a well, and could also receive supplies carried by yamabushi supporting Prince Moriyoshi, who was on the other side of Mount Kongō, shogunal forces were unable to cut the castle's supply lines. Moreover, Kusunoki Masashige used guerrilla warfare tactics, making continual raids and surprise attacks, or using straw dummies to trick the shogunal army. As the siege continued, the shogunal army, trapped in the narrow valley, began to experience supply problems, along with a considerable loss in prestige in being unable to defeat their numerically much smaller enemy.
While this battle was occurring, Emperor Go-Daigo managed to escape from Oki and raised a new army. The Kamakura shogunate dispatched another large army under the command of Ashikaga Takauji; however, Takauji defected to the imperial side and soon afterwards, at the Battle of Kamakura, the shogunate was overthrown. After the Kenmu restoration, Chihaya Castle became the residence of Kusunoki Masashige, and his sons Kusunoki Masatsura, Kusunoki Masanori and Kusunoki Masatoki. However, in 1392, the castle was attacked by Hatakeyama Motokuni of the Northern Court, who defeated Kusunoki Masanori's son, Kusunoki Masakatsu, and the castle was abandoned.[2]
During the Meiji period, a Shinto shrine to Kusunoki Masashige was erected on the site of the Ni-no-maru in castle ruins, with the shrine offices in the San-no-maru.[2]