The castle was built and named by Albrecht von Perg (1120–1168), from the family of the Lords of Perg and Machland, who was married to a Countess of Schalla and brought property in the Pielachtal into the marriage. Albrecht had bailiwick rights over the monastery of Sankt Pölten and was sub-bailiff of the Melk Abbey. Albrecht von Perg lived in the castle with his wife and his two sons, Siegfried and Meingozus von Perg und Albrechtsberg. The later history up to 1400 is closely connected with the area around Kilb, where the Lords of Perg also had properties.[2]
After the Perger family died out, half of the castle passed directly to the Babenbergs, and the other half to the County of Plain or its successor, the County of Schaunberg.[2]
The castle was built in 1581 in the Renaissance style over a medieval castle. In 1606, Ludwig von Starhemberg became the new owner of Albrechtsberg Castle. As he took part in the Protestant aristocratic uprising, his castle was set on fire by imperial soldiers during the fighting. The castle was expropriated and the chapel was re-consecrated to the Catholic Church. In 1704, another fire destroyed the chapel tower.[2]
At the end of the 19th century, extensive renovation work was carried out under the Auersperg family. Between 1925 and 1936, Karl Anton Rohan published the European Review in the castle. In 1945 at the end of World War II, the castle was partially destroyed. Since 1995, it has been owned by the Weinberger family and is being gradually renovated.[3]