The oldest, ruinous construction section of the castle - seen all the way in the front of the mountains's top - was built during the last quarter of the 12th century. Its old bridge arch, connecting both bulwarks, can still be detected among the constructional elements. After 1188, a ministerial lineage assumed the name of the castle.
In 1460, the principality war between Margrave Albrecht Achilles von Brandenburg and Duke Ludwig von Baiern-Landshut destroyed castle Rabenstein. According to the 1495 chronogram at the exterior gate, the castle was again reconstructed after six years. In 1570, Daniel von Rabenstein had the castle significantly reconstructed adding a three-wing Renaissance structure and a round tower whose ruins werde standing until 1829. Its rounded foundations were detectable until 1974 in the basement underneath the castle garden. Another tower lead to the rooms of the so-called Rittersaal. Today, the coat of arms of Daniel von Rabenstein and his wife, Margarete von Kerppen, are still decorating the castle's entrance.