On the site of Wittelsbach Castle stood the castle of Kelheim, which is first mentioned in 1050 as being in the possession of the Wittelsbachs. According to tradition, the later Duke Ludwig the Kelheimer is said to have been born here in 1174. Kelheim was the center of his rule. After his murder, the Wittelsbachs moved their residence to Landshut. In 1476, the old castle was demolished. The two round towers and the Duke's Granary were built from its material. Inside, the massive ashlar stones, remains of the keep, can still be seen.