The origins of the castle date back to the 12th century. The task of the castle was to secure and safeguard the important crossing over the river Danube. The keep was built during the reign of the House of Hohenstaufen and explicitly mentioned in a document from 1384. The Gothic spire was added at the end of the 15th century. The original modest Romanesque castle was extended in the 16th century into a renaissance palace. In 1525 during the German Peasants' War the castle was damaged by revolting farmers of the Baltringer Haufen.