Built during the tenth century by the Banu Hud dynasty of the Taifa of Zaragoza, Aragonese forces captured the castle in 1089 when it was conquered by Sancho Ramírez. In 1143 the castle passed to the Templars who added walls, towers, stables, a dining hall and dormitories. King James I of Aragon lived in the fort under Templar protection during his childhood .