The castle was founded by Ranulf le Meschin at the beginning of the 12th century. In about 1170 the square stone keep known as Caesar's Tower was built. The castle was in royal hands when the Scottish king, William the Lion, invaded the Eden Valley in 1174. The constable of the castle surrendered without a fight. In 1203 the castle was granted to Robert de Vieuxpont by King John. In 1269 it came into the possession of Roger de Clifford and remained in the ownership of the Clifford family for nearly 400 years. and was responsible for the restoration of the castle.