The castle was originally built in the 11th century in timber, with a square keep added in stone in 1107. The castle was of primarily local significance, being relatively small. During the Anarchy of the 12th century, the castle was seized in the early years of the conflict by Robert, Earl of Gloucester on behalf of the Empress Matilda. In 1142 King Stephen seized the castle in a surprise attack, setting fire to it afterwards.