It was built as a motte and bailey fortress in the late 11th or early 12th century, founded by Peter de Valoignes. In 1136, Roger de Valoignes built a stone castle with a keep. In 1177, King Henry II ordered that the tower be demolished but in 1192 the castle was in use again. It was finally destroyed in 1212, after Robert Fitzwalter rebelled against King John. Only the foundations of the keep and some earthworks now remain. In 1832, George Proctor added a neo-Norman gatehouse, summerhouse and curtain wall.