It is likely that the first castle was built after the Norman invasion. Then, in the 12th or 13th century, it was rebuilt in stone. The owner, John Giffard, 2nd Lord of Brimpsfield, rebelled against King Edward II and was executed in 1322. The castle was then destroyed. Today the remains consist of a mound with an outer bank and ditch and the foundations of a gatehouse.