It was built as a circular blockhouse with a three storey central keep in 1540 using stone from Beaulieu Abbey. The outer walls were lowered in 1774 and the gatehouse was rebuilt in order to provide more living space. The poet Caroline Anne Bowles (1786–1854) spent childhood summers there when it was owned by a military uncle, Sir Harry Burrard. The south east battery was added in 1895 but has since been demolished. The castle was in use until 1956. It is now owned by English Heritage and visitors to the castle can park in the adjacent leisure centre car park.