A succession of buildings has been on the site. The first was a Norman castle built by the Bulmers, which was rebuilt by the Nevilles in the late 14th century. For many years the castle was owned by the Neville family until in 1569 it was confiscated by the Crown following the family's involvement in the Rising of the North. The castle is now privately owned by the Dobson family, the castle having been purchased by Margaret Dobson, wife of publisher Dennis Dobson, in 1978.