A stone castle was built on the site of an earlier defended position from around 1095 to 1125 by Guy de Balliol. Between 1125 and 1185 his nephew Bernard de Balliol and his son Bernard II extended the building. Vane decided to make Raby his principal residence and Barnard Castle was abandoned and its contents and much of its masonry was removed for the maintenance and improvement of Raby. The castle is in the custody of English Heritage and is open to the public. Of particular interest are the ruins of the 12th-century cylindrical tower and the 14th-century Great Hall and Great Chamber.