It was built by the bishops of the Diocese of St David between the 12th century and 14th century. The first castle on the site was constructed in 1115 by Bishop Bernard. Only the moat and the earth bank from this period survive. The vast majority of the ruins seen today date from the construction commissioned by Bishop Adam de Houghton between 1362 and 1389. This was much grander, with two suites of residences situated on the first floor. The gate house shown in the picture was probably added at a later date.