A wooden castle was built by Geoffrey de Mowbray, Bishop of Coutances in the 11th century, clearing houses to make room for it. Juhel of Totnes (Judhael) later lived in the castle and established a priory just outside its walls. The castle's first stone buildings were probably erected by Henry de Tracey, a strong supporter of King Stephen. In 1228, the Sheriff of Devon had the walls of the castle reduced to a height of . By the time of the death of the last Henry de Tracey in 1274, the castle was beginning to decay. The fabric of the castle was used in the construction of other buildings and by 1326 the castle was a ruin. The remaining walls blew down in a storm in 1601. Now only the tree covered motte remains.