A motte and bailey castle was built by Roland, Lord of Galloway, in the 12th century, half a mile upstream from the later Norman castle. The castle passed by marriage to John de Balliol through the heiress Dervorguilla of Galloway, who built the Norman castle. Robert de Brus, 5th Lord of Annandale, captured the castle in 1286. The castle was in English hands in 1296. King Robert the Bruce captured the castle in 1313 and it was given to Sir James Douglas, Lord of Douglas. The castle came into Edward Balliol's hands in 1332, before being given to Archibald the Grim, 3rd Earl of Douglas, in 1372. The castle remained in the hands of the Douglasses until 1456, when the castle reverted to the Crown. The castle was later in the hands of the Maxwells and later the Gordons. The castle was slighted