The original castle was built by George Gordon, 2nd Earl of Huntly in the 1470s and enlarged by his grandson. John Adam (architect) was commissioned with the French architect Abraham Roumieu to redesign the castle in 1764 but this did not come to fruition. Eventually the commission fell to the lesser-known Edinburgh architect, John Baxter, who rebuilt it in 1769 for Alexander Gordon, 4th Duke of Gordon. The central four storey block incorporated a six-storey medieval tower called the Bog-of-Gight, and was flanked by a pair of two-storey wings. The main facade was 568 feet (173 metres) long. Most of the castle has been demolished, but the Bog-of-Gight and one of the wings—now a detached medium sized country house in its own right—survive.