The castle was originally associated with Clan Fraser, and was perhaps named for Oliver Fraser, who gave lands to Newbattle Abbey as recorded in its register. The lines of descent from Oliver and his nephew Adam are uncertain, but the Frasers continued to exert power from Oliver Castle with Sir Bernard Fraser and Sir Gilbert Fraser, who held in turn the hereditary office of Sheriff of Tweeddale. A descendant, Sir Simon Fraser of Oliver and Neidpath, Knight Banneret, fought in the Wars of Scottish Independence. Clan Tweedie was a sept of Clan Fraser and by this time the Tweedies had already taken over the Fraser title to Drumelzier. In the 14th century, William Tweedie, a younger son of James and Katherine Tweedie of Drumelzier, obtained the lands of Oliver from the preceptor of Torphichen.