Simon was born in 1726, and his Clan was a prosperous one. His father, Simon "the Fox" Fraser, Lord Lovat, had recently been pardoned for his role in supporting the Jacobites during The Fifteen. He grew up learning, and became fluent in, Gaelic, French, and English, and probably Latin as well. At the Battle of Culloden, Simon, along with Charles Fraser of Inverallochy, commanded several hundred Frasers, who were front and centre in the Jacobite lines. General Hawley reportedly found Charles lying on the field, and ordered a young James Wolfe to shoot him. Wolfe refusing, Hawley found another common soldier to do his dirty work. Simon, on the other hand, escaped the field complete with life and limb, and went on to serve in the British Army.