Born in Ireland, Duffy grew up in Edinburgh, Scotland after moving there with his family as a child. According to his own later account, Duffy participated in cross-country races in Scotland, winning many of them. In 1911 he emigrated to Canada, where he worked in Toronto as a tinsmith and stonecutter. In his spare time he visited the Central YMCA, the director of which quickly recognized his talent.