Lusk was born and raised in Norwich, Connecticut. Lusk, the fifth generation of his family, was the son of Sylvester Graham Lusk and Elizabeth Freeman Lusk (née Adams). Lusk attended a school run by local Reverend Albert Spooner in preparation for attending Yale College. However, his uncle believed that learning the old version of grammar from Spooner would get Lusk rejected from Yale, and for the winter of 1853–1854, he attended Anthon's Grammar School at Murray Street, New York City. For the winter of 1854–1855, Lusk was sent by his mother to Russell's Military School in New Haven, Connecticut to gain physical toughness through gymnastics.