Sargent was born in Gloucester, Massachusetts and graduated from Harvard College before the Revolution. He spent some time at sea, as captain of a merchantman owned by his father. He enlisted in Gridley's Regiment of Massachusetts Artillery on July 7, 1775 as a lieutenant, and later that year was promoted to captain lieutenant of Knox's Regiment, Continental Artillery, on December 10. He was with his guns at the siege of Boston, as well as the battles of Long Island, White Plains, Trenton, Brandywine, Germantown, and Monmouth. He was promoted to captain in the 3rd Continental Artillery on January 1, 1777, and brevetted major on August 25, 1783.