James was born in 1745 to Thomas Tilton, a farmer in Kent County, Delaware. After attending an academy in West Nottingham, Pennsylvania, he attended the College of Philadelphia (which was later the University of Pennsylvania). He graduated in 1768 and earned a degree as a Doctor of Medicine in 1771. He started a medical practice at Dover, and also served as an infantry Lieutenant in Dover's Company of the Kent County militia.