James was born in Athens, Georgia. He is the son of James F., and Susan E. Ponsoldt, and grandson of graphic artist William Teason. His father is a retired law professor, formerly at the University of Georgia, and he has one sister, Katherine. He grew up in Athens, and attended Cedar Shoals High School. He received his bachelor's degree from Yale University and graduated from Columbia University with a MFA in directing. James married Megan Holloway in September 2010.