Branch was born in Enfield, Halifax County, North Carolina to Major Joseph Branch and Susan Simpson O'Bryan Branch. His family moved to Williamson County, Tennessee but his mother died when he was five and his father died in 1827. His uncle, Secretary of the Navy John Branch (then serving as Governor of North Carolina) assumed his guardianship and took him back to North Carolina. Branch moved to Washington City with his uncle when the latter took the position as Secretary of the Navy and he was tutored by Salmon P. Chase. He pursued a preparatory course under a private teacher in Washington, D.C., before going on to train at North Carolina's Bingham Military Academy. He also attended the University of North Carolina at Chapel Hill for a short time and, in 1838, graduated first in his class