Berry was born in Jackson County, Alabama to Isabella Jane (née Orr) and James McFerrin Berry. The family moved to Arkansas in 1848. Berry attended Berryville Academy in Berryville, Arkansas for one year. The Academy was named after his family. Berry studied law and in 1866 was admitted to the Arkansas bar. In 1865, Berry married E.Q. "Lizzie" Quaile. They had six children: daughters Nellie Frank, Berl, Jennie, and Bessie; and sons Elliott and Frederick Hugh.