Brassey was the son of the railway contractor Thomas Brassey, by Maria Harrison, daughter of Joseph Harrison, a forwarding and shipping agent. He was the brother of Henry Brassey and Albert Brassey. He was educated at Rugby and University College, Oxford, and was called to the Bar, Lincoln's Inn, in 1864.