The eldest son of Samuel Ballance, a tenant farmer, and Mary McNiece (a quaker), Ballance was born on 27 March 1839 in Glenavy in County Antrim in Ireland. He was educated at a national school, then apprenticed to an ironmonger in Belfast. He later became a clerk in a wholesale ironmonger's house in Birmingham, where he married. Ballance was highly interested in literature, and was known for spending vast amounts of time reading books. He also became interested in politics, mostly due to the influence of his parents – his father was active in conservative circles, while his mother was a liberal. It was from his mother that Ballance gained many of the ideas he was later to promote. Having witnessed religious rioting when in Belfast, he became committed to the principle of secularism.