Brown was born on November 28, 1879 in Joliet, Illinois. He attended high school in Elgin, Illinois, graduating in 1898. He played football and baseball in high school and also competed for the Elgin Athletic Club football team in 1898 and 1899. In 1900, he enrolled at Dartmouth College where he played football for the Dartmouth Big Green football team as a halfback and fullback. He attended Dartmouth from 1900 to 1903.