Rafferty was born at Pittsburgh, Pennsylvania in August 1879. He was the son of Gilbert Thomas Rafferty and Harriett Martin Oliver. He attended preparatory school at Andover. He played two years at the end position for Andover's football team.