Clifford was born on June 19, 1889 in Chilton, Wisconsin. He was the son of Jeremiah M. and Catherine Connelly Clifford. His father was employed with the Chicago, Milwaukee & St. Paul Railroad and the family lived in Iron Mountain and later Escanaba, Michigan. After graduating from high school, Clifford traveled throughout Europe, then returned to attend the University of Michigan, receiving an LL.B. in 1912. He began his practice of law that same year in Green Bay, Wisconsin, where his father was now regional superintendent.