Ruffo grew up in Tacoma, Washington. In 1927 he moved to San Jose, California to attend nearby Santa Clara University, where he played football, and graduated with degrees in political science, electrical engineering, and literature. After graduating in 1931, Ruffo taught in the university's school of engineering and coached the freshman football team to help pay for attending law school. In 1936 he graduated at the head of his class from Santa Clara University School of Law. After passing the bar exam, he continued to coach football with the university as the assistant varsity coach under legendary SCU head coach Lawrence T. "Buck" Shaw and helped lead the team to victory in the 1937 and 1938 Sugar Bowls against LSU.