Dorfman was born in Detroit, Michigan in 1923 to a working class family and attended Marshall High School in Chicago, Illinois. He enlisted in the US Marines and won a Silver Star at the Battle of Iwo Jima. He attended the University of Illinois and taught physical education there. He was the stepson of Paul "Red" Dorfman, who was head of the Chicago Waste Handler's Union and a kingpin in the Chicago Outfit.