Fisher, fourth of seven children, was born in Philadelphia, Pennsylvania, the son of Russian born Jewish immigrants Kate (née Winokur) and Joseph Fisher. His father's surname was originally Tisch or Fisch, but was anglicised to Fisher upon entry into the United States. To his family, Fisher was always called "Sonny Boy", a nickname derived from the song of the same name in Al Jolson's film The Singing Fool (1928).