According to the Social Security Death Index, Zwillman was born on July 27, 1904, in Newark, New Jersey. He was forced to quit school in order to support his family after his father's death in 1918. Zwillman first began working at a Prince Street cafe, the headquarters of a local alderman in Newark's Third Ward. However, in need of more money, Zwillman was eventually forced to quit, later selling fruits and vegetables in his neighborhood with a rented horse and wagon.