Accardi was born in Vita, Italy, in the Sicilian province of Trapani. He emigrated to the U.S. shortly before World War I, and soon became involved in gas-stamp black marketeering with Future Gambino crime family Bosses Paul Castellano and Carlo Gambino. During Prohibition, Accardi served as a gang enforcer both in New York and North Jersey. He later founded his own criminal organization with mobsters Joseph Sica, Willie Moretti, Joe Adonis and Abner Zwillman. Accardi was suspected of several gangland killings, but was never charged for them.