Born Antonino Gaggi to Angelo and Mary Gaggi, Gaggi was the youngest of three children. Gaggi had a sister Marie, and a brother known as "Roy". Angelo emigrated to the United States from Palermo, Sicily and ran a barbershop on the Lower East Side of Manhattan. Mary worked as a seamstress until Gaggi's birth. Gaggi dropped out of school during the eighth grade and followed his father into the barber business. Gaggi also earned extra money delivering flowers, which he used for gambling. It was at this age that Gaggi learned the profitability of loansharking to gamblers.