Rastelli was born and raised in Maspeth, Queens in 1918. He had three brothers (Carmine, Marinello, and Augustus) and two sisters (Justina Devita and Grace Iacomini). Rastelli was married to Connie Rastelli. Rastelli was heavily involved in loansharking, extortion and drug trafficking activities before joining the Bonanno crime family. Rastelli also had a lunch wagon business. After moving to Greenpoint, Brooklyn where he lived until his incarceration, he met and became close friends with Dominick Napolitano, Carmine Galante, Joseph "Bananas" Bonanno and Joseph Massino.