Rossi was born and raised on Long Island, New York, in the town of Commack. As a child, he was friends with a young Emma Taylor, who he refers to as the "one who got away." He was also close with Ray Finnegan, who eventually grew up to be a prominent local mobster. Rossi, however, avoided the lure of organized crime, and joined the Marine Corps, rising to the rank of Sergeant Major. Despite growing up on Long Island, Rossi is apparently a Chicago Cubs fan, judging by the Cubs gear in his Quantico office.