James was born to first generation immigrants from Lombardy, Italy. Growing up in the Little Italy neighborhood of Manhattan, New York, Ida was inducted into the Genovese crime family in the late 1970s. Ida was placed in the Little Italy based crew of captain Matthew Ianniello, frequently serving as Ianniello's bodyguard and chauffeur. Ida's younger brother Joseph also served in Ianniello's crew.