Richards was born Emilio Radocchia in Hartford, Connecticut. He began playing the xylophone at the age of six, and while in tenth grade, he joined the Hartford Symphony Orchestra and performed with various jazz musicians around New England. After serving in the First Cavalry Army Band as assistant band leader, he graduated from the Julius Hartt School of Music (now part of the University of Hartford) and later from Hillard College.