Rao was born in New York City to Charles and Francis Rao. Joseph had two brothers, Vincent and Louis Rao. He was the cousin of Lucchese crime family consigliere Vincenzo Rao. Rao was married to Lena Stracci; they had a daughter Francis Rao and two sons, Robert and Charles Rao. Rao would tell people that he was a house painter. He stood 5'7" and weighed 190 pounds with brown eyes and black curly hair that he slicked back on his head. His home was in East Harlem.