Al Hirt was born in New Orleans, Louisiana, the son of a police officer. At the age of six, he was given his first trumpet, which had been purchased at a local pawnshop. He would play in the Junior Police Band with the children of Alcide Nunez, and by the age of 16, Hirt was playing professionally, often with his friend Pete Fountain. During this time, he was hired to play at the local horse racing track, beginning a six-decade connection to the sport.