Gardner was born in Okolona, Mississippi and grew up in Austin, Texas. After graduating from the University of Texas at Austin, he earned a Master's Degree in German from the University of Wisconsin–Madison, where he was a teaching assistant from 1938 to 1940. He was a linguist and professor of German at the University of Akron when the United States Army's Signals Intelligence Service recruited him to work on breaking German codes. Soon after, he started working on the Japanese codes instead, mastering the Japanese language in only a few months.