Ballard grew up in Pacific Beach, San Diego, California to a mother of German heritage and a father of British heritage. He has attributed his early interest in underwater exploration to reading the novel Twenty Thousand Leagues Under the Sea, living by the ocean in San Diego, and his fascination with the groundbreaking expeditions of the bathyscaphe Trieste.