Bayer was the first of two children born to Major General Stephan Bayer (1816–1893) and his second wife, Julie Henoch (1839–1888). During his early life the family moved often between Italy and Germany, living in Pisa, Florence, Bagni di Lucca, Viareggio, Baden-Baden and Gotha. While living in Baden-Baden, the nine year old Bayer suffered from a severe case of diphtheria of the eye. The family moved to Italy again between 1883 and 1886, and it was during this time living in Capri and Venice that he became familiar with Italian language and culture.