Daugherty was born in Emeigh, Pennsylvania on September 8, 1915. Though Daugherty would later become known as "the Irish pixie, short and stocky, a man of endearing charm, with smiles and jokes," both of his parents were Pennslylvania natives whose parents were immigrants from Scotland. His father, Joseph Daugherty, was the manager of a general merchandise store at Susquehanna in 1920. By 1930, the family had moved to Barnesboro, Pennsylvania, where Daugherty's father was working as an adjuster for a compensation and insurance company. Daugherty had two older brothers, John and Joseph, Jr., and a younger sister Jean.