Robert D. Lilley was born near Greenville, Virginia to a military family. His father, James M. Lilley (1802–1875), a prominent landowner and patented inventor of surveying instruments, was commissioned in 1838 as a colonel in the Virginia militia. Lilley studied at Washington College before beginning a career selling surveying instruments invented by his father.