Packer was born in Pennsylvania in 1842 to the shoe-making family of James Packer and his wife Esther Griner (or Crider). He went off to fight on the Union's side in the American Civil War as quickly as he could to avoid making shoes. He was discharged the following December because he suffered from epilepsy. He then enlisted in an Iowa Cavalry Regiment but was discharged again after an epileptic seizure. He decided he'd go West and try to find some gold.