Miller was born in Ithaca, New York, but moved to Louisiana with his family while still a young child. He attended Louisiana College. He was a veteran of the Mexican-American War, serving in the army of Zachary Taylor and was awarded by the government for his service with of land in Florida. He studied law, passed his bar exam, and engaged in a private practice in Santa Rosa County. He was engaged in the timber and lumber business in northwest Florida prior to the Civil War. Miller and his men finally surrendered in Tallahassee in May 1865.