Ellet was born on June 1, 1843 in Philadelphia, the son of the noted civil engineer Charles Ellet, Jr. He was studying medicine at Georgetown University when the Civil War began. He served as an Army Assistant during 1861-62. In the spring of 1862, when his father established the U.S. Ram Fleet, an Army unit of river steamers converted to rams, Charles Rivers Ellet transferred to that organization. Promoted to the rank of colonel later in the year, he commanded the ram Queen of the West during her daring independent operations below Vicksburg in February 1863.