Ellerbee was born Linda Jane Smith in Bryan, Texas. She attended River Oaks Elementary School, Lanier Middle School, and Lamar High School in Houston. She also attended Vanderbilt University in Nashville, Tennessee, although she dropped out in 1964. Ellerbee traveled around the country for some time afterward, working itinerant jobs in radio. In her own words: