Norris was born in the small town of Lewes, Delaware, on March 13, 1944, the third of four children. His father, Ellsworth Q. Norris, worked at a warehouse where toy cars with chipped paint were stored until they could be shipped back to the manufacturer for repainting (toys with different defects were stored in a separate warehouse across the street). His mother, Pearl Norris, was known for making "the best darn tootin' apple cobbler in Lewes."