Patterson grew up in Rozel, Kansas and played football at Dodge City Community College and at Kansas State University. Patterson is married to Kelsey Patterson (née Hayes). He has three sons: Josh, Cade and Blake. He received his bachelor's degree in physical education in 1983 from Kansas State University, where he became a member of the Acacia Fraternity. While coaching at Tennessee Technological University he earned a master's degree in educational administration in 1984. Outside of coaching, Patterson plays guitar and performs at charity events around the Dallas-Fort Worth area during the off season.