Lusher's earliest musical career was interrupted by service during World War II, serving in the Royal Artillery and sometimes performing in military bands. After demobilisation, he found work here and there before joining Ted Heath's big band in the 1950s. The association continued through recordings, concerts, and Heath's radio series (sometimes backing guests like Frank Sinatra and Johnny Mathis), and television. When Heath died in 1969, Lusher took over as the band's leader.