O'Brien was the third son of musician Haskell "Hack" O'Brien, a well-known drummer for several big band era ensembles. When he was an infant, his mother thought he resembled a bear cub, and tagged him with his lifelong nickname. He has two older brothers, Haskell O'Brien, Jr., a trumpet player, and Warren O'Brien, also a drummer.