Caffery played for the St. Michael's Buzzers Junior "B" and the St. Michael's Majors Junior "A" teams from 1950 to 1954. He then turned pro in 1954-55 with the Leafs and played until 1958. He was a good professional baseball player and juggled his time between the two sports until 1958, when he turned to baseball full-time. Caffery played two more seasons of hockey - 1960-61 with the Springfield Indians and 1963-64 with the Greensboro Generals.