Ruffins was a founder of the graphic design studio/collective Push Pin Studios, although he departed in 1963. He then partnered with Simms Taback for over thirty years. He contributed to The New York Times and did print advertisements (including for IBM), book covers (The English Novel, Aspects of Love), album covers, and posters. He illustrated a series of books and calendars by Jane Sarnoff (mostly riddle themed) as well as his own children's book My Brother Never Feeds the Cat (1979). In later years, he illustrated a number of multicultural children's books mostly based on traditional folktales.