Just because one doesn't have a devil fruit ability doesn't mean that they are weak in comparison. Look at Zoro, for example. He has no fruit power, but is incredibly strong due to his hard training. Shanks being another example, has one arm, but can hold his own in blocking Whitebeard's attack. Same can be said for Roger and Garp, even though the D's carry a trait as being incredibly strong. As far as Whitebeard is concerned, look at his size. He is huge, and to think he fought on EVEN grounds with a guy as small in comparison as Roger is a feat in itself.