No, I don't think so. It would only really matter if only by a small amount, if the two were fighting at normal human strength like in the real world. There have been many instances in the Dragon Ball series which can support this. For example, in the 23rd World Martial Arts Tournament when Goku and Piccolo Jr. fought, there was one point in the fight in which Piccolo grew to a huge size dwarfing Goku by a decent amount. Despite this, Goku managed to hold his own. Keep in mind though, strength isn't the only thing that matters. In Judo, a real life martial art, you learn to use your opponents size against them.