Common sense, huh?
I've got a younger cousin over 30 who's maybe 5'7" and weighs about 148 lbs. I've seen a guy 6 inches taller and 70 pounds heavier crack his jaw with a closed fist in a parking lot and watch my cousin get up, laugh at the guy, and tell him he hits like a faqquit.
I've got a friend who's about two years older than my cousin, who runs about 6'1" and 250 lbs. I've seen a fat guy about 5'9" and 260lbs pop my buddy in the face, bust his lip open, and the poor ******* responded with nothing but tears.
How a human being responds to being hit is maybe 20% physical and 80% mental. Obviously, you don't want a physically frail player at any position, but give me the smaller guy with the tough as nails mentality who will get back up, tell the dude that hit him he isn't ****, and stand tall in the pocket over the bigger guy with a frail mind who will be bracing for the impact.