The non P-5 teams you mention are 4-2 vs P5 teams.
The average win total of those teams they beat is 5, with none of the teams being over .500.
They are 0-2 vs the Top 25.
Put teams like Appy State, Memphis, Cincy, and Navy in any P5 division and they'll be middle of the road at best at this...