I lived through the Manny Pacquiao stupidity. Floyd never entered the ring without his PEDs, just like all competitive fighters to this day. Instead, he wanted to duck and dodge Manny based on steroids claims. It was the whiniest moment of a long line of ducking in his fighting career.
Paul Williams, Miguel Cotto, Antonio Margarito, Shane Mosley, Joshua Clottey, and Manny Pacquiao were all in or near their primes and fighting and beating each other around 2007–2009ish. Although he ended up fighting three of these later, it's fair to say he avoided them during their prime years.
The unbeaten record means less when you realize he's avoided fighters at their most dangerous. Great fighters like Muhammad Ali, Joe Frazier, Sugar Ray, Hitman Hearns, Hands of Stones Duran, etc, etc, didn't just fight the best of the best when they were the best but all of them had fighting trilogies or even rematches with each other that fight fans still talk about today. The public wanted to see rematches when they were great fights, it's what we do. Floyd had one rematch in his entire career. He had fights that the public believed he got beat in, he never once wanted to fight that opponent again. Floyd was always playing defense. He was a runner.