There is an incredibly easy answer to this.
The tipping point was the 2004 season, because by this point you had lost a lot of the Butch recruited AND COACHED players and you also lost Chudzinski as the OC. Coker wasn't perfect, but at least he had 2 UM alums as his OC and DC and once Coker...