There's the rub. Politics.
If people thought it was embarrassing to see a half full stadium with a capacity of 65,000, wait until they see that same thing in a smaller stadium. Other colleges will still be able to say no on-campus stadium and point out how they can't fill even a tiny stadium...