There were two things that really clued me in early on (by the end of year 2, specifically). It is not something that happened in a specific play, or against a specific team, but personality and mentality traits that simply were never going to work that revealed themselves after a satisfactory sample size had been observed. Traits that make it almost impossible to get to the very top of any competitive field such as multi million dollar sports.
1) Golden is not cut throat. The games against lower level teams were allowed to get embarrassingly close, embarrassingly late in the games.
2) Golden is not able to evolve/be flexible. He was going to force his style on the players, not put a style into place that fit the players.
You can reach your personal ceiling with these traits, and you may be a better person for them. But for this type of person, which I fully believe Golden to be, that is likely a 7 or 8 win mid west middle of the big 10 team.