I think obviously there is a degree of luck involved. Crapshoot if you will. But I think more of it is similar to analyzing companies for stock investment.
The best way to know what works, is to find shared characteristics of your past coaches. You then can eliminate candidates that don't share those common characteristics.
For example, none of our 4 great coaches came from the college coordinator ranks. All either were college HCs, or NFL coordinators. So right off the bat, in a coaching search you eliminate every single coordinator.
Jimmy, Howard, and Butch all coached under LEGENDARY, Hall of Fame level guys: (Chuck Fairbanks, Don Shula, and Jimmy Johnson) respectively.
They all (except Dennis) were championship winners as coaches or players.
Find a guy who fits all that, and my guess is you have yourself a winner.
Tom Herman fits the bill completely, for example. Pagano has a ton of the qualifications. Butch obviously does. Fuente comes close.
Of course nobody doubts that Chip Kelly would succeed bc he already proved he can. Trying to find the "next" chip kelly is where the risk comes into play. If you want to make a safe hire, chose a PROVEN winning elite coach, or hire a guy who fits the above qualifications.
My only concern with Herman is the small sample size. He's only been a head coach for all of 7 games. You can't really judge anything in such a short time. How's he going to recruit year after year? No one really knows. Recruiting is the lifeblood of any program.
Thats not to say that he won't be successful. At this juncture we just can't afford to roll the dice. We need someone proven.