There is no down side.
Everything mentioned is just a natural consequence of any coaching change no matter when you make it. You will always risk losing a player here or there but it is irrelevant to the long-term goals. When it is time to make a change, you make it. Delaying for reasons like...