I sense your sarcasm which is unfortunate.
Some quick research may give you further information on the phrase. The phrase is not directly nor indirectly tied to race like you inferred. I would venture to guess that the majority of inmates at asylums (typically for the mentally ill) are not...