Mangos are related to poison ivy, so its not unusual for the skin to irritate those that touch it. However, the actual fruit should not be an irritant.
p.s. I LOVE mangoes and almost all tropical fruits, I have multiple trees in my backyard, including Mango, Nispero, Bread Fruit, guava and...