Well, I think another point to be made in discussions like these (and an awful lot somewhere in the chain on the fanfic post about some Big Bang Theory Fic set in Haiti) is that people seldom intend to marginalize or hurt non-privileged groups - but it doesn't change the result, or make it better. How "big" it is has a lot more to do with the impact than the intention, unfortunately, and all you can control is the response when someone points out that you've given offense.

I don't think anyone imagines (with a first offense, anyway) that the other party is twirling a waxed mustache chuckling over how they've shown that [insert group here]. Rather, they want to point out that regardless of intent, it's hurtful, and either raise awareness for the future or at least receive an acknowledgement of error - as opposed to an "apology" that amounts to "I'm sorry you're so sensitive."

So, you're in the position, generally, of saying "It was a mistake, I apologize for hurting you, and I will [correct it / do my best to be correct in the future]."

I think I've just had this drilled into me pretty well. I'm a large, relatively strong guy, and owning the responsibility for accidentally hurting people is something I've been doing from a fairly young age.