I think it's partly because there isn't a word in the english dictionary to allocate white people in a negative way, like you have with blacks, asians and latino's. In their language there certainly is, but not in ours, and they are using our language, english, white people's language, so they've basically already lost the argument when they say something like cracker. In no country does there exist a word that describes the majority of it's inhabitants disparagingly, it doesn't make sense to. This could also explain why some people get annoyed or insulted when people use a different language.