Think about how stupid most people are.

I'd be willing to be there are plenty of old men and women who see a young kid who shaves their head for the style of it and put them in the Nazi column because they are shaving their head, and that means they are a skin head Nazi despite no other evidence.

Think about how stupid people like to jump to these insane conclusions and then keep them forever. There are still people who think playing DnD is some sort of cult ritua for crying out loud.

Now think about how many of these little beliefs there are and how many little communities think differing sets of them... madness.