They hate him because he doesn't see race. As he himself put it 'when I was shot and dying on the floor, people blacker than me were stepping over me to get away, but it was a white man named Uncle Bob that picked me up and took me to the hospital'. It was apparently that event that solidified his views on how race didn't matter, actions did.