Short answer, no. It's not libel to call someone a nazi.

Firstly, libel only applies to published statements. Actual publishers get around this by not calling you a nazi. Instead, they publish that someone ELSE called you a nazi. That way it's not libel, it's "news".

Libel laws are pretty weak and a big chunk of a journalism degree is all about how to carefully word your way around libel laws. That's why they always toss "alleged" on everything.

Spoken false statements that unjustly harm your reputation would be defamation, but that's even more difficult to gain real legal traction with. You have to prove definitive harm (say loss of customers at a store you own), and prove that the speaker knew their statement was factually false, AND prove a connection between the loss of customers and the factually false public statement.

So there's a major hurdle in arguing that the statement was not a statement of opinion or something that the speaker believed to be true. What is or isn't a "nazi" in common public vernacular is weakly defined and thus likely to be ruled opinion. Even if it weren't, then the speaker merely has to claim that they really do believe that you're a nazi (it's not illegal to be wrong).

Then you have the hurdle of proving the statement directly caused material harm such as loss of income. So if you get fired as a direct result, possible, but just your neighbors thinking less of you and not inviting you to parties, probably not.

Basically, you have to have something as clear cut as someone standing outside the door of your business telling people that are about to walk in that you're a convicted child molester AND be able to prove that they know for a fact that you aren't.

Of course, if you've got the money, a lawyer's job is to find a way to work around the limits of the law. If you could just go by the clear cut obvious definitions, then there'd be no need for lawyers.