We call those brown wasps dirt dobbers around here.

You can tell them apart from the red bastard wasps by looking at their stinger area, dirt dobbers stinger is connected to their body with a thin tube-like extension. Where the red wasps stinger is directly connected to their body.

Also dirt dobbers build their nest from mud, where the red wasps build theirs from paper.

The red wasps are highly aggressive and should be delt with before their numbers grow to much.