Good for you! I also stayed home (my boys are older now) and I have no regrets. What I do have is a lifetime of memories (field trips, plays, parties, etc) that I would have missed out on had I decided to go back to work. As long as this is what you and your family want, ignore all the idiots. In the end, they don't matter.

ETA: I'm sure your kids love that you're home. It provides a lot of emotional security for them.