We have a right to confront our accusers (so to speak). If you're going to downvote, without an intellectually honest reply, then you're just a coward that doesn't want to backup where they're coming from.
There should also be an option for a "reply to all" if there is more than one downvoter, so you can confront all the downvoting shills....or, be proven wrong by them and delete your post, once you've realized you were in the wrong.
People don't understand what the purpose of the vote is. Agree or disagree, good or bad, an upvote is given to help with exposure. We read from the top down. Naturally the posts at the top get read most. An upvote means that the information is worth reading not that the reader necessarily agrees with the information.