It shouldn't matter. 1 vote or 100 votes. If fraud or even irregularities are found, one side gets to contest and challenge those lots of votes with an audit. If they can't come to an agreement, through court arbitration if necessary, throw them out. Yeah it can be abused, which is why we have a process for vacating or filling offices regardless of the election results.

Of course that is a really extreme example so I don't know why it's even being brought up here. We have ample evidence of a loss of chain of custody over entire counties.