The reason that you think that it shouldn't be possible for both to act in self-defense is likely because you apply an assumption often used in economics, that of perfect knowledge for those involved.

If people are assumed to have limited knowledge of situations, then it stops being impossible.

Both arguments can be true, so the question becomes, was the McMichaels justified in attempting a citizens arrest?