The Vatican has always been my main problem with Roman Catholicism: religion is about the relationship between man and God. Introducing some other man as the End-All-Be-All middle-man of that relationship just leaves an easily-infiltrated/-exploited central hub for any con artists or wannabe-dictators to work towards.

If you don't like a priest in one church, you can go to a different church.
If you don't like the Pope, you have to bite your lip for potentially decades until the Pope retires/dies, and hope that the next one is better.