You are not wrong, but the problem I mentioned still stands. All your message would be routed by the servers that are owned by "them" and they can choose to shut you out whenever they want. Even if they are not able to read your message, they can shut you down at their whim.
The best (only) way is to build a web of trust, through exchange and signing of keys when you physically meet someone.
Read, and check out the whole manual. The same concepts would apply to Signal, I suppose, though I haven't used it.
You are not wrong, but the problem I mentioned still stands. All your message would be routed by the servers that are owned by "them" and they can choose to shut you out whenever they want. Even if they are not able to read your message, they can shut you down at their whim.