AFAIK it wasn't about E2E encryption, but P2P calls, and mega-peers which would serve as ad-hoc "servers" for coordination of other clients. That was dropped when Microsoft bought Skype and instead switched it to using central servers. However, I notice these days that calls are P2P, meaning I see in my PC's connections the IP of the person I'm talking to, so not sure what's going on there...
My guess is that the signaling servers are centralized. You need those for connection establishment between two peers. Once they 'know' each other, they may aswell talk to each other directly. Routing all calls through central servers would yield an enormous bill every month. On the other hand, it is sometimes necessary as direct communication between peers may not be possible (NAT, firewalling). IIRC, a central server is also required for multi-party calls.
Yes, signaling is definitely centralized, before it was done by mega-peers. P2P contact does work behind NAT if it's a cone NAT. Symmetric NAT would require that the signaling server assign a forwarding server to handle data from and to each peer. In general CGNAT applications with cone NAT, UDP hole punching can be used to communicate between peers, organized by the signaling server.
