On what platform are you seeing this? Element is three different apps on iOS/Android/Web and they implement scrolling differently. There is no architectural reason in Matrix why it should be slower than TG.
They are probably talking about the Matrix.org instance being slow at fetching messages for you, additionally to the delay needed to get keys for them in encrypted rooms.
This is far nicer on smaller home servers, but matrix.org is the "default experience" of element and that's what people are judging it by.
Probably because the people developing these clients aren't being bankrolled by multi-million dollar companies with private interests. The previous poster was correct; Matrix is a protocol, and so your performance really comes down to whatever client you're using it with. Element is a web app, so it will inevitably be pretty slow. If you want a client that won't slow down, look into Fractal, a GTK Matrix frontend written in Rust. If that doesn't iron out your performance issues, you just have a slow machine/connection.
Fractal is actually responsive, I am actually impressed. It's not bankrolled by multi-million dollar companies though and Telegram web app is way faster so something doesn't add.