Yes it really would be. It's the reason I use Linux in the first place, to have freedom to do whatever I want. I don't need linux to protect me from Windows Recall. I chose linux because it already doesn't have stuff like that. And the system (used to) treat me like an adult.
it's where the security boundary is. The system should protect me from unwanted software getting on the system. Once a piece of software starts running, I do want to give it full access to the system. If not, I'll run it in a container. All the protections wayland offers are to protect against a piece of software someone had to already have access to install. If you are that paranoid, set up selinux properly. Which nobody does, cos it's a pita. Enforcing permissions for everything on everyone is a similar pain in the ass.