Anybody is free to agree or disagree with that as they see fit. You yourself qualify them as "trans women", after all. However, you can accept them as "real women" without that automatically entitling them to participate in women's sports. It is justified by the importance of women's sports to the well being of the vast majority. They ought to pick a different hill to die on.

A lot of ethics is about when general norms like "greatest good for the greatest number" ought to be suspended. These are complex questions.