That's Matt's whole argument, no? That anyone can grab it from e.g. Github, but he's not compelled to provide access to the wordpress.org servers or the WordPress trademarks under the GPL.
Not a lawyer but my opinion is the law shouldn't be used to compel anyone to provide unpaid services. With that said, I still think it's terrible for the community. If other package repositories like npm, PyPI, and others decided to randomly block people over grudges, it would create a lot of dysfunction. The consequences should come in the form of Matt no longer being reputable as a provider of services to the community