That's true. The hope is that users will favor generalist / unbiased relays (less fragmentation by design) rather than heavily biased / restricted ones. Maybe even fund them: I will pay you as long as you don't start banning large swathes of the network just because you don't like what they say.

Users you follow can also advertise relays behind the scenes, so it's more probable that, if you follow a coherent set of users, you will converge on a coherent subset of relays that doesn't really feel fragmented.