I think the problem is if/when AGI enables "someone else" to not need human employees for ~anything. The people that own physical capital (land, farms, mines, etc.) would have robots and GPT-N to extract value from it. The people who survive based on their labor are SOL. I think it is reasonable that many people won't be excited about that kind of automation.
Social/economic stratification (to a certain degree) makes sense as long as there is a reasonable amount of social mobility. AGI paired with advanced robotics seems as though it would all but eliminate social mobility. What would your options be? Politics, celebrity, or a small number of jobs where the human element is essential? I think the economic system needs to dramatically change if/when we reach that point (and ideally before, so people don't suffer in the transition).