The idea that AGI doom scenarios are really late-stage capitalism is interesting and strikes me as fundamentally correct. The difference between the AGI takeover and capitalism is just the choice of metric to optimize on.
But, I think, it's the act of trying to optimize on a metric itself that is the source of the destruction. Unmeasurable human values can't survive an optimization process focused on measurable ones.
Our Gross National Product, now, is over $800 billion dollars a year, but that Gross National Product - if we judge the United States of America by that - that Gross National Product counts air pollution and cigarette advertising, and ambulances to clear our highways of carnage. It counts special locks for our doors and the jails for the people who break them. It counts the destruction of the redwood and the loss of our natural wonder in chaotic sprawl. It counts napalm and counts nuclear warheads and armored cars for the police to fight the riots in our cities. It counts Whitman's rifle and Speck's knife, and the television programs which glorify violence in order to sell toys to our children. Yet the gross national product does not allow for the health of our children, the quality of their education or the joy of their play. It does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short, except that which makes life worthwhile. And it can tell us everything about America except why we are proud that we are Americans.
It makes me question if I've ever seen a metric employed by a company that didn't have a detrimental effect on the happiness of the people involved. We are all unique but we're also really, really similar and it's in those uniquenesses that we find real happiness. But if we're only a number of sales/dollars/tickets/loc then the distinction is a zero-sum game; I only stand to lose a dollar that you make.
Business is not a zero-sum game, especially not technology businesses that constantly innovate. All sides of the transaction can benefit. Optimizing for profit or similar metrics doesn't inhibit those benefits.
I personally have never seen a metric that was actually beneficial for a company. Because the moment a metric exists, all rationality and reason is thrown out the window in favor of optimizing that metric. It doesn't matter if it destroys the company or customer trust in your product or runs out valuable employees. None of those are part of The Metric.
It's the whole 'we increased shareholder value by 1% even if we won't exist next year' scheme. The notion of any sort of sustainability or long term planning has long vanished.
> The difference between the AGI takeover and capitalism is just the choice of metric to optimize on.
Not specific to capitalism. The same thing occurs with any government/economic structure that is optimizing for a metric. Think of a communist government using “happiness” to drive decisions that just silently kills or drugs everyone who isn’t happy.
The fundamental difference of capitalism vs the other instrumental forces you are describing or intimating is that capitalism is self-reproducing. Through the explosion of the profit motive and the harnessing of wage labour it continually opens new markets, and demolishes limits to growth.
"AI" in the current hype cycle (LLM generative ML yada yada) is just classic capitalist industrial automation, applied to one of the few areas that still had echoes of the pre/proto-capitalist "craft economy"
Prepare, nerds, to be put through the same process as seamstresses, weavers, blacksmiths, lumberjacks, etc.
So much of what people identify as post-modern or late-stage or whatever is just the continuation of the same intrinsic process that has been happening since the end of the 18th century.
But, I think, it's the act of trying to optimize on a metric itself that is the source of the destruction. Unmeasurable human values can't survive an optimization process focused on measurable ones.