I don't think it's the X-risk AI threat that should primarily concern everyone. I'm not suggesting it should be ignored; some like Scott and Eliezer are working on trying to reduce that risk (or more accurately, trying to figure out ways to understand the alignment problem well enough to develop ideas to reduce the risk), but it's out of the hands of most of us. Either AGI instantiates and kills (most) all of us, or it doesn't. And, as Scott points out, humans—due to various technologies and psychosocial problems—may be running headlong into a great filter in the near future without AGI, so an AGI might be an improvement in that case.
But what happens to society when all non-face-to-face communication becomes saturated with unlabelled AI-created and AI-assisted content, which may or may not be trustworthy or correct, but which we can't separate from human-created content? What happens to economies, governments, countries, and geopolitical stability? You don't have to believe in a large-scale AI replacement of human work and massively increased unemployment to see the problems AI is already creating, and LLMs will only get better.
I agree with the saturation, It's potentially the end of information, makes me wonder, how do we stop Wikipedia entries being rewritten little by little, what sounds like valid context changes, referencing unaware AI fake content, that is then read by the next AI model, so the "pattern" is kindof preserved and on and on it goes... To me it sounds like a future where archive.org would be the only valid source of information, the rest is saturated by garbage. It's a problem of processing power. The needle in the haystack has a new added factor, fake needles. Where is our red stop button that turns the Internet in read only mode because these garbage generators are running wild? Or what was the name of that project where you would have CDs with information on how to rebuild civilization?
But what happens to society when all non-face-to-face communication becomes saturated with unlabelled AI-created and AI-assisted content, which may or may not be trustworthy or correct, but which we can't separate from human-created content? What happens to economies, governments, countries, and geopolitical stability? You don't have to believe in a large-scale AI replacement of human work and massively increased unemployment to see the problems AI is already creating, and LLMs will only get better.