AI safety is an apocalypse cult and we should recognize that everyone who has ever been in an apocalypse cult thinks that "no, this time it's really the end of the world". Everyone who believes crazy shit thinks they're being reasonable, so just because we're super sure we're being reasonable doesn't mean what we believe isn't crazy.
Do we believe that we alone, of all people who have believed in the coming end, have it right this time?
There have been people for a long time who think the world is going to end. We can use your argument to dismiss anyone who fears any existential risk like asteroids, nuclear war, climate change, AI, etc...
And since the world exists every group will be wrong except one.
I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.
The issue is that the effective altruism crowd are focused exclusively on AI safety, at least in what they publish online. If they talked about all possible threats to humanity and how to solve them, from AI, to climate change, to nuclear bombs, to asteroids, to disease, they would be an interesting and important group.
They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession.
Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important.
This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.
I've never been on this website before, but am I correct in understanding that the "biosecurity" categoery is about a month old?
It would be interesting to understand how much of the "existential risk" category is devoted to biosecurity or pandemics, and how recent those posts are.
Pending the above, my tentative takeaway from this is that COVID - arguably the greatest threat to human health and wellbeing in the past several decades - seems to have been a minor/non-existent blip on their radar until it actually happened. This raises serious questions in my mind about the community's predictive abilities and/or bias.
> am I correct in understanding that the "biosecurity" category is about a month old?
It looks like the Forum is confusing here: it has "Biosecurity & Pandemic Preparedness" [1] as one topic, and "Biosecurity" [2] as its largest subtopic. But when you click on the parent topic it's not showing the subtopic posts; I've filed a bug.
Thanks for the reading. The 2017 talk/article is interesting in light of COVID:
"Some of the reasons I'm skeptical of natural risks are that first of all, they've never really happened before. Humans have obviously never been caused to go extinct by a natural risk, otherwise we would not be here talking. It doesn't seem like human civilization has come close to the brink of collapse because of a natural risk, especially in the recent past.
"You can argue about some things like the Black Death, which certainly caused very severe effects on civilization in certain areas in the past. But this implies a fairly low base rate. We should think in any given decade, there's a relatively low chance of some disease just emerging that could have such a devastating impact. Similarly, it seems like it rarely happens with nonhuman animals that a pathogen emerges that causes them to go extinct. I know there's one confirmed case in mammals. I don't know of any others. This scarcity of cases also implies that this isn't something that happens very frequently, so in any given decade, we should probably start with a prior that there's a low probability of a catastrophically bad natural pathogen occurring."
I wonder if this is a case of extrapolation fallacy? Modern human dynamical behaviour is significantly different to both animal behaviour and less modern behaviour. Viruses spread more easily in the age of global travel; those that used to be too deadly to spread very far suddenly have more opportunity.
EDIT: Reading this more carefully, the speaker does actually address globalisation, but seems to dismiss it as a counterargument and I'm not really sure why.
Anyway I read through that article. There's a lot of guff about Newcomb's paradox and Eliezer, and a single paragraph referencing COVID, with two links supporting your statement. I've clicked through those links and have come out unimpressed.
Scott rightly highlights how the media and institutions got things wrong, but conversely gives only a few examples of "generic smart people on Twitter" getting some things right in the early days of the outbreak. He confesses that he himself did not predict the seriousness of COVID.
The second link is used to support the claim "the wider tech community, were using masks and stocking up on essential goods, even as others were saying to worry about the flu instead": https://putanumonit.com/2020/02/27/seeing-the-smoke/ .
The linked article was written at the end of February, when panic-buying had already firmly set in across the US.
There is nothing here about the reasoning or predictive abilities of the rationalist or EA community specifically. Nor is there any compelling comparison of its response compared to the wider public.
Climate change is unlikely to kill everyone on earth. Other risks, such as an engineered pandemic, asteroid impact, or an AI apocalypse have the possibility of killing ~everyone. This is not saying that climate change is not a real issue.
The AIpocalypse seems incredibly unlikely. I'm a lot more worried about the nukes, and even if we end up with robotic overlords, I'd bet they'll be a whole lot better at administration than meat -admins have proven.
I'm with you on engineered superviruses. Feasible, likely, and incredibly high impact.
What I kind of keep coming back to is the risk profile of all this stuff. That magic vector of likelihood * impact. Global warming is happening now. And it's real bad - worse than I think we give it credit for.
What I worry about is that we're the proverbial frog in the pot. Things get just slightly hotter each year, so we'll miss it when we actually boil.
So what you're saying is that if a risk can totally upend our society, destroy most of our cities, make most of our farmland unusable, destabilize geopolitics in a way that's almost certain to lead to war between nuclear powers (who also have the ability to engineer deadly diseases), and massively disrupt every ecosystem on earth, and that there's enough evidence to say with almost complete certainty that this will come to pass without massive societal and political change and technological intervention... but it's unlikely to kill everyone... then it doesn't really deserve mention?
The cascading risks you mention are certainly real and serious, and are worthy of our best and urgent efforts to solve. Effective altruists are rightly concerned about these effects e.g. https://80000hours.org/problem-profiles/climate-change/
My comment was written with the summary of that article in mind (I didn't make this clear):
> Climate change is going to significantly and negatively impact the world. Its impacts on the poorest people in our society and our planet’s biodiversity are cause for particular concern. Looking at the worst possible scenarios, it could be an important factor that increases existential threats from other sources, like great power conflicts, nuclear war, or pandemics. But because the worst potential consequences seem to run through those other sources, and these other risks seem larger and more neglected, we think most readers can have a greater impact in expectation working directly on one of these other risks.
There is an excellent chapter on the existential risks associated with climate change in Toby Ord's book The Precipice, which you can get a free copy of at https://80000hours.org/the-precipice/
The way the EA forum handles parent topics is confusing, but this is only counting posts tagged with the top-level label. It has several subcategories, and there are hundreds of posts when you include them:
* 218: Biosecurity
* 176: Covid
* 54: Pandemic preparedness
* 40: Global catastrophic biological risk
* 39: Vaccines
* 20: Biotech
* 16: Life sciences
* 11: Dual-use
* 10: Biosurveillance
(Posts can have multiple labels, so the above list double-counts a bit. I don't see an easy way to extract the actual count from the forum, but it's at least 218, the count for the largest category.)
> The issue is that the effective altruism crowd are focused exclusively on AI safety.
How many EA people do you know?
That seems to get the most clicks, but I usually associate them with things like mosquito bed nets to help prevent malaria, because that's a far more common topic in my experience.
I'm sure you can find particular people who are all about the AI risk stuff that seems pretty wild to me, too.
But when you say "focused exclusively" I feel like you may have mostly seen news articles about wild AI stuff and not the average EA enthusiast who spends their time talking about picking efficient charities.
As someone who is not an EA, but has been peripherally aware of them for years, the bed nets stuff was their big thing until maybe three years ago, but it genuinely is the case that a lot of their energy had recently redirected to AI safety and other X-risks.
That isn't to say that the malaria stuff isn't still a big topic, but there has been a shift.
Yep, I think the community is pretty evenly split amongst "longtermist"-focused people, and this includes all the X-Risk stuff, as well as the more classical "Global Health & Happiness" people. At this point they are actually relatively distinct groups I'd wager. One thing missing from the article is that, at least in my experience, the vast majority of the opportunists are in the longtermist camp.
Longtermist topics are the perfect combination of high-prestige and smart sounding with a complete lack of accountability, because the impact of research or projects won't materialize for decades. Ironically, in practice it directly contradicts the founding tenants of EA, despite being a logical conclusion of the underlying moral philosophy.
AI safety and "longtermism" (I feel slightly more stupid just by writing this) is an escape route for the group's realization that (a) they cannot really make a difference and (b) even if they could, it would require a lot of actual work. The person who goes out in the middle of a cold night and cooks soup for homeless people can quite easily count how many mouths he fed; sounds quite effective and quantitative to me. Or maybe the volunteer who is helping fight malaria in a desert somewhere outside of America/Europe; actually, they don't need to count, their exhaustion at the end of a hard day is proof enough of their altruism.
But since the real goals of the EAs are not to do any of that (it's just a bunch of socially-awkward geeks looking to feel accepted in some kind of community and find sexual partners; in other words, trying to feel like normal humans), they fooled themselves into believing that such short term actions are "ineffective"; actually, any actual measurable hard work has to necessarily be ineffective, or else it would actually have to be done. So they keep pushing their agenda to longer and longer term issues, more abstract and less well-defined as possible, which cannot be actually measured, so they can continue to build this weird, meaningless, ridiculous fantasy that they live in (and which supports their social hierarchy) without having to actually do anything concrete.
>The issue is that the effective altruism crowd are focused exclusively on AI safety.
That's really not true. There's a huge overlap in these groups, but the top EA charities are not shoveling all their money at AI alignment orgs... even if one can argue that they should.
Arguing that AI safety is needed only demonstrates that it's altruistic, but you have a lot farther to go if you want to say funding research in this decade can be effective.
I think they're worried about it because they honestly do care about existential risk. If you can come up with some more likely existential threats than AI, I'm sure they'll worry about that too.
> They don't. They hardly talk about anything else except AI.
Even if that were true, there are lots of people already looking at those other existential risks. The EA crowd was the only group concerned about AI for a long, long time. Seeing an existential risk that nobody else does leads to evangelism and hyper focus so people start paying attention.
> AI is much more compute bound than algorithm bound
Any reason to believe this? The human brain runs on 20 watts (!). Obviously biological hardware is not directly comparable to silicon, perhaps silicon is intrinsically less efficient by some factor such that in order to obtain 20W of human brain compute we need to expend K times more in silicon. But is that factor really so large that we are compute-bound? How big would K have to be for us to be compute-bound anyway? If you could have Einstein for 200kW (i.e. silicon is intrinsically 1000x less efficient than brains) would that be compute-bound? It seems much more likely that we are algorithm-bound and simply not using our available compute efficiently.
Yudkowsky's hard take off mentions "solving protein folding", nanomachines, persuading everyone to let them connect to the internet, infinite adaptability, etc. I think those are unrealistic, but obviously they still have a non-zero chance.
More probably bad possibilities are AI-Hitler (monomaniacal populist absolute leader) or AI-Stalin (manipulative, smart, absolutely paranoid, rising through the ranks) ... so something that's human-like enough to be able to connect with humans, to manipulate them, but at the same time less affected by the psychological shortcomings. (Ie. such an AI could spend enough time to cross-interrogate every underling, constantly watch them, etc.)
And yes, a very efficient immortal dictator is very bad news, but still bound by human-like limits.
And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?
Can it increase the number of its input-and-output channels while maintaining its human-like efficiency?
Because it's very different to run a fast neural network that spits out a myriad labels for every frame of a video stream (YOLO does this already, but it's not 20W!) and to integrate those labels into actions based on a constantly evolving strategy.
Sure maybe the hypothetical AI will simply run a lot of AlphaZero-like hybrid tree-search estimator-evaluator things ...
Anyway, what I'm trying to say is that our 20W efficiency comes with getting tried very fast, and using "fast mode" thinking for everything. (Slow mode is the exception, and using it is so rare that we basically pop open a glass of champagne every time.)
I agree that the most likely way an AI would take control involves social/political engineering, but that doesn't mean it will have human-like morals making it keep humanity alive once it doesn't need us or that it will have human-like limits.
>And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?
Even if its mind wasn't truly directly scalable, it could make 1000 short or long-lived copies of itself to delegate those tasks to.
Asteroids, nuclear war, climate change, and AI. As the old song says "One of these things is not like the other". We know that asteroids exist and many large ones are uncomfortably close to our home. We know that nuclear weapons exist and have seen them used. We know that the climate is getting hotter (*).
AI is...well, people losing their shit over ChatGPT (**) aside, AI is not going to be real enough to worry about for a few more decades at least.
(*) Anyone who's about to regurgitate some fossil-fuel industry talking points in response, just save your breath.
Would you rather we wait until AIs are actually posing a threat for us to study ways to align them with human values? Tons of money already goes into fighting climate change and basically everyone on earth is aware of the threat it poses. The AI safety field is only about a decade old and is relatively unknown. Of course it makes sense to raise awareness there
You seem to agree that if at some point in the next few decades AI will be something we need to worry about, so I'm trying to figure out exactly what it is you oppose.
Would you have opposed research into renewable energy in the 1970s since global warming was still a few decades away from being something we needed to worry about?
What’s the upper bound for climate change’s existential risks? The end of human existence and society as we know it except for a relatively small number of survivors living in a world those of us alive today can barely imagine?
The scenario described in this article comes the closest, detailing a mechanism that may have been responsible for the end-Permian mass extinction, wherein warming oceans become anoxic and begin to release huge quantities of hydrogen sulfide gas. The gas is directly lethal to most life in the oceans and on land, and destroys the ozone layer as a bonus. It's hard for me to imagine any way in which an agricultural human civilization could survive that scenario.
By "climate change" people mean the sort of change that will be realistically caused by humans within the next few hundred years, not the sort of extinction caused by massive volcano activity that takes 10,000 years and happens once every 100 million years. Obviously if the climate changes drastically enough fast enough then everyone dies but nobody is suggesting that's the case for preset day human activity.
In terms of timeline, the key sentence would be "The so-called thermal extinction at the end of the Paleocene began when atmospheric CO2 was just under 1,000 parts per million (ppm)". The end-Permian was likely around 2500 ppm[1]. It doesn't really matter whether that carbon comes from supervolcanoes or from human emissions.
When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like.
Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine?
"It doesn't really matter whether that carbon comes from supervolcanoes or from human emissions."
It matters a lot. Whilst I was eating dinner I've now read the article you cited, the paper that it cites (Kump, Arthur & Pavlov, 2015). I also cross checked the argument and numbers against several other papers. Ward's argument is extremely slippery. This is sadly what I'm coming to expect from academics with unverifiable models. You have to read everything they write adversarially.
"In [Kump2015]'s models, if the deepwater H2S concentrations were to increase beyond a critical threshold during such an interval of oceanic anoxia, then the chemocline separating the H2S-rich deepwater from oxygenated surface water could have floated up to the top abruptly. The horrific result would be great bubbles of toxic H2S gas erupting into the atmosphere."
Observe that his argument starts with a "critical threshold" for H2S levels, not CO2, and that he doesn't tell us what this critical level is. The obvious questions are thus: what is this level, is it realistic for our present day oceans to reach this critical level and if so, how? To get to those levels of H2S he hypothesises that:
"if ancient volcanism raised CO2 and lowered the amount of oxygen in the atmosphere, and global warming made it more difficult for the remaining oxygen to penetrate the oceans, conditions would have become amenable for the deep-sea anaerobic bacteria to generate massive upwellings of H2S."
In other words in this theory the supervolcanoes have to come first, triggering a sharp drop in oxygen levels in the atmosphere, which in turn then causes the chemocline to move, which then causes the H2S upswellings. It's all caused by a massive loss of oxygen, not increase in CO2 levels, which is simply another result of the volcanisms. Kump2015 also makes it clear that their hypothesis requires a truly massive drop in oxygen levels to occur. We'll look at how much in a moment.
But just a few paragraphs later Ward has forgotten all about the volcanoes and oxygen levels. Suddenly it's all about absolute CO2 levels - not even rates! Cause, effect and unrelated side effects have become entirely muddled, probably because he knows nobody would care about his article unless he ties it to global warming armageddon somehow and because hey, this entire field is nothing but assumptions, suppositions, and playing with numbers that can never be verified anyway so why not?
His argument is brittle in other ways. He asserts without any backing argument that whilst supervolcanoes explain all the other non-asteroid mass extinction events it doesn't for the Permian because, apparently, supervolcanoes are really great for plants on land which can "probably" survive the warming. And there was me thinking that large scale volcanic activity is supposed to be very bad for plants because it dims the atmosphere:
"Plant growth is restricted and mass extinction can be caused."
Back to the H2S claims. Turning our attention to the Kump2015 paper we find immediately that it's got an annoying structure in which they work backwards from their desired scenario to calculate the level of H2S that could trigger it:
"Thus, if the H2S of the deep sea increased during an anoxic interval beyond a critical value (1 mmol/kg), upwelling regions of the world ocean would become sulfidic, even with the modern [oxygen level at atmospheric pressure] .... A slightly more sulfidic ocean with H2S = 3 mmol/kg ... could sustain ... 2000 times the present-day flux and a critical value for the atmosphere (see following)."
But these critical values are never placed clearly in context. Is 3 millimols a lot or not much? Note that the 3 mmol/kg value is the absolute "best" case for their scenario; it can only be that low in (they estimate) 0.1% of the world's oceans because normal ocean requires levels 20x higher. They do admit that:
"The [H2S] condition is extreme, and thus likely to have been rarely achieved in Earth history. Is there any evidence that such conditions have occurred in the geologic past?"
How extreme is it? The value of the normal ocean would be important to have here but they don't give it to us. The paper "Hydrogen Sulfide in the Black Sea" by Volkov & Neretin does give values though, and bear in mind this is by far the most anoxic basin in the world: at a depth of 1km the value is ~314 micromols/kg. So these H2S levels that are claimed to trigger this process are literally many, many orders of magnitude higher than even the Black Sea.
The levels of oxygen drop needed is something they also don't seem to directly share, but in one section they're kicking around a figure like half of all today's oxygen having vanished, or for a different event, 99% of all oxygen having gone.
Even putting aside that these papers are just piles of completely unverifiable suppositions stacked like a jenga tower, there is absolutely nothing even remotely close to realistic about this scenario happening to us. It requires oxygen levels to drop so much that if it were to ever become an actual threat we'd all have died of oxygen starvation long before.
The solubility of gasses in water decreases with rising temperature. For ocean absorption of O2 from the atmosphere, the surface temperature is particularly important. This is why even if atmospheric O2 does not decrease, this could still be a concern -- albeit one with a different threshold than a scenario where the ocean warms and atmospheric O2 concentrations decrease at the same time.
Still, I certainly do hope that you're correct! And thank you for the thoughtful and well-considered reply!
Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!
I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions.
Since very smart human beings have in history done a lot of damage, but never ended the species or anything, we can conclude that the world has an "intelligence safety margin" that extends up to Alexander the Great or Napoleon. There have been some incredibly smart people in human history and none of them have ruined everything, a few countries at most.
I'm pretty sure they have stock arguments against what you said, even from people like Nick Bostrom who I think have a bit better handle on the situation. In that vein though, I think making the world more robust in general might be a way forward. Having multiple broad spectrum anti-virals for every class of virus ready to go and stockpiled, having flexible on-shore manufacturing infrastructure, with supply chains that are on the same continent or at least have massive component stockpiling on national levels, and having smart and vigilant people running these systems from the very top down to the janitors and security guards. Make a world where the machine stopping because a few agents, human or otherwise, doing something unexpected, much less likely. At least to the point EAs can start buying mosquito nets again.
Alexander the Great or Napoleon have been limited in how much damage they can do, because they're human too. They don't live for hundreds of years, they can't copy their brains to new hardware, and they can't modify their own minds to remove that pesky conscience when it slows them down. Perhaps most importantly, Napoleon's happiness, power, and continued survival depends on having a civilization of surviving humans working under his command, whereas that might only temporarily be true for AI.
At best, history serves as a counterexample to the idea that if an AI goes bad the attendants would just unplug it, seeing how often it is that dictators don't get stabbed by their aides as soon as they start causing mass deaths, instead often receiving broad popular support as the world burns.
>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions.
That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.
From what I've seen, the AI safety arguments basically rely on assuming that the singularity is not merely real, but guaranteed: that we will create a full-fledged general AI, that it will be smarter than us, and that it will be fully capable of upgrading itself to be far beyond our control before we have any ability to prevent it from doing so.
Every step of this relies on assumptions that are not merely questionable, but unfalsifiable.
Whether AGI is possible or not, regardless of anyone's personal opinion, is as yet unprovable and unfalsifiable.
Assuming that AGI itself is possible, there is no way to tell whether we, as humans, can create an intelligence that is "smarter" than we are.
Assuming that we can create an AGI that is "smarter" than we are, there is no way to determine whether it would be able to upgrade itself to become exponentially smarter than that, and beyond human understanding and control.
If you have a hard time coming up with arguments against these things, maybe it's because they're fundamentally unfalsifiable, which makes them useless for trying to build any framework of understanding on.
Do you think it's a strike against these points to call them unfalsifiable? They seem no more so than any claim of technological possibility, such as "nuclear fission is possible", "fission chain reactions are possible", "nuclear weapons are possible", "nuclear fusion is possible", "nuclear fusion can be made economical", etc. Unless a technology violates laws of physics it's very hard to prove through math or experiment that something can't be done.
But the AI safety groups don't just assume these without any justification. The existence of the human brain itself is either a proof and very strong launching point for most of these. In 1930, maybe you could have convinced someone that a nuclear bomb is impossible or at least an unfalsifiable worry, because one had never yet been built, but your reasoning is like trying to cast doubt on the possibility of artificial digestion, when there are already billions of stomachs roaming the earth.
The idea that a computer can't possibly be made to accomplish whatever a brain can is losing plausibility with every passing day. And as for surpassing it: a human brain is powerful, but has so many surmountable limitations, like: requiring 20 years of education to get up to speed with existing experts; dying after 70-90 years; not being able to run copies of oneself in parallel; etc. We only need to imagine removing these limitations, and doing so violates no known scientific principles.
It's like having a 50-megaton warhead in your lab, and meanwhile the rocket scientists are gradually figuring out how to make rockets fly, and you're saying "yes this is a big bomb, but the idea that one of these could be more dangerous if mounted on a missile is an unfalsifiable assumption!"
Nuclear fission, nuclear fusion, digestion—all of these are reasonably well-understood phenomena. The physics and chemistry behind them are clear. Making nuclear fusion energy-positive and economically viable is an engineering problem.
Consciousness, intelligence, sapience—these are not well-understood phenomena. We don't know what makes us conscious. We don't know if other animals are, or to what degree. It's not even possible to determine with any scientific certainty that another human being is conscious.
As things stand, "we can build an AGI" is not a scientific statement. It is not grounded on a foundation that allows clear reasoning about it, one way or the other.
Your arguments are not incorrect; however, they do not bridge the gap to "and so we can definitely make a conscious, sapient, intelligent computer." Being able to replicate particular capabilities of the brain is not the same thing.
And, again, that's only the first unfalsifiable proposition that must be satisfied in order for the purported AI threat to be real. They also have to be capable of breaking free of our control, decide we're a threat to them for whatever reason, and have the means to carry it out.
Artificially sustained nuclear fission was, in 1923, not-well-understood and thought impossible; radioactive decay had been observed but the idea of chain reaction had not yet been conceived (it would take another 10 years). Natural nuclear fusion we currently know to be possible in the sun, but we don't yet know how to sustain it artificially today. Digestion was not-well-understood in 1823; although they knew that it was somehow similar in input-and-output to combustion, which could be imitated artificially, they did not know how to replicate digestion on the chemical level. Now we do.
Consciousness is irrelevant, as it's clearly not necessary for AGI nor even human intelligence; otherwise you wouldn't say "it's not even possible to determine with any scientific certainty that another human being is conscious."
On what grounds do you believe that there are capabilities of the brain that cannot be replicated by a computer? When I look at the 1.5 kgs of matter in a typical human brain, I don't see anything that jumps out and says "my operation is not computable!"
At least with fusion, the high temperatures and pressures are a clear barrier. Our brains don't need to be held at 3.8 trillion psi and 15 million Kelvin in order to enjoy poetry.
I can certainly argue on the other points as well (breaking free / deciding threat / acquiring means) but you need to pick your goalposts one at a time. I think the first step would be asking yourself: if you were a super-genius but being held by guards in solitary confinement on a remote island with only a supercomputer connected to the internet, how would you earn money?
Those conditions clearly would not have stopped even an ordinary Satoshi Nakamoto from gaining control of sufficient resources to hire private military contractors to arrange his escape. I'm not sure what a superhuman would do, but that's a human baseline.
> the AI safety arguments basically rely on assuming that the singularity is not merely real, but guaranteed
The argument goes like this: If you want to save the world, then doomsday scenario S is the best place for you to invest your resources if S has a nonzero probability, and currently we are investing less resources in S than in other existential risk scenarios (per basis point of probability).
"AI risk" is a pretty good candidate for S (especially back in 2010-2015 when the movement was just starting.)
> Whether AGI is possible or not, regardless of anyone's personal opinion, is as yet unprovable and unfalsifiable.
"AGI is impossible" is certainly falsifiable: All I have to do is build an AGI and show it to you.
Further, there is no theoretical reason AGI is impossible. Rather the reverse; consider these Well Settled Scientific Facts:
- (1) Human Mind = Human Brain
- (2) Human Brain obeys the laws of physics
- (3) We understand the laws of physics well enough to simulate them in a computer
If you accept these three facts, then in theory AGI is possible: You could implement AGI by building a machine implementation of the Human Brain by fully simulating the underlying physics.
> Assuming that AGI itself is possible, there is no way to tell whether we, as humans, can create an intelligence that is "smarter" than we are.
> Assuming that we can create an AGI that is "smarter" than we are, there is no way to determine whether it would be able to upgrade itself to become exponentially smarter than that, and beyond human understanding and control.
An AI safety person would say to this: "You're right, we don't know -- and that's exactly the problem."
What we can do is assign a probability based on our confidence. How likely do you think it is that we can do those things? 20%? 2%? 0.000002%?
If you say "0.000002%", what makes it so extraordinarily certain it's impossible? If you say "2%" or "20%" then as a matter of self-preservation, shouldn't our society be devoting a lot of money and smart people's time and attention to figuring out how to make sure it doesn't happen?
Most of the people that could have better things to do than to respond and give it more credibility.
As well, while the arguments are logical, most of them rely upon large assumptions to move between steps. If any of these assumptions fail, the entire thing fails. Especially the hard take off assumption.
As well, the assumption that AGI is happening in the next 3-10 years. I’d say most prominent people in the AI research space don’t think we’re much, if any, closer to AGI. Yet you have Yud and LW screaming that we will all be dead in a few years and AGI is right around the corner.
When people like Chollet and Ng say we aren’t close to AGI, I’m more likely to believe they’re right, vs. Yud who hasn’t contributed to any actual developments within the field besides theorizing about alignment and how AGI can go wrong.
Yeah, I can't really argue against the whole idea. My "P(doom)"(language from the article), at least as related to agentic AI, is pretty low. About 7 or 8 percent before I die, and I'm not really old. There's probably not a lot I could do about it anyway, given that most AI jobs are alleged to just make it worse. That said, I can't really go around telling people with different numbers they should be working on antiviral research or physical manufacturing, just because I (personally, without their massive and cohesive "evidence" (i.e. gamed out scenarios) bases) think that's more important.
I'm not one of those "all value in the universe" people, P(doom) for me does not have to be that bad. I would actually consider the inability to manufacture or unwillingness to manufacture certain things at scale in the US to count. Generally, my point is that agentic AI is not something I can really do much about, and other things have a combined P(doom) probability higher than it. The unwillingness in certain groups to use what they consider to be toxic or "forever" chemicals in the manufacture of physical products at scale that can make them world a better and less fragile place is something that could be worked in instead.
My argument against is that we can't do much about it. We can't predict how machines will interpret our rules if they are smarter than us. Nor do we have a shared set of values we can articulate. Nor can we control rogue users making rogue AI.
So unless one is proposing a global crackdown on AI research, AI safety is a lost cause.
Any uncontrolled group that creates an AI can either find a body of safety research accessible to them, or not. Preparing the former is hardly a lost cause.
AI safety reminds me a lot of regulatory capture, and I don't think it's a coincidence that its top proponents are in charge of the dominant players in the space. It's easy to preach about AI safety when you're sitting on all the GPU power.
There are many, but the biggest one is just that the possibility space is vast and unknowable and while drawing a line straight through it to AI doom is of course possible it doesn't say anything at all about likelihood, and it can't. It makes a ton of assumptions and expects everyone to just go along with them to reach its desired conclusion.
AI certainly has risks, I don't think any reasonable human being doubts that, its just that the AI doom cult seems to think the worst outcomes are near certainties without really backing that up.
> AI doom cult seems to think the worst outcomes are near certainties without really backing that up
What are you basing this on? There have been tons of arguments written about why these worst outcomes are likely. Read Bostrom's Superintelligence for example, or Yudkowsky's Intelligence Explosion Microeconomics.
Many conversations with AI doomers. They gloss over and make assumptions about intelligence that aren't really backed by priors and when this is pointed out they hand wave and say "but computer".
> Read Bostrom's Superintelligence for example, or Yudkowsky's Intelligence Explosion Microeconomics.
I don't really have any interest in doing so, and if I'm honest have a particularly unfavorable read of Yudkowsky as a person based on his cultish following.
The ai safety arguments rely on something called the “orthoganality thesis” which is a huge, unintuitive assumption. In real life, intelligence is associated with picking better goals. Generally, an entity with higher intelligence will pick different goals than a being with lower intelligence.
The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable. But by using complex language, EA cultists hide the ridiculous assumptions their system has so that they can maintain their feelings of superiority while gaining real power that enables them to abuse others.
No goals are intrinsically stupid. The only reason that you might think some particular goal is stupid is that it goes against your goals.
You could conceive of a super intelligent AI that came into existence with the goal of terminating itself. That would be an "stupid goal" from our perspective, since we have the goal of self-preservation really ingrained in our brains.
But for a being that self-termination is the absolute best thing ever, it's not stupid. It makes perfect sense, since, well, that it's goal. It doesn't care about self-preservation, it doesn't care about becoming more intelligent/rich/powerful, other than as an instrumental goal to help achieve self-termination, if it's not able to do so in its current state.
And most importantly, no amount of getting more intelligent would change this fundamental goal, just as humans getting more intelligent has not overridden our fundamental goals of "breathe, feed, have sex". It may have given us other goals as well, but those are very much still there.
Not only that, but we've managed to subvert our reward functions in exactly the way AI safety people fear. Evolution tried to get us to reproduce as much as possible, and we came up with ways to get the reward without producing offspring.
That’s a pretty naive understanding of sex. In human history sex has often played as important a role in bonding, community, and social hierarchy as it has in child production. These things are beneficial for survival. It’s very common for evolution to result in redirected drives.
I’m not sure what you mean by “stupid” but it’s definitely not the colloquial meaning. A human seeking suicide is not stupid. In fact humans often do seek suicide because humans are playing social games where the maximizing move is to commit suicide. Supporting these people requires changing their context so that they have a wider and better variety of options available.
This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.
What do you mean by stupid? I used that word only because you used it first.
And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.
>In real life, intelligence is associated with picking better goals.
"Better" according to what metric?
It may be the case that there is a tendency for high-intelligence humans to pick "more enlightened" goals. Perhaps there is a natural "enlightened goals" attractor for our species.
However I don't think we can extrapolate from that to a fundamentally alien AI.
I think even if this statistical tendency exists, it has clear counterexamples -- consider that 2 genius chess players may have opposite goals, of beating one another. And we shouldn't bet the future of humanity on this statistical tendency extrapolating outside of the original distribution of human species.
Here are some intuition pumps on how diverse goals can be even across intelligent species:
As soon as you phrase goal selection in terms of metrics, you’re assuming that goal selection is based on some other goal - that is you’re already assuming the orthoganality thesis. Your logic is fully circular.
One thing that’s interesting to note about all of the examples you picked - every single one of those species shows cooperative behaviors. They share many other behaviors that are more similar than they are different. To reject the orthoganality thesis it’s sufficient to show that there is an empirical general association between intelligence and certain goals - then we can extrapolate an AI although of course it will function differently and may have many unusual behaviors will tend to follow those goals more directly. For instance intelligence is associated with : cooperation, empathy, inter and intra species communication, curiosity, etc. all of the species you mentioned exhibit these more than less intelligent species. Meanwhile something like “hunting to eat” is observed across the intelligence spectrum.
I've known lots of intelligent people who pursue stupid goals.
I maybe agree with you that there's an this belief that a maximally intelligent creature will blindly follow maximally obviously stupid goals, and that belief is under-argued, but your phrasing above isn't the slam dunk argument that you seem to believe.
Remember the term here is “associated”. If you spend a lot of time with people from diverse mental backgrounds, which is much harder to do than you might think, you’ll easily see that people who lack education, common sense, or the necessary context to be intelligent to pick much worse goals than people who aren’t in that position. Of course none of us humans are particularly intelligent, and all of us are stupid. Being a human is not intrinsically about intelligence - being a human is about being a social mammal. It’s funny seeing how rationalists often fail to exercise while emphasizing their own rationality - a perfect contradiction demonstrated by the difference between imagination and bodily realities.
> The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable.
To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense.
We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider."
> The orthogonality thesis is an unproven assumption
A safety mindset would suggest that rather than disregarding it until proven true, we should worry about it until proven false.
> meaning an intelligent being can pursue stupid goals.
It would pursue very intelligent instrumental goals, but the terminal goal is a free variable and I don't think there exists any measure by which terminal goals can be considered smart or stupid. It would be whatever is implied by its programming.
> States like that, it’s obviously wrong and laughable.
Perhaps not so obviously wrong nor so laughable as you think?
> There’s no clear delineation in real entities between instrumental and terminal goals.
In healthy individuals, yes there absolutely is. Terminal goals are the ones you pursue for their own sake; instrumental goals are the ones you pursue as part of a plan to pursue a terminal goal, or another instrumental goal which connects to a terminal goal. Most people go to work in the morning not as a terminal goal, but as an instrumental goal; employment is in service of another instrumental goal of earning money; earning money is in service of a terminal goal of not starving to death. This is not exactly controversial stuff here. Some people do get so focused on an instrumental goal like "earning money" that they develop tunnel-vision and forget what terminal goal that money was originally in service of, but that's something most of them will eventually realize and then write a self-help book about.
Anyway, it takes intelligence to decide what your instrumental goals should be, such as whether there's perhaps a cleverer way to make money than by going to work for your boss each morning, but there's no way in which intelligence will help you choose your terminal goals. For the most part they aren't something you can even consciously choose.
Yes, you can stretch your model to try to explain why humans go to work.
In reality, people do not need to go to work to “not starve to death” as you say. There are a myriad of ways to survive without working a daily job.
Humans have to be socialized and trained to work a 9 to 5 job - there’s an entire education system structured to help create humans who view that as an acceptable goal.
No what you are saying may not be controversial in your little community but the AI panic is mostly isolated to a small community in a small corner of the USA.
Between "I go to the grocery store because I've been socialized and trained that going to the grocery store regularly is a Good Thing, and I have adopted it as a terminal goal to which I know I should dedicate efforts", vs "I go to the grocery store because I'm out of carrots, my dinner recipe calls for carrots, and I think I can get some there", the latter model is not the one that strikes me as being stretched to explain human behaviour.
As for there being "a myriad of ways to survive without working a daily job", congratulations! Your intelligence has allowed you to identify alternative instrumental goals that provide a path to your terminal goal; now you can rank them and choose the best option. You can also grow carrots in the garden or ask your neighbour if they have any, or ask your spouse to pick some up on the way home. Your intelligence will do the work and find a way. But your intelligence isn't what will guide you toward preferring carrot soup over parsnip soup, and preferring parsnip soup over fasting.
You’re arguing again from a position that assumes that entities have clearly defined terminal versus instrumental goals - which is precisely the position I reject. For instance in your example of “terminal goal is groceries” versus “terminal goal is hunger” neither of these describe how actual humans make decisions. Instead there’s a process, part biological and part environmental. The human checks the fridge then thinks “oh I feel hungry”. Is hunger the terminal goal or the fridge? That question doesn’t even make sense - it’s an interaction between the agent and the environment. Do Pavlov’s dogs have a terminal goal of “salivating to bells” or “salivating to food”? Again the question doesn’t make sense- the agent has built a habit from within a certain environment and the salivating is not goal directed. That’s why training works for dogs, and putting the fridge out of site reduces hunger in humans.
Think more carefully about the implications of multiple ways to survive here. Why do people pick one over the other? In a terminal/instrumental goal model, agents would pick the instrumental route that maximizes the return on the terminal goal. In reality we see that instead humans adopt habits, processes, and heuristics that guide them through daily life even when those do not lead to any specific goal.
Yeah, that's because re-evaluating your entire life plan and belief structure every second is expensive, and heuristics are cheap. People certainly have flaws in their thinking, which is why we fall prey to pyramid schemes, gambling, responding to pointless comment chains on HN, and so on. I don't disagree with this, and again, it's why we publish so many self-help books. But I believe it's our weakness and stupidity, not our superior intelligence and clear thinking, that traps us in bad habits.
So remind me of your original point? I believe you said it's "obviously wrong and laughable" that "an intelligent being can pursue stupid goals". Now here you are trying to convince me that humans are the ones who, like Pavlov's dogs, "pursue habits, processes, and heuristics that guide them through daily life even when those do not lead to any specific goal". Even when those habits involve repeatedly re-opening a fridge that you already know has no carrots in it, or salivating at a bell when you already know no food is coming.
So I'm confused how that proves your point about AGI. If I accept your view, it seems that if an AGI does merely no better than a human on this metric, I should anticipate all sorts of strange and irrational behaviour, including the pursuit of goals that would appear stupid, such as addiction to a reward channel. That does not seem to undermine the orthogonality thesis.
And the smarter the AGI gets, presumable the less it should lean on Pavlovian heuristics and the more it should make use of clear thought, which puts it more in my camp.
So that would apparently put the lower bound at "the AGI takes unexpected and irrational actions because it's not a rational agent and doesn't think coherently", and the upper bound at "the AGI takes unexpected and dangerous actions as rational steps toward an unaligned terminal goal".
I'm not sure where in this chain of thought it becomes laughably obvious that intelligence and goals are correlated, such that an AGI's increasing intelligence will tend it toward actions that we humans approve of, because anything else would be a "stupid goal"?
Humans + AI will always be stronger than AI alone. So why be afraid of AI? Nothing has changed, humans still have all the agency and remain the #1 tangible threat to other humans which has been the case throughout almost all of human history.
What have they actually done for AI safety? Written essays and held symposiums in grand Wytham Abbey. Attracted followers and forum posts. Where's their git repo?
"Apocalypse cults are always wrong, AI safety is an apocalypse cult, therefore it is (probably) wrong" is an argument which really only betrays the ignorance of the speaker regarding the specifics of the AI doomer argument. Why are apocalypse cults always wrong? Do the same specific reasons apply to the doomers here?
Humans are attracted to certain types of mythological stories - and there’s nothing wrong with that. It becomes a problem when the stories and reality are conflated, like is happening with AI. The idea that apocalypse cults are always wrong comes from two source: (1) empirical evidence, every time there’s been one it’s wrong and (2) understanding human sociology, humans are susceptible to making certain types of thought errors as groups and apocalypse cults such as the AI one demonstrate those errors everywhere.
So yeah an apocalypse cult could be right, but it would necessarily be for the wrong reasons. Just like a broken clock that matches the current time 2 minutes of the full day.
I am well aware of the AI doomer argument, but you appear to assume that argumentation can be trusted just because it is convincing*. I believe you are not factoring in the possibility of epistemological error.
*for what it's worth I do not find it convincing but I am arguing "even if I found it convincing, I wouldn't believe it"
What's your approach? Do you only believe arguments that sound unconvincing? Or just ignore all arguments, and try to adopt the beliefs that the popular kids in the schoolyard talk about?
The first step would be to recognize the error bounds on epistemological error and redo one’s estimates, but the tl;dr is that one’s certainty on edge case predictions should drop by a lot.
Having low certainty on edge case predictions is fine, but it also should result in correspondingly larger updates when those edge-case predictions come true, which one had previously doubted. For me, that was the case with AlphaGo, AlphaFold, and now ChatGPT. In all three cases I was highly skeptical that, in my lifetime, AI would ever beat humans at Go, adapt the same architecture to problems like protein folding, and blow the Turing Test right out of the water.
I've had to update accordingly, and now I'm less skeptical that the barriers ahead will be harder to break than the ones behind us.
If there was an apocalypse cult in ancient Pompeii predicting the mountain would explode and bury everyone with ash, they would not be here. And there easily could have been.
This is not an argument, it is a tautology powered by the anthropic principle.
While you make a reasonable point in terms of human psychology, when you create an AI, you are leaving the realm of human psychology and dealing with something new that doesn’t have to play by the same rules. I’d argue that’s a big difference.
We’re creating a fundamentally alien intelligence from scratch and basically just crossing our fingers and hoping to hell it cares about keeping us alive.
On the other hand, we and we alone live in a time where a grapefruit-sized weapon could obliterate a reasonably sized town, AND when such weapons are increasingly in reach of non-state entities. I would imagine the chance of apocalypse rises as fewer people in agreement would be required to enact it.
Just because the parts of the community in the Effective Altruism / AI Safety have developed aspects of a cult doesn't mean that their arguments are wrong. It just means that they're humans doing human things. It sounds like their main mistake was to not realize that the cult-like behavior of other cults was sociological, due to the fact that they contained humans, as opposed to from their belief systems; and thus didn't take any steps to try to prevent their own movement from developing cult-like aspects.
That said, if you have a movement of thousands (?) of people all trying to figure out how to make AI safe for over a decade, and you still don't feel like you're any closer to that goal at the end of it, then you should probably step back and ask yourself what you're doing wrong.
There are top level scientistis warning of this apocalypse. Sam Altman which was the CEO of the very site you are posting and the CEO of OpenAI has warned starkly that there is a probability that humanity will go extinct from AI in his latest interview. Have you seen it? You think he is somehow bluffing or is he part of the same cult?
Sam Altman isn't a scientist but even if he was, so what, scientists have a reputation for constantly claiming the world will end unless they get more grant funding. Fear is a great motivator and makes people feel important.
I'm sure a lot of people are basing their belief on AI on sci-fi interpretation.
If you're one of the folks doing this at least acknowledge the story would have been quite boring if it was 800 pages of "everything went well, the AI turned out really great, very helpful. Would use again"
If you're not blindly afraid of zombies you shouldn't be blindly afraid of AI. Give it thought.
If you're still scared after dismissing sci-fi bias, cool, I'm all ears.
The AI safety community has many people who were previously extremely optimistic about the sheer world-changing potential of AI, but then realized that getting AI right might be harder than getting it wrong, and that figuring out how to get it right must be a priority before a superintelligent AI is made. It's not people who just saw Terminator and made up their mind.
Can you back up those claims? I don't believe all of the nuance is contained in your comment.
I'm looking into MIRI, if that's not "the AI safety community" please do correct me.
I've become very aware ever since ChatGPT started saying inaccurate things confidently that humans have a much worse hit rate.
Seriously, keep an eye out for it, you'll see it everywhere. At least ChatGPT will double check if you ask if it's sure. People tend to just get annoyed when you don't blindly trust their "research" haha.
Eliezer Yudkowsky and the LessWrong forum popularized AI safety/alignment ideas. (The Effective Altruism community was originally mostly populated by people from LessWrong.) I think this article is a little awkwardly infatuated with LessWrong, I say even as a fan, but it does fit this discussion conveniently well about where they're coming at the subject from: https://unherd.com/2020/12/how-rational-have-you-been-this-y...
While MIRI is prominently connected to Yudkowsky, I wouldn't treat them as defining the AI alignment community. There are many people not involved with it who make substantive posts and discussions on LessWrong and the Alignment Foundations forum. There are other organizations too. OpenAI considers alignment important and has researchers concerned with it, though Yudkowsky argues the company doesn't do enough to prioritize it relative to the AI progress they make. Anthropic is an AI company prioritizing AI safety through interpretability research.
I think I'm at the edge of my ability with AI as I'm noticing I'm trying to argue against the usefulness of the concepts rather than the concepts themselves. At the very least I'm not smart enough to casually read these sites (lesswrong, AI alignment) at this time.
I remember feeling like this (brain CPUs pegged at 100%) trying to slog through HPMOR the first time too in fairness, it's just too many concepts to take in in one sitting. I'll get there eventually if I keep at it but not on my first read.
I'll consider my opinions on AI safety void due to lack of knowledge for now, always try to jump over the first stage of competence. I'll start with the Wikipedia page for AI alignment, haha.
Thank you for your responses in any case, I'll dig into this further!
Oh, I think my last post was more about the people concerned with AI safety rather than the topic itself. If you want to get closer to the actual topic itself, this article is a surprisingly great resource: https://www.vox.com/future-perfect/2018/12/21/18126576/ai-ar...
IMO AI Safety is to AI as HR is to Employees. You can't just learn it out of a book and even then it's a pretty bad experience for everyone involved. The world is not a corporation and the existence of "AI Safety" just guarantees that no "mere mortals" will ever get to touch AI.
Waiting for the day our Overlords make Graphic Card 2.0 and refuse to sell any to the plebs 'cause the overlords "know better"
You’re vastly overestimating the risks but you’re also not crazy - humans naturally respect authority figures and follow their beliefs which is in fact good for a healthy life.
My biggest problem with AI safety is that, simply, the problem they envisage doesn't exist yet (generally, at a minimum, relying on the existence of "AGI"). Hence discussions about it have to make a huge amount of assumptions about a whole range of aspects of what the AI threat will be - what the AI will be capable of, what it's impact will be - before getting on to what possible solutions might be relevant to preventing it. But given the first two are so undefined, the later is pure speculation - one that is difficult to criticise directly, because any specific critisms can usually be easy deflected by adjusting any of the above assumptions without making a substantial change to the "inevitable" conclusion.
That's why it feels like an apocalypse cult to me - it's a conclusion, that has little strong evidence today, stacked on top of a constantly shifting set of assumptions, allowing adherents to avoid backing their arguments with evidence.
For the freseable future the real danger I see from AI is inferior basic statistical models being jammed into production pretending to be some sort of all seeing all knowing AGI - a class of product we are not really that close to.
I don't think there is anything safe about ChatGPT regurgitating and hallucinating false information about someone's business based on real legal proceedings that it had been trained on. [0]
As long as it cannot also explain its own decisions transparently [1], then there there will always be a need for AI safety.
I don't have any solid arguments either, but I just get the feeling that after the first very public tragedy, like a killbot rampaging in New York City, we'll see a very, very sudden tilt in the public eye towards never wanting to touch this technology again. People are terrified of killer robots.
One of the likely outcomes like that is even stronger pivoting towards walled gardens and end to end stuff than we previously had on the Internet. Can't train a model on someone else's encrypted database without decrypting it first.
Several of the accounts turned out to concern the same person, Michael Vassar, who was central ~10y ago but kicked out for this and other harmful behavior, and now describes himself as anti-EA.
On the other hand, the comment treats the cases they can't identify as not concerning from a community perspective ("it's very unclear from this what actions anyone could or should have taken") and I don't endorse that.
EA is at least partly a moral and ethical pyramid scheme. You get to be noble and far-sighted, do relatively little real work, and you’ll never be judged on the outcomes because of the long timelines. And so you rope in more and more acolytes who both increase and accrue these same benefits. It seems unlikely that the fall from grace of prominent figures will affect the movement, given that it’s partly based on cognitive dissonance as it is.
All that said, the amount of energy we expend on AI safety probably shouldn’t be zero. But I choose to believe that if Roko’s basilisk is going to torture us over anything, it’ll be that we didn’t raise up millions of AI scientists from the cotton fields and sweatshops. Credit to the less prominent altruists doing that, or who at least have some form of compound interest built into their utility function which compels them to be kind, now.
Roko's basilisk makes no sense. Sans magic physics time travel - there's no motivation. One thing all A.I. is, no matter how nefarious, is goal directed. It might have poorly aligned goals, absurd goals, dangerous goals, goals it misunderstands, goals that contradict etc. But ultimately it proceeds toward some form of goal directed reward function. There's zero reward in punishing someone for past behaviour - except in the context of altering their future behaviour.
Since the A.I. would have to already exist (and there's no reason to believe it would be incentivised to even encourage humans to create other, rival A.Is, with potentially differing goals) before it could punish anyone - Q.E.D.: Roko's basilisk has essentially zero likelihood.
Worse - it's an essentially christian eschatological idea. What is this hypothetical A.I. but an avenging angel, a righteous judge at the end of time? It's pure religion. Dark, manipulative, blood sacrifice religion at that.
> There's zero reward in punishing someone for past behaviour
A similar argument regarding Roko's Basilisk is that it is effectively a prisoner's dilemma, where the Basilisk torturing us and us not helping it is the Defect-Defect scenario. It makes no sense for either party to defect.
> except in the context of altering their future behaviour.
It really seems like a very contrived version of Pascal's mugging.
People in EA donate lots of money to causes that help people today. AI is just one of the more interesting subjects they often talk about.
AI safety isn't about Roko's basilisk, or the idea that AI will take revenge on us for our moral choices. It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being.
> It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being.
We already have those and they're called corporations. They've done significant real damage to the world already and they are still working hard to do more damage.
It makes little sense to me to focus on this potential future problem when we haven't even agreed to deal with the ones that we already have.
I'm not sure these issues are in conflict. Corporations cause lots of harm despite being held back by their dependence on people who control them, who can occasionally put the brakes on some of their worst excesses, and by their inability to recursively self-improve their intelligence. Removing these two handicaps on corporations would allow them to do significantly more damage, so preventing this from happening is important.
> inability recursively self-improve their intelligence
It is not for lack of trying, corporations modify themselves all the time. That they are quite often unsuccessful at making meaningful improvement should be taken as a prior against the idea that an AI will inherently be better at it.
I think I was misunderstood. My point is that not only do we need to consider the cases where the AI is smarter than us, but also the cases where it is dumber than we expect it.
An aligned agent will not fall into stupid errors, thus solving the lower bound of performance is a necessity for complete alignment.
Ohh, right on. Definitely agree now that I've re-read it right. I was a little confused by the first sentence of your other post and thought you were putting forward a disagreement in the second sentence.
It's more about the unintended consequences, the implied/contextual information and getting it to do what we want in a way actually fulfills our goals instead of merely satisfying a condition.
A task such as "bring me a cup of tea" has lots of implied information such as not making it too hot to burn the person, not breaking anything in the process, not harming other people in the process and so on.
I think that's a fair way of putting alignment. The concern is that we might make an AI that's capable of outsmarting us before we're good at making it care about doing what we want instead of some other goal that disregards what we really care about like our safety.
If you'd like to read more about the roots of Effective Altruism, ranging from old mailing lists, LessWrong, bitcoin, cults, AI and the surprising connections to many well known figures in tech, I highly recommend the 7-part Extropia's Childen series from Jon Evans [1]
This feels like the endless debate about whether America is forever tainted by the sins of its founding fathers.
I’m pro-American because I believe in its stated values. Moreover, I believe that despite original and continuing failures to perfectly reflect those values, there is large and concrete progress in aggregate.
I feel the same way about EA: I aspire to their ideals (including the continued discussion and reconsideration of those ideals) and believe the weight of thousands of people quietly moving concretely in the right direction far exceeds the damage of bad actors (who would have twisted any ideology they were immersed in).
So the bad actors should be punished. The parts of the community system which failed should be changed. AND the many others who are earnestly trying to live according to these values should be judged according to their _own_ impact.
I can’t imagine having the optimism you have. My rude mind is thinking this person is naive but you know I’m pretty naive too haha!
I disagree and think both the US and institutions like EA are too tainted. The US hasn’t done anything about slavery except make things worse still. That’s not on the founding fathers any more only. It’s on the US since the civil war. EA will take money from future multimillionaire and billionaires. Much like the US, they are both completely controlled by capitalism. I am in the US but not pro US.
Golly. I'm on part 2 and it sounds like they're about to invent Scientology.
> The full post goes into more depth and nuance, but it's worth stressing how heavily Leverage relied on “debugging,” based on Anders's “Connection Theory.” Debugging consisted of — to paraphrase and oversimplify — opening up one's psyche, history, self, and emotional core to maximum vulnerability, generally to one's hierarchical superior, and then doing all you could to follow their suggestions to “fix” your mind.
Incredible letter, I had no idea about the SWATing. Tbh the Epstein donation seems like a nothingburger compared to the other allegations in that letter.
Admittedly I don’t follow this rationalist thing on the internet at all, so I don’t know what’s going on with them. But that was hell of a read. I am no stranger to people being dicks, but still, that all is just sick.
This is all cult-like behavior, reminds me of the much documented Lyndon Larouche 'creative mentation' stuff, and pretty much any cult in their stage of grilling people to take down their preconceived notions.
Now with this info, that's all I need to know about these characters.
According to a study of 700 ML researchers (not AI safety) the median chance that that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)" is 5%. And 48% of respondents gave an answer of 10% or more.
With great power comes great responsibility. If we start plugging ML everywhere - which we are currently doing - we better be sure that it is behaving as expected.
So far we have seen no power at all from ML. So spending large amounts of money making it "safe" (not that anyone knows how or even what it would look like) seems like an over reaction.
With no power, comes no responsibility.
But of course, if ML is a person's pet interest, they will likely think we should increase spending on it. That doesn't make that a smart decision...
I believe that you conflate “power” with capabilities wrt AGI.
I’d argue that power is in relation to reach and the capacity to influence. In this regard, ML has quite a bit of power these days, from deciding whether you are allowed to take a loan, to driving engagement in social media, to predicting and influencing what you will watch, to the way our language is even used.
There are also quite a few implicit ways that ML models have influenced society, eg GPGPUs with emphasis on ML, and ML accelerators everywhere. The effectiveness of certain algorithms made us use them more often which influenced NVDA and Google towards developing hardware to accelerate those common cases thereby creating a feedback loop where our algorithms are chosen based on works well on our GPUs (cf transformers).
As such systems become more prevalent and influential, ensuring safety and explainability will help us prevent pitfalls that put humans at risk.
We can argue back and forth about the definition of power. Right now, ML has had very little actual impact. And it is very far from the sort of "skynet" general intelligence that people are afraid of. Those are just facts.
Personally I view it as no different to the hype over crypto or the fact we get a huge breakthrough in fusion every 3months but somehow never get any closer to an actual commercial reactor...
Meanwhile, we KNOW as a fact that climate change is going to be devastating. And there are numerous areas with nuclear wars brewing from far east to Europe.
I get that a super AI is a more interesting thing to worry about. But we need to deal with out actual, serious, issues first IMHO.
Given that ML has been exploited to drive polarization which fuels said wars and causes instability, can you claim that there is no impact with respect to said wars?
Power is not just about being more intelligent than us. An isolated airgaped super intelligence is powerless.
I had previously heard of EA, and knew about paperclip maximizers, trolley problems, unintended consequences, and other risks associated with unregulated AI. But I wasn't aware of the connections between the two communities / families of ideas until recently.
Because I don't consider myself sufficiently educated in these matters, I default to using GiveWell and their mosquito net recommendations as an effective low-effort approach to making monthly contributions to charity.
Overall, is there anything about this approach which is objectionable? How do philosophers go from something like GiveWell to something like fending off the AI apocalypse? Is this ideological expansion inevitable?
EAs premise includes basically "all human lives are equal. Saving one stranger's life is just as valuable as saving your father's life (more valuable actually if the stranger has more years left than your dad), so you should donate your kidney to a stranger, not your father"
"All human lives are equal" then expands to "and there are more unborn lives than there are currently living lives, so it's more important to save unborn kids in 200 years by diverting AI catastrophe now".
I personally don't think unborn lives are worth anything, and as such I don't follow that jump. I don't feel a need to justify it either.
I think there's two additional easy ways to veer off the EA course.
The first is "how do we know EA is right about the charities?" The answer you'll get is basically "Sure, you can donate a kidney to your father and know it saved his life, you can verify that... but the people verifying your EA kidney donation does more good are smarter than you. They're rationalists. They do math. They know selling your kidney, which would have only saved one life, to a black market and getting enough money for 50 mosquito nets, which will save 20 lives, is the right deal. The spreadsheet says so."
I do not buy that the people running and choosing EA charities are actually the smartest people, especially when it comes to having a positive impact on lives. Sure, they talk a big talk, but I know for a fact that when I give money to a local shelter, the money gets used for good. I know for a fact that when I donated to a certain EA charity I won't name, the money went into "AI research" which is to say into the pockets of a rich white dude. I know for a fact that the average EA rationalist I have met does not come off as a caring human.
A second way that one can veer off the EA course is by deciding that one values some lives more than others. For example, if I live in a small city, I may decide I value the lives of orphans in this city more than the lives of the orphans in some third-world country. Sure, feeding an orphan in africa is cheaper so EA would say do that, but I may decide that the intangible value of improving my local community is worth more than "getting the most bang for my buck". I think this is also a totally reasonable argument, and in fact I would much rather live in a world where everyone tried to improve their local community than one where everyone used a spreadsheet to try and maximize their "best improvement per dollar".
How they value unborn future lives also veers into absurdity when you realize that the logical goal from a utilitarian perspective then is to populate the universe with trillions upon trillions of humans, just as long as each human at least has some infinitesimally small net happiness. You don't need them to be particularly happy, just happy enough to barely not want to kill themselves.
I think I'd rather prioritize making the billions on earth we have right now very happy instead.
Not all utilitarian roads lead to the repugnant conclusion. Ex. being "in favor of making people happy but neutral about making happy people" is another valid answer. A pure negative utilitarian is the other extreme - we shouldn't make any more lives at all, because every new person (& animal?) will inevitably suffer during their existence.
That's a logical consequence of one possible view in population ethics, which is orthogonal to utilitarianism.
There are equally counter-intuitive consequences of discounting future lives or not valuing them at all. What makes this specific point in time so much more important than every other point in time? Can we do anything we want to make our lives better regardless of future consequences?
Many EA people are into forms of longtermism but AI safety arguments don't rely on longtermism at all. Many people worried about AI safety think it will be relevant in well less than 200 years. (Though uh, would you really not care at all if the world ended in 200 years? This sounds much more alien to me than any form of longtermism, and I think most people would agree that society should make decisions to sustain itself.)
> How do philosophers go from something like GiveWell to something like fending off the AI apocalypse?
It's simply that writers like to write, forum commenters like to comment, philosophy academics like to publish papers, and there's only so many ways you can rehash The Life You Can Save.
Advocating for mosquito nets and praziquantel is like advocating that programmers should use version control - it may be true, but it's not going to drive traffic to your blog or impress people with how forward-thinking you are.
The people who haven't expanded their ideology in debatable directions aren't getting to the front page of HN.
It's just super obvious by this point that none of the AI safety LARP is really about safety it's more about justifying why a technology that to hire the team to work on it requires that work to be published (Apple tried to do it without publishing and struggled to get the talent) but by publishing you lose part of your advantage so the only way to lock that advantage up is to make it so use of the models has to go through you and also trying to push for reasons for others to not have access to the tech required to train their own model.
If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100.
| If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100.
Fuck, really? I hadn't heard about this. If true, it's the godfather of all ladder pulls.
Not to mention, it's categorically awful to silo this stuff and grant it to an org that's so opaque. Good luck separating the wheat from the propaganda chaff without access to the tools that produce it.
I'm leaning towards this as a big reason. PhDs really want to publish and they tend not to put as much effort into or not find interesting work that cannot be published. The trend in academia is towards making your code and data available but now you can now get around that by saying AI safety.
EA as a concept is totally fine and the risk of runaway AI still seems rather fantastical to me. Clearly a more pressing problem is runaway egos latching onto these ideas to inflate themselves to deadly girths. I'm sure some people can practice a totally constructive form of EA. But if you structure your entire identity around it, clearly you've drifted somewhat from rationality.
YMMV. I tried to finish it based on a strong trusted recommendation, but could not get past the insufferable author-inserts, and the unpleasant and unbelievable hero character.
I too didn't finish it - assuming it's "Harry Potter and the Methods of Rationality".
I got caught up by the multiple times the main character made sweeping assessments of things with insufficient information, and just happened to be right by accident (or author fiat I guess). The main characters figure things out by Thinking Really Hard, and experimentation is only ever done to prove they were right all along to other characters.
That's an arrogant character who keeps falling into the right answer, not a scientist.
It went above their head because it was poor writing in their view? Why would you think taste in writing has anything to do with their ability to understand the points the writer was trying to make? This seems unecessary and you are probably capable of coming up with a more interesting response.
It's commonly thought (and obvious) that Yud wrote the book about himself so I doubt it. It doesn't really fit the rationalist playbook either, they take themselves excessively seriously.
He's too fragile and prone to overcompensation for that, nice try. Of course he's going to back track after getting mercilessly mocked for it and then claim it was just a joke.
The first few chapters have a song scene parodying ghostbusters and a joke about the protagonist biting a teacher in a tantrum as a young kid. There's plenty of elements of crackfic parody in it.
At the end Harry is basically a Messiah that can order the magical world to provide emergency medical aid to everyone in the whole world and turns Hermione into basically an angel; not really seeing the story itself saying he did anything wrong...
Nah, HPMOR just sucks, plain and simple. There was no need for a HP fanfiction in the first place. Also, if you ever ask anyone that reads fanfiction, no one will ever have a positive opinion of HPMOR. There are millions of HP fanfics, and there are many that are much better than HPMOR. My guess is that it was set in the HP universe just so it could be popular.
My strong belief is that HPMOR is a Harry Potter fanfic not for any calculated reason, but just because the author was in a demographic maximally interested in making Harry Potter fanfic.
he has literally stated that's why he did it as a Harry Potter fanfic. "There's a large number of potential readers who would enter at least moderately familiar with the Harry Potter universe." He was also reading a lot of HP fic himself, so he knew the stuff.
I think that Eliezer is very well-situated by both age and social niche to find HP fanfiction vastly more appealing than most human beings do. I think that he genuinely finds the idea of "Harry Potter but more as a hard fantasy with a hero bent on breaking the setting by using everything without regard for genre convention" interesting as a story concept. And when he set out to write up his ideas about rationality in the form of something other than dry nonfiction essays, I think that HP fanfiction sounded good to him.
I don't think that he correctly found the medium that is maximally appealing to the potential target audience for his ideas (like, I definitely do not want to read a super long Harry Potter fanfic, and I think I'm joined in that by essentially everyone who's not a very geeky millennial), and I don't think that he gritted his teeth and wrote a HP fanfic in order to increase the audience for his ideas despite not liking HP fanfics.
> I definitely do not want to read a super long Harry Potter fanfic
This hits the nail on the head for me.
I mostly enjoyed the books as a kid, but didn’t really maintain being a Harry Potter “fan” into adulthood. Out of the dozen or so “hardcore” millennial nerds that I know really well, there’s exactly one person that I think would attempt to read something like that, and based on the reviews I’ve read, I would bet that she would struggle to finish it, if at all.
There is a somewhat interestingly vocal group of internet adults that insist that the default mode for nerds in their 30s is to be Harry Potter super fans, some of whom lapse. Like we were all raised Catholic and some stray.
I was a big fan of Animorphs at that age too, but you won’t see me going around reading or writing Hork-Bajir/Andalite fanfic. Because I enjoyed it as a kid and then went on to read other books.
wait, really? I scrolled right past someone posting about "Harry Potter rationality fanfiction" and thought, "huh, oddly specific burn but it's chuckle worthy".
it's an actual thing?
This is a joke, right? tongue in cheek or something. Or should I go get some Brawndo to water my basil...
It's not really about rationality past the first four chapters or so, after that I don't really know what its about but when part of the plot is helping Voldemort break people out of azkeban then rationality is clearly not an important theme.
I think the first chapter is really fun, then it loses steam and by something like chapter six it has fukky jettisoned the promise chapter one sets up and it is strictly in "ok to terrible" territory for the rest of it.
I disagree, Atlas Shrugged was much better, used an original premise and had an interesting phrase throughout ("Who is John Galt?").
HPMOR is really garbage-tier. It's parasitic on another series, doesn't have much in the way of a good story, and quite legitimately makes shit up. There are much better HP fanfics (and indeed books) written by people that are not as "smart" as Yudkowsky and also didn't have an agenda to push.
This article is such a mess. If you are alleging something, then say it clearly.
Judging by my participation in lesswrong and others, there are both feminists and sexist people, just like all other online forums. People are more free to be sexist online and every close knit online community has fair share of sexists.
The EA people seem to have more than a spurious relationship with polygamy, drugs, orgies and odd-ball ascetic leader figures, doomsday prophecies and psychotic behavior. Taken from the article
" MIRI employee Jessica Taylor had a job that sometimes involved “imagining extreme AI torture scenarios,” as she described it in a post on LessWrong—the worst possible suffering AI might be able to inflict on people. At work, she says, she and a small team of researchers believed “we might make God, but we might mess up and destroy everything.” In 2017 she was hospitalized for three weeks with delusions that she was “intrinsically evil” and “had destroyed significant parts of the world with my demonic powers,”"
This is like what, one bad week away from AI Jonestown? At some point this isn't a community but a cult, and women usually tend to draw the shortest sticks in any cult environemnt with bizarre moneyed, male leader figures.
Angel here would be an "aligned" AI: an AI whose behavior is in line with human values. A demon is simply an unaligned one.
The key point would be that it needs not be actively hostile to human values, but indifferent to them. Think of a Lovecraftian Elder God, not AI Hitler.
The supporting hypotheses would be that it's much easier to create an unaligned AI than an aligned one, as an aligned one is an unaligned one with additional constraints, and that unaligned AIs will wreak massive destruction.
Could a kind user please have a second look at the comment I posted earlier in this thread and which has been flagged to death? That was meant as a riddle.
Are riddles not OK for HN? Or should I have made it more clear that it is a riddle?
It looks like a hit piece. Someone is against rationality regarding the question of AI safety (any real practical consequences is to censor AI under _pretense_ of AI safety, Perhaps, the articles are the preventive strikes to discredit the AI safety experts that might point out the obvious).
That's a rather specific accusation to make without evidence. Possibly, a journalist might have other reasons to cover a story? Not necessarily good ones, but still...
Effective Altruism, like and philosophy is (a) only as good as it's practitioners and (b) often used as a cover for charlatans. That doesn't make it wrong perse. Just (like all ideologies) buyer-beware.
Some of that is just typical cult behavior. Scientology and Ayn Rand followers have had similar problems. Drugs usually make it worse.
AI safety is a real issue, but look at what it looks like so far.
- "Machines should think, people should work". That's Marshall Brain's "Manna". It's also working at an an Amazon fulfillment center, where the computers tell the humans what to do.
- "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations.
Those are the classic problems. As the technology advances, we're discovering new ones in the large language model space. Those issues are well known here.
>- "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations.
Corporations are not capable of having the "fast takeoff" which is an integral part of the doomer narrative here. Furthermore, a corporation being a paperclip maximizer is an abstraction that obviously falls apart in the extremes, if for no other reason than because at some point destroying the world for the sake of more paperclips is going to have a negative ROI....
Meanwhile, the (AI) paperclip maximizer is actually trying to maximize paperclips and does not have the sort of internal checks that would apply to real-world orgs. That is the crux of the problem.
Corporations, when first developed, took out large chunks of Asia before anyone could stop them. The main difference is that corporations have to be powered by people while an AI could theoretically work with complete independence.
I think the argument is that due to emergent behavior and possibly perverse incentives corporations are indistinguishable from slow-moving AI -- they can be treated like independent agents.
Elsewhere in the thread, it's brought up that AI could move faster than corporations generally move. That might be true, but there are still physical limits: the speed of light, moving mountains of steel takes energy, etc.
At some point it had more soldiers and controlled a greater subject population than the government of Britain itself.
I'm not sure it was ever consistently profitable, though, which is the main paperclip corporations ostensibly maximize. Which itself speaks to the problem: superficial objective functions don't always determine outcomes or behavior.
Beic was only a corporation in the most technical sense of the word. It became so powerful because the Queen had given it a government enforced monopoly from the start and the government kept extending that monopoly in return for loans and shares of the profits. The best modern equivalent would be Saudi Aramco, which also has its own "army", but it's meaningless to compare these sorts of organisations to modern uses of the word consideration. They're much closer to government departments.
What exactly is the argument that corporations are incapable of unbounded exponential growth contra a possible future AI? Is there just something magic about computers, or am I missing something obvious?
You can find discussions on this by googling "AI Foom" - the key idea is that an AGI that can recursively self-improve will be able to rapidly escape human control before any entity could check it, and very likely without humans even knowing that this had happened. Such an AI would likely consider humans an obstacle to its objectives, and would develop some sort of capacity to quickly destroy the world. Popular hypothesized mechanisms for doing so involve creating nanomachines or highly-lethal viruses.
That's key to the story of the paperclip maximizer - the paperclip maximizer will go about its task by trying to improve itself to be able to best solve the problem, and once it's improved enough it will decide that paperclips would be maximized by destroying humanity and would come up with a plan to achieve this outcome. However, humans may not realize that the AI is planning this until it's too late.
> What exactly is the argument that corporations are incapable of unbounded exponential growth
Individual corporations aren't capable of unbounded exponential growth because they can't keep the interests of the humans that make them up aligned with the "interests" of the corporation indefinitely. They develop cancer of the middle management and either die or settle into a comfortable steady-state monopoly.
Market systems as a whole can and do grow exponentially - and this makes them extremely dangerous. But they're not intelligent and so can't effectively resist when a world power decides to shorten the leash, as occasionally happens.
Corporations innovate through the work of human minds. Corporations improving doesn't cause the human minds to improve, so there's no recursive self-improvement. Corporations today still have the same kind of human brains trying to innovate them as they did in the past.
A human+ level AI would be able to understand and improve its own hardware and software in a way we can't with our own brains. As it improves itself, it will get compounding benefits from its own improvements.
Top level scientistis warning of this apocalypse. Sam Altman which was the CEO of the very site you are posting and the CEO of OpenAI has warned starkly that there is a probability that humanity will go extinct from AI in his latest interview. Have you seen it? You think he is somehow bluffing or is he part of the same cult?
A recent poll of published AI researchers showed a median estimate of a 5% chance AI causes human extinction, with half believing the chance is greater than 10%.
it's a little annoying that the "AI Impacts" paper seems to be three MIRIs in a trenchcoat, considering that this article is about EA's AI agenda and by extension Singularity/MIRI. There is a slight conflict of interest here - as the publishers of the paper are also taking funding specifically to "deal with the threat" they're publishing on.
Yes if we have scientists that are getting paid to research a certain risk it’s inevitable that they have conflict of interest. Otherwise how would it be done?
I'll admit, there isn't actually a good solution to this. It's one of the core failures of the way we organize society and its resources.
I find this kind of fascinating, really. Because of the way we sort work and create institutions, there's a structural hook that ties in the persistence of the problem (or the creation of a perception of the problem) with the longevity and motivation of the institution / entity that is attempting to solve it.
Yeah, you could contract out to a third party (which would be better), but you're still establishing a relationship where their future earnings are dependent on coming up with a certain result, unless everyone involved is incredibly self aware and conscientious and is willing to work to put themselves out of a job. Not impossible, but a rarity.
Take the DEA, for example. Every indicator points to deprecation as the way forward. The science says that treating drugs like a crime is ineffective and creates all sorts of externalities, and there isn't even broad public support for the War On Drugs. It's expensive, ineffective, dangerous, wasteful. And it keeps an absolute fuckton of people employed, at all levels. These people have a vested interest in maintaining the status quo, if for no other reason than their positions afford them power, quality of life, etc. So they lobby to keep stuff illegal and go on campaigns attempting to convince people that they'll peel themselves if they get high.
I'm not suggesting that EA and this AI thing is comparable to the DEA, mind you. If given the choice to keep one I'd be hanging out with you guys. That said, I'm ever leery of surveys in general, and there's a recursiveness to using your own data to back up your claims that doesn't sit well. If it's truly something that warrants money / time / attention, there's gotta be external sources that corroborate - Which I don't necessarily doubt, but that's not the thing we're looking at, here.
It’s interesting to me that the comments of the HN crowd seem to focus on Effective Altuism and AI safety.
A ton of the article is about immoral behavior of men in positions of power towards women. And the misogyny.
> The underlying ideology valorizes extremes: seeking rational truth above all else, donating the most money and doing the utmost good for the most important reason. This way of thinking can lend an attractive clarity, but it can also provide cover for destructive or despicable behavior.
I doubt very much that the poorest people on this planet care much about AIs. People living in absolute poverty in Africa, India, China etc. I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world.
Places where AIs aren't used to command peoples lives will probably not even realise that an AI apocalypse had happened. Perhaps a case of western navel-gazing?
> I doubt very much that the poorest people on this planet care much about AIs.
Whether people know or care about a risk doesn't mean that the risk won't affect them!
> I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world.
A rogue AI might as well just kill everyone rather than just the Western naval-gazers, as this would bring to 0 the possibility that a human could ever turn it off / thwart its plans.
Get access to the internet, get money. Money buys it power to get people to do things in the real world. Or it can hack through systems and blackmail and coerce people. Now it can act in the real world by getting people to do things it wants.
As one example it could design, synthesize and spread a highly contagious and lethal pathogen.
By that logic, uncontacted tribes shouldn't be negatively affected by the modern world, and the Sentinelese don't need to concern themselves with sea level rise even if their island is completely swallowed.
In order to make an iPhone there was the basis of cell phones, and telephony and radio in general. And the mining of materials that such tribes may not be in proximity to or be able to mine at such intensities given their low populations and tech levels.
For some reason you seem to expect a civilization with the technological sophistication of globalized society with 10,000+ years and billions of people and countless global resources fueling them to suddenly emerge in the small geographies and populations of uncontacted tribes? If you were trying to be witty, you came up short.
Edit: or were you trying to play off "negatively affected by the modern world" as, "well they don't have iPhones so they don't have the Social Dilemma, everything's peachy" instead of the blatantly obvious "we destroyed their world and they had no clue" angle...
I'm assuming you read it, so why does Harry Potter (a 11 year old in 1991/1992) think polyamory might (or probably would be, looking at the text right now) be a good idea in a future where people live a long time, in said fanfic? I think it's pretty dubious, but the other 11 and 12 year old girls in the same fanfic also seem to be fans of the idea of yaoi like situations, so all semblances of actual historical events are probably gone at that point. This is the UK in 1992, not the US West Coast. To give my personal impression, when I read some science fiction that had some suggestions of polyamory in it in a similar situation, I did not think it was a good idea. I know the whole thing is essentially a morality story and for various plot and author-community communication reasons the "Harry Potter" in the story does not behave like J. K. Rowling's Harry Potter with different adoptive parents nor like any person, child or otherwise, could behave in 1991 and 1992 within the realm of probability, but I think the whole thing could be done a lot better even given the various constraints. How about he actually does science experiments for more than two chapters instead of going on stupid adventures that should probably be left to the JKR books. Cut the time travel as well, figure out some other reason for constraint solving to show up.
Yes, spoilers though, but I don't actually think that explains it. Are there any indications in cannon that Tom Riddle was ever interested in polyamory? I'm pretty sure voldemort is not interested in other people at all after his soul gets all divided, but I don't know about before that. Riddle is not from the US west coast either, so it's kind of dubious.
I don't think the story is trying to take the characters from cannon, but take the world and completely reimagine the characters. The original characters just serve as foils for their reimagined versions. I never really understood complaints about fanfic authors not sticking to cannon character traits. Isn't the point to create new stories that are loosely linked to the original? Anyway, HPMOR is more pop philosophy novel/religious text that attaches itself to a well known world for publicity than any traditional sort of fan fiction.
I guess so. As I said before it's essentially a morality story, but in that sense I think it's evidence for the author having a bias toward polyamory, because there's very little or no in-universe reasons, and there's no broad social acceptance problem in not having polyamory mentioned at all. As an example, I think there would be social acceptance problems if the story had no women or clothing in it. Polyamory is no where near that level.
I have always felt that the embrace of "AI can be dangerous" comes from an elitist fear of capitalist class that the "unwashed masses" will have too much access to technology and will make capital, as well as labor, obsolete.
I mean if everybody has a super-smart personal assistant in their home, who thinks independently, why would they need or want a ruling class, when we can simply have democratic socialism?
What about a future where only energy and resources are important leads you to think we’ll have less inequality? Brains are the only thing that don’t flow to the wealthy right now
Human nature. If you're a part of the common tribe, people are actually pretty good. There doesn't seem to be an upper limit on the size of the tribe (although some people just wanna hate somebody).
You can also look at it this way. The only real objection against socialist societies was that people are lazy, and don't care enough about producing stuff or resolving economic inefficiencies. But with AI, this objection is gone. Nobody really wants to be in a constant "resource competition" or whatever, it's an effort. People generally want to have fun and enjoy life.
Humans ability to push others out of the circle of concern is near limitless. We literally bought and sold people. I think there is reason for hope. We stopped doing some of those terrible things and only partially for economic reasons, but ai still fills me with some great fear. It shrinks the circle that the wealthy absolutely must keep in their circle, in their tribe, to go on and live happy lives and people of the past have been uncomfortably good at excluding people from the circle in the past. Basically it might not destroy us, but I certainly don’t see it as inevitable that it will bring us to utopia
> Humans ability to push others out of the circle of concern is near limitless
In theory, yes. In practice, the circle has been continually expanding with technological progress.
Stated differently: What's the point of being powerful if you cannot rule over other people? What's the point of having thousands of empty palaces, when no other humans see them or care?
> In theory, yes. In practice, the circle has been continually expanding with technological progress.
Only if by continually you mean in sum. The holocaust and the genocide that have come since were pretty huge back slides and the tran Atlantic slave trade was a fairly new innovation in scale and kind even if slavery had existed before. Both of those seem if anything enabled by technology. And certainly the rich and powerful are unlikely to to kill absolutely everyone else, but I thought earlier you were arguing that AI would see the end of the rich and powerful. Now you're just saying they'll want to have pets
I think that's dubious, most people's motivations are only loosely coupled with the goal (if there really is one) of making the world a more efficient place. I guess that's what's good about capitalism, at least in the End of History sense[0]. Gets more people more aligned with general efficiency. Anyway, guess what's not efficient, at least in the standard economic sense? Stockpiling for a pandemic. Got to have a bit of a balance there.
[0]: i.e. the system in the US and western Europe that prevailed over the USSR at the end of the cold war, technically not "real" capitalism.
Right, but one of my points above is that AIs would quite possibly miss things or fail in general to make the comprehensive and sophisticated tradeoffs between efficiency and fragility on one side and inefficiency and robustness on the other. They could also miss cases where being less efficient is more fragile and being more efficient is less fragile. I really don't think leaving everything to AIs is a good idea, and having a whole lot of them could cause weird interactions to develop that would need to be monitored by humans.
Ever heard of reinforcement learning, AlphaGo, AlphaStar? AIs can already make that trade-off and rather well. AlphaGo was actually better than humans to play robust moves that cement its victory than riskier moves that increase the winning margin.
This seems a bit too cynical of a take to match what I've seen. I see mainly two distinct "AI can be dangerous" camps: the EA-adjacent camp centered around LessWrong and Eliezer Yudkowsky, and the anti-generative-art camp worried about the threat to artists' livelihoods. The latter seems to be the closest to wanting an outright ban on the technology despite not being anywhere near the capitalist class. The former seems to take the stance that "we NEED a plan, and no one seems to have one that plausibly works", and is calling for more research into such plans rather than stopping the technology itself (partly because they see stopping it as an impossible feat). It hardly seems like they're doing this out of fear of anything but actual catastrophe.
They have a business model problem. A century ago, recorded music obsoleted low-level musicians. Over the last two decades, the World Wide Web obsoleted most journalists and printers. Artists now face a similar problem.
An insight from ChatGPT is that "creative" is easy, but "accurate" is hard.
The answer there is simply that they make more money by keeping it closed. Why release something publicly if it means you lose your cash cow? No fear of the poor making money needed.
* Founding and scaling GiveWell, synthesizing and analyzing global poverty research to make recommendations. This has helped guide about $1B+ in really valuable work, the largest portion in anti-malarial bednets. https://www.givewell.org/about/impact
* Convincing Dustin Moskovitz and Cari Tuna to allocate their giving along these lines, building Open Philanthropy which has made a lot of grants that look really valuable https://www.openphilanthropy.org/grants (this is in addition to the ones where they granted at GiveWell's recommendation -- don't want to double count).
* Michael Kremer won a Nobel prize in econ (shared with Esther Duflo and Abhijit Banerjee who I'd describe as EA-sympathetic but probably don't consider themselves EAs?) for global poverty research.
* Built an "AI Safety" subfield of computer science, with the 2016 paper "Concrete problems in AI safety" having 1.8k citations.
* Founding and scaling 80,000 Hours, helping people figure out how to use their careers to do more good. This includes looking into potential career opportunities, writing up public advice, and 1:1 advising, and 1k+ people have cited them as part of why they changed what they were working on. https://80000hours.org
(Not especially representative; there's been thousands of people working along these lines for years and I'm sure I'm missing a lot of valuable work.)
Purchasing large mansions[0], chateaus[1], and spending lavishly on administrative kickbacks[2] because longtermism can justify any present action if you extend your horizon far enough. EA is effectively two disconnected ideas at this point.
Thanks for sharing that reference. I started out being intrigued, but he totally lost me with some pretty unsubstantiated absolutist statements, e.g. "anyone smart and independent-minded will have views on race and sex well outside the liberal mainstream" and "men are sure to be a disproportionate share of leaders for biological reasons".
The first statement is categorically false or otherwise implies the author's engaging in "no true Scotsman" fallacy; and, in the second quote, how he can disentangle biological from cultural and social causes with such absolute conviction is mystifying. Stating opinions as facts in such a way is a kind of sophistry, and ironically this tendency limits one's ability to reach deeper truth.
The photo of Aella in a bra with the caption "The rare rationalist female" wasn't a red flag? More seriously, yes that piece is dripping with sexism.
>Women will never make up a significant portion of self-made billionaires or outstanding philosophers, but they can get a majority of BAs. Academia practices affirmative action in everything from admissions to journal submissions and citations practices, artificially reducing the natural male advantage at higher levels of ability.
>how he can disentangle biological from cultural and social causes with such absolute conviction is mystifying.
He doesn't have to disentangle the two completely, he just strongly believes that there's a non-trivial biological contribution to these outcomes. Which already puts his views "well outside the liberal mainstream".
> Does this mean that Rationalists are the only people uniquely free of status concerns and therefore able to discuss important issues in intelligent ways? To a limited extent, yes.
Lol. “Only people who agree with me are capable of rational thought and discussion.” The philosophy of and for internet trolls.
I'd submit that this is actually the universal philosophy of every cult that's ever existed, rationalism included. Suspension of belief because of some special property, maybe the argument is that the leader is actually god, or simply free from status concerns thus a better and clearer thinker.
I read Hasania's article hoping for some solid material to make me feel better about the point a few years when I first discovered Less Wrong.
Instead, I just read one of worst articles I've read in at least a month, so full of not just poor (and transparently polemical) writing, but poor reasoning and blind dismissal of opposing viewpoints, regurgitation of unsubstantiated bullshit, and more.
If that truly represents the best anyone can do in defense of Less Wrong / EA, then it would appear that the complaints about both are substantive.
Caring about one thing substitutes for caring about other things. Shocking.
Most charitable foundation on the left and the right may focus on one problem but make nods to it's connections to others. I know a lot of lesswrong people acknowledge climate change is real and Covid is a problem but raise the point that "this is so important we have to ignore other potential problems" but by doing this they alien those who might care about their important problem.
It's funny 'cause Eliezer and other have realized that the way corporations go about things means they won't care about safety here, just as they don't about safety anywhere else (say East Palestine). Yet these people have burned all bridges with those who are generally concerned about corporate behavior.
(I'm personally not as concerned only because I'd see AGI as further than they imagine but if GPT-X could unsafe, then Eliezer's correct you might as well dig a hole somewhere).
>Yet these people have burned all bridges with those who are generally concerned about corporate behavior.
Well, because there's the obvious risk that EA/rationalists just become another blue tribe identity group if the focus on actual effective altruism is replaced with unquestionable dogmas, which is really the only way you get stuff like people believing that climate change is going to lead to civilizational collapse by the end of the century or that donating to BLM is a good use of charity dollars or whatever. I don't know where the margin of optimal pandering is here, but I'm pretty sure that bending the knee to whatever the blue tribe's moral panic du jour is is going too far.
>>Every AI safety researcher knows about the paper clip maximizer. Few seem to grasp the ways this subculture is mimicking that tunnel vision.
>Lol really?? You're comparing some bad PR incidents to the literal unintentional destruction of life on Earth
Your reply is a perfect example of the blind spot the quote from the article is referring to.
"some bad PR incidents" being sexual harassment and assault, facilitated by the in-group doctrine, and the Paperclip Optimizer being a central tenet of the in-group, while for the real world it's a quaint and somewhat interesting hypothetical about a far off future.
So we have real, actual harm hand-waved away by reference to a hypothetical.
>So we have real, actual harm hand-waved away by reference to a hypothetical.
Yes, because that harm is not even close to the same order of magnitude as the harm that rationalists are focused on, even if you discount it by a moderate likelihood factor. It doesn't matter if the "real world" sees AI doom as a "quaint and somewhat interesting hypothetical about a far off future" - unless you're actually agreeing with me that this is primarily a PR issue.
When rationalist sexual harassment starts threatening the survival of the human race (in a way other than raising the probability of AI doom via bad PR for rationalists), the tunnel vision critique may become more than laughable.
It's like "the ends justify the means" except the means have nothing to do with the ends at all. The supporters just keep repeating themselves about saving the world or something.
Yes, well, I guess I just think that a rogue AI inadvertently destroying the world is a somewhat more worrisome form of tunnel vision than having a "blind spot".
The real and the only relevant question is: do EAs rape/sexual assault more than the the average population (and if the answer is no (it is), then we need more EAs).
It encourages mortal men to play God at levels of complexity that they can't possibly comprehend.
It should be banned for the greater good
>> “earning to give,” the idea that people like Bankman-Fried should do whatever it takes to make a lot of money so they can give it away. To amass his billions, Bankman-Fried allegedly defrauded his customers, and critics have said his downfall shows that EA is vulnerable to a myopia that allows the ends to justify illegal means. Among EAs themselves, however, the most salient criticism is subtler: that living at the logical extremes of the ideology is impractical and a recipe for misery.
Utilitarianism is false, but I think it is a very useful heuristic, where I think it being used (as a heuristic) more often would probably be beneficial at the margins.
There is no canonical way the aggregate utility between persons, and people probably don’t actually have utility functions. And, there are moral laws which should have at the least some weight when weighed against a “but this action seems to have better outcomes overall, on average”, and possibly infinite weight when weighed against such things.
