EA is at least partly a moral and ethical pyramid scheme. You get to be noble and far-sighted, do relatively little real work, and you’ll never be judged on the outcomes because of the long timelines. And so you rope in more and more acolytes who both increase and accrue these same benefits. It seems unlikely that the fall from grace of prominent figures will affect the movement, given that it’s partly based on cognitive dissonance as it is.
All that said, the amount of energy we expend on AI safety probably shouldn’t be zero. But I choose to believe that if Roko’s basilisk is going to torture us over anything, it’ll be that we didn’t raise up millions of AI scientists from the cotton fields and sweatshops. Credit to the less prominent altruists doing that, or who at least have some form of compound interest built into their utility function which compels them to be kind, now.
Roko's basilisk makes no sense. Sans magic physics time travel - there's no motivation. One thing all A.I. is, no matter how nefarious, is goal directed. It might have poorly aligned goals, absurd goals, dangerous goals, goals it misunderstands, goals that contradict etc. But ultimately it proceeds toward some form of goal directed reward function. There's zero reward in punishing someone for past behaviour - except in the context of altering their future behaviour.
Since the A.I. would have to already exist (and there's no reason to believe it would be incentivised to even encourage humans to create other, rival A.Is, with potentially differing goals) before it could punish anyone - Q.E.D.: Roko's basilisk has essentially zero likelihood.
Worse - it's an essentially christian eschatological idea. What is this hypothetical A.I. but an avenging angel, a righteous judge at the end of time? It's pure religion. Dark, manipulative, blood sacrifice religion at that.
> There's zero reward in punishing someone for past behaviour
A similar argument regarding Roko's Basilisk is that it is effectively a prisoner's dilemma, where the Basilisk torturing us and us not helping it is the Defect-Defect scenario. It makes no sense for either party to defect.
> except in the context of altering their future behaviour.
It really seems like a very contrived version of Pascal's mugging.
People in EA donate lots of money to causes that help people today. AI is just one of the more interesting subjects they often talk about.
AI safety isn't about Roko's basilisk, or the idea that AI will take revenge on us for our moral choices. It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being.
> It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being.
We already have those and they're called corporations. They've done significant real damage to the world already and they are still working hard to do more damage.
It makes little sense to me to focus on this potential future problem when we haven't even agreed to deal with the ones that we already have.
I'm not sure these issues are in conflict. Corporations cause lots of harm despite being held back by their dependence on people who control them, who can occasionally put the brakes on some of their worst excesses, and by their inability to recursively self-improve their intelligence. Removing these two handicaps on corporations would allow them to do significantly more damage, so preventing this from happening is important.
> inability recursively self-improve their intelligence
It is not for lack of trying, corporations modify themselves all the time. That they are quite often unsuccessful at making meaningful improvement should be taken as a prior against the idea that an AI will inherently be better at it.
I think I was misunderstood. My point is that not only do we need to consider the cases where the AI is smarter than us, but also the cases where it is dumber than we expect it.
An aligned agent will not fall into stupid errors, thus solving the lower bound of performance is a necessity for complete alignment.
Ohh, right on. Definitely agree now that I've re-read it right. I was a little confused by the first sentence of your other post and thought you were putting forward a disagreement in the second sentence.
It's more about the unintended consequences, the implied/contextual information and getting it to do what we want in a way actually fulfills our goals instead of merely satisfying a condition.
A task such as "bring me a cup of tea" has lots of implied information such as not making it too hot to burn the person, not breaking anything in the process, not harming other people in the process and so on.
I think that's a fair way of putting alignment. The concern is that we might make an AI that's capable of outsmarting us before we're good at making it care about doing what we want instead of some other goal that disregards what we really care about like our safety.
All that said, the amount of energy we expend on AI safety probably shouldn’t be zero. But I choose to believe that if Roko’s basilisk is going to torture us over anything, it’ll be that we didn’t raise up millions of AI scientists from the cotton fields and sweatshops. Credit to the less prominent altruists doing that, or who at least have some form of compound interest built into their utility function which compels them to be kind, now.