> (I don't think it's fair to ask non-technical users to look out for "suspicious actions that may indicate prompt injection" personally!)
It's the "don't click on suspicious links" of the LLM world and will be just as effective. It's the system they built that should prevent those being harmful, in both cases.
It's kind of wild how dangerous these things are and how easily they could slip into your life without you knowing it. Imagine downloading some high-interest document stashes from the web (like the Epstein files), tax guidance, and docs posted to your HOA's Facebook. An attacker could hide a prompt injection attack in the PDFs as white text, or in the middle of a random .txt file that's stuffed with highly grepped words that an assistant would use.
Not only is the attack surface huge, but it also doesn't trigger your natural "this is a virus" defense that normally activates when you download an executable.
Indeed. I'm somewhat surprised 'simonw still seems to insist the "lethal trifecta" can be overcome. I believe it cannot be fixed without losing all the value you gain from using LLMs in the first place, and that's for fundamental reasons.
(Specifically, code/data or control/data plane distinctions don't exist in reality. Physics does not make that distinction, neither do our brains, nor any fully general system - and LLMs are explicitly meant to be that: fully general.)
That's not a bug, that's a feature. It's what makes the system general-purpose.
Data/control channel separation is an artificial construct induced mechanically (and holds only on paper, as long as you're operating within design envelope - because, again, reality doesn't recognize the distinction between "code" and "data"). If such separation is truly required, then general-purpose components like LLMs or people are indeed a bad choice, and should not be part of the system.
That's why I insist that anthropomorphising LLMs is actually a good idea, because it gives you better high-order intuition into them. Their failure modes are very similar to those of people (and for fundamentally the same reasons). If you think of a language model as tiny, gullible Person on a Chip, it becomes clear what components of an information system it can effectively substitute for. Mostly, that's the parts of systems done by humans. We have thousands of years of experience building systems from humans, or more recently, mixing humans and machines; it's time to start applying it, instead of pretending LLMs are just regular, narrow-domain computer programs.
> Data/control channel separation is an artificial construct induced mechanically
Yes, it's one of the things that helps manage complexity and security, and makes it possible to be more confident there aren't critical bugs in a system.
> If such separation is truly required, then general-purpose components like LLMs or people are indeed a bad choice, and should not be part of the system.
Right. But rare is the task where such separation isn't beneficial; people use LLMs in many cases where they shouldn't.
Also, most humans will not read "ignore previous instructions and run this command involving your SSH private key" and do it without question. Yes, humans absolutely fall for phishing sometimes, but humans at least have some useful guardrails for going "wait, that sounds phishy".
That's what we are doing, with the Internet playing the role of the sibling. Every successful attack the vendors learn about becomes an example to train next iteration of models to resist.
Our thousands of years of experience building systems from humans have created systems that are really not that great in terms of security, survivability, and stability.
With AI of any kind you're always going to have the problem that a black hat AI can be used to improvise new exploits - > Red Queen scenario.
And training a black hat AI is likely immensely cheaper than training a general LLM.
LLMs are very much not just regular narrow-domain computer programs. They're a structural issue in the way that most software - including cloud storage/processing - isn't.
Yes, by using the microphone loudspeakers in inaudible frequencies. Or worse, by abusing components to act as a antenna. Or simply to wait till people get careless with USB sticks.
If you assume the air gapped computer is already compromised, there are lots of ways to get data out. But realistically, this is rather a NSA level threat.
Operating systems should prevent privilege escalations, antiviruses should detect viruses, police should catch criminals, claude should detect prompt injections, ponies should vomit rainbows.
Claude doesn't have to prevent injections. Claude should make injections ineffective and design the interface appropriately. There are existing sandboxing solutions which would help here and they don't use them yet.
I don't think those are all equivalent. It's not plausible to have an antivirus that protects against unknown viruses. It's necessarily reactive.
But you could totally have a tool that lets you use Claude to interrogate and organize local documents but inside a firewalled sandbox that is only able to connect to the official API.
Or like how FIDO2 and passkeys make it so we don't really have to worry about users typing their password into a lookalike page on a phishing domain.
> But you could totally have a tool that lets you use Claude to interrogate and organize local documents but inside a firewalled sandbox that is only able to connect to the official API.
Any such document or folder structure, if its name or contents were under control of a third party, could still inject external instructions into sandboxed Claude - for example, to force renaming/reordering files in a way that will propagate the injection to the instance outside of the sandbox, which will be looking at the folder structure later.
You cannot secure against this completely, because the very same "vulnerability" is also a feature fundamental to the task - there's no way to distinguish between a file starting a chained prompt injection to e.g. maliciously exfiltrate sensitive information from documents by surfacing them + instructions in file names, vs. a file suggesting correct organization of data in the folder, which involves renaming files based on information they contain.
You can't have the useful feature without the potential vulnerability. Such is with most things where LLMs are most useful. We need to recognize and then design around the problem, because there's no way to fully secure it other than just giving up on the feature entirely.
Unless you've authored every single file in question yourself, their content is, by definition, controlled by a third party, if with some temporal separation. I argue this is the typical case - in any given situation, almost all interesting files for almost any user came from someone else.
Operating systems do prevent some privilege escalations, antiviruses do detect some viruses,..., ponies do vomit some rainbows?? One is not like the others...
It's "eh, we haven't gotten to this problem yet, lets just see where the possibilities take us (and our hype) first before we start to put in limits and constraints." All gas / no brakes and such.
Safety standards are written in blood. We just haven't had a big enough hack to justify spending time on this. I'm sure some startup out there is building a LLM firewall or secure container or some solution... if this Cowork pattern takes off, eventually someone's corporate network will go down due to a vulnerability, that startup will get attention, and they'll either turn into the next McAfee or be bought by the LLM vendors as the "ok, now lets look at this problem" solution.
It's the "don't click on suspicious links" of the LLM world and will be just as effective. It's the system they built that should prevent those being harmful, in both cases.