"Test the tests" is a big ask for many complex software projects.
Most human-driven coding + testing takes heavy advantage of being white-box testing.
For open-ended complex-systems development turning everything into black-box testing is hard. The LLMs, as noted in the post, are good at trying a lot of shit and inadvertently discovering stuff that passes incomplete tests without fully working. Or if you're in straight-up yolo mode, fucking up your test because it misunderstood the assignment, my personal favorite.
We already know it's very hard to have exhaustive coverage for unexpected input edge cases, for instance. The stuff of a million security bugs.
So as the combinatorial surface of "all possible actions that can be taken in the system in all possible orders" increases because you build more stuff into your system, so does the difficulty of relying on LLMs looping over prompts until tests go green.
I believe the author explicitly suggests strategies to deal with this problem, which is the entire second half of the post. There’s a big difference between when you act as a human tester in the middle vs when you build out enough guardrails that it can do meaningful autonomous work with verification.
I'm just extremely skeptical about that because I had many ideas like that and it still ended up being miserable. Maybe with Opus 4.5 things would go better though. I did choose an extremely ambitious project to be fair. If I were to try it again I would pick something more standard and a lot smaller.
This is so relatable it's painful: many many hours of work, overly ambitious project, now feeling discouraged (but hopefully not willing to give up). It's some small consolation to me to know others have found themselves in this boat.
+1... like with a large enough engineering team, this is ultimately a guardrails problem, which in my experience with agentic coding it’s very solvable, at least in certain domains.
Like with large engineering teams I have little faith people will suddenly get the discipline to do the tedious, annoying, difficult work of building good enough guardrails now.
We don't even build guardrails that keep humans who test stuff as they go from introducing subtle bugs by accident; removing more eyes from that introduces new risks (although LLMs are also better at avoiding certain types of bugs, like copypasta shit).
"Test your tests" gets very difficult as a product evolves and increases in complexity. Few contracts (whether unit test level or "automation clicking on the element on the page") level are static enough to avoid needing to rework the tests, which means reworking the testing of the tests, ...
I think we'll find out just how low the general public's tolerance for bugs and regressions is.
You have two poison pills (`import "server-only"` and `import "client-only"`) that cause a build error when transitively imported from the wrong environment. This lets you, for example, constrain that a database layer or an env file can never make it into the client bundle (or that some logic that requires client state can never be accidentally used from the stateless request/response cycle). You also have two directives that explicitly expose entry points between the two worlds.
The vulnerabilities in question aren't about wrong code/data getting pulled into a wrong environment. They're about weaknesses in the (de)serialization protocol which relied on dynamic nature of JavaScript (shared prototypes being writable, function having a string constructor, etc) to trick the server into executing code or looping. These are bad, yes, but they're not due to the client/server split being implicit. They're in the space of (de)serialization.
> You have two poison pills (`import "server-only"` and `import "client-only"`) that cause a build error when transitively imported from the wrong environment.
Sure, but even though it's "build time" it feels more like runtime (indeed I think the only reason it works that way is your code is being run at build time). I should be able to look at a given piece of code and know locally whether it's client-side, server-side, or polymorphic, rather than having to trace through all its transitive imports.
This code doesn’t exist in `react` or `react-dom`, no. Packages are released in lockstep to avoid confusion which is why everything got a version bump.
The vulnerable packages are the ones starting with `react-server-` (like `react-server-dom-webpack') or anything that vendors their code (like `next` does).
Note that the exploits so far haven’t had much to do with “server code/data getting bundled into the client code” or similar which you’re alluding to. Also, RSC does not try to “guess” how to split code — it is deterministic and always user-controlled.
The vulnerabilities so far were weaknesses in the (de)serializer stemming from the dynamism of JavaScript — ability to hijack root object prototype, ability to toString functions to get their code, ability to override a Promise then implementation, ability to construct a function from a string. The patches are patching the (de)serializer to work around those dynamic pieces of JavaScript to avoid those gaps. This is similar to mistakes in parsers where they’re fooled by properties called hasOwnProperty/constructor/etc.
The serialization format is essentially “JSON with Promises and code chunk references”, and it seems like there’s enough pieces where dynamic nature of JS can leak that needed to be plugged. Hopefully with more scrutiny on the protocol, these will be well-understood by the team. The surface area there isn’t growing much anymore (it’s close to being feature-complete), and the (de)serializers themselves are roughly 5 kloc each.
The problem you had in Opa is solved in RSC with build-time assertions (import "server-only" is the server environment poison pill, and import "client-only" is the client environment poison pill). These poison pills work transitively up the module import stack and are statically enforced and prevent code (eg DB code, secrets, etc) from being pulled into the wrong environment. Of course this doesn’t prevent bugs in the (de)serializer but it’s why the overall approach is sound, in the absence of (de)serialization vulnerabilities.
The problem we tried to solve with Opa was more general than RSC, probably too general.
// Opa decides
function client_or_server (x, y) { ... }
// Client-side
client function client_function(x, y) {= }
// Server-side
server function server_function(x, y) {... }
Without the optional side inference (which could also use both), it seems we had similar side constraints, and serializers/sanitizers. Probably with the same flaws as the recent vulnerabilities... Like all the OWASP AppSec circa 2013-2015 range of exploits in browser countermeasures when the browsers where starting to roll out defense in depth with string matching :)
For what it’s worth, I’ve just built an app for myself with RSC, and I’m still a huge fan of this way of building and structuring web software.
I agree I underestimated the likelihood of bugs like this in the protocol, though that’s different from most discussions I’ve had about RSC (where concerns were about user code). The protocol itself has a fairly limited surface area (the serializer and deserializer are a few kloc each), and that’s where all of the exploits so far have concentrated.
Vulnerabilities are frustrating, and this seems to be the first time the protocol is getting a very close look from the security community. I wish this was something the team had done proactively. We’ll probably hear more from the team after things stabilize a bit.
RSC is not a protocol, that is probably one of the reasons it is bad and affected only NextJS - most other server framework struggled and gave up this mistake that was React Server.
I like to think of Server Components as componentized BFF ("backend for frontend") layer. Each piece of UI has some associated "API" with it (whether REST endpoints, GraphQL, RPC, or what have you). Server Components let you express the dependency between the "backend piece" and the "frontend piece" as an import, instead of as a `fetch` (client calling server) or a <script> (server calling client). You can still have an API layer of course, but this gives you a syntactical way to express that there's a piece of backend that prepares data for this piece of frontend.
This resolves tensions between evolving both sides: the each piece of backend always prepares the exact data the corresponding piece of frontend needs because they're literally bound by a function call (or rather JSX). This also lets you load data as granularly as you want without blocking (very nice when you have a low-latency data layer).
Of course you can still have a traditional REST API if you want. But you can also have UI-specific server computation in the middle. There's inherent tension between the data needed to display the UI (a view model) and the way the data is stored (database model); RSC gives you a place to put UI-specific logic that should execute on the backend but keeps composability benefits of components.
Thanks for the comment Dan, I always appreciate you commenting and explaining in civility, and I’m sorry if I came a bit harsh.
I understand the logic, but there are several issues I can think of.
1 - as I said, SSR and API layers are good enough, so investing heavily in RSC when the hooks development experience is still so lacking seems weird to me. React always hailed itself as the “just JS framework”, but you can’t actually write regular JS in components since hooks have so many rules that bind the developer in a very specific way of writing code.
2 - as react was always celebrated as an unopinionated framework, RSC creates a deep coupling between 2 layers which were classically very far apart.
Here are a list of things that would rather have react provide:
- advanced form functionality that binds to model, and supports validation
- i18n, angular has the translations compiled into the application and fetching a massive json with translations is not needed
- signals, for proper reactive state
- better templating ability for control flows
- native animation library
All of these things are important so I wouldn’t have to learn each new project’s permutation of the libraries de jour.
> React always hailed itself as the “just JS framework”,
I've literally never heard someone say "React is just a JS framework". They've said React uses JSX over templates. And that React has introduced functional components. But never heard someone say what you're claiming.
> but you can’t actually write regular JS in components since hooks have so many rules that bind the developer in a very specific way of writing code.
This is wild. Yes you can. You can write regular JS in components. I can go build a component right now that uses JS (either with or without hooks). You're conflating the rules of hooks with the ability to use Javascript. Yes, there are rules. No, that doesn't mean you can no longer can write JS.
> i18n, angular has the translations compiled into the application and fetching a massive json with translations is not needed
Tradeoffs. Now each update needs to be rebuilt and redeployed. I don't have that problem in React.
> better templating ability for control flows
Better templating? React uses JSX. Are you saying there exists a better way to control flows than if/else?
> signals, for proper reactive state
This has been debated ad-nauseum in the React community and everything has a trade-off. I wish people would stop saying this as if it's categorically correct. React is UI is a function of state. Singlars would totally break the current mental model of React. Data flows down. This change would come with tradeoffs.
I’ve heard, especially during the first few years when react was introduced, that you don’t need templating, compiler, or anything special to write react, “it’s just JS”.
Of course you CAN write anything you want inside a component, but then it breaks, or has awful performance. To write components the proper way you can’t use any control flows with state management, you need to keep remembering which are the correct dependencies to recreate state, it makes components 20% BL and 80% react logic.
You can’t use if-else in JSX, since only expressions are allowed. So you need to create nested ternaries, which are hard to read, or using JS anomalies like having a condition expression return the last truthish evaluation.
And regarding signals, preact is using it and it doesn’t seem to break anything there.
Function of a state has a nice ring to it, but eventually this was solved a long time before react, every templating engine is a function of a state. The hard part is composing the state easily which react has never been able to achieve.
> that you don’t need templating, compiler, or anything special to write react, “it’s just JS”
This is still true. I don't currently use any of those things. And the existance of a compiler does imply you can't write Javascript. Totally different concepts. Also, pretty sure they had compiler plans for like years now.
> but then it breaks, or has awful performance.
You're gonna have to be more specific. I could repeat that sentence for every programming language/library on the planet and without specifics it would make sense.
> You can’t use if-else in JSX,
I don't need to use if-else in JSX to control flow. I can write if(condition) return html;
> which are hard to read, or using JS anomalies like having a condition expression return the last truthish evaluation.
See the sentence I just wrote before this. I can use if/else to control flow and return early without templating. How is that not ideal?
> And regarding signals, preact is using it and it doesn’t seem to break anything there.
It's not about literlaly "breaking" something. They could implement htem if they wanted to. It's about breaking the mental model.
In React, data flows down. That's a constraint, but not always a bad one. I know exactly where to look for data (up). With signals, that's throw out the window. And now, it's not just about what the component accepts via props/context (which again, is down) it now needs to turn itself on it's head.
I used Angular for years before React and I do not miss having things talking to each other throw multiple lateral levels.
> Function of a state has a nice ring to it, but eventually this was solved a long time before react, every templating engine is a function of a state.
> Function of a state has a nice ring to it, but eventually this was solved a long time before react, every templating engine is a function of a state. The hard part is composing the state easily which react has never been able to achieve.
This is incredibly misleading (and wrong). Templates don't compose. And React is quite literlaly the king of composition.
It's starting to feel like you've never actually used React, but instead are basing your opinions on what you see other people say (who have also not used React).
As merely a systems engineer sometimes having to create a Web app I really much appreciate the experience of building a well-separated app without layers of trivial but flaky boilerplate layers that is a REST API in a dynamic language. The Next app I built last year using heavily RSC is one of the most legible and easy-to-maintain apps I have created so far.
We'll see if the magic can be trusted on or if we need more explicit solutions to this, but the Next/RSC experience was vastly superior compared to writing another REST API that is never to be used with anything else than the accompanied React app, and I'd love to use it or something similar to it in the future.
The reason is probably that a REST API for a "BFF" is in many cases quite tightly coupled with the frontend, and trying to detach those in the system architecture does not separate them in some higher scheme of things. Even if the two parts could separated but would never end up used without another, the separation probably just makes an unnecessary barrier.
I mean the different aspects of my Next app are now clearly separated, but they do form functional units. The separation between frontend and BFF is gone, but that was a wrong boundary in small scale apps to begin with.
You either die a hero or live long enough to see yourself become the villain. The amount of time I've spent debugging other PRs (and mine) around hooks is just unruly, then React turned its attention to the server, something that I (most of us? we?) never ever asked for; but I guess that's what Meta, a company of cancer needs. I sure don't need it. Never have I imagined during the last 15 years that I'll be happy to say I'm using the mountain of enterprise spaghetti called Angular, but now I am. For years I hoped I'll be able to get back to React projects one day; that hope is long gone.
reply