Right, this is why I (author here) close the article mentioning that product design needs to keep the humans in the loop for these models to be useful.
If the product is designed assuming humans will turn their brain off while using it, the fundamental unreliability of LLM behavior will create problems.
Yeah, product design and marketing, for sure. As I said, I wish the marketing material was more like your blog post than what it is now. Obviously tough to get nuance in short-form copy but promising the world is a big mistake seemingly all these companies are making (…on purpose.)
If the product is designed assuming humans will turn their brain off while using it, the fundamental unreliability of LLM behavior will create problems.