I use claude sonnet for coding and it's better than GPT4 most of the time. Something I am realising is that LLMs doesn't have any moat. Today OpenAI, tomorrow someone else.
I agree. My personal experience is that 80% of the time Opus is better than GPT-4 on coding.
Honestly the only thing that keeps me to sometimes prefer GPT-4 now is the UI. I like being able to Edit my messages, and to Stop the model if I gave it the wrong prompt. Please improve Claude's UI!
The interoperability between LLMs right now is amazing. When I write a program I can quickly test it with each of GPT, Claude and Gemini to see which work better for what I'm doing. Here's to hoping nobody figure out how to create a moat any time soon!
"Tomorrow" your desktop computer might be twice powerful but at the same time the "good model of tomorrow" will be four or ten times larger - I'd expect that the gap between what can be done locally versus what is offered as a service will grow, not shrink.
The diminishing returns from model scale means that if your personal computer improves twofold in the same time a datacenter improves fivefold, you may still have lessened the gap in terms of quality.
That doesn't mean you'll be able to run the best model, but I'm relatively optimistic about the gap not growing out of control.
Well, sure, one thing is that the absolute numbers do increase, so for any given notion of "good enough", every device at some point will reach the level where it will be able to run it.
i run a home media server, can't wait to be able to add my own LLM service. it's just a matter of time for it to be something i can install over the weekend with proper hardware
Do you have an IDE integration? What I find so great about Copilot/GPT-4 is how it's integrated into VSCode/Jetbrains and can use the context you're in. Like knowing what line you highlighted, what documents you have open, etc. Do you copy paste into whatever chatbot you're using?
I had the same problem of copying and pasting code into LLM web UI so I built a small tool to streamline the process and add source code into the prompt: https://prompt.16x.engineer/
You can't rely on IDE auto context since the entire codebase is too large to feed into LLM (maybe Claude 3 200k context token can take it, but too expensive). And RAG is not smart enough to figure which part of code is relevant.
> You can't rely on IDE auto context since the entire codebase is too large to feed into LLM
It's not feeding in the entire codebase but whatever you selected or whatever files you have open, at least that's what the UI suggests. So I ask it "what does this line do" and I get an answer that uses the whole file to explain what the line does.
Yeah for context within the single file, GitHub Copilot is good, I use it all the time.
But if you use it to do something across multiple files (DB schema, service, controller, HTML, JavaScript), then it becomes less accurate (precision vs recall problem) as like you said it uses your open windows or some heuristics to decide what's the context.
With an IDE as an interface, it is just not intuitive, UX wise, to "open tabs" to signal to GitHub Copilot that the files should be included in the context.
That works, but for more complex questions taking into account different files and the whole architecture of the app, CoPilot fails. I've been trying to RAG my repos to accomplish this, but the parent comment said that's not possible.
It won’t answer questions that are not somehow related to code or computing, I usually don’t need anything else so I didn’t really test the limits of that so far.
Using copilot in both vsc and vs2022 . I see vast differences but usually down to the language I'm working with.
I've noticed that in visual studio (IDE) copilot gives better answers if I physically view an interface or implementation, then I get "okay" results. But it stuggles with larger more abstract projects.
Vscode is better for sure, but usually I'm working in smaller projects or interpreted languages.
The vim copilot extension is probably the better bit again I'm not working with dotnet in vim.
You have contextual IDE integration with the main LLMs including OpenAI and Claude within Cursor (a VSCode "AI first" derivative), though I haven't tried it but heard good things.
Is generative ai easier (which isn't to say easy) than I assumed it'd be, and it's more limited by training data and training hardware than model complexity?
Edit: although on some level the training only gets you the general capabilities of the model. How you fine tune it to specifically be a useful bot is a very important element. That’s not really model complexity so much as design thinking and experimentation
Their moat right now is developer tooling. They allow fine tuning + an easy API to use their llm.
Noone else does that right now. By the time others do, so much tooling and infrastructure has been built around OAI that the switching costs will be a lot.
It will get to the point that if you want your llm to beat oai on the market it won't be enough that you are as good or even better. You need to be vey very significantly better than OAI. For an extreme example of this see Windows. The network effects keeping it together are so strong that the platform becoming an abandoned adware hasn't been enough to push users to significantly better platforms like Mac.
Now I've fine tuned the hell out of gpt-3.5 and I'd love to see how my app would performed on a fine tuned Opus. I went to their website and I can't seem to fine tune their model yet. Meh. My guess is that by the time they make it available I won't have a strong reason to even try anymore.
For coding I've found ChatGPT4 a bit better than Claude 3 Opus because it tends to understand my intentions more and I trust it to make better suggestions for code changes.
I think Claude has better writing style, and it's refreshing not having to fight with the thing to get it to give me full code snippets. I also hate how difficult it is to get the arrow keys to scroll up or down on chatgpt.
It might be language or task dependent. I'm writing C++, and Claude is more likely to suggest things which simply won't compile. It feels like Claude 3's knowledge of C++ is a bit lacking comparatively.
For me GPT4 seems to suggest more generic unittests in python. They're much more "Put tests here", or "path.to.dependency".
Claude3 opus (and often Sonnet) actually fills in the full dependency paths, actually makes tests, and just overall seems to "know what I want from it".
This tends to be my experience, but mainly when it comes to areas that neither model know about. Claude 3 Opus will happily make something up, but ChatGPT4 will point out where it's lacking.
I wonder how well ELO score handle the edge case where your most important games are against yourself. There are 4 GPT4's in the top 10 (both #2 and #3) and 3 Claude's.
(To their credit, they count anything where the 95% confidence intervals overlap as a tie)
GPT4 feels so much more confident when coding because it is so fast. Claude Opus is generally better for me but takes a long time to reply. It also has much lower limits and sometimes rejects medium size files (75k) as too large. I find the most power comes from bouncing between them. When one stalls hand the bad code over and ask for fixes.
Is there something that can make transcripts of Youtube videos without subtitles in Hungarian? Preferably not requring log in.
I tried to do that today but all the bots either claim to have no access to Youtube or try to bs me.
It's enticing to switch to Claude, however it doesn't look like it includes any image generation/Internet browsing Plugins, makes it more difficult to switch tbh.
There's rumors that Sama has been holding back new versions until companies catch up to the latest OpenAI version. "The king is dead" quote feels like it's meant to bring forth GPT-5.
