From a friend who uses LLMs to write code…
Current state of affairs according to me:
OpenAI is at the frontier. There is no reason to use Terra, as there is a Luna or Sol model at an effort level that matches it for cheaper. You can get quite a lot of inference out of the $20/mo plan. $100/mo gives you 5x, but $200/mo gives you 20x, not 10.
Claude Fable is marginally the best, but quite expensive and falls back to Opus on anything it considers remotely questionable, while still charging Fable prices.
Gemini’s latest is 3.5-flash and is well off the curve. The latest non-flash is still 3.1-pro.
GLM 5.2 is the best open weight Chinese model, but its price per token is misleading as it eats up reasoning tokens like crazy.
Best local coding model is probably Qwen 3.6 27B. Gemma 4 is a good local all around model to talk to, but not as good at coding.
Where will these various models run? New York State has banned data centers, which is forward-thinking but I prefer to reflect on the exquisite timing of the New York political elite in shutting down their massive nuclear electricity plant just one year before the AI/data center boom began (launch of ChatGPT in 2022):
How about AI data centers in space?
Loosely related:
I’ve noticed a failure mode where Claude will correctly identify a relationship but then get the direction wrong. E.g., higher rates of evapotranspiration will prevent rainfall by drying the atmosphere, or something similarly silly. Repeatable across other domains like medicine and economics.
As for New York, they are just copying Europe’s self-destruction. Both were early movers in banning fracking, and both are now early movers in banning AI. Progressives against progress.
I did extensive testing by having models take a large code base and find problems with it. Grok 4.5 and Opus 4.8 were good. Fable refused to do the work because it might find a security problem in my code.
I tried Qwen Coder up to 80B, and it found 7x fewer problems than the top models.
Gemini Pro 3.5 was much worse than GLM 5.2.
GLM 5.2 was the only model that I was able to both run locally and get good results. Sadly I can only get 17 tokens/sec on my home computer (due to it being 744 Billion parameters) – so I don’t really use it.
> GLM 5.2 was the only model that I was able to both run locally and get good results
Wow, how much GPU memory you have? Or was it M4/M5? Was model quantized?
Linus gradually came around to accepting AI generated code of some kind in the kernal, similar to how he was big on Rust, 2 years ago. No mention of how much moderation is imposed though he emphasized bug fixes, which means most of the AI output is valgrind style analysis, which means it’s manely an increased workload of filtering through junk. His focus lately shows signs of the kernal being a solved problem & engagement being the new priority.
Why is there such a strong correlation between A.I. code slop tools coming online and the whole freaking ‘net being borked? Archive.org, my email provider, both out today with 500 http status and even less helpfully “unknown error”. Oh, I know what the error is.
I’m trying to keep pace with A.I. and training my brain to take internal prompting. “Brain, draw a picture of a naked hot chick from the ’90s directly into my visual cortex.” Gets around Mrs. Hippy’s internet “filters” and gives me tattoo free (well maybe a little butterfly tattoo or a tramp stamp) honeys for a change.
[I’m Neo Hippy, self-elected president of the Phil Greenspun Blog Commenter’s Union and I approve this message.]
I had Fable generate a small project, at the end of which it got confused and built some sort of strange testing framework for the code, and was unable to fix it the right way (by installing Xcode properly).
Opus 4.8 however did the right thing and got rid of the Fable mistakes.
Net output: something like 800 lines of code.
Net time: about 32 minutes (25 minutes of Fable). Somewhat more than 1000 tokens/minute.
I used 4.8 after Fable to clean up, just to be clear.
How is this friend possibly doing the testing required to come to these conclusions? Doesn’t it take a crazy amount of time, if not money for tokens? Or is he just passing along leader board data?
Meanwhile, my anecdotal testing with whichever reasonably new Anthropic or OpenAI model seems affordable shows that there is a huge benefit to me. Of course, I’m just a 1x programmer and the friend is likely 10x.