From a friend who uses LLMs to write code…
Current state of affairs according to me:
OpenAI is at the frontier. There is no reason to use Terra, as there is a Luna or Sol model at an effort level that matches it for cheaper. You can get quite a lot of inference out of the $20/mo plan. $100/mo gives you 5x, but $200/mo gives you 20x, not 10.
Claude Fable is marginally the best, but quite expensive and falls back to Opus on anything it considers remotely questionable, while still charging Fable prices.
Gemini’s latest is 3.5-flash and is well off the curve. The latest non-flash is still 3.1-pro.
GLM 5.2 is the best open weight Chinese model, but its price per token is misleading as it eats up reasoning tokens like crazy.
Best local coding model is probably Qwen 3.6 27B. Gemma 4 is a good local all around model to talk to, but not as good at coding.
Where will these various models run? New York State has banned data centers, which is forward-thinking but I prefer to reflect on the exquisite timing of the New York political elite in shutting down their massive nuclear electricity plant just one year before the AI/data center boom began (launch of ChatGPT in 2022):
How about AI data centers in space?
Loosely related:
I’ve noticed a failure mode where Claude will correctly identify a relationship but then get the direction wrong. E.g., higher rates of evapotranspiration will prevent rainfall by drying the atmosphere, or something similarly silly. Repeatable across other domains like medicine and economics.
As for New York, they are just copying Europe’s self-destruction. Both were early movers in banning fracking, and both are now early movers in banning AI. Progressives against progress.
I did extensive testing by having models take a large code base and find problems with it. Grok 4.5 and Opus 4.8 were good. Fable refused to do the work because it might find a security problem in my code.
I tried Qwen Coder up to 80B, and it found 7x fewer problems than the top models.
Gemini Pro 3.5 was much worse than GLM 5.2.
GLM 5.2 was the only model that I was able to both run locally and get good results. Sadly I can only get 17 tokens/sec on my home computer (due to it being 744 Billion parameters) – so I don’t really use it.