To your slow point: I did a quick eval for a data viz task to do a qualitative comparison and Fable was much faster (by 4x), but I struggle with the "token inefficiency" as a sort of whatever metric.
Kimi was half the cost and produced a near identical output.
It's kind of ridiculous how good these are getting. 3.5 Flash lite is pretty comparable to Opus 4.8 (at least for the couple tests I did) while simultaneously being 6x faster and 19x cheaper.
Not for me personally. While setting up a custom website, 3.5 Flash introduced tons of bugs that Claude had to fix. The website has about 3,000 lines of code spread across multiple files, and Gemini somehow couldn't do frontend changes without breaking things. Sharing my 2c, but I've stayed on GPT 5.5+ and Claude Sonnet/Opus 4.6+. Anything past that from those two have been bug-free, but Google's latest hasn't been.
Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.
Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.
By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.
It's a free site, so I was trying to limit both the privacy and risk exposure.
Making the content auto expire after a short period of time greatly decreases the attractiveness of the site to lots of SEO spammers and other types of abuse, and if someone were to get something malicious or vile posted it will clean up after itself without me having to wade into things.
I built a simple (free) eval tool for my own uses (Github Gists + Model Outputs) after not being able to find a suitable one in the market.
The market's being split into
1. Longitudinal LLM observability tooling
Most eval startups have gone down the route of something more like being an observability platform for LLM inference. They want to be in your stack and running the inference to collect data on performance of it.
They collect things like how often a model returns JSON that's out of spec or returns values that aren't expected as well as general timing and cost info.
2. Safety Limiting / Pentesting
Say you're doing something in the medical field or that's sensitive in some way and you want to figure out what model has the best outputs for your task that won't fly off the guardrails.
3. Simple cost + performance + quality swapping
This is what my tool does, basically lets you test if you _really_ need to be running that frontier model in a loop across a million records or if you'd be better with an older model or something else.
Cool project! I haven't seen that OpenRouter workflow yet (sign into OpenRouter and it creates an API key that your app can use), that looks like an interesting pattern to investigate.
My company recently built a tool that is closer to your first category, but it's an API so it doesn't have the security (supply chain) concern of being embedded in your application.
It's built to help people manage the risk of LLMs changing underneath them and drifting from their designed behavior. Traditional deterministic testing probably won't be sufficient for apps that provide nondeterministic output, like a chatbot backed by an LLM.
The point in the linked article about the challenge of selling developer tools to developers is a good one. I think the first reaction to coding agents is "let's build everything ourselves!" but the long tail of maintenance is still there and the pendulum will probably swing back to "let's stick to our knitting."
Something that's improved my life has been buying a sticker sheet of those LED darkening dots. They're only a couple bucks and look much cleaner than other solutions I've tried while still allowing for _some_ light to come through.
Thanks. You should. I remember something I read about a teacher who asked kids to write little interactive fiction pieces using Inform 7 set in different parts of the school and then he stitched them all together into a large story and they could all play it. Everyone's imagination comes to life. You wander into the parking lot and some kid has created a portal there into a mirror universe. I don't know if I'm misremembering or the work involved but it's definitely feasible and not something you can do with a regular programming language easily in such an open ended way.
I've always loved interactive fiction. https://en.wikipedia.org/wiki/Get_Lamp is a great watch. "Before the first person shooter, we had the second person thinker." :)
Which models were you using under this? If you used the quality default as exists in the interface, it makes sense that it was ~4x the cost as it'd be 3 frontier models judged by one of those.
The idea would be to use fusion with simpler, cheaper models.
yeah its really counterintuitive i think; i.e, getting the right framework and structure for this to work probably isn't trivial, models really hate playing well together. i wonder how their version would fair in real world use.
A counterpoint to this is that we have some real different definitions of AI.
If you consider things like the machine learning filters in your smartphone camera and Google's AI Overviews for searches it's entirely plausible that the US is currently at 75%+ of AI usage.
I think it's your last point that's actually the strongest.
There's always gaps between theoretical and practical, but to see China investing so hard in the future while the US digs in it's heels is infuriating.
The price of energy sets a floor on the price of all manufactured goods. By kneecapping the cheapest sources of energy, the regime kneecaps all domestic manufacturers.
China's aggressive buildout of cost effective energy production isn't because they're 'woke,' it's because it makes them more competitive. Every product they export at low prices is in part due to the their extremely cheap energy.
It's like the regime looked at the UK's collapsing manufacturing industry due to their high energy costs and said "I want that for us!"
Kimi was half the cost and produced a near identical output.
https://fr4geiw93g.evvl.io/
On a fairly simple coding task Kimi was 2x the cost and 7x as slow (and gpt-5.6-sol was even cheaper).
https://r26pakjmhv.evvl.io/
reply