GLM series has made it very practical to self host. If the new update for Deepseek flash holds up, I think it would be silly for some companies to not self host.
It’s wild how broken search has become, LLM made it worse but it was already downhill prior to that. Even Reddit forces you login to search within a subreddit and displays the annoying “Best place on internet” overlay after staying on the site for 2 min. I just use AnythingLLM with DuckDuckGo + local LLM model to do searches now.
iPhones are iPads have a milled CNC “unibody” as well… Being able to dictate where to have more thickness helps a ton on the luxury feel. Most of the chips are recycled anyway into new billets.
Unibody is a very appropriate technique for a small portable device that needs to be strong from force in many directions. It is overkill for a keyboard.
Overkill is the point. Same thing for iPads and iPhones. It gives a luxury feeling. That and Chinese manufacturing have made milling a very cost effective manufacturing method.
You cannot infer this because they only show the tokens per second per user. One way to get a higher number is to have fewer users per chip.
I'm pretty sure Cerebras has a confidentiality agreement with OpenAI, and this press release was carefully constructed to avoid leaking details about the model weights. For example, the graph of tokens per second vs. tokens per second per user doesn't have any numbers that would allow you to translate between the two. (And in any case the relationship depends on the model.)
This was also interesting: "CS-4 delivers more than 1,000 tokens per second on models exceeding 10 trillion parameters." Was it known that there were 10 trillion parameter models in use?
I think the frontier providers keep the size of their models carefully hidden.
You don't really need to train a 10T model to test cerebras against a 10T model. You can feed it an untrained (randomly initialized) model and benchmark it. Result will be gibberish but performance the same.
I'm confused. I thought Mythos 5 and Fable 5 were exactly the same model just with a different security layer in front of it.
Could they mean the Mythos 5 Preview?
> I believe this report has confused Opus (which is known to be around 5T) and Fable.
5T for Opus feels quite high though. DeepSeek V4 Pro is a mere 1.6T and often described as a match with Opus in overall quality. Even the largest open models in common use are around 2.8T.
If this is true, it's even more impressive that some of the open weight models that are <3.5T in size, approx 33% of its size, are within a few points of it in the artificial analysis leaderboard.
Not necessarily, there could be diminishing returns on mere parameters count .
There is nothing to say for example a 1 Quadrillion parameter model will be vastly more intelligent than current SOTA especially since new training data is largely synthetic today
I read it as it is impressive because smaller models 2.5T are squeezing similar returns as 10T models despite being 1/4th size not that there beyond 2T today the number or parameters do not have much meaning
I do not rely on any LLM of any size for general knowledge baked into the weights, they all hallucinate and that is the wrong way to hold them imo
I think there is some merit in that smaller models cannot memorize so much of the training data, i.e. that they are less likely to do copyright infringement, and by analogy not having memorized SDK / API surfaces that have since changed from the training data
> I do not rely on any LLM of any size for general knowledge baked into the weights
You have to rely on it to a certain level for agentic/coding work, presuming that's the general subject we're talking about here... For instance I recently encountered a project where it would have been a lot worse if the LLM didn't already know "what is" xterm.js and a bunch of its associated npm-related/node related software. If it was still smart but had to google and find results for everything it would have been a lot more time consuming and risked sending it down a wrong path.
for sure, there is a minimum size and knowledge base that is required to be useful
at the same time, search may find newer or better alternatives, and you can always specify specific technologies you want to use, I typically do this when starting a new project
But these labs distill off the larger models. Both officially at the labs with the big ones, and unofficially. We need the giant models to get the smaller models.
The fact that Musk claims Opus is 5T to justify why Grok is far behind should be taken with a massive grain of salt given he's a recidivist mythomaniac.
Honestly if Opus is 5T parameters while being matched by the biggest open models that are at least twice smaller, it would mean that the US is already behind China in the AI race, despite a significant edge in compute.
Even if they don't match current-day Opus in everything, they do beat 6 month old Opus, which we have no reason to believe it was smaller than the latest version.
Yes. And Opus goes a very long way compared to Fable, Anthropic isn't doing any favour, it's clearly just 2 models with a very different amount of parameters.
The raw margins on proprietary model inference are rumored to be quite high though (they have to successfully defray the entire investment into model training and datacenter capacity for inference, which is massive enough). The API cost you're paying for the model includes that raw margin.
> The cost to train and infer that would be insane, even by today's standards.
This assumption is likely what has led to the erroneous failure.
Enterprise compute per rack has scaled multiple fold in the last 3-5 years. Alongside the training efficiency gains & datacenter scale increases, even 50T+ is well within reach at the top end.
pretty sure 10 trillion parameters is now the norm among closed ai labs, given that nvidia also references the same 10 trillion number for their nvl72 racks
Memory capacity on the WSE is the same as before, but access to off-wafer memory is much slower, so the sweet spot is a given fixed balance of memory and compute. They have announced a partnership with AMD in which CPU/GPU hardware is used for part of the workload and the WSE-3 machines are used for inference for specialized smaller models, but I'm not really sure of the details on that.
And there is, of course, the educated guesses about what WSE-4 will be, one being adding a LOT of stacked SRAM or DRAM to tip the balance towards memory (which could also be done by having a few different tile designs with various configurations of compute and memory capacity). I am curious about which way they'll go.
I think the hardest part is how to define “waste”. Are you “wasting” your time if you go to a school event for your kids during the weekday? Is it a waste of time walking your kids to the bus? Is it a waste of time reading to my kids before they sleep? I argue it’s extremely hard to draw that line and a personal question for sure but being “efficient” is hard.
It varies widely based on a bunch of factors. With this specific model at 8-bit quantization and MTP, it starts out at about 25 t/s for basic chat, but for agentic tasks with long context it slows way down to something like 12-15. I don't see a big difference in token rates based on any config changes I have tried, or going to the smaller 6-bit quantization, so far, though I haven't spent a ton of time on experimenting.
If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
Update on this: When I enable tensor parallelism in llama.cpp, I see 25-33 t/s. With reasoning effort set to medium, Qwen 3.8 finished the same task that previously took 11 hours in a little over three hours, which is still more than three times what most of the large models required including Opus 4.8, and nine times what GPT 5.5 (the fastest of the models I've used) needed for a similar task. So, it's still not fast enough for comfort, but it's much faster than the first run. And, I guess, faster than writing the code myself.
Thanks for the update! I wonder if a Blackwell GPU would be noticeably faster. Which vendor did you end up using? I want to get a gigabyte one but thats be OOS for months.
The Blackwell and Strix Halo will be similar to each other and much slower than the number I'm getting on the dual V620 setup (I see about 10-15 t/s on my Strix Halo with this model at 8-bit quantization depending on context). Prefill is generally quite a bit faster on the DGX Spark and token generation slightly faster on the Strix Halo, as I understand it. But, there are better software efficiency improvements for the Spark line.
This model is far from usable on current AMD or Nvidia 128GB AI machines, IMHO, they just don't have the memory bandwidth, especially since it chews so many tokens for any task. If you want to run this specific model, two (or more) 32GB GPUs with decent memory bandwidth is the right way to do it. It doesn't benefit from the larger memory of the Strix Halo. There's enough room for full context and 8-bit quantized model in 64GB. But, it's really a terrible time to buy hardware. MoE models are a much better fir for the Spark and Strix Halo; you can run Laguna S2.1 (slowly) or one of the Qwen 3.6 MoE fine-tunes (pretty quick). Ling 3.0 Flash also looks promising. Nemotron 3.5 Lightning in the MXFP4 quantization absolutely flies on the Strix Halo at 65-80 t/s, but it's dumb. But, all of those are weaker than Qwen 3.8 27B for coding.
The issue with “AI” is how toxic it feels. Feels grimy like social media. Just like social media, I haven’t introduced any of it to my young kids nor do I plan to until they’re teens. At most I give them access to local AI through home assistant. I have a coworker who have kids of similar age and had to get rid of all his echos when Alexa started feeding his son new Lego sets as “Christmas gift ideas” to tell his parents.
when I was a child I read a book about a luddite family who avoided getting their kids neurolinks and the oldest child decides to go and get one put in at you know, coming of age, and then dies from the implant. I have no idea why I remember this but. Godspeed to you.
It's the circular saw on the job site that makes a bunch of manual jobs quicker. But at the same time it can feel inappropriate, like people are in love with their circular saws so much that they bring them to concerts, and try and do jobs that don't call for a circular saw, like painting walls or digging a hole.
I think it also helps that it's basically the default platform for any software that AI writes, unless you tell it otherwise. And JavaScript is one of the most widely used and well known languages in the world, so there's that, too.
1. it's built for async
2. runs everywhere
3. interpreted, making it fast to iterate on
4. decent performance
5. most popular language, llms are decent at writing it
JVM apparently has the disadvantage that nobody under the age of 40 wants to touch it anymore. I admit I haven't worked in it in 20 years, but I do think it's a marvel of engineering and unfairly maligned. It used to be my career but I wanted to be closer to the metal.
Having Oracle's tramp-stamp on it may have been the final kiss of death in terms of totally-superficial "coolness" factor.
The JVM has a fixed size heap which for me it is wasteful.
IMHO, Microsoft made the correct approach on .NET.
For LLMs, I prefer C# and C++ instead of TypeScript, JavaScript or Python as the static + compiled language factor keeps the coding agents on track. Plus, they have a true threading/async implementation.
It's only appearing wasteful if you're not understanding how memory management works on modern operating systems. It's not wasting any RAM at all if you pay attention to RSS vs VSS.
The actual physical RAM is still entirely available to other applications. It's just made the OS know it might want that many pages. Until there's data in the pages, they will not count towards total RSS.
It's the kind of things some sysadmins used to gripe to me about and I would question whether they should be in charge of a machine at all.
To repeat: just because an application mmaps a large region doesn't mean the OS has actually given it all that physical RAM. It's merely made sure the pagetable knows about it.
I know how mmap works. The JVM is/was terrible on freeing allocated memory though.
If the program is actively using that allocation, that's fine. My problem is with the runtime hoarding RAM when it should have been freed after GC back to the OS.
Then there's also the JVM not handling peaks well because it hit the max heap size, while you still could rely on the OS doing its job to shuffle stuff to swap temporarily. I still see JVM OOMs in my $dayjob's product while the OS has plenty of free physical memory. It is stupid.
I mean, we have malloc() and free(), they are in the stdlib for a reason :)
The JVM seems to follow a philosophy where it assumes it is the only process running besides PID 1, which is valid for some scenarios, but not for others.
Well, you either spend energy freeing stuff or you waste some memory. It's a tradeoff, that you can trivially set with a single flag in Java.
In pretty much every other runtime's case you are stuck with whatever their GC uses, and almost every other GC is far less advanced than the JVM's implementationS. And manual memory management is not free of tradeoffs either, e.g. RAII can have pretty long destruct chains in both C++ and Rust.
Even when I was at Google (2011-2021) it was slowly falling out of favour and by the time I left, Go was fairly rapidly taking its place as the "garbage collected managed language" choice.
Not saying that's a good or a bad thing. I still think the JVM is remarkable.
and you could say the same if not better from C#. But those are now becoming niche languages and ecosystems. One for people around microsoft, azure, etc. The other around oracle solutions. There are still pretty interesting projects around it any of them, but they seem to be losing mindshare against other languages.
An additional benefit of interpreted, I think, is to make plugins easier to distribute and incorporate. With a compiled language you’d need message passing or something.
That actually don't like you're describing Python. I've been working on a couple JS/TS projects and it's like the models I use (Claude Sonnet and DeepSeek v4 Flash) continually struggle to do coherent work; I have to always keep close watch to reduce sloppiness. I go to Python and it's smooth sailing with minimal prompting (and reduced token burn) for acceptable outcomes.
aren't 3 and 4 a tradeoff though? Yes you have 3 but "decent performance" cannot be an extaled value as compared to "runs everywhere". If its used as a counter balance to 3 then it shouldn't be its own unique point basically saying 4 is true despite 3 in this case.
This line of thinking I feel like assumes it's the only program running on your computer. Using less of my CPU and memory means my computer can do more things in parallel, or even run more instances of the harness. My laptop is sweating when I got 5+ claude code sessions running.
I haven't monitored CPU usage so closely, but seems to get heavy with basic tool calls and editing. Memory usage definitely is out of hand, have an idle session right now eating 500MB
yeah fair enough, my entire point is not about the application itself but the contradiction on using superlative terms for all points but a compromising/normal term for one. Like if performance is not revelant why include it in the list of benefits.
decent performance, lol! compared to what? a shell script? "i'll only take up 200MB of disk and 4GB of RAM to output flickering text on a terminal. boy this is high performance"
fast iteration is for POCs. once you have the app built and working, you need performance and stability much more than fast iteration
Probably for the ease of coding extensions — which strikes me as outdated thinking: if it’s open source and you’re outsourcing the coding to LLMs, why not use a compiled, safe language?
There’s an interesting counter example for DeepSeek called CodeWhale, though:
I don’t think so. The ChatGPT app was, which is the “Classic” app now. The Codex app that they’re carrying forward is an Electron app and if you forget to quit it before you walk away it’ll make even your M5 Max unresponsive eventually. Sad days.
Reasonix (which has been seemingly the most recommended harness for using DeepSeek, as it is designed around maximizing caching in DeepSeek) is now a Go app, but still installed via npm. Which feels ugly, but I guess everyone has npm already, and it handles binaries, so I guess it's a reasonable choice.
I'm not sure why specifically Javascript instead of something like Python or other options, but using an interpreted environment minimizes the friction for implementing extension systems, which are an important feature in AI harnesses.
Python basically requires containers unless you are OK with it bit-rotting every six months or so. At least, this used to be the case for trivial python, and recently was the case for stuff that uses cuda.
I stopped paying attention the third time they redefined matrix arithmetic semantics. That happened to be around the 100th time I was sent a script and it only ran on the author’s machine. Maybe they will fix it some day. When they do, I will not believe it.
In contrast, TS has a much nicer type system and better async support. It runs well on web, mobile, desktop and server. Yes, sometimes you have to ship node.js or a whole web browser, but the tooling for that is slightly less insane than the analogous tooling for python.
Its language interoperability story is slightly nicer too (invoke native code, or use wasm). It’s UI story is much, much better since it reuses all the web stuff.
Pip practically invented the supply chain attack; npm perfected it. That’s probably a draw.
Of course, if you care about performance, then other choices make more sense. If you’re training a model then python probably still wins, but very few customers have a $1M+ machine.
`uv` helps but it's new, and I don't think it has the same mindshare yet on "I just globally want to install this thing that needs an interpreter/runtime", so Python probably just doesn't come first to mind.
I'd put it on this. In my experience Python is fine for scripting your own machine but an obnoxious platform to distribute code on. It's very fragile to version changes, in both directions; I don't know how many things I've seen that only run on 3.10, not 3.9 or 3.11. Its packaging system is global by default which only compounds this because everything needs a specific version but they're all dumped in the same place. And it tends to have a lot of native code as dependencies, leading to all the issues of needing to either have the right build environment or a runtime environment that's already been built for.
i also got quite a lot of requests to provide the code in non-golfed form to make the implementation more approachable and idiomatic in each language (will do!)
TypeScript's type system is extremely expressive while still allowing you to retain the flexibility of a scripting language. v8 and JSC also have decades of performance tuning across basically every consumer device.
> But for CLI, it never made sense to me. Especially when Python and Go exist.
But why? Not saying node is better, just want to know where you are coming from for my own knowledge.
Bc I would have picked typescript + node too. It has types (where python just has type hints) and a lot of developers know it already (where go is more niche).
I guess the reasons are that I feel (no data to back this up) that having a Python runtime available is much more probable than having a nodejs runtime around.
I was going to say you cannot easily distribute a nodejs based CLI app, but that’s of course not true. devcontainer-cli is a nodejs app and so are many of the coding agent harnesses.
Yeah, thanks for pushing back. I guess my view was irrational.
