>What stops them <..> simply use cheaper model for every Nth request.
That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.
To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.
If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
Prefill is generally faster than generation, but not by much on older Mac processors. I get around 70-60 tps in prefill on my M1 Max for Muse Glimmer (not sure about the generation speed, probably between 15 and 30).
They allegedly improved this by “up to 7x” with M5 but I’m not sure about the exact numbers here.
Often Deep Seek V4 flash or Qwen should be enough.
I wanted to see whether Kimi runs at all on one machine with the full record published, and for long multi-table finance reasoning I wanted the strongest model I could keep on the machine.
I did some tests against Deep Seek v4 flash results on my reports and Kimi definitely has some advantages.
I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency).
Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurrent use, while vLLM/SGLang easily crash on non-common setups (things like, if you do pipeline parallelism for RTX5090+RTX4090, they will randomly crash with RAM caching enabled or select wrong kernels because they usually assume that every rank is the same device type; they also don't support Q5-Q6).
For me what's most interesting is to optimize inference for lack of good datacenter hardware and how to optimize for it best. I've been running an AI server in the office, and so far I've find these techniques most important for concurrent use on cheap hardware: pipeline parallelism (to accomodate for PCie), RAM caching (to quickly restore contexts into VRAM), speculative decoding (including domain-specific ngrams, they already can speed up code generation considerably without the overhead of a draft model), good kernels highly optimized for a specific device, support for Q5-Q6 (almost as good as Q8), FP8 contexts (more context to fit), paged attention (for better VRAM utilization), prefix caching, continuous batching (this is the default everywhere).
So far the main bottlenecks have been llama.cpp's poor VRAM utilization for contexts (you either have fixed-size slots, or use unified KV cache where each request attends to attention from all other requests and then unnecessary portions of attention are masked out), and lack of decode/prefill segregation: when a request starts prefilling a long context, all decoding threads slow down to like 5 tok/sec. On the other hand, vLLM/SGLang feel superbuggy if you don't run them on some officially approved node like 8xH200
Yeah, there's this thing called the curse of knowledge. If an engineer has a deep understanding of something, it's not a given that they can explain it well. For them, the topic feels so simple, and they've done it so many times that they may have forgotten other people aren't as knowledgeable. They will throw terms around without explaining them, etc.
In agentic scenarios, an LLM has to read far more tokens than it outputs. I think focusing only on the decode speed is somewhat misleading. 14 tok/sec for decode is actually okayish. 93 tok/sec prefill is what's abysmal, my RTX 5090 goes above 2000 tok/sec with 5 bit quants.
Qwen 3.8 has the same architecture and the same parameter count as Qwen 3.6. Something is not right with the GGUF if it's 2 times slower. The post says "The hybrid attention architecture is new" and says the author's older Llama build from a "couple weeks ago" failed to run Qwen 3.8 because it did not support Qwen35 architecture, but both 3.6 and 3.8 are based on Qwen35 which was released in February 2026. The post doesn't make any sense.
That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.
To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.
If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.
reply