> $20k workstation, best case: $15k M5 Ultra 512GB, 36-month amortization, ~$440/mo. Runs a GLM-5.3-class model at ~30 tok/s. Saturated 24/7 it produces roughly 58M output tokens/month.
For agentic coding, ~90% of the cost comes from cached input tokens. This cost increases quadratically with the session length. If sessions go near 1M context, the number of cached input tokens can easily exceed 1B in a day.
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...
If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.
> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)
But this failure mode can still happen.
Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.
So, perhaps the lesson here is "don't use vllm at home"?
Let's say DeepSeek is being forced to use the CANN stack, and the new pricing reflects the cost when 100% of inference is done with Huawei chips. Then, I suppose we can infer that:
* CANN stack is 1.5x~2.3x less efficient in compute
* CANN stack has 6x lower inter-connect capacity
> computer chips once again become a commodity
Ascend 950 is going for $7k to $9k with mediocre looking specs. $16k for RTX Pro 6000, $6k for RTX Pro 5000. This is not looking good.
It would have to be reproduced on a CUDA stack because AFAIK Huawei don't sell the Ascend 950PR (for inference) to anyone, rather, they operate them as part of the Huawei Cloud and only allow select customers (such as DeepSeek) to rent them.
The CEO of DeepSeek recently revealed to investors a lot about the resources available to DeepSeek, and the gap between Huawei and NVIDIA. Select quotes from the transcript (translation is a bit patchy on the source website though):
"We currently have roughly 20,000 H-equivalent compute cards"
"Huawei 950—right now Huawei gives us 16,000 cards, this should be publicly stateable."
"Like Huawei gives us roughly 16,000 cards of capacity, internet giants maybe get a hundred-something thousand, we get ten-something thousand—I think this ratio is also relatively... but this is probably just how much capacity Huawei has."
"16,000 Huawei 950 cards only equal 4,000 B-series cards."
"Huawei’s supernode, Huawei’s 950 supernode, in performance and price can completely substitute for NVIDIA’s GB200, GB300. The price is definitely more expensive, but limitedly so. Fifty percent more expensive, a hundred percent more expensive—a hundred percent more doesn’t matter, two hundred percent more doesn’t matter. For example, a hundred percent more expensive—I think it can already be considered a price-level substitute."
"I think domestic hardware might need a few years."
"I don’t quite believe that five years from now, we’ll still be stuck on the production capacity problem. Right now we’re definitely stuck on the production capacity problem—this year, next year, the year after, I think we might still be stuck on the production capacity problem, but five years later, I think maybe not necessarily—I’m still relatively optimistic."
Objectively speaking, the 2 bit quant from antirez has very low accuracy. Meanwhile, his 4 bit quant does have decent accuracy, but is a bit pointless by being bigger than the full precision MXFP4 quant. Anyway, they all work fine in practice.
reply