IQ3_XXS (~3.2 BPW). For me this is an option because my Mac studio is only used for serving LLMs, so I can afford to dedicate most of its RAM to this. I can run with 256k context and only uses ~117G, with the remaining (up to 125G which I can allocate to VRAM) being used for prompt caching and context checkpoints.
I'm making my own quants, though the Vision-Exp version is outdated and won't work on llama.cpp master branch (I built it before llama added support):
Don't use my Vision-Exp GGUF though. As I said I built those GGUFs before llama.cpp supported, and they can't be loaded on current master (require my own branch).
I already have new GGUFs but haven't uploaded yet. If you want Vision-Exp, maybe use bartowski or unsloth's GGUFs.
Side note:
As an alternative to deepseek v4, you might want to give it a shot at qwen 3.8 flash next. I have IQ4_NL GGUFs that can be loaded fully into 128G, or Q5_K GGUFs that can offload the PLE to disk (use --load-mode none --lazy-mode on for that): https://huggingface.co/tarruda/Qwen3.8-Flash-Next-GGUF.
llama.cpp master is still somewhat bad in Qwen 3.8 next performance, but I was able to achieve 40tps tg and 600 tps pp on my private branch.
I'm curious if you've tried dwarfstar and decided to move to llama.cpp and 3 bit quants or what made you go that route instead? I've been using ds4 for months now and it's already got support for the new vision model, haven't tried it yet, still on 0731 but it's been very solid for me.
I tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar.
Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actually get much better metal performance on my llama.cpp branch than on dwarfstar (plus all the extra llama.cpp features). Here it is in case you want to give it a shot: https://github.com/tarruda/llama.cpp/tree/qwen4exp-dsv4-opti...
Okay, but the anecdote states that every model repeated the pseudo-factoid about Foobar square, not just the 4 GB open source model equivalent of a tabloid.
I think the key phrase here is, "an obscure small town." There may only be a single mention of this place, hence the only one on which a response can be based. This says more about the user's understanding of LLMs than it does about LLMs.
This says more about the user's understanding of LLMs than it does about LLMs.
"Tell me everything you know about (obscure small town), (state). Only what's unique to (town), not commonly-known facts" is an excellent way to test for hallucinatory tendencies in a new model, in my experience. Likely the best I've found.
Quality of results is almost linearly proportional to the size of the model in many cases. The largest models like K3 and GLM 5.3 will either confine their responses to known true facts about the town and its surroundings, or admit they don't have enough information to answer. Smaller ones will reliably make up hilarious or downright-strange things.
Another good test is https://whatever.scalzi.com/2025/12/13/ai-a-dedicated-fact-f... , which still works on the newest models. Of the open-weight models available, only Kimi K3 will consistently admit it has no idea who Scalzi's novel is dedicated to. The rest still make up random stuff and present it confidently.
TL,DR: progress is possible, and it has been made, but it's happening slower than many people think.
N/RVO works by (at the machine language level, of course) rewriting the function signature to return void and take an extra pointer parameter, which is written to before returning. If you're returning a newly-constructed object, the compiler can rewrite that into calling the constructor on the pointer, but if you're returning a named object, the class may have a non-trivial destructor that needs to run after the move, such that it's not possible to rewrite uses of the local object into uses of the pointer.
I'm not too confident on that last part, because such an implementation would mess with semantics in case of an exception, so anyone feel free to correct me on that.
NRVO does not affect the ABI of the function. It cannot affect the ABI, for whether or not it kicks in depends on the body of the function, and affecting the ABI would make it impossible to use it if only the declaration appears in a header.
The correct explanation is this:
In C++, classes with nontrivial destructors or copy/move constructors are considered nontrivial for the purposes of calls and are passed via pointers rather than via value. By passing via pointer, the class has a stable address and thus 'this' pointer. Returning such a class means the caller allocates the storage for the class on the stack before calling the function, and passes the pointer to that storage to the function as an extra parameter. This is based solely on the definition of the class itself; this happens whether or not NRVO kicks in.
Usually, when you declare a variable, the abstract machine of C++ requires you to construct a new object and call the copy/move constructors or assignment operators and the destructors at various times as appropriate. With nontrivial versions of these special functions, it is possible to observe whether or not they were called (these things still happen with trivial classes, but it's not so easy to observe). Returning a value requires constructing the storage space for that object--with all the attendant abstract machinery that involves.
What NRVO does is to say that, under certain conditions, rather than constructing storage space for a given variable that is normally required, the storage space that is allocated for the return value by the ABI is used instead. In essence, you are promoting a given variable to the return value hence the name 'Named Return Value Optimization'. What makes this annoying to implement is that you have to track at the AST level, before doing any code generation at all, whether or not a given variable is eligible for NRVO, and then use that information to control the code generation for allocating storage space.
Despite its name, NRVO is not actually an 'optimization' in the compiler. The optimizer plays no role in it, since the optimizer is fulfilling the requirements of the abstract machine. Instead, it is a set of conditions that allows the frontend to omit calls to copy constructors, etc. under specific circumstances.
> Despite its name, NRVO is not actually an 'optimization' in the compiler. The optimizer plays no role in it, since the optimizer is fulfilling the requirements of the abstract machine. Instead, it is a set of conditions that allows the frontend to omit calls to copy constructors, etc. under specific circumstances.
That's semantics. The way the compiler is structured and in which component the transformation is implemented has no bearing on whether something is an optimization.
> N/RVO works by (at the machine language level, of course) rewriting the function signature to return void and take an extra pointer parameter
This sounds wrong, are you sure? Would you mind demonstrating with an example on godbolt? Whether NRVO applies or not, the ABI should be the same, AFAIK.
Yes, it works exactly like this, this is a demo on godbolt [0]. rdi stores the pointer in both cases, makeS1() uses RVO, makeS2() takes it explicitly and constructs with placement new.
I will say before testing this i didn't realize the RVO calling convention was to return the pointer you pass in, but apparently so. If makeS2() returned void, it's just a tail call to the constructor, but makeS1() has to spill rbx and use it to save the pointer.
No, all you're showing in that example is that a pointer is passed as part of the ABI. You're not showing that RVO relates to that in any way whatsoever. If you write the same function in a manner that (N)RVO can't kick in, does the pointer no longer get passed?
The reason this should sound dubious is that you're suggesting the caller needs to know the callee's body in order to know how to call it, but it should be possible for the two to be compiled entirely independently, and in fact mutual recursions should be fine too. After all, the callee knows where the return value has to land either way, and the caller similarly knows where to expect it, regardless of when/how the object is constructed or destroyed.
Yes, of that I'm sure. This optimization is only possible if the compiler has control of both sides of a call. If the function may be callable from other translation units or modules I imagine it generates a thin wrapper that's externally callable.
The optimization is often possible even if the computer does not see the call, because most (all?) ABIs have always required hidden pointer parameters for class types with non-trivial destructors.
https://godbolt.org/z/9WvnEvEYh
Note how `std::unique_ptr<int>` effectively passed as a `int**`; and that the by-value unique_ptr is not destroyed at the end of the function -- destroying parameters is instead the caller's job (and commonly only happens at the end of the full expression containing the call -- though this choice is implementation-defined).
But that can only work if the caller can see the updated value of the parameter (to avoid double-free for `clear`) -> thus the need to pass the parameter by hidden pointer.
I don't know why I wrote the comment about parameters earlier -- (N)RVO is about return values.
Those have a different reason for being passed behind a hidden pointer: the class type might have self-referencing pointers, so there must be an explicit move/copy constructor call whenever it changes address, to give the class an opportunity to update those pointers. This cannot work when returning in a register: the callee doesn't know the target address, and the caller doesn't the source address, so neither can call the move constructor.
Thus, all ABIs must pass a pointer (or let caller+callee agree on a memory location in some other way) for types that aren't trivially copyable.
> Unfortunately, I don't think there's getting away from just understanding value semantics to get the correct and/or performant behavior.
This is not specific to C++ though. It just so happens that C++ developers who feel this topic is important are those invested in performance optimization. For them, C++ offers them these types of tools.
Meanwhile, those who don't have a pressing need to go through great extents to optimize performance can simply fall back to the compiler generating most special member functions and then pay the performance tax of doing deep copies by default. For these cases, which I'd say corresponds to most C++ floating around, it's far more important to know the rules of when to define our own custom constructors and assignment operators.
> pay the performance tax of doing deep copies by default.
For most code that performance tax is not worth worrying about. There are almost high performance priorities. It is almost always the case that your code runs "fast enough" long before you start worrying about the few nanoseconds a deep copy of a few bytes costs.
> For most code that performance tax is not worth worrying about.
Indeed, I agree. It's possible to go a long way in terms of performance with a basic understanding of passing by reference, without even having ro bother with move semantics. So these topics end up being dominated by language lawyers and the types that enjoy debating "aktualy" topics, who also contribute to making things sound far harder than what they actually are by pretending that this sort of trivia is very important stuff.
> So these topics end up being dominated by language lawyers and the types that enjoy debating "aktualy" topics,
I'm not sure about that.
Even though most people never need it, a small minority really do, and those types have to become expert in those weird details. Well you pretty much have to be a language lawyer to get these optimizations right, but the goal really is the ultimate performance in some place where it really matters.
AI will definitely not take over translation for a long time. Machine translation has improved to the point it's mostly no longer incoherent, but for non-formulaic content it's still very hit-and-miss, not that much better compared to 10-20 years ago. For example, something they still struggle with is consistently translating terms coined within the text with consistent phrases.
>A budding painter or a musician could support themselves off commissioned / commercial work while working on their grand opus... but now, the customers just prompt gen AI.
Are you joking? The only people I see using generative AI are either companies, shitposters, and spammers. Who are all these people who have stopped paying commissions and moved over to prompting AIs?
There's another translation use case worth considering. For non-native English speakers, AI can help express ideas they already have in a language they're less comfortable with.
Especially on forums like HN, the AI isn't generating the person's views - it's helping them say that what they already mean more clearly in English.
But what share of translation dollars are non-formulaic content? To me it seems that there are tons of manuals for dishwasher and such to translate (where from my experience quality basically doesn't matter) and only a limited amount of diplomatic communiques.
Manuals have been machine-translated for ages, well before algorithms were anywhere close to ready for the task. No change there.
>a limited amount of diplomatic communiques
With zero data to support it, I'd bet good money the vast majority of semi-professional translators (that is, those not employed by publishers but still making some money off of their work) work on fiction, translating comics, subtitles, etc.
Actually, come to think of it, mixed media like those will be the last where machine translation will be able to fully take over humans, just because the text doesn't contain the entire relevant context for the job.
Keep in mind that 99% of translators are freelance. Translating something takes only a fraction of the time it takes to write the original, so publishers only have full time project managers and editors and hire translators per job as needed.
Engines like Deepl do give very good results on some single sentences. After all, they are huge databases of previous human translations, they are bound to nail it here and there. Where it falls apart is in keeping a good average and a consistent voice, so either you leave it all as messy AI sludge or you rewrite it substantially (at which point it's a glorified dictionary)
First, as others have said, a lot of translation work doesn't involve high-brow fiction. It's reference material of all sorts. Technical books, manuals, software localizations, etc.
Second, you vastly overestimate the desire that book publishers have to pay for good translations. I've seen books from reputable European publishers that were (badly) machine-translated by a contracted translator keen to dig their own grave. Yes, Harry Potter will get a good translation, but 1,000 lesser books won't.
>It's reference material of all sorts. Technical books, manuals, software localizations, etc.
Yes, I covered that under "formulaic content". As I've already said, a lot of this was already done decades ago. East Asian manuals and copies using machine translation are kind of a meme. Human translators are not losing any work from this. These are all businesses that were never going to employ a translator to begin with; it was either machine translation or nothing.
>Second, you vastly overestimate the desire that book publishers have to pay for good translations. I've seen books from reputable European publishers that were (badly) machine-translated by a contracted translator keen to dig their own grave. Yes, Harry Potter will get a good translation, but 1,000 lesser books won't.
Well, you're not citing any examples, so you you don't give me much to work with. I haven't read a translated book in ages (I don't think), but I do move in circles where translations, human- and machine-made, are freely distributed, and people kick up a fuss over machine translations that they didn't even have to pay for. I can't imagine a paying reader being any kinder upon realizing that they're traded their hard-earned money for machine-produced nonsense. You yourself admit that the translator who does this is digging their own grave. How is this not a problem that corrects itself?
Most paid translation work is just businesses that want to cover their ass so they don’t get blamed for a wildly shitty Google Translate. LLM translation is good enough for that. They’re not looking for art.
Theyll certainly go for something they believe is reliably mildly shitty, yea. LLM translation has largely solved the biggest issue machine translation had which was proper nouns. Its good enough for most business cases now
The thing is, there is no real evidence to back up the classification. The assignment between words and objects is arbitrary. A consistent definition for "planet" can be given and immediately after define the Solar System as being comprised of nine planets, with the understanding that a "planet" and a "planet of the Solar System" are similar but distinct concepts. The same is already done in biology with paraphyletic groups, with for example "reptiles" not including birds.
(I say this as someone who doesn't care either way whether Pluto is a planet or not.)
Yep, it’s a word. We can make more of them. We use them differently depending on the context. It’s strange that armchair scientists seem unable to understand how words work. They seem to use words fine enough? But they struggle with the abstract concept of what words are and how their meaning comes from how they are used rather than from committees.
I vote we cut science funding for universities since the humanities are clearly going through difficult times.
The Sun's gravitational pull on the Moon is always stronger than the Earth's. (Do the math and see--on average the Sun's pull is more than twice as strong as the Earth's.) So the net acceleration of the Moon is always towards the Sun, never towards the Earth.
So it's an exponential with a negative exponent? The thing about those kinds of curves is that since they have asymptotes to zero, it's hard to tell when you've plateaued.
>The current LLMs are making it quite easy to seperate software design from implementation.
I don't really agree. This kind of division of labor has always been possible, with architects doing the design and engineers/programmers doing the implementation. Architects who design systems with lofty requirements with no regard for the cost of their decisions are kind of a meme in the industry. I don't believe it's really possible to separate design from implementation, unless the person specifying the design is really knowledgeable about the problem space.
The Architects you're describing aren't reviewing the code and making suggestions for changes so your comparing apples and oranges. What I'm describing is more of a team lead to team members relationship.
I understand that. That review process is a bridge that imposes a maximum separation between design and implementation (that is to say, they can be separated so far, and no further). The two can't be fully decoupled.
reply