So the interesting thing is that this shrinking “linguistic diversity” is fundamental to how an LLM works.
The LLM is a big probabilistic statistical trick. It picks the next token based on certain words are simply “the best” because they are specific and well connected to other tokens. The is gives them a great overall cost function. (Basically a good score on “will it make sense in context” while also having specific meaning that makes it better than other options, unambiguous in common use and being a single token rather than several).
You can trim those tokens, but then you just get other tokens that are “the best” tokens (and you’re worse off because the output became less clear).
The cool thing is this seems to get worse the more powerful and accurate your model is, because it is picking technically / statistically perfect tokens, not tasteful ones.
It's not fundamental at all, that's what randomized sampling is for. Try tinkering with a base model and you'll be surprised how diverse it is. The semantic collapse happens in post-training that is using the current methods.
Yes, although admittedly I haven't tried recent ones which have a bunch of synthetic data in them (mainly to aid the reasoning), and are usually only available after mid-training. One look at the logits/output distribution and it's clear the base is pretty different.
that's a very longwinded claim that inference providers are sampling with temperature T=0, but is that even true? a sufficient explanation would be merely sampling at a lower temperature compared to human sources providing similar content
The whole point is that we’re TikToking the software industry and no one will know what to tell the ai to get the software written in a way that works. It’s in principle the same as idiocracy where no one knows how to make burrito covers anymore because shit won’t grow
You can train and run small models on an old gpu. That’s what I’m doing now at, well, much older than 17. Does it produce a useful model? No. Not even remotely.
However, I do learn stuff about models that takes it from “magic” to “useful tool I understand the limitations of.”
Do I do it for that reason? No not really, I’ve never had luck learning something because it would be good for my career. I do it because at my core I’m a bored teenager who wants to make the computer do cool shit.
On the one hand we have Claude doing my work for me, on the other hand it is laced with unoriginality, on the gripping hand everyone will get all references.
All witty stuff disappearing and turning into LLM mush is kind of the maximal expression of capitalism for capitalism’s sake. We don’t care if things are novel or interesting, we’re producing “content” to fulfill an imaginary content demand that we can feed to make money.
The best way to feed it is to create stuff that’s just barely enough degrees of quality from a viagra spam email that it can pass for original thought.
reply