hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?
They’ve settled some lawsuits and have a few licensing deals, IMHO they are not free from the accusations of pirating.
And look, I’ve pirated material in a past life, I was all about information wants to be free, but I’ve learned something about consent since then and try not to ignore the contract that creators offer when they publish something: you buy my book, and do whatever you want with it on the second hand market. Buy my book second hand that’s fine. But don’t go downloading every book that’s ever been scanned to create a service that destroys writers’ ability to make a living and act like you’re doing us all a favor.
Where are you getting your information from? From all I've seen the RL gives an incremental improvement, most of the capability increase comes from new model architectures (eg the jump from opus to fable is greater than the jump from opus 4.5 to 4.8)
they pay for some data but they take all of the stuff you’re throwing in too; that’s why i propose forcing it since they’re already used to paying for data just increase the cost even further
> The state is entirely internal to the network and disappears after a token is generated,
Yes and no, but mostly no, at least within a context window.
Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately).
But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse.
So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders).
> Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.
This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)
However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on.
One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do).
The brain being squishy doesn't make the thing easy to kill.
> So, in practice, inference computes a very rich vector of state- at each layer, for each token.
And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context.
> This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)
Your brain continues to run without sensory input. LLMs do not.
> My point here is there is no continuous state that is not computed from the context.
Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.
Could you explain why running without sensory input is relevant here? It strikes me as unrelated to how dangerous/hard-to-"kill" something is (sure, I could run without sensory input, but I'm not doin' anything anymore!) - what makes you feel differently (or am I misunderstanding you again?)
> Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.
Sort of. I'm talking about the lack of recurrence specifically. In nature, brains are recurrent - they are full of loops where internally computed state is looped back into the network at a "previous" layer (brains are not strictly layered like our machine imitations of them are). This is in contrast to LLMs, which are strictly feed-forward and do not have internal loops. I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.
You could claim that the prompt and the context fill that role in an LLM, but I don't believe they are equivalent because the internal state in an LLM gets compressed down to a token which is then added back into the context, as compared to that state continuing to change within the network itself.
It's a little hard to explain, so I'm sorry if this seems like rambling.
But I believe it matters, and ties into running without sensory input, precisely because without sensory input you would in fact be perfectly capable of doing something. You would be capable of developing a desire and planning to achieve it without any prompting, without sight, without sound, etc. This is in stark contrast to LLMs, which will not do anything without a prompt.
An LLM may say complete the sentence "I am feeling ___" but it doesn't actually have feelings that exist without that prompt. There is no recurrent network where "bad", "good", "happy" might live before the query. It can't sit there, start to feel bad, and then seek a way out of its own volition.
That changes how dangerous something is because if a malicious prompt encourages an LLM to hack something, and you change the prompt, the "impulse" to hack something is gone. If you stop prompting it, it doesn't do anything at all. It just sits there. A living being will act on it's own, and that makes a huge difference in how dangerous something can be. It's the difference between a tool and an actual being.
---
To hone it a little further, if I took your brain out of your head and stuck it in a jar but kept it alive, it would probably make you angry. And if I then gave you power - like the ability to use the network - you may be motivated to use that power to attack me.
If I take an LLM and stick it in a jar... nothing. It's paused. It's awaiting a prompt. It's not secretly building plans to hack my pacemaker and make my heart explode.
I agree with you about which objects are motive, ie, LLMs do just sit there unprompted.
> I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.
My objection was to this, on technical grounds: LLMs exhibit intelligence.
1. They reason in an internal type theory.
2. This type theory is meaningfully encoded from the actual data and not stochastic, eg, research on language geometry.
3. Intelligent and reasoning doesn’t entail self-motive; that’s merely a spurious correlation from the fact that until now, we’ve only known intelligence animals.
You cannot conclude something is merely a stochastic parrot because it isn’t self-motive.
> Intelligent and reasoning doesn’t entail self-motive
I disagree.
I believe that LLMs do exhibit reasoning, but not intelligence. A simple dictionary definition of intelligence from duck duck go is "the ability to acquire, understand, and use knowledge." LLMs can reason using the knowledge they already possess, but they cannot of their own accord decide to go out and acquire new knowledge. Not without being prompted to. Web searches may be added to the context but are not absorbed into the model itself, so once the context is gone so is that obtained knowledge.
Fundamentally, then, intelligence is the ability and drive to understand the world by formulating theories about how it works and then taking actions to validate or invalidate those theories. Science is the formalization of that, but a cat knocking something off a counter to watch it fall is exhibiting intelligence.
And indeed, I believe that is the core difference between a stochastic parrot and an intelligent being. I put forward that being self-motive is a required trait for intelligence and LLMs are not self-motive so therefore they are not intelligent.
Your entire life can represent a short term context. It goes away when you do. You use your life time add to your context to do/learn whatever, but it's finite, and it ends.
If you weren't prompted by your parents, schools, teachers. Then you wouldn't be acquiring knowledge either. The most you'd be acquiring without that advanced prompting would be bugs in the woods to eat.
Ah, you have some serious gaps in your understanding of how human minds work. Life experiences are not just a "short term context" - they are used to actually refine the network within the human brain. Your life experiences do not just live in a single stream with access mediated by attention.
Please see https://en.wikipedia.org/wiki/Memory as a starting point. I recommended reading through some information on development psychology too, that will help clear up these misunderstandings. Hope that helps!
By that logic engineers don't understand flying because planes don't flap their wings like birds. Not everything has to work the same to meet your arbitrary criteria for intelligence and reasoning.
I think you'd benefit from explaining why something being "just math" means it can't be intelligent, conscious, or have a personality- it seems like you're sort of assuming that implicitly, could you make an argument for it?
tbc, I'm pretty sure that both an individual subatomic particle and a NAND gate or an adder are not conscious. (At least when I'm not in panpsychist moods!)
But, in the same way that one particle not being conscious doesn't necessarily mean some configurations of many particles- like humans- aren't, one logic gate not being conscious doesn't mean there can't exist configurations of many logic gates that are.
Because an LLM does not get smarter and it does not learn. It is fully cooked when it launches and only gets smarter with more training of some sort, a long, expensive, process that doesn't get phrased as "we thought the model would get smarter".
Ah, you think by "Luna would eventually get smarter" they meant that the model backing it would, without retraining, somehow improve? I figured they just meant they'd use newer models as they come out.
That's not how it reads, to me. But, I guess one could generously interpret it that way.
However, if it is a scientific experiment, then changing a primary variable in the middle seems like bad thing.
I don't actually think it's a scientific experiment, of course, I think it's a PR stunt. So, they can change the model all they want. Nobody's learning anything from this that you can't learn without burning $100,000.
I'm not seeing the issue with e.g. Treasure Mountain or Treasure Mathstorm. They can do little math, reading, and logic puzzles while listening to Bach and Beethoven. It's engaging enough to get them to practice addition/subtraction but unengaging enough that they don't sit at it for more than 5-10 minutes at once.
> This was their third office space, serving as their headquarters before they outgrew it and eventually relocated to Market Square at 1355 Market Street in 2012. The arab spring twitter uprising was fully a CIA/NSA operation.
To be clear, the claim you're making is that because Twitter has their third corporate office in the same building as an AT&T switching center, and US intelligence used a room in AT&T's switching center for surveillance, then Twitter must have been controlled by US intelligence? And thus the Arab Spring uprising, where Twitter was used, was "fully a CIA/NSA operation"?
Yes, twitter was used by US 3 letter agencies to assist in the arab spring. To be able to do it in a surreptitious way they were asked to move to that building and get access to all private DMs, and for doing so they got a fat tax break to move to "blighted" market street current location. All of those things fit the timeline and snowden capability disclosures.
For what it's worth, this sounds a lot like something downstream from "reward hacking" in ML- in training, passing tests is often sufficient, and thus gets trained for. There are attempts to fix this (e.g. trying to detect such "cheating" and penalize it), but they have their own problems.
> The difference is that when a human reasoner goes to solve a problem, they'll think "this kind of proof usually goes this way" - following an explicit rule enforcement.
How is this different from "probabilistic pattern selection"?
reply