> In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.
It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode.
Overall I think this all makes me appreciate humanity a little more. Sometimes the truculent dev who stubbornly refuses to go with the flow produces very valuable insights, as a small example, discovering things the status quo thought unlikely.
I agree - I think one of the biggest reasons memory systems fail in LLMs is that they have poor theory of mind - they're terrible at considering how others will react. Both humans yes, but also future versions of itself. When asked to give advice to itself, it pontificates at length about trivial stuff it already knows and fails to emphasize the stuff that was new or interesting
Alfred North Whitehead talks about the notion of "Importance" as fundamental to the human (and all other living things) way of being. Living creatures first and foremost select information that is important to them from the broader environment, and then make decisions and take actions. (Of course at a physiological level it's much more complicated than this, but it's a sound philosophical description of how living things work.) LLMs lack this entirely. They have no selective filter because they weren't designed to have one (interesting question if you could even do that) and they're not evolved beings with a survival imperative. When they enter a self-conscious or other-conscious mode like you're describing, they just emit text that looks like the thoughts of a self-or-other-conscious person. They can't direct a stream of attention or hold a concept in the forefront relative to other concepts or (to your main point) think about what matters to the other person/being because they don't experience "matters." All they can do is emulate the verbal output of beings that actually experience these things, and given that I don't find it surprising they get trapped in loops over trivial things.
It will be interesting to see if LLMs “evolve” importance as they run out of ram and storage to think.
For humans with limited space, “Importance” is an output of a first pass “of the available infinite amount of information, what do I need to consider to solve this problem”. And it’s not necessarily a good algorithm. People misidentify “importance” all the time.
This part was really interesting to me. You could imagine future models using theory of mind to reason - "I want to do this strategy, which means the other agents (who think just like me) will want to use the same strategy, which means I should change my strategy to account for that."
Maybe they did? Or maybe they don't realize they're playing against other agents.
Of course, if agents running different models are competing in these 'games', I wonder how much of the theory of mind would translate.
(N.B. - I don't think they're all defecting from the first turn, although it's not clear. It just says 'they all defect at the same time'. So if they're playing for 10 iterations, they might all decide to defect after turn 6, but since they all do it together they don't get the benefits. I would expect these models know that optimal strategies in repeated prisoner's dilemma start with cooperation.)
Ooof. There's a whole lot of research related to multiagent Systems and Epistemic Logic (and plenty of other logics) from when the "original" MultiAgent Systems where studied. Im very familiar with van der Hoek and Wooldridge work (vd Hoek was one of my PhD supervisors).
A lot of it went over my head as way too theoretical, but I recommend a lot for whoever wants to delve on the logical side of agents interactions.
In VC investing there is the saying that most of the value comes from the outliers. I think it’s the same for a lot of domains. I read an interesting article recently on LLMs homogeneity when writing fiction: https://arxiv.org/abs/2604.03136
It seem that at scale LLMs output is the average of their dataset, they all cluster around the same space, where human creativity comes with more variance, exploring way more of the space
Can't the LLMs build scripts/tools etc that help generate creative ideas?
e.g. have the LLM generate multiple lists of characters, themes etc and then have a script hooked up to a RNG pick from the lists to create more "creative" ideas.
I think there’s merit to this approach, particularly to highly parallelizable tasks.
Rather than giving many agents the same prompt, introduce random variations that lead each agent in different directions. For a single bug, you might fire three agents, and later select the best result:
“Fix this bug. The solution is a trivial typo.”
“Fix this bug. The solution centers on correcting a bad assumption.”
“Fix this bug. The solution will require a complete redesign.”
You could follow the same idea with varying the input context, or by adding artificial constraints to the solution. Like telling each agent to “fix the bug, by only modifying file a/b/c”
I fear there is a strong sentiment that people are somehow special and magical and AI will always be a pale comparison.
The truth is likely that people are simple heuristical machines. There are already studies that strongly suggest our consciousness is a story our brains make up and we don’t really know why we do some things.
My fear is by rejecting the idea that AIs might in-fact become as capable as people, because “people are awesome!” We will be blind to real danger. Because of our hubris.
People are indeed special, in the sense that what we consider creativity, art, etc are human concepts. They are an expression of humanness, and are evaluated against human experience and human expectations. It’s what comes from a collective human consciousness.
"A lot of creative work is remixes" in no way conflicts with "That's pretty low-value creativity."
Hiphop made using samples a fashion. You can think of it as choosing to restrict one's in order to explore something deeper, like a painter deciding to paint with only two pigments to exercise their skills at shading at patterning. Or you can think of it as a cheap way to make demo tracks in your bedroom.
If you want to find actually creative things, look outside of mainstream, Disney, etc.
I'll leave you with this link. You tell me if you think it is just a remix of something earlier:
This is roughly the same as putting a group of random people and telling them to think outside the box which in my experiences never yields much of thinking outside of the box
> This is roughly the same as putting a group of random people
In the book Range by David Epstein [0], he mentions that research has shown that groups of people with very mixed backgrounds leads to much faster rates of problem solving and innovation. The reasoning is that someone will have an experience or piece of knowledge that directly applies to the problem.
As an example, two different groups were given the same problem to solve. One was a homogeneous group of academics. The other had a mixture of academics and non-academics. Both had to solve a problem that involved flowing liquid. The mixed group solved it faster due to one of them having had a father who was a plumber.
i mean i agree diversity of opinion is good for problem solving, but the comment you were replying to is saying telling people to think outside the box doesnt really do anything. The e.g. also seems really cherry picked, im sure after a 100 samplings, the academics would solve the problem faster most of the time
> im sure after a 100 samplings, the academics would solve the problem faster
Repharase thinking:
Some problems are better for academics, other problems require other background information.
The point of the GP isn’t that they are all academics. It’s that one solution group is homogeneous and the other is not.
The claim is: the breadth of a heterogeneous groups is more likely to encompass a good idea than the depth of an academic group.
Also, I have to say that academics have (imo) little experience working together cooperatively, so even on academic problems you’d be better off with some non-academic people in there to act as buffers.
every good team I was ever on used this line of thinking during the hiring process. we whiteboard a problem we worked on in the past and discuss it with the candidate and look for anyone whose thinking about the problem raised our eyebrows!
Another (semi serious) possible explanation of this is that LLMs were trained on a huge volume of text from Reddit. I wonder if you'd see similar inhuman-looking behavior if you did an iterated prisoners dilemma with a group of Reddit power-users as the participants?
I had the same thought. Also, a significant amount of human knowledge/context/communication is done privately on the internet. Imagine if DMs across all platforms made it into training data...
Wouldn’t it be better with different models working together? To me it feels intuitive that this type of degradation would be avoided with different models.
> It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode.
We need better words to describe this than "self-awareness" or "consider". These words mean fundamentally different things when speaking about humans or clankers.
Why? Unjustified linguistic analogy is a huge part of AI companies success. If we want to actually understand how LLMs work and differ in their operation from humans we need to develop appropriate language, not latch onto some totally unproven equivalence that makes Dario's eyes water with all the free boosting we are doing for him.
Lay people might talk about traditional computers "thinking" but there's a reason actual computer scientists developed precise language around what they were actually doing instead.
We should be critical and use language appropriately and try to describe things correctly. LLMs do not have plastic brain like humans do, which already fundamentally changes what "self awareness" can means for them. There's also the small problem of LLMs not really having an identity beyond the stateless computation that is a chat session which is just some computer code executed on a server. lol
> Lay people might talk about traditional computers "thinking" but there's a reason actual computer scientists developed precise language around what they were actually doing instead.
> not latch onto some totally unproven equivalence
Precisely. This is akin to my dad, back in 1995, telling me to stop moving the mouse cursor in MS Paint because the computer was "thinking" (he heard the HDD buzzing). That I can understand.
But surely we can expect more from people here than from a carpenter in a tiny mountain village who bought his son a computer in the 90s. (The second one in town!)
Before the mouse pointers started getting hardware-composited, needing to redraw the pointer in a new position and redraw the underlying contents to its old position actually used to slow a computer down enough that you could eyeball the effect of vigorous wiggling in something like a Mandelbrot renderer.
> In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.
It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode.
Overall I think this all makes me appreciate humanity a little more. Sometimes the truculent dev who stubbornly refuses to go with the flow produces very valuable insights, as a small example, discovering things the status quo thought unlikely.