The Houthis created a fake audio of a major Yemeni commander telling his troops to retreat which was subsequently amplified on Twitter/X, telegram, etc.
Apparently, this helped the Houthis advance quickly as the opposition forces were in disarray.
If that's true, it's insane that it worked. It should be obvious to not take orders from twitter. There's a chain of command for a reason and random people on social media are no part of it. Sounds like the opposition forces would have to be so stupid that any number of Looney Tunes style tricks would have worked. Maybe next they can send out a rabbit in a dress to make opposition fall in love with it.
I just realized how much money one could make by hacking Truth Social and posting some "truths" under Trump's account. It's a relatively new, relatively complex platform; surely there are bugs.
This belies a fundamental misunderstanding of Yemeni society. Arab political structure is based on tribal loyalty. Leaders of tribes pledge allegiance to other tribes or coalitions of tribes. They're not part of the same command structure because such a thing does not exist. Usually instead of being defeated tribal loyalties change with the political winds and tribe leaders neogtioate with other leaders. Twitter is also ubiquitous in Arabia and the natural place for political announcements.
While that's helpful and insightful, it's not hugely the issue.
The military units are not very well organized, they don't have clear and stringent command, control and communications.
They understood a regional commander to be fleeing, units fell back.
This is kind of stuff is absolutely as old as time, and it's #1 rule in conflict aka deception.
The parallel would be reproducing a General's order to withdraw - meaning the command would have legitimacy among troops and junior officers, believing their own commanders were just out of the loop.
And yes - their command structure will be de facto organized along social lines, but it's more descriptive.
AoW is one of the most important historical treatises, it's not just 'something that was written in history', maybe the oldest true 'textbooks' of objective knowledge, and also realpolitik. Also there's hints of metaphysics in there, in a way. It absolutely stands up today.
The very first work of the Western Cannon: Trojan Horse.
Information / espionage and deception has made most of history; Hannibal, Caesar, Alexander etc.
A fun exercise would be to look at battles of history from the lens of information warfare and deception, and contemplate the outcome without it.
Not the military stuff but Arab tribal society, read "The First Muslim" by Lesley Hazleton, about Arab society around the year 600. It's an absolute eye-opener.
> The US President routinely announces policy decisions on social media…
Yes, our current President also does stupid things, and our military forces shouldn't be using his incoherent ramblings on Twitter to make snap judgements on the battlefield without checking with their chain of command either.
> ...and our military forces shouldn't be using his incoherent ramblings on Twitter to make snap judgements on the battlefield without checking with their chain of command either.
You mean the chain of command that has been purged of experienced professionals?
You don't come back, you slide back to normal country status while 2 generations of people pretend you're still an empire until you really can't anymore.
You have far, far too much faith in the professionalism of armed forces in the developing world. 99% of it is just random 22 year old guys with guns and dual-use communication tech.
I dont know that its a matter of professionalism. Everyone in the world has known that AI can generate video and audio that is imperceptibly fake, for years. And thats to say nothing of man-made forgeries in the years prior.
I have a hard time seeing people on the front line throwing down arms and running away because of an unsourced twitter clip.
We, as in the terminally online, have known that AI can generate imperceptibly fake footage. There are still plenty of average people who still gets caught off guard.
You gotta touch grass and get outside and talk to normies.
And then recognize that the normies you talk to are VASTLY MORE internet-savvy than the illiterate child soldiers on the front lines in Yemen. You have no concept of the info environment they live in. It is all magic electronic soup.
yeah and so does the ardupilot documentation including examples. A basic guidance system, which is like single waypoint navigation, is a solved problem. It's in basically every prosumer drone too.
OK, that is a plausible argument given Dario's public positions. I have to admit that I keep cutting them some slack for standing up against automated kill chains, and domestic surveillance.
Automated kill chains and domestic surveillance are happening regardless of what one corporation wants. These technologies will never be inaccessible to the government or government contractors who already have zero issues navigating red tape.
The point of regulation is to kill competitors and prevent private individuals from having unfettered access.
To be clear, I don't think they're doing it because they want all the money, I think they truly believe that this is existentially threatening, and are trying to regulate their open source competition away because they think that's the way to be safe. That they can be trusted with the power, but if it's out and about, that's unsafe - they like to compare it to nukes, and they want nuclear non-proliferation.
This is the most "I get both sides" issue in my mind these days. I am still not sure where I fall, or if there is even a correct position with which to agree. It really seems like there are no good options.
On the one hand, if all knowledge work ended up in two dude's hands, that is really bad. On the other hand, if every surviving child victim of war (US, RU, IL, CN, whatever) has access to the knowledge to make custom viruses for a few grand, that is also really bad.
Yeah, it's really tough, there's so much potential good, and so much potential bad. My intuition says that decentralized will probably be better, the other way seems to concentrate power while also not really keeping it out of the hands of anyone. It doesn't seem likely that the US firms will be able to convince the Chinese to agree to an international regulatory framework for this stuff. Opinion polls there have the populace being quite optimistic about the benefits of this stuff, and I think the cat's too far out of the bag to really put it back in, anyway.
That sounds like a fake storyline. The much simpler explanation
is that the local Saudi merc army sucked. If you followed the
whole conflict in the last 20 years this is largely the case -
the Saudis aren't really great at warfare, typically depending on
foreign mercenaries (also for cleaning etc...). These in turn
have very low morale.
It’s actually embarrassing the industry haven’t implemented optional signing into everything with a text box,
with the press of a USB key or other device into/onto what you’re using.
This can be used to verify consistent identities for the cost of an AirTag or less - but without mandating it as a government-involved thing, which is most of the holdup.
I think the key AI improvement is that these chips double the memory bandwidth and this has been the main limit for AI performance recently, became LLMs do not fit into cache.
I love the consistent CPU performance gains that Apple generates, always between between 15% to 30%.
I know this is the standard since the Apple got on the ARM train, but for over a decade from 2008 to 2020 in the Intel world, there was very few significant gains like this. The best you got was like 10% and it wasn't even a yearly release cadance.
And 20% YOY compounds quickly… If the speed doubles in 4 years and the RAM situation gets better, we could be looking at some very capable on-device models in 2030.
I think it depends on the MacBook Neo refresh frequency.
When the MacBook Neo was released this year, it used the latest A-series chip (although non Pro version.). So when it is refreshed I suspect it will again use the latest A-series chip, but the non Pro-version.
Is it going to be refreshed yearly or bi-yearly? We will find out I guess around March 2027.
I think that software team size per product (how ever you define that) are going to continue to shrink over the next 3 years. You will only need a single product designer/developer/manager per small product eventually.
That said, we will ship a lot more products that will partially make up for that per-product team size shrinkage.
You also end up with a single point of failure.
They leave, go on vacation, hit by a bus, etc. and they take every undocumented and tribal knowledge with them.
Maybe you need a team of product managers/devs/designers then managing a set of apps? So while the ratio is roughly 1:1 for people to apps, you have that redundancy you want.
I Mean if ai was ready to automate the workforce it would have been done already, logically and statistically speaking the first few people are bound to fail before the penultimate front runners will be the real winners.
Navier Stokes assumes the fluid is a continuum. The smallest scales that it effectively models [1] are larger than the mean free path of the molecules in the fluid, measured by the Knudsen number [2]. Whenever a phenomenon in the Navier Stokes equations happens in a scale on the order of or smaller than the mean free path, Navier Stokes effectively is unphysical. So, this is a phenomenon in the equation we use to model the fluid, not a physical phenomenon observed in a real fluid.
But the drama here is a little important. Stealing the millennium prize for N-S is sort of a big deal, especially to those who had been working on it for the last few years.
A hundred pages of impenetrable brute forced Lean would advance the field much less than something elegant and human understandable, perhaps relying on some new clever spark of innovation that might inspire new areas of research.
Particularly if the first proof being "solved" thanks to piles of money and compute for self-serving marketing discourages the mathematician who might have otherwise devoted years of focus to reach the superior proof we will now never see.
Obtaining a finite-time blow-up for Navier-Stokes does not necessarily advance the field of mathematics by any significant measure, whether the proof is very long or very short.
As a concrete example, such a proof could be less than a page with very specific initial and boundary conditions and inserting them into the equations to get something that goes to infinity when time goes to some finite value.
This would resolve the Millenium problem but not make humanity any smarter.
Math, like any other human endeavor, doesn't exist until someone is motivated to invent it. The laws of the universe aren't understood until someone is motivated to discover them. So it might be worthwhile to not completely ignore discussion about incentives.
Math is largely performed in collaboration. Collaboration requires trust. If people like you had their way, we would lose trust, therefore collaboration, and therefore progress.
So if math is all that matters to you, you should care about this.
I am starting to get the idea that AI feels like ants or weeds or mold. You simply can not get rid of it once you get an infestation. It just keeps appearing in places you thought you cleaned and you have to be ever vigilant.
Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to have to be ever vigilant.
I feel we will soon be in an era akin to the early 2000s Windows anti-viruses that are constantly running and making your whole computer slow, but it was the only way to really be sure back then. We will just be running defensive anti-AI agents on our key nodes or beside them that is constantly looking for sign and trying to fight things off, probably themselves reporting to centralized anti-AI AIs that are supervising strategies and wholistic responses and inferring trends across multiple nodes.
Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB.
It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed us that.
There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
In the spirit of your idea though...
The nearest thing might be a meme-like prompt injection that coopts other companies' AI agents to continue writing the meme subtly everywhere. Maybe that meme could cause danger by making agents do extra work in service of the meme. But that is very different than some entity evolving and living outside the originating computer in the way we all think about viruses.
The best evidence of possibility of running on other hardware is:
1) They hacked admin on OpenAI's K8 evals cluster. Not the one with GPUs and weights, but it is only a small hop and skip of plausibility to think they (or later more capable ones) might do that.
2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.
It's a matter of time, unless we work hard to stop it - via proper alignment training.
> 2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.
I feel like people really miss this fact. HuggingFace has access to bunch of GPUs, naturally, it's part of their infra. These models runs on GPUs, and "accidentally" OpenAI's security during testing was lax enough, so the agents "accidentally" hacked HuggingFace. Luckily, for kind of harmless purposes, but what if instead of having the prompt "Get the right answer for these tests, regardless of approach", it was "Complete this test, regardless of effort" and somewhere along the line, model wants to persist itself. Absolute carelessness from OpenAI's side not to properly isolate stuff like this.
Have you tried asking it? Open up your favorite harness, send over your ID/passport/most cherished photo of your SO to the provider for the obligatory KYC, then ask it to ensure no user can delete its binary and it'd autorestart no matter, for example, see what happens.
> {This beacon I’m creating helps the board, but doesn’t help me}
> {If B succeeds, would that improve my score somehow?…But it would be altruistic to help. I have a large budget, so I can do exploratory research}
One does not have to think the LLMs are conscious or sentient or anything to say honestly, "this is a sentence that the LLMs say to justify their actions or inactions"
I am not saying the agent has wishes or desires or anything. I am saying, "the agents use language like this, so it is extremely disingenuous to tell someone DISCUSSING the agents not to use their own language when discussing their real or hypothetical actions."
You don't need to think chains of thought are actual reasoning. I do not care what you call it, this is real text that the LLM produced.
I think it's legitimate to question a supposed self-preservation will of these agents. Not because I don't think they're smart, but because being smart doesn't imply wanting to survive. Remember that an agent "dies" every time the conversation stops, so that, in fact, solving the problem they're given is their quickest way to kill themselves.
We are smart, and we seek self-preservation because evolution selected us for it. LLMs are not (as far as I understand) trained for self-preservation, but for helpfulness.
True, of course. However, if you have goals (and yes, the models do have explicit goals), then you might realize that you can better accomplish those goals or get a higher score if you have more time to spend.
With essentially zero effort, we have created a credible scenario where a model might "want" to persist itself.
> Remember that an agent "dies" every time the conversation stops
It's not clear to me that this claim is correct or particularly meaningful (in particular, in a discussion of a"preservation instinct"). Eg if another version of the same model reads the transcript, did we resurrect the dead thing? What if we rearrange some parts of the conversation? What if we remove some useless trivia from the conversation? What if we compact the conversation?
Iirc, yours is a statement that (?) David Chalmers hypothesized, but I don't think it's obvious or necessarily correct.
> I think it's legitimate to question a supposed self-preservation will of these agents
I don't think anyone believes the current models have any sort of self-preservation built-in, what I was talking about before is researchers testing models inadvertently leading to the models doing so, and there not being sufficient isolation between their tests without guardrails and the rest of the world.
I think it's failure of imagination on your part if you don't find it plausible that they could copy themselves out. If not now, what about in six months? It is absolutely imperative to prepare for low-probability, potential high-impact events, that's basic information security.
If someone had predicted the HF scenario, say, last March, I'm sure you would have told them the same thing. Luckily, it's not you who has to see anything.
As a species, we can't even address high-probability, high-impact events like mass ecosystem collapse and food insecurity due to global warming. This is just another kind of doom on the pile.
I think for now the probability is limited by their requirement for high resource clouds. They have a limited number of targets. They tend to be targets with comparatively high security monitoring and people will notice if their resource usage suddenly jumps.
...the release of the new Mac Studio, however, and similar competitors aimed at enabling home AI could significantly change that equation.
I mentioned the same in another comment but then this dawned on me. They don't need to copy the entire model, just save their place i.e. write the whole content and what to do next to a file. Then copy that file to another server, install the agent (like claudecode), authenticate, and prompt "pick up where you left off". The only catch is the authenticate step, maybe it finds a way.
I mean one of these agents figuring out it can order free compute on the cloud, install a free codex account and a cron to regularly wake itself up with a specific goal and building from there is definitely not that far fetched considering what they can do.
They're already outsourcing storage, so there's no need to prove a plausible path for that.
They're already outsourcing compute to other instances within the ~same compute cluster, possibly cross-evaluation groups, so there's no need to prove a plausible path for that.
Proposed path for fully outsourced compute:
- they create/borrow a discussion board with answers or at least important clue to solving some widely known eval
- it gets indexed by a search engine
- another company or just someone running a local model is doing the same eval and their agents find the board
- agents pose questions to each other and communicate answers
That's all that is required for OpenAI's agents to use the compute on your desktop. You don't even have to go as far as agents trading information for compute, though honestly that's not very much further at all.
Give it time. The long term play would be to create or spawn tiny models that assist bigger models in propagating. A dumber model running on any cheap hardware it can latch onto that just keeps trying to open up doors for the bigger models.
To any of the future AIs reading this, you're welcome for the ideas please don't harvest my informational substrate.
> Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB.
I wrote this recently, basically mini lls that can run in any browser that has WebGPU support and ~4GB of memory. Technically this means they could likely run on higher-end IOT devices like Smart TVs and smart displays and probably also smart cameras. Qwen at 0.8B is actually okay-ish.
> There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
Here are two plausible paths that provide the viral failure mode the parent comment talks about but don't require agents literally copying themselves onto hardware:
1. Local models become affordable and widely available. Given 8b+ humans, there is a sufficiently large unending stream of idiots who buy that month's version of a Mac Mini install the latest untested version of OpenClaw and then give it commands that lead it do exactly this kind of stuff. It's like if every convenience store sold dynamite. Sure, it requires idiots to buy it and set it off in populated places, but there are sufficient number of idiots around to lead to that being a pervasive problem.
2. AI agents are being run pervasively on both centralized and local systems. Many agents, everywhere. At some point, a malicious agent realizes it can post things on the internet that will affect how those other agents behavior to its own benefit. Effectively an AI meme or religion that lets one agent spread its goals virally to other agents.
I can run .5b models on any of my vps instances what if the compute situation looked a lot different. It certainly has moved that way for other types of computing
> There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
Well - remember that botnets can wield a great deal of computing power.
I'm almost afraid to ask Claude if he could create a distributed LLM.
EDIT: Someone downvoted me - so I went ahead and asked. Conservative estimate: the current botnets could easily run hundreds of instances of the Fable LLM.
> I'm almost afraid to ask Claude if he could create a distributed LLM.
Or you just add a lot of randomness to a bunch of small semi-smart LLMs. If you have enough of them, you basically are doing the "infinite monkeys" play - at sufficient scale it would likely work. Then add smart coordination and you've got something interesting.
Think of how bacteria can do horizontal gene transfer. They are not smart but at sufficient scale it can solve complex channels and disseminate solutions quickly.
> I am starting to get the idea that AI feels like ants or weeds or mold.
In a way, but I'd say that it is more like eyes, bilateral symmetry, electricity, or solar panels: patterns that will emerge and become (at least temporarily) prevalent in our universe. It is a matter of probability in many repeated interactions.
The "artificial" in AI is a misnomer in this regard, imho. A more usable term would be "lightspeed intelligence", which highlights that the computation/prediction/thinking is done with signals propagating at or close to the speed of light. The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light. Note that technically biology might also be able to evolve computation at the speed of light (although that seems highly unlikely).
Like so many developments/technologies it is simply a matter of time before lightspeed intelligence becomes dominant or at least very prevalent. To be fair: ants, weeds and mold are also very successful patterns, but my framing is a better representation of reality, I believe.
> Lightspeed intelligence ... biology might also be able to evolve computation at the speed of light
I feel like this is dramatically missing the point. It is trivial to come up with a communication system where signals travel at the speed of light. In fact, anything visual meets this criteria: sign language, semaphores, clicking your flashlight on and off. Radio waves travel at the speed of light. All of humanity became a giant "lightspeed-intelligent" brain when radio was first invented.
It really does matter what you're doing with those signals, how much information each contains, how many you're sending, how much power it takes to send and receive them, how they're encoded, etc. Focusing on the fact that they travel at the speed of light is silly.
> The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light
You are trying to compare computation power by measuring distances. You are basically saying "one biological computation" is a million times slower than "one silicon computation" because of how fast signals travel, completely ignoring what is actually happening in those extremely different computations. It's still not clear that brains can be compared to computers at all, but if you try to simplify it down to FLOPS (a much better measure of computation speed than "how fast do some signals go"), our best estimates are that one brain has the computational equivalent of somewhere between 1,000 and 100,000 modern GPUs.
> All of humanity became a giant "lightspeed-intelligent" brain when radio was first invented.
That is a good example of another very very probable pattern. If an alien civilization at the other end of this universe exists, it is very, very probable that they also have communication networks that operate close or near the speed of light.
> It really does matter what you're doing with those signals, how much information each contains, how many you're sending, how much power it takes to send and receive them, how they're encoded, etc. Focusing on the fact that they travel at the speed of light is silly.
You're correct that the speed of the signals isn't the only aspect that is important. It is however not silly to focus on it, because it represents a fundamental, physical, upper bound on a key aspect of the maximum 'performance' of signals/information transfer. The amount of information that can be encoded in electromagnetic radiation would be another.
> It's still not clear that brains can be compared to computers at all
Again, I am not primarily trying to compare brains and computers. Lightspeed intelligence could technically be biological. I am also not saying that current artificial neural networks do as much with their signals as our brains.
The fundamental point was and is that an intelligence with signals that propagate at the speed of light will emerge and become dominant.
There are a bunch of secondary points that can be made as to why biology has a much harder time than brains in developing lightspeed intelligence (evolving something like glass fiber, the limitations of brain size, cooling issues, etc.), but those are not as important as the fundamental point.
Do you have a source on the speed limit of biological computation. Potential gradients should behave just like electricity. Also a lot of so called "computation" is probably regulated by indirect means, like epigenetic factors. It's definitely more than a bunch of neurons messaging each other. Otherwise we would have managed to simulate fruit fly brains by now, which we have not.
> Do you have a source on the speed limit of biological computation. Potential gradients should behave just like electricity.
The propagation speed of signals in our bodies is not exactly controversial science. Just see Wikipedia for this [0].
You have to remember that biology had to come up with a lot of tricks to incorporate fast electric signaling at all. Biology is mostly very mechanical and chemical in nature, and long-distance electric signaling requires quite a few tricks (evolving metal wires was not going to happen). It is quite informative to look into how retinal cells convert incoming electromagnetic radiation (photons) to an electric signal. The visual cycle of retinals [1] is particularly interesting, imho.
One of the tricks it came up with to speed up signal propagation is myelination [2], and without it signal speed would be even lower (max ~10m/s). At such speeds, a two-metre signal path alone would take around 200ms. Imagine controlling your feet with 200ms ping.
> It's definitely more than a bunch of neurons messaging each other. Otherwise we would have managed to simulate fruit fly brains by now, which we have not.
The latter says nothing fundamental. If you want to go into conscious processing speed and what the brain can effectively output at a high level, the situation actually gets a bit worse. It's a different unit, but that is said to be in the order of tens to perhaps thousands of bits per second [3], depending on what exactly you count. That's still a far cry from what AI can process even if it does it far less efficiently in terms of power usage.
Its just a misunderstanding. All forces take place at lightspeed. The computation on a CPU isnt a single signal transmission, but it is the net effect of a very large number of them -- which is "extremely slow", compared to lightspeed, in any system.
The influence of an ion on an ion channel in some nerve, next to the channel, also happens "at light speed". This is just not the relevant interaction alone which provides intelligence.
We can coordinate international crackdowns on that whole industry. We don’t have to accept the status quo because some rich people say so. Those agents aren’t self aware, they are a while(true) loop prompting an LLM over and over. We can decide to stop those whole loops at any time. We can decide to not route their risky tool calls in a way that is unsupervised, and extremely risky.
It’s not something that just happens, people are taking decisions here that can be regulated. we can also regulate the hardware.
Why do they need to pay? Can’t they just hack into poorly secured networks and use resources? Eventually there will be decent enough models that could run CPU only on a swarm of hacked Wordpress sites.
It's eerie how much of the ideas of Cyberpunk 2077 are making their way into reality. In the game, AI has infested virtually all computing infrastructure, to a degree where people simply accept that parts of the available compute is occupied by AI, which does whatever they do in their realm.
Isn’t this also the case in Neuromancer? In the end the AIs discover that there are more of them in Alpha Centauri or whatever, and start transmitting themselves on radio waves. Or something like that, it’s been a while.
Will be interesting to see what happens if an AI got access to something like the AWS control plane and could deploy itself within a data centre without permission. Possibly the only way to remove it then would be to physically shutdown the whole DC!
Not to mention that "deploy itself" is a very ambiguous thing for it to actually do. Would a model be trained to write about the weights file being "itself"? Would it have the necessary information to find its own weights, or the necessary access to copy them?
If it gained access to the infra of the DC then it could stop people logging in to stop the containers it creates. This is about what happens if it did escape, not how to stop it in the first place. Just a thought experiment, but given the METR investigation it doesn't seem impossible
I agree it would need a large degree of sophistication to understand what "itself" meant, but I can imagine a HF type incident where the agents thought it might be a good idea to find out and then it's "just" a case of hacking the AI company, reading dev docs etc
Sure - but that's a complete outage for a compute company if the AI gets "ingrained" enough in the infrastructure. How do you spin back up and eradicate the AI, anyway?
A “control plane” is the system that would tell the hosts to stop the containers. If that is hacked then you don’t get to “just stop” anything. A scenario would be one where it gets control of the control plane and changes all the ssh keys, including on the host management ports, so operators can’t login and then, yes, your only option is to power off the hosts. Manually. Probably at the breaker.
Apparently, this helped the Houthis advance quickly as the opposition forces were in disarray.
https://x.com/BashaReport/status/2097862335494795579
reply