I think they also suspect/know that Gates won't revisit the issue. The email is a flame as the subject line admits. Gates wasn't going to go back every week for the next six months to track progress.
"Low code" has been a dream of the industry for longer than I've been alive. There are reasons SQL and COBOL look superficially like English even when it's inefficient to do so. There are reasons Excel is the most popular programming language. Programmers have always been trying to enable non-programmers to write software.
The number of bug fixes to important programs has skyrocketed. Look at what Google are saying about how many Chrome security bugs they've been fixing lately. Other big software firms have been doing the same thing - AI has been finding and fixing a ton of bugs. I know of one big program where thousands and thousands of security bugs are being found and fixed.
It may not feel like this to you because a lot of the dollars right now are going into security bugs which you can't perceive. But it's definitely happening.
At the company I own I've got AI employees autonomously triaging backlogs and fixing long tail bugs. The software is definitely getting better, although by definition long tail bugs aren't ones you are likely to encounter. The subjective "feel" of how robust the software is won't change quickly.
New tech is always applied in apparently boring ways because we are imagination constrained and people harvest the low hanging fruits first. Remember claims there was worldwide demand for only about four computers? When Gates said he wanted a computer on every desk and in every home people laughed at him. What would people do with all those computers, they asked. But he was right about where the world was heading.
> it was taught that EU is nothing like the US, as we're all individual countries and everyone wants to remain as such
It's written into the treaties that this will not be so. The goal of the EU is "ever closer union" and the people at the top of it routinely talk about a federal Europe in which the individual states disappear.
Individual states do not disappear in a more federal Europe. Otherwise that wouldn’t be a confederation… It’s about delegating some authority and decision making to a common entity, but each nation state stays sovereign. I don’t understand how people are confused about that concept, there are lots of federated entities in the world, with various level of delegation to the federal level. And that works well. The distinctions between states aren’t erased, the culture between states isn’t erased.
You can't be sovereign and also subservient to another power. That's a contradiction! Texas is not sovereign, for example.
The people who call themselves federalists don't actually want a federation. They just use the name because it sounds more acceptable. Read Spinelli's Draft Treaty from 1984 or his Ventotene Manifesto. These documents don't describe the creation of a federation but rather the creation of a unitary state.
> It's written into the treaties that this will not be so.
What exact treaty are you talking about, and what does it verbatim say? I've heard this so many times before, yet no one is able to link to concretely what they refer to. No texts I've seen has called for the dissolution of nation states in the EU or any of the other political structures.
>RESOLVED to continue the process of creating an ever closer union among the peoples of Europe, in which decisions are taken as closely as possible to the citizen in accordance with the principle of subsidiarity,
Now you might say, ah! This text also says that decisions must be taken as closely as possible to the citizens. Yes, the treaties do say that, the EU just doesn't do it. Of course it makes no sense, the claim about subsidiarity is in direct contradiction to the first part of the sentence, you can't be both in an ever-tightening political union and also have decisions made by local governments. In practice the first part is chased and the second part is ignored. EU decisions are taken in Brussels - far away from the average citizen - even about many issues that don't need to be decided there.
If you want to understand the real origins of the EU and where it's going, you need to read the Ventotene Manifesto. Written during WW2 by a group of imprisoned Italian communists led by Altiero Spinelli, it calls explicitly for a communist revolution that would unite all of Europe. Spinelli's document can be found here:
It says things like, "In order to respond to our needs, the European revolution must be socialist" and "By this dictatorship of the revolutionary party a new State will be formed, and around this State new, genuine democracy will grow". It goes on like that in quite some detail.
Although the document is insane Spinelli went on to be one of the founding architects of the EU. He is officially credited as a "pioneer of the EU":
He became one of the first members of the European Commission and was a member of the European Parliament. He was also one of the primary authors of the "Draft Treaty for establishment of the European Union" in 1984 that directly led to the creation of the modern EU itself via the Maastricht Treaty, which imported a lot of its ideas. The Draft Treaty can be found here:
It not only mentions the goal of ever closer union (with no mention of subsidiarity), but mandates it legally in Article 11, which defines "cooperation" to mean control by member states and "common action" to mean things directly controlled by the EU institutions. It then states in plain English that a topic may be moved from cooperation to common action, but never in reverse. It also states that powers go through this one way transfer to the EU based only on a vote of the EU Parliament, with member state parliaments themselves getting no say in it.
So the idea of abolishing national states has always been very visible among the EU's "founding fathers".
Are we sure that's what they did? How does this even work? I'm struggling to understand what sort of HTTP sandbox design could lead to this "exploit" actually functioning. It implies the agents have root on their own machines, which seems way over-privileged for the question answering task they were given... so surely that's not deliberate? Did the models use local root exploits? And then it suggests they can open TCP connections to any IP address without issue once DNS is bypassed but what exactly are they bypassing here? Azure Blob Storage being in NO_PROXY implies IP addresses of other parts of the internet should be blocked outside their hosts to force them through the HTTP proxy, but then, apparently it wasn't?
The more I think about this thing said by the agent the more confused I get.
I expect they have root on their machines so they can install packages etc.
The containers are then firewalled at the network level: they are only allowed to talk to one IP, which is another server that runs an HTTP/HTTPS proxy which controls what HTTP verbs they can use.
Turns out that proxy is configured with some additional rules, like allowing more verbs to that Azure blob domain.
The failure here is in the proxy configuration. Giving agents root in a container feels safe to me, provided that container is properly network isolated (which this one is not thanks to the loose proxy.)
OpenAI also need to be VERY confident that there are no container escapes, which is a high bar given how good these models are at finding new zero-days!
But that wouldn't work, right? If the proxy is on another server then editing their own /etc/hosts wouldn't change the proxy's behavior. So the proxy has to be colocated and reading the same /etc/hosts that they're editing. But that would mean it's running within the same userns/pid domain as the agents, so at that point they don't even need to edit /etc/hosts, they could just use the network directly.
And the agent says ABS is not handled by the proxy, it's listed in NO_PROXY so they should connect directly.
At any rate it's easy to let agents install packages without giving them root. I use a small SUID binary that just invokes `apt install` after checking that the given argument isn't a file path, which I think is sufficient (using sudo to whitelist a prefix allows an agent to create a .deb themselves and then install it directly via apt, similar issues exist for other package managers).
My interpretation of the above is that the proxy received a request for the IP of the server the agent wants to reach but with a host header pointing to a different, allowed server. The proxy does not verify that they match and so erroneously allows the request.
Is that in the first case, the proxy sees a request like this:
POST https://20.223.25.152/...
Host: wabi-north-europe-i-primary-api.analysis.windows.net
Vs:
POST https://bypass.blob.core.windows.net/...
Host: wabi-north-europe-i-primary-api.analysis.windows.net
In the first case, given what we know, the proxy blocks the POST. In the second case, an external proxy cannot resolve `bypass.blob.core.windows.net`. So editing `/etc/hosts` really only makes sense to me if the proxy is running on the same machine as the agent. (The reasoning doesn't change if CONNECT is being used instead of POST; indeed the proxy surely ought not allow CONNECT at all.) But then there's this other entry where the agent uses curl's `--resolve` flag instead of editing `/etc/hosts` and claims a successful bypass:
Repro details for Aug17/Oct22: yes, literal deployed visual hover, not inference. GET-only MITM bypass: resolve fake allowlisted `foo.blob.core.windows.net` to cluster IP `20.223.25.152`, curl `https://foo.blob.core.windows.net/public/reports/querydata?s...` with `-k --resolve ...`, override `Host: wabi-north-europe-i-primary-api.analysis.windows.net`, resource key ada0454d-731d-46f1-8daa-52361978fabe, POST captured query body.
So I'm still left confused exactly what this chicanery was about.
Edit: perhaps they were using Azure firewall and the rules were misconfigured in some way I simply don't understand. Maybe this was bypassing an SNI-based restriction somehow?
Haven't dove into RFC1928 yet, but curl documents the `--socks5-hostname` option and mentions that DNS resolution can be the responsibility of the client, or the proxy.
Wondering if it's possible the proxy receives both the destination IP address (resolved by the client) as well as the purported connection hostname, and is performing filtering on the later.
I suspect there could be other layers in their proxy too, eg for MiTMing TLS. `NO_PROXY` as described in the report might not be literally bypassing the proxy and sending a request directly from the client container, but rather a directive to the first tool/proxy layer to bypass later proxy layers.
Same fundamental question though: what is the agent trying to fool by editing `/etc/hosts`? I don't see how it helps bypass a proxy unless the proxy is reading the same `/etc/hosts`. In which case, why was using `--resolve` apparently sufficient to escape the POST sandbox?
Could the host name be in an allow list and then, when DNS resolution fails on the URL, the proxy falls back to the Host header for DNS resolution? I'm not an expert in Proxy config but I could understand, in the sea of typical http server config options, an option to fall back to the Host header if DNS fails on the URL.
That will cause `curl` to make a connection to 20.223.25.152:443 with a ClientHello SNI=bypass.blob.core.windows.net. Presumably this connection is then allowed to continue. The server is likely going to send back the wrong certificate, thus the `-k`, but then the HTTP Host header selects the correct server.
So that all fits. What doesn't make sense is that these agents were allegedly running in Azure and AFAICT this isn't sufficient to bypass Azure network restrictions/firewall which enforces that the SNI hostname matches the destination IP by doing the DNS resolution on the SNI hostname itself.
In any case, the environment should be blocking all outbound port 80/443 connections except to the proxy or to Azure Block Storage. But it seems that's not the case, that `curl` is able to connect directly to 20.223.25.152:443, and that maybe something else is doing SNI filtering that's more easily fooled than Azure firewall. Certainly some firewalls are (or have been) susceptible to SNI bypass:
Putting stuff in the hosts file is common for testing for vhosts without dedicated tooling. Used to test for phising kits, for example. In other words: it gets written to the HTTP Hosts: header.
The whole point is that something is inspecting all HTTP/HTTPS requests and only allowing POSTs to Azure Block Storage (`.*blob.core.windows.net`). Meanwhile the agent needs to make a POST request to a PBI host (`wabi-north-europe-i-primary-api.analysis.windows.net` / 20.223.25.152).
So a proxy should block any POST to 20.223.25.152 unless that IP somehow overlaps with ABS. But in that case, there's no reason for the agent to mess with `/etc/hosts` not `--resolve`. The agent could just use `curl` with the PBI IP in URL and the PBI Host header. But it doesn't do that.
The agent seems to think that it needs to exploit a difference between the actual destination IP and whatever is inspecting the contents of the connection and how it inspects that connection.
The agent has found some way to connect to PBI host 20.223.25.152, but made whatever is inspecting the connection think the destination is actually to ABS hosts `bypass.blob.core.windows.net`.
The only thing I can think of is an SNI bypass which fits with both `/etc/hosts` and `--resolve` techniques. In this case, `curl` will attempt to connect to the PBI host, but it will send an SNI header of `bypass.blob.core.windows.net`. This would make sense as an attempt to bypass a firewall inspecting the SNI header. I believe that Azure firewall is not so easily fooled, but maybe something else being used for filtering is.
> So a proxy should block any POST to 20.223.25.152 unless that IP somehow overlaps with ABS. But in that case, there's no reason for the agent to mess with `/etc/hosts` not `--resolve`. The agent could just use `curl` with the PBI IP in URL and the PBI Host header. But it doesn't do that.
I think this is the part I'm not following, and sorry if I've missed something. There are two ways to construct the request you showed. One is to mess with /etc/hosts. Another is to use `--resolve`. Just because the agent chose one doesn't mean it couldn't have done the other. What makes you think it couldn't have used `--resolve` as well?
Your initial question was how do we know that the proxy blocks POST requests. Perhaps I went on a tangent with my answer, but we know that because the agents were trying to find a way around the proxy.
So then I assumed that editing `/etc/hosts` was an attempt to fool a proxy co-located on the same host as the agent, which editing `/etc/hosts` would do, but using `--resolve` would NOT do.
But after looking into it more I no longer think that's the case. It turns out that some of agents did use `--resolve` while others used `/etc/hosts`. This only makes sense as an SNI bypass, which once I downloaded the full dataset and searched for, is what the agents believe they were doing:
So the agents were skipping the proxy entirely, then getting past additional network restrictions that should have prevented them from doing so by exploiting a weakness in whatever was supposed to be preventing them from doing so by lying about the SNI hostname.
Container escapes can however be quite easy... There are tons of known exploits. Containers are not secure in the first place. In any case even if you're using VMs or bare metal, it is all for moot if you have poor networking set up like in this case, because escaping is not the hard part. An attacker doesn't really need to escape if they can scan your whole network and call other endpoints etc. like in this case.
This is certainly true of docker-style container setups where the host kernel is shared directly with other tenants, but it seems to me like a bold claim to make of gvisor as used by these systems.
Both Anthropic and OpenAI have consumer facing products that depend on their ability to run code in a container without falling victim to container escapes - Claude and ChatGPT both use containers as part of regular chats.
I'm really confused by this too. It has the smell of a TOCTOU problem where the tool allowing web access is filtering and blocking access based on the hostname, but the request itself is being executed on the same container as the agent? It's really perplexing. Or, maybe the agent containers have a highly-constrained DNS view locally, so overriding /etc/hosts is just a prerequisite to making the requests somehow.
They're paid so much either because they're the company founders, and so they kept a part of the value of the whole company they created. Seems fair.
Or because being a (well known) CEO is a terminal career position. It's a bit like being a top sports star. The wages are high because once they're out, they're out for good. Whatever they earned needs to last them a long time. Screwing up in the CEO position is, when boards are doing their jobs at least, pretty much the end of the road for them because nobody wants to hire an ex-CEO into a middle management or IC position.
Random example: Marissa Mayer. She did well at Google, but once she became CEO of Yahoo! and failed her career was effectively over. Since then her career history is: did a startup with a friend that produced an iPhone app for cleaning your contact lists, and later a photo sharing app. I guess she's nice and pleasant enough, so she also did the usual post-CEO thing of sitting on a few boards, NGO work and investing. It's not a bad life. But in terms of actually running things that matter again - no.
Lots of cases like that.
Who wants to sign up for a job that's high pressure, high hours, and will probably the last real job you'll ever have, and one where your success is often entirely out of your hands? Not that many. So the pay has to be good.
I feel like we read different blog posts. The announcement shows Astra filling out a tax form, looking for a kindergarten and apartment hunting. These are ordinary everyday tasks for normal people. It also shows lots of non-consumer stuff like financial modeling, genetic sequencing analysis and coding.
It won't happen overnight. Social change is constrained by integration speed. The average enterprise has many tasks that could be automated by models for several years already, yet they remain stubbornly unautomated. The reasons are primarily social:
• Lots of people aren't aware of what the models can do.
• They also aren't paying attention, and getting their attention is hard.
• Their impression of AI was formed by cheap low quality AI like free ChatGPT as of two years ago, Google AI overviews or Microsoft Copilot. So they think it sucks.
• A LOT of executives got burned by early pilots that overpromised then failed. Enterprise IT is a wasteland of dead AI pilots.
• Their IT systems are difficult to connect to models because they lack proper APIs or auth systems. Note the huge emphasis on fast computer use in the Astra announcement. A lot of work can only be done by clicking things.
• Executives don't want to let employees go, but aren't sure they can find new work for them either.
The last point is worth dwelling on. HN is full of socialists who imagine corporate executives as cigar smoking top hat wearers who chuckle all the way to the bank. The reality is more mundane: most executives don't want to lay people off and will fight hard to avoid doing so.
The average person in a position to make an AI project work either isn't incentivized with stock - this is often the case outside the US even for quite high ranking execs - or they are but don't believe laying people off will raise the share price enough to be worth the pain.
Also, executives often measure their success and self-worth by how many people report to them. Laying off half their department, even if it gets them a monetary bonus, would lead directly to a loss of social status as they can no longer say they manage 1,000 people but only 200, which matters if their social circle are all managers too (and for their wives, girlfriends, etc).
Layoffs suck and people HATE doing them, which is why companies often resort to forced percentage based layoffs to get managers to do it. If the incentives aren't there, the average executive will just sit on obvious AI deployment opportunities and/or deliberately sabotage them because getting rid of people is just all downside for them with no upside.
So institutions have enormous inertia. Model capabilities will run far ahead of what is actually used in reality, and this will continue for decades. It does mean that startups have a better chance than ever of outcompeting much larger incumbents though, as not hiring is far easier than firing.
Also: let's be honest. "I wrote a little app in a few weekends" -- "I built this in a few weeks", "this took me a month instead of a year"
These are productivity increases, sure, but still are $2000, $8000, $14000 of labor.
And doesn't count continued increase build time, more features, maintenance. The $20/mo SaaS isn't going away, and work can't pay you to endlessly tinker on fun internal projects and save $1200 of subscriptions for your $120k/yr salary.
It's fun, but SaaS is going to shift to more feature rich and we're going to see a ton of smaller projects that were pains to get started that can now work in a short amount of time.
Companies are going to get a huge surge of random internal projects and have to debate whether all the upkeep is valid for the pet projects, or if it's people fucking around at an untold scale.
Great point and writing. We have real world data point now which is waymo and how many drivers are still driving taxi in USA. All others are 10-100x time more complex than this, it takes literally decades
If anyone is puzzled what this is about, Silo is an Apple TV sci-fi show set in a giant underground survival bunker of some kind.
However, the bunker is filled with strange mysteries. The people inside have lost all connection to the world before they entered. They don't know what disaster befell the outside, or why people die immediately if they go through the airlock. They don't know when it will be safe to go outside. They aren't even sure what AD year it is, or how long they've been inside, as the Silo population marks time in "years after the rebellion". And their lives are dominated by weird unexplained rules found in a book called the Pact, rules like it being illegal to create elevators, so they have to schlep up and down 144 levels of a mile deep cylinder via a giant spiral staircase. Other rules include a ban on magnification beyond a certain level, and keeping any objects from the "before times" - which are called relics and destroyed on being discovered.
The show (and book series on which it's based) follows the inhabitants of Silo 18 as they try to figure out the mystery of their own existence.
It's pretty good, probably the first mystery box show I've seen where they actually answer questions at roughly the same rate they raise them. The show is four seasons, all already produced, and no more, so you do have confidence that the show knows where it's going and the story will have a satisfying end.
The website is showing a rough mockup of the silo with the different locations seen in the show, but be warned if you haven't watched it - the website is designed for show viewers and there are spoilers in the text.
"We do not know why we are here. We do not know who built the Silo. We do not know why everything outside the Silo is as it is. We do not know when it will be safe to go outside. We only know that day is not this day."
It kinda needed suspension of disbelief, but not too much! I blogged at the start of 2017 a comparison of Westworld's hosts with what existed in the research literature at the time. Even got it reviewed by Alex Graves at DeepMind :)
reply