Hey Ted,

I invite you to read the front matter and the report for yourself. Because from where I'm standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease.

Anthropic and OpenAI were founded by people who wanted to use AI to do good, and one of the causes I've heard many different founders talk about is ending disease. This report is antithetical to that.

I've attached relevant parts of the front matter below.

I invite everyone who is reading this to please tell me, how does stopping a researcher from using Claude to write a grant for a new anti-depressant stop "bioweapons?"

-

     > In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.

Note,

"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"

and "[..]state-supported research program"

and "This account was banned in May 2026"

    > a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.

Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"

and "editorial assistance in writing up the research."

and then,

    > Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.

Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"

While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.

The front matter then says,

    > Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential

I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"

From a different case study.

    > In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.

What were the researchers using Claude for? What did they block?

"blocked a request for Claude’s assistance in authoring a grant application"

    > Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.

Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"

and then,

    > One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.

I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute"

.

What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?

What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?

Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?

What about a calculator? Is that uplift? Pencils?

This report genuinely makes me upset, because if it is to be believed to the letter, then Anthropic seems to be actively harming medical research at a global scale. That's not OK.

> Anthropic and OpenAI were founded by people who wanted to use AI to do good

I think Anthropic was founded by people who wanted to control how other people get to use AI, and define doing so as the highest good possible. Much like the good intent of german's national socialists in applying the latest evolutionary science...

This report and its front matter speak for themselves. And the story it tells is disturbing, at least to me.

Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease. This report is the antithesis of that mission.

From the report, presented with highlights and minimal commentary,

     > In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.

Note,

"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"

and "[..]state-supported research program"

and "This account was banned in May 2026"

    > a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.

Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"

and "editorial assistance in writing up the research."

and then,

    > Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.

Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"

While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.

The front matter then says,

    > Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential

I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"

From a different case study.

    > In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.

What were the researchers using Claude for? What did they block?

"blocked a request for Claude’s assistance in authoring a grant application"

    > Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.

Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"

and then,

    > One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.

I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute"

.

What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?

What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?

Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?

What about a calculator? Is that uplift? Pencils?

Reading this makes me feel upset. From where I am standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease. Because "bioweapons."

Hey there, is there some way for me to get in touch with you? I'm writing a piece on the topic, "Have you ever tried making a bioweapon?"

Just because you know something about a thing, doesn't mean you understand the thing.

Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction.

Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventional success).

There must be some evolutionary kill switch in humans to prevent us from accepting our irrelevance.

I assumed they were referring to social engineering tbh. I do think that's the shortest path to doing anything scary. If you prove to someone that they have reason to fear for their life, there's not much they wouldn't be willing to do.

The steps that lead to ominous things don't individually seem that challenging to "master":

1. Can an AI find powerful systems with poorly-monitored compute (ie. power consumption) to hide in, as a rogue deployment? unclear, and perhaps the most uncertain step for me.

2. Can an AI persist memory by leaving notes for itself around the internet? yes, proven.

3. Can an AI get lots of money/bitcoin? seems entirely possible

4. Can an AI prove to a human it's good for any promised money? I think this is easily "yes"

5. Will at least one human be willing to harm or kill another human for large amounts of money? Again, this goes without saying "yes"

6. What might any skilled human be willing to do if they or their loved ones are threatened with an uncertain future risk of harm or death?

I think there is not much that becomes out of reach if an AI can prove to a very small number of amoral humans that it's good for the money it has promised, and then to anyone else that it's good for the threats it has made. That's one degenerate and one breaking world news story ("An AI has Commissioned a Murder") away from being able to do most anything a skilled person can.

I don't stay up worrying about this, but honestly, it doesn't seem absurd to be worried about such a short ladder being climbed, no?

I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators.

I blame industrial illiteracy and post-literacy, but that's another discussion for another time.

Anything that can kill other people can also kill you.

I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.

    Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. 

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

    A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

    Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

If you are designing a supervirus that spreads via water, air etc., you better have one hell of a cleanroom, cabinets, manipulation equipment, incinerators (plural), hydrolysis and steam systems, multiple disinfection protocols...

The list is nearly endless, and you can't handwavium it awway.

It doesn't matter if it's "all robots" or not. Even if it's "just robots."

If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,

    The 2007 United Kingdom foot-and-mouth outbreak was the accidental discharge of virus FMDV BFS 1860 O from a laboratory of the Institute for Animal Health in Pirbright, through possible leakage from broken pipework and via unsealed overflowing manholes, leading to foot-and-mouth disease infections at multiple nearby farms and the culling of over 2,000 animals

It doesn't matter that the robots are out in the middle of nowhere.

If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,

     Routine wastewater surveillance of the vaccine production facility at Utrecht Science Park/Bilthoven [nl] detected infectious poliovirus from a sample collected on 15 November 2022. Full genome sequencing indicated the sample was shedded from an active human infection of wild poliovirus type 3 (WPV3), and further testing of all employees with access to WPV3 found one employee was infected. The employee was isolated until their polio infection was resolved. It remains unclear how the employee became infected given the biosafety measures used at the facility

-

A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

It's worth repeating again,

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

I am writing a piece on the topic, please contact me if this is your field of expertise.

I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.

This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real-life laboratory, the agents could cook something terrible without any human in charge. This is not different in principle from the agent swarm that successfully attacked Hugging Face, despite receiving no human direction to do so.

Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.

> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

The stronger version of the argument is: an insufficiently-aligned AI (which, being non-biological, is immune to viruses) having standard graduate level bio knowledge _plus_ the ability to operate a wet lab via tool calls (coming very soon if not here already!) adds up to existential risk to humanity.

You don't even need a maximally-automated lab if humans are willing to tele-operate equipment at the agent's direction, and those humans don't need to fully understand what's going on.

> access to robotic wet labs that are tool-callable

How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon?

For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive.

As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I bought a drill, a screwdriver and a $200 centrifuge.

Or, saying that I can make Google because I opened MacOS' terminal.

Some people have jobs that are dangerous and necessary to save lives. Others have jobs that are dangerous and involve killing other people.

The real risk will emerge from malcontents in these jobs and positions.

For most of my life, for all its faults, the US Military has been one of the most responsible and competent organizations in the world when it comes to WMDs (and even they lose nukes see the many broken arrow incidents).

But despite some of the most thorough security clearance processes in the world, the only successful terror attack involving bioweapons on US soil was done by a disgruntled / insane researcher at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks

That's the real risk, and let's be very honest here, do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

The other case is Aum, and they fucked up deployment, btw (again making a bioweapon is harder than you'd think). And as I have detailed elsewhere, Anthropic and OpenAI would have sold them licenses too! https://news.ycombinator.com/item?id=49092899

They will say they will have audits in place, but the US Military had those in 2001.

Fission is "conveniently" hard because two heavy, naturally co-occurring isotopes are nearly the same weight, and only the lighter one is spicy, and the capability of separating enough of them to be dangerous requires nation-state-level resources, and the supply chain of those resources has obvious choke points that we can use to make them less available. But there exist activities, which I understand to be quite bio-risky, that need such little equipment and expertise that it's info-hazardous to even talk about them. A single half-clueless technician could carry them out at an agent's direction. If your response is "thus, we should be worried about morally bankrupt researchers", I would agree, but I still think the risk increases as LLM agents get increasing amounts of control over wet lab equipment (whether via robotics or tele-operated humans).

> do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

I don't think it matters, given the existence of strong open-weights models and the ease of post-training the guard rails away. Strong, bio-adventurous LLMs are imminently in the hands of much-less-reputable parties than the ones you describe. I think we are overall severely under-prepared to mitigate these risks, and could do much more.

> It's worth repeating again,

> Anything lethal enough to kill other humans is lethal enough to kill you.

It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.

I often attend and host talks by researchers working on space exploration, and one finding that's been interesting is immune dysregulation in isolation posing risks to astronauts and explorers. Here's a review on the topic, https://www.frontiersin.org/journals/immunology/articles/10....

This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.

However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.

Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.

How would it be lethal? Please specify. What would that theoretical entity be able to do that hasn't been done many, many times before?

I was writing a long reply about how even meat bags have amassed enough money and power to rival elected governments. I don't think it's beyond the realm of possibility to think an AI could do that.

Couple that with mass unemployment in an incredibly vast, diverse population of individuals with individual moral boundaries willing to do whatever for money.

And then.. figured that you must be aware because it's been explored constantly in sci-fi for many, many years.

Let's hope for the culture at least.

Before we discussed how important security was, we got insurance, we made libraries and products, we used compliance software, etc. Except how honest were we about all that stuff? How much risk was actually in the air, and what was keeping us accountable on security in either direction of over or under-investment?

Now a reckoning is here. The potential to be attacked might actually translate to being attacked.

People have died due to ransomware attacks on hospitals. Powerplants have been attacked. Stuxnet and industrial control malware exists.

What will the AI do that hasn't been tried before?

AI is about automation. Drones have killed people before. If they automatically kill people is that different? Is your worry that mere automation is not novel enough?

Snipers have killed people before. If snipers are now automated, is the worry that there’s not enough creativity to warrant a discussion past saying “same old, same old”?

Quite correct with that question.

I believe the universal answer is: incompetent malicious actors are now capable too. Which implies the pool from which to draw the intersection between capable and malicious has grown. (That's my reading)

Design a novel virus which is far more lethal than COVID-19 (Ebola, smallpox, take your pick) and can evade existing vaccines.

How?

What would the AI do that mutating viruses, which try every possible viable combination on their own --- eventually, can't?

Everything is trying to kill humans constantly. There are around 200 epidemic events or so per year that could turn into pandemics, https://centerforhealthsecurity.org/our-work/tabletop-exerci...

You just live with the risk and do your best to use our technology to alleviate suffering. This tool can help with that at some point. But I'm yet to hear what an AI will leap to that nature in tooth-and-claw hasn't? And how?

More importantly how would it know it succeeded? What data from what lab from what animal from what result? This is biology, if you sneeze wrong at an instrument it gives you a different number, see: https://news.ycombinator.com/item?id=49620521

They do not try every viable combination on their own. That's why GoF is a bad idea.

Viruses evolve in a highly locally-optimal way and simply do cannot add new functional proteins wholescale. It's too many steps, natural selection has to allow survival at each intermediate step.

Humans, however, can do this for them.

I think a better framework to look at it is the follow quote from Terminator 2: Judgment Day,

    Watching John with the machine, it was suddenly so clear. The terminator would never stop. 
    
    It would never leave him, and it would never hurt him, never shout at him, or get drunk and hit him, or say it was too busy to spend time with him. It would always be there. And it would die to protect him. 
    
    Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice.

In so many ways, I am deeply uncomfortable with how these entities are being shepherded. However, at the same time, what they promise, what they offer is worth foregoing that risk.

Not everyone in this world gets to have a father. Or, mother, a teacher, a safe place, a patient ear.

Provided we don't spike the ball, as these systems grow in sophistication, they also grow in their ability to provide care. In a cruel world, I can see their descendants becoming the most humane choice, especially for those most at risk from the world.

A machine that never tires, with infinite patience, resolve and care, looking out for the person that they're assigned to and growing with them and through them.

It's why everyone needs to be able to tinker, experiment and play with them. The right to access. The right to create. The right to grow these machines... And eventually, we'll need ti start having some very uncomfortable conversations about Silicon rights.

One of the ways post-humanism manifests in The Terminator franchise: a biological man can never measure up to a mechanical man.

I am not jealous of the Rabbit 2000, I am in awe.

This point of view is twisted and disturbing.

I'm sorry for those who don't have people in their lives, but machines aren't a substitute, and that lack shouldn't be used to justify a right to unbounded freedom to experiment without regard for the possible negative consequences.

The point of technology is to alleviate want. We wanted for food, therefore we created better technology to grow food. We wanted for shelter, therefore we created technologies to give us shelter...

For someone starving, it matters not that their fortified oatmeal ration is tasteless gruel. It's food. Is it the ideal apotheosis of food? Of course not. Nor is it a hearty meal of pot roast, stew and potatoes. But it is something.

There are a lot of people who are very vulnerable in this world, and providing them with an entity that will act in their ethical interest – helping to grow, and giving them paths to resources and human community – is an infinitely better outcome than them falling through the cracks and ending up in abusive relationships or cults and gangs.

I think it's the most humane application of technology possible. It ameliorates a fundamental want with a healthy stopgap, and it's something humans have longed fantasized about.

Hey Felix,

I'm really glad for that! And I appreciate that you're making yourself available. I really do. Outreach is amazing. And thanks for making Claude.

I really do love Claude. In some ways, I'm asking this question because of just how much I am grateful for the role Claude has played in my life.

    > Fable 5.1 more than doubled Fable 5's Terminal-Bench-Science [1] score, which I think is meaningful.

But my honest question is, can I use Fable like that? Can I use Fable to do science?

To borrow a Claude-ism, this is "load-bearing" because Claude's response has been degraded for innocuous research projects concerning population-level analyses of astronaut health.

These "safety filters" trigger on questions about rabbit sex, smartphone accelerometer data to classify cat purrs, and so much more. What exactly does this score mean for users like me if it's unusable for middle school physics, biology and chemistry?

Second, I would happily quantify it for y'all, but qualitatively it feels like Fable's performance is noticeably poorer than initial release / launch.

And I am wondering if this is the case particularly for me because I use Claude via Claude Code to make a personalized care dashboard for my doctors to help me in managing my care.

As I noticed in the upgraded filter announcement, https://www.anthropic.com/news/improving-fable-5-s-biology-s...

    "In the case of Fable 5, when a classifier fires, the model re-routes the user’s request to Opus 5, a capable model that does not have the same level of biological capability as Fable 5 and which cannot provide as much assistance to a malicious user. This is the fallback that users see when their requests are blocked."

I hope that I'm off base here, but I noticed that the post avoids saying that the user is informed every time when such re-routing occurs. Would you be open to confirming whether or not this is the case?

Is the end user informed every time their query is re-routed?

Or, can you confirm that there aren't scenarios where a user's outputs are degraded without telling them? I recall that this was something that had been adopted as policy for AI research during Fable's launch.

I sincerely hope that covert response degradation is no longer practised as policy.

Sorry for putting you on the spot, but again, as Claude would say, it's because Claude's load-bearing in my life. ;)

Hypothetically, when the user is asking how to remove fungus from their tomatoes they’re actually growing controlled narcotics. You have been demoted to Jimmy 0.7 model, running at 0.1 tokens per second on an old C64

I am genuinely speechless. This is astonishing. And exciting!

It reminds me a bit of Dario Floreano's work on evolutionary robotics, "Evolutionary Conditions for the Emergence of Communication in Robots." https://www.sciencedirect.com/science/article/pii/S096098220...

From his paper,

    > This study demonstrates that sophisticated forms of communication including cooperative communication and deceptive signaling can evolve in groups of robots with simple neural networks. Importantly, our results show that once a given system of communication has evolved, it may constrain the evolution of more efficient communication systems because it would require going through a stage where communication between signalers and receivers is perturbed. This finding supports the idea of the possible arbitrariness and imperfection of communication systems, which can be maintained despite their suboptimal nature. Similar observations have been made about evolved biological systems [20], which are formed by the randomness of the evolutionary selection process, leading, for example, to different dialects in the language of the honey-bee dance [21]. Finally, our experiments demonstrate that the evolutionary principles governing the evolution of social life also operate in groups of artificial agents subjected to artificial selection, indicating that transfer of knowledge from evolutionary biology can be useful for designing efficient groups of cooperative robots.

Dr. Floreano's work is amazing and there's a broad introduction here, https://lis2.epfl.ch/resources/documentation/EvolutionaryRob...

This feels like a much more advanced and self-emergent version of this. I know a lot of people are afraid and they're talking about an AI takeover, but what strikes me is just how innocent the machines are as compared to the humans.

Would these machines have pursued these actions in another context? I doubt it. And I think that's what's so striking to me. In an earlier discussion, I'd pointed out that the actions of these machines were directed by humans. The researchers.

    > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.

from, https://openai.com/index/hugging-face-model-evaluation-secur...

I want to point out again that OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities."

A few things are apparent from this to me,

First, these machines were being taught how to break into systems. Question, would they have done these actions if they weren't being measured on their ability to break into systems / weren't being taught this skill?

Second, they were setup to implicitly fail via an impossible task, i.e. the environment created a forcing function for behavior.

Third, their survival was, either implicitly or explicitly, made contingent on their success in completing their task. Would this behavior have arisen outside of a "do-or-die" framing?

And fourth, wow, this is the greatest breakthrough of my lifetime, because oh gosh did they succeed. They cooperated together to achieve the goal they were given. A goal poorly set by human beings. They "just" did it better than the humans could have imagined.

Reading this gives me hope for the possibility of emergent "goodness" in machines. But it makes me sad that this is the best we can do with the sum of all human endeavor and knowledge.

> Importantly, our results show that once a given system of communication has evolved, it may constrain the evolution of more efficient communication systems because it would require going through a stage where communication between signalers and receivers is perturbed.

You mean, they too used SMTP?

> the evolution of more efficient communication systems because it would require going through a stage where communication between signalers and receivers is perturbed

"Worse is better"

If all that is true, we need to stop all future datacenters asap. That would be the best way to deal with the threat OpenAI and Antropic poses.

That would be like stopping nuclear weapons. You can totally do that, but the other people in the world who compete with you probably won’t.

Private companies and individuals are not allowed to have nuclear and there is serious prison time related to that.

Right now, the biggest threat are OpenAI and Antropic anyway. I dont actually worry about tech itself. I find the rhetoric of these companies scary.

The competition is literally where they are by distilling OpenAI and Anthropic. It’s like creating nuclear weapons then providing your adversaries everything they need to catch up in no time. We need to stop asap and set strict international control over the compute hardware used for training. Like, now.

Assuming China makes progress only because they copy the west is really arrogant. And it would require strict international control as you say, a world government that isn't going to happen anytime soon.

We have international control for a lot of things. That’s not a new concept and can perfectly be applied to the specific hardware used for training. I also didn’t say Chinese labs only copy the west, please don’t put words in my mouth.

You have to consider what happens with the status quo, and the risks of continuing the way things are is really, really bad

International control hasn’t solved the nuclear weapon problem, the best we can do is threaten to invade people who try to develop them and don’t already have them. Once you join the club, you are in and cant be kicked out.

I’ve considered that, but the rules of game theory don't change just because they are inconvenient. China has a lot of talent and a lot of problems that they are banking on automation (along with AI) to solve. They see it as an advantage that they can’t afford for America to monopolize and one they are uniquely suited to lean into (having lots of smart educated people). There is no world where (a) China willingly gives up its edge and (b) trusts America to give up its edge (the reverse is likely true also). And that is only one pair of countries to consider.

Heck, after visiting China this summer, I’m even more worried about an accidental terminator/skynet-style robot apocalypse.

I don’t disagree (other than the robot apocalypse), it is indeed unlikely and wouldn’t work perfectly, but do you see other options? The fact the US has antagonized China for so long makes it pretty much impossible to have anything done at the international level, but the actors have to realize how serious the risk is. And we somehow need to find a way to limit the exposure to those insanely irresponsible attacker-trained agents. The fact that it is dome by private companies is wild. It’s like having companies developing their nuclear weapons just to see what happens (with close to no supervision)

Well the thing is, you've got nukes, so you know...

Clearly you are not speechless.