On reliability, killer robots, and classrooms.

What Anthropic's statement tells us about LLMs outside of the military.

Mar 01, 2026

On Friday the U.S. government designated Anthropic as a supply chain risk — an action historically reserved for foreign adversaries, and it sounds like OpenAI will be filling the void. The short version of the story is that the Pentagon said they’ll only contract with AI companies who accept “any lawful use” of their technology. Anthropic pushed back saying that their technology cannot be used for (1) mass domestic surveillance or (2) fully autonomous weapons.

The mass surveillance piece is getting most of the attention, but the second point seems to be getting lost, probably because it’s about operations and reliability and is less exciting. Taking a position against mass surveillance is ultimately a values argument, and I’m grateful they are making it. But their point about fully autonomous weapons is about RELIABILITY — and that is a point that is acutely and immediately relevant to our lives and our kids right now. Let me explain.

Anthropic’s CEO explained their position on fully autonomous weapons like this:

Some uses are also simply outside the bounds of what today’s technology can safely and reliably do. [...] without proper oversight, fully autonomous weapons cannot be relied upon to exercise the critical judgment that our highly trained, professional troops exhibit every day. They need to be deployed with proper guardrails, which don’t exist today.

Read that again... The CEO of one of the world’s leading AI companies had the spine to say the truth, publicly: LLMs are still emerging, they are not reliable enough for this context and use, and they need to be deployed with “proper guardrails.”

This is a very VERY important point, especially for those of us who are wrestling with how LLMs are being used in our lives, our kids’ classrooms, and our workplaces.

Anthropic has drawn their lines at mass surveillance and fully autonomous weapons. Good, thanks. But ultimately the issues they are flagging apply to some degree everywhere else too. LLMs are an emergent technology. It is not reliable, and it is not consistent. Even the people who create it cannot fully explain how and why it does what it does. This is true everywhere, not just in a military setting.

While the stakes may be different, they are also high in all sorts of other environments: hospitals, emergency response, classrooms and education, caretaking of all kinds… And yet, leaders everywhere have decided to integrate what is ultimately *emergent and experimental technology* into products, systems, and environments that we and our kids engage with every day. To be clear: while I wish companies cared about this, I do not expect them to… because they are companies. As I said 2 years ago:

Of course, it is not surprising to find companies chasing the market. What is baffling is that people and organizations working in the public trust are scrambling to integrate — not just experiment with, integrate — technology that is, in the words of [Sam Altman], “a barely useful cell phone” into some of our most foundational and critical systems: education, the military, caretaking, and healthcare.

The basic technology is the same — what differs is how we think about or understand “...what today’s technology can safely and reliably do,” and our tolerance for associated risks. How do we define safe? How do we define reliable? The answers to these questions vary depending on context, and we cannot expect for companies to think through this for us. These are questions that some combination of public officials, and leaders considering adoption and integration of LLMs need to consider with the seriousness it deserves. So here’s my question:

If AI is not reliable enough for fully autonomous military action, who determines what is reliable enough for clinical settings? For educational settings? For operating rooms? For classrooms?

I have long had concerns about rushing to integrate LLMs into high-stakes environments for years. Two years ago I wrote a piece about how the potential of technology breaks down in its implementation. In it, I said:

So much of how a technology (or anything) works in the world is affected by a combination of its design, human behavior, and the environment that it is embedded in. That includes how the technology will work in the environment, and how humans will use it. But something we often overlook is the capacity of institutions and organizations to effectively implement and manage the technology.

I went on to quote Nancy Levenson (Professor of Aeronautics and Astronautics at MIT who specializes in software safety) who once said that “... the problem is that we are attempting to build systems that are beyond our ability to intellectually manage.” I closed the piece by saying that “we are adopting technology beyond our ability to implement safely or even effectively.”

This is precisely what Anthropic just told us. LLMs need “proper oversight” and must be “deployed with proper guardrails, which don’t exist today.” This is not only true for Anthropic’s technology… it’s true for all LLMs. And while the stakes may be different, this is just as true for combat as it is for the classroom.

It cannot and should not be the job of companies to be our moral compass, and our public institutions are… not very helpful at the moment. We cannot wait around for others to advocate for our kids. If we are concerned about LLMs being integrated into the products and systems that we and our kids use, we must ask:

What is the goal / intended outcome of integrating LLMs? What problem is it solving, and FOR WHOM?
Why is an LLM an improvement on the current state? If it is a bandaid for a deeper issue, then the LLM will only mask (and even amplify) the problem it seeks to solve.
What can LLMs safely and reliably do here / in this context?
Who determines what is safe and reliable “enough” here?

**And remember: Anthropic had the spine to say this out loud. But their statement is not an indication of moral authority—it’s just a clear-eyed awareness of risk and the limitations of its technology. And there is a sea of competitors with products who are silent… and their products are not as good.

Know someone else who would appreciate this? Please share Home Screen with them.