Amazon Web Services (AWS) says AI agents wrote and tested all of the code in AWS Emulator, a free tool it released on 11 November that imitates its cloud services on a developer's own machine. According to AWS, an engineer built the scaffolding the agents run in, and engineers design the guardrails around their work. Matthew Goldberg, the AWS engineer behind the announcement, writes: "Our team hasn't written or reviewed any of the code."
For engineering leads at small software companies starting to let AI agents write code, this is a public example of what AWS put in place of human review: tests checked against the real service, an automated gate, and a public list of what does not work. It also shows where that stops. AWS's own documentation says not to use the emulator to test security, because it does not check permissions the way AWS does.
What happened
The emulator runs on your own machine, on a development server or in continuous integration (CI), according to its README. A post on Werner Vogels's blog says it covers more than 200 services.
Goldberg describes three teams of agents. One writes test scenarios and checks each against real AWS. One runs those tests against the emulator and files each difference as an issue. One fixes the issues. Every change then passes a merge gate: it must compile, and agents check it against coding standards, for security, and that it does not make the emulator match AWS less closely. The Vogels post adds checks that stop agents from touching code they are not supposed to.
AWS says hundreds of agents work at once and fix thousands of issues a day. These are AWS's figures. Help Net Security repeats them, while SiliconANGLE describes "dozens" of agents. Neither reports testing the tool, and we found no independent review of the code.
The README says not to use the emulator for production or to test security. Its security page adds that it never denies a request signed with its own key and does not check storage-bucket or encryption-key policies: "Passing tests in the emulator don't show that your permissions are correct or safe."
Why it matters
The method uses the live AWS service as its reference, which SiliconANGLE calls the "ground truth". AWS does not say whether the agents that fix issues can also edit the tests. In the Vogels post, guest author Swami Sivasubramanian describes an agent rebuilding the replication engine of Alf, a distributed lock service he helped build at Amazon. He supervised it for four nights before seeing it had no way to check its results. With a spec and a test environment, the rebuild took about two hours.
AWS's case is that where such a reference exists, agent work can be checked without a person reading each line. Where it does not, as with permissions here, AWS sends users back to real AWS.
Something to build on
In our view, this is not a case against AI-written code, but for keeping a person in the loop. AWS's own security page lists what passing tests in the emulator cannot prove: that permissions and other security controls are correct, and permissions decide who can see the data. (AWS gives no reason for its separate warning against production use.) For code that does go into production, a person who reviews and approves each change lets AI write much of the code without lowering the bar. Before accepting agent-written code, a team can name four things:
- Ground truth. What the code is checked against that the agent cannot edit: a real service, recorded real responses, labeled examples. If one agent can change both the code and its tests, there is none.
- Gate. The checks every change must pass before it merges, written down, including the files the agent may not touch.
- Published gaps. What is known not to work and what the checks do not cover. AWS keeps a public things-that-do-not-work folder, and its README points to fidelity scores, published with each release, that show how closely each service matches AWS.
- A person who approves. Who reviews and signs off each change before it merges, starting with what the checks do not cover.
In our view, for systems holding personal or clinical data, the gaps list matters most where the ground truth stops. Here, that is permissions, which decide who sees the data.
We use AI heavily in our own development and testing, without access to user data, and its work passes the same tests and validation before it is merged (how we build with AI).
What you can do
- Engineering lead: for one component your agents write, write down its ground truth, its gate, where its known gaps are listed and who approves changes, one line each. Fix any empty line first.
- Engineering lead: confirm the agent cannot change the tests or gate settings it is judged by, for example with branch protection on those files.
- Developer: if you try AWS Emulator, put only test data in it and test permissions on real AWS, as AWS's security page asks.