Building Software That Can Prove Agents Wrong

Recently I've been building a workflow where coding agents take a task through triage, planning, implementation, verification, and finally opening a pull request. As the workflow and models got better, the thing that surprised me was that implementation stopped being the part I worried about most.

The agent could make code changes surprisingly fast. The harder part was getting it to verify those changes and decide what evidence was enough to consider the task complete.

A verification step could exist in the workflow and the agent could run the application in a browser, reproduce the behavior, execute tests, and return screenshots or other evidence... but that raised a question I hadn't initially designed the workflow around:

What kinds of mistakes was the agent actually capable of detecting?

Consider a payment checkout change. An agent adds a new payment path, runs the application, completes checkout, presses "Pay", and then sees a toast: "Payment successful".

That is success evidence but it's a weak one. The UI can say "success" while we have two orders persisted. A retry logic can charge twice. An event can be emitted twice. The browser verification only proved one thing: the happy path looked right. It didn't prove the absence of several failures that actually mattered for us.

Once I started looking at agentic workflows this way, I stopped asking only:

Can the agent use the product?

I started asking:

What kinds of mistakes can this product actually expose to the agent?

That question stopped feeling like a workflow-design problem and started looking like an application-design problem: the product itself was shaping what the agent could verify.

§ 1Some systems are easier to prove wrong

To make it easier to understand, imagine the same checkout feature in two different projects.

In the first, an agent can open the application and click through the flow but to understand what happened underneath, someone has to inspect a database manually, search through logs, reconstruct state, or explain how a retry should behave.

In the second, the agent can reset the application to a deterministic state, execute the payment, inspect structured order state, assert a balance diff, retry the operation, and query the trace.

The user-facing experience might be identical. The model might be identical. But the amount of autonomy you can safely give the agent isn't.

Our first instinct is to attribute that difference entirely to the agent harness: the tools, context, constraints, evaluators, and feedback surrounding the model. Harness engineering has become a useful way to think about coding agents: the tools can include application UIs, logs, metrics, traces, and isolated runtimes, especially when human QA becomes the bottleneck.

But the harness doesn't have access to all the information it needs, and eventually, it reaches the application boundary where the application itself determines what can be observed and monitored. It determines whether an important state is observable, whether operations can be done through scripts, whether some specific scenarios are reproducible, and whether failures / errors contain enough data for the agent actually act on.

The system (application) being built supplies some of the sensors for the system (harness) building it.

That is the part I find interesting.

§ 2Development starts to look like a feedback loop

That sensor layer matters because once the agent can act on what the application exposes, the work starts to look less like a handoff and more like a feedback loop.

A simplified view of the old workflow might look like:

With a more robust coding agent / workflow, more of it can become a loop:

In my workflow, that meant the agent could change the code, run the product, verify the change, collect evidence, determine if it looks correct or not and try again.

This looks less like one-shot code generation and more like a control problem. And a control system needs useful sensors.

For a coding agent, the sensors are not just tests; they are the things that let it observe what happened: types, runtime state, logs, metrics, traces, browser output, tests, and independent evaluators. It also needs ways to act on the system (the actuators) such as: browsers, CLIs, APIs, and scripts.

What I kept running into, though, was a more specific problem: verification was only as good as the information the software exposed. If the agent cannot see a state, event, or invariant, it cannot check it either.

This overlaps with what Birgitta Böckeler has called harnessability: how well a codebase supports this kind of agent-driven work. But I'm using that lens more narrowly here: not just whether the agent can work in the codebase, but whether the application exposes evidence that can cheaply and independently prove the agent wrong.

§ 3The verification layer has to be able to see the bug

That sounds obvious, but it's easy to miss when designing an agent workflow: adding more ways to verify work only helps if those tools can expose the failure types we actually care about.

I mean, a screenshot can tell the agent that a toast rendered correctly, but it cannot tell when an operation was idempotent. A browser agent can complete checkout successfully without noticing that two orders were persisted. A green test suite can still pass if the agent wrote both the implementation and the tests from the same mistaken interpretation of the requirement.

This pattern shows up beyond checkout flows. Achint Mehta11arxiv.org/abs/2608.28795 recently measured a version of this effect across more than a thousand agent-generated web applications where different verification mechanisms helped with different failure modes: startup checks found apps that wouldn't launch, screenshots exposed visible UI errors, and performance problems only showed up when the evidence measured performance.

The main question I take from this isn't: "Did the agent verify its work?" It's: "Could the evidence available to the agent have revealed the relevant failure if it existed?"

That is a much harder standard.

§ 4Build software that makes incorrect states cheap to expose

That standard changes how I frame verification: there is a subtle difference between asking an agent to confirm that its code change works and asking what evidence could prove that it doesn't work. The first encourages confirmation while the second forces us to be explicit about the failure.

Using the checkout requirement as an example: "Clicking Pay should create exactly one order and deduct the correct amount from the balance."

A weak loop looks like:

But a stronger loop would look like:

The second loop isn't stronger because it contains more steps. It's stronger because those steps expose more ways for the implementation to be verified as wrong.

That changed the question I started asking myself:

Can this software make important incorrect states cheap to expose?

When I started applying that question to the agent loop, I kept coming back to a set of boring but useful properties:

• Deterministic, reproducible scenarios mattered because the agent could get back to a known state instead of debugging whatever the last attempt left behind.
• Explicit rules that must always hold mattered because they gave the agent a concrete condition to check against.
• Queryable runtime state mattered because the UI often only showed the end result, not the state transition.
• Structured failures mattered because the agent needed a clear failure reason before it could choose the proper fix.
• Fast isolated environments mattered because every slow verification step becomes part of the time we're waiting for the agent feedback.

And, to be clear, none of these are new engineering ideas; they overlap with testability, TDD, observability, and operability. What changed for me was their economic value: once an autonomous agent consumes the feedback loop, these properties lower the cost of each failed attempt and make it safer to let the agent keep iterating.

§ 5Evidence speed becomes part of system design

We already build for testability, observability, and operability, but the agent loop changes the economic value of those properties: they determine how quickly the agent can get useful evidence back.

A trace is no longer only something an engineer opens after production gets broken. It can be an input to the agent. A CLI is no longer only for developers. It can also be used by the agent to set up state, perform an action, and inspect what happened during the execution.

Shopify's work22shopify.engineering/back-to-native on mobile agents is a good example of this design pressure: once implementation is fast, the cost of getting evidence back starts to be the main part of the loop. The problem they describe wasn't that agents were too slow at writing code. It was that the feedback loop around the code was too slow: an agent could make a change in seconds, then spend minutes waiting on simulator interaction and to verify that the goal was achieved.

They also describe making business logic runnable headlessly and exposing it through a CLI so agents could inspect state, navigate, and perform operations without paying the simulator cost on every iteration.

The CLI is not the interesting part because it's something that have been around forever. The interesting part is that evidence speed became important enough to change the shape of the system.

§ 6What this looks like in practice

To be honest, I don't think this requires inventing an "agent architecture". The practical move is simpler than you should think: audit the system through the eyes of an agent that needs evidence capable of showing that its own change is wrong.

Can it cheaply create a known state?

If reproducing a bug requires a human to create accounts, click through onboarding, wait for asynchronous jobs, and explain what happened, the feedback loop is expensive. We should have deterministic fixtures, seeds, snapshots, and resettable environments. It brings that cost down.

Can it inspect the state that actually matters?

A UI / screenshot exposes only the final state of a much larger state transition. Having structured access to specific state, events, traces and diagnostics lets the agent check what changed, not just what was shown on the screen.

Are important business assumptions easy to test?

"This should never create two orders" helps, but a check that runs automatically and fails when two orders exist gives the agent evidence instead of something it needs to guess.

Can behavior be tested without unnecessary UI work?

The browser is a valuable sensor and actuator, especially for requirements that are actually visible to the user, but some questions are better answered through more deterministic ways like a CLI, API, headless mode, or domain test helper. Using the product through a browser and debugging the system are different tasks that require different ways of working.

Are failures machine-legible?

An error message saying "Something went wrong" is bad feedback for a human and even worse for an autonomous loop. If we want to reduce the amount of ambiguity and interpretation for our agents, we should ensure we have structured errors, traces, and explicit failure reasons.

How quickly can the loop restart?

Slow builds, shared environments, manual authentication, flaky setup, and hard-to-reset state aren't just developer-experience problems anymore. That's basically what adds real friction to how quickly an agent can learn from a failed attempt, adjust its plan and perform the next move.

And finally:

What important claims still have no reliable mechanical check?

Those gaps should be explicit and the point here isn't to automate every judgment. It's to know where the evidence ends.

The difference now is that these practices affect not only maintainability and developer experience, but how much implementation work an agent can safely do before a human has to step in.

§ 7More verification isn't necessarily more trust

It's important to notice that more verification doesn't necessarily mean more trust. This is just another trap.

Suppose an agent:

1. Interprets a requirement
2. Writes the implementation
3. Writes the tests
4. Runs those tests
5. Reviews the code

It looks like several layers of verification but they can all contain the same mistake. If the agent misunderstood the requirement at step one, it can faithfully propagate that misunderstanding into both the implementation and its tests. Five checks created based on one mistaken assumption may be weaker than one independent rule.

I still think that coverage still matters but that's why a useful verification has at least two dimensions:

• Coverage: could this check observe the failure?
• Independence: is this check based on the same assumption that might be wrong?

The independence part is about where the check comes from: a pre-existing rule that must always hold is more independent than a test generated after implementation. Queryable runtime state is useful for the same reason: it can show what actually changed, rather than what the model believes its code did.

And sometimes the only verification layer that can answer the question is a human.

This is why I don't think the goal should be to create "self-verifying" agents in some absolute sense but it's to give them stronger ways to prove themselves wrong before requiring human judgment.

§ 8Where humans still matter

That leaves a narrower, but still important, role for humans: judging claims the system cannot turn into reliable evidence.

Some software claims have clear checks an agent can verify:

• Was exactly one order created?
• Was the rule still true after the change?
• Did this request exceed the latency threshold?

Others don't:

• Is this interaction actually good?
• Is this what the user meant?
• Is this risk acceptable?
• Does this feel trustworthy?

Adding another agent doesn't magically make those questions deterministic.

So the point here isn't zero human involvement. It's using humans where human judgment provides information the agent cannot get and shouldn't infer.

That distinction matters more as agents get faster at implementation. Because if an agent can generate a 1,200-line change faster than a senior engineer can read it, asking the engineer to figure out whether the change is correct (based on the diff) is a poor scaling strategy.

Over time, I expect more reviews to start with evidence:

• What behavior did the agent test?
• What rules were checked?
• What state transitions were observed?
• What failure cases did the agent try to reproduce?

And, maybe most importantly:

What was the agent unable to verify?

The code still matters, but reviewers should also look at the evidence, not only inspect the diff line by line.

§ 9The application boundary matters

With all that being said, if reviews are going to start from evidence rather than diffs, the next question is where that evidence comes from.

If we take a look at the first wave of improvements in coding agents, we'll realize that it came from better models. And a lot of the current improvements of the second wave are coming from better harnesses: context, tools, instructions, evaluators, browser control, isolated environments, and feedback loops.

These tools matter a lot, but eventually the harness reaches the application boundary and has to ask the application a few questions:

• What happened?
• What state changed?
• Did this rule still hold?
• Can I reproduce this scenario?
• Can I perform this operation without manually navigating six screens?

This doesn't mean every product needs a dedicated API for agents. A more observable system can create coupling, security risk, and maintenance cost, so an agent-facing surface is only worth building when the behavior it helps inspect is important enough to justify that cost. Sometimes, the right answer is an existing test, trace, CLI, or human review.

But the design pressure is still real: software is increasingly being modified by systems that operate through feedback. The better systems can expose its real state to agents, the more capable the agent becomes without changing the model at all.

The model didn't improve. The environment did.

That change points us to a different question:

How easy is it for the software to prove that the agent is wrong?

The best software for coding agents may not be the software that is easiest to generate. It may be the software that is easiest to prove wrong.