ARBT: how the agent decides what to test, and proves it

Part 3 of 5 in a series on autonomous software delivery. Part 2 covered AATB, the flow where our agent Ralph merges its own pull requests. This part covers the testing and evidence behind that decision.

An agent that writes its own code, writes its own tests and reports its own success has plenty of room to fool itself. A green build helps, but it does not tell us whether the agent tested the behavior we asked for. Our answer is ARBT: Autonomous Risk-Based Testing, which gives Ralph explicit rules for what to test, when more testing is needed, and what evidence must accompany a completion claim.

Start with the request

The work item’s acceptance criteria define the minimum test surface. Every criterion that can be covered by a test gets one, and the tests stay in the repository as regression protection. Take the export bug from Part 1: because the request was to fix an export with an active filter, checking whether a file downloads is insufficient. The test must check that the file contains the matching records and excludes the others.

If that test fails, narrowing it to “a file was downloaded” changes the question until the answer is green. The agent needs to fix the behavior or explicitly revisit the requirement, so that a passing test still answers the original question.

Eight testing steps, chosen by risk

Ralph uses eight named testing steps. Four are required on every delivery; four depend on the change. “Always” means running the relevant suite or checks, with new tests added for the behavior being changed. Unit and component integration tests also run in CI as required merge checks.

Before the merge

1. Unit tests. Always. Individual parts in isolation.
2. Component integration tests. Always. Our own components working together, using local or disposable infrastructure.
3. Live integration tests. When external integrations change. Real calls to external services, retained in the regression suite but excluded from the default CI gate because of cost and latency.
4. Local system test. When runtime behavior changes. Start the application in the session and exercise the affected UI, API or command, including after refactors.

After deployment to the integration environment

5. Smoke test. Always. Check that the deployed application starts, responds and can reach its dependencies.
6. Acceptance test. Always. Demonstrate the work item’s acceptance criteria on the deployed application.
7. Full regression suite. When integrations or shared integration code change. Run the accumulated integration tests, including live tests beyond those written for this change.
8. Full system test. When broader system behavior changes. Exercise important flows across the deployed system through its real interfaces.

Acceptance testing focuses on the request, while full system testing checks the wider behavior that the change could affect. Running every expensive testing step for every change would consume time and external-service budget without necessarily adding useful confidence. The agent makes that risk judgement and reports every testing step: ran, with evidence; skipped, with a reason; or blocked, with what is needed to proceed.

Reporting makes those decisions open to review: “No external integration changed” is a claim a reviewer can challenge, whereas an omitted testing step is much harder to notice. Missing access belongs under blocked, with the actual error and a request for the permission Ralph needs. That verification remains incomplete until the test runs successfully.

The screenshot that became a gate

One purely visual change taught us why naming the testing steps was not enough. Ralph passed its unit tests and skipped starting the application locally, only to deliver a visibly broken result. One look at the running screen would have caught the defect; instead, fixing it cost another PR, CI and deployment cycle.

We moved the check to the moment before opening a PR: for a UI change, the workflow now tells the agent to find a screenshot of the changed surface, captured from the locally running application. A missing screenshot sends it back to verify the change; an application that cannot start requires it to report the blocker before merging. The check still relies on the agent following the workflow, but it makes verification concrete at the decision point and produces evidence the human can use later.

Show what happened

For each acceptance criterion, evidence has three parts: which testing step verified it, where it ran, and what was observed. For the export example, a useful result would explain that an acceptance test on the integration environment confirmed the downloaded file contained all matching records and no excluded records with the filter active. Linking the run and attaching the exported file makes that claim checkable. A screenshot of the download button alone would not establish what the file contained.

“All tests pass” leaves out what they demonstrated, and a link to a test file only shows that a test exists. Connecting the requested behavior to an observed outcome is how ARBT supports the human review in Part 2. Alongside CI, branch protection and the revert path, it gives the reviewer a running feature, evidence against the request, and an explicit account of verification gaps. Larger changes need another decision before implementation: agreement on the design, covered in Part 4.

The agents write the code. Engineering the loop is the job now.

Ralph is built at Rubicon, part of BBTG. AATB and ARBT are our working answers, not industry standards. Yet.