There is a line in my job now that was not there two years ago. Read what the machine wrote, and decide whether to put my name on it.

Nobody added it to a job description. It arrived anyway, the week the flow started producing output faster than I could produce it myself.

The team I work with runs an agentic QA flow that I put together. A ticket goes in. Reviewed, formatted, house-style test cases come out, checked against the test management tool for duplicates and ready to import. It saves hours every sprint and nobody wants to go back.

It is also nowhere near able to do the job without me. The interesting part is that the flow itself is built on that assumption. Every serious decision point in it ends at a person.

What the Flow Actually Does

Seventeen specialised agents, fourteen orchestrator pipelines. For a test case request the chain runs roughly like this:

  1. Analyse the ticket. Pull out requirements, acceptance criteria, risk areas, scope.
  2. Run a risk analysis over it, using SFDIPOT, to decide which scenarios deserve a test case.
  3. Update the knowledge base if the ticket introduced a module or flow we have not documented.
  4. Design the test cases.
  5. Generate the import-ready CSV in the exact schema the test manager wants.
  6. Rewrite the steps into house style. No passive verbs in action steps, verifications only in expected results, the right markup for the tool.
  7. Run a deliberately harsh review pass over the result and score it.
  8. Look the ticket up in the test manager. Do these test cases already exist, or are they new?
  9. Dry run the import. Show exactly what would be created and what would be updated, per step.
  10. Stop. Wait for a human to approve. Only then write.

What that replaced was me reading the ticket, opening the template, writing four to six cases by hand, fixing the formatting, searching the tool for something similar, fixing the CSV when the import complained, and importing.

I have not instrumented it properly, so I am not going to quote a percentage at you. Hours, not minutes, per sprint. That is the honest version.

Now the part that matters more.

1. A "No" From a Machine Is Not Evidence of Absence

Step eight above is a small agent with one job: given a ticket, does a test case for it already exist. It answers CREATE or EDIT, and when it answers EDIT it pulls the current steps so the designer revises in place instead of producing a duplicate.

The branch I care about is the third one. UNKNOWN.

When it cannot reach the tool, it does not guess. It reports UNKNOWN, says so plainly, and notes that the dry run at the end of the pipeline is the real safety net. I wrote that branch in, and writing it forced me to admit something about the other two answers as well.

If the agent says a test case exists, I can verify that in a second. The key is right there, I open it, it is real. A positive claim carries its own proof.

A negative claim carries nothing. "No test case exists for this" is a statement about a search, not about the world. The search used a ticket key. Somebody may have written that coverage under a different ticket last quarter, or attached it to an epic, or named it something the query never matched. The agent is not lying. It genuinely did not find anything. That is a different sentence from "there is nothing there."

And duplicate test cases are not a cosmetic problem. They are two versions of the truth in a system somebody executes from. The one that drifts is the one a new tester runs six months from now, on the assumption that it is current.

So the QA still searches. Not always, not for everything. But the habit of doubting a negative, and knowing which negatives are worth the thirty seconds, is judgement. It is not a step you can put in a pipeline, because the answer to "is this worth checking" depends on what you know about the module.

2. Guardrails Written in Markdown Are Requests, Not Walls

The flow has hard rules, and they are good ones. Never write test cases from a ticket ID alone. If the ticket system is unreachable, halt and say so rather than inventing ticket content. Never claim a test passed without running it. Four to six cases per ticket by default, never silently exceed it.

Those rules are the reason the output is usable. They are also, almost all of them, sentences in a file.

I ran into this while reviewing my own handoff notes for a refactor of the pipeline. I had written a checkpoint into the procedure, and then, underneath it, this:

Honest limit: this is a doc-level instruction to the executing agent, not an enforced gate. It holds only as well as that agent complies. The human reading the reported numbers is the real gate.

I wrote that to keep myself honest about a refactor. It turned out to be the whole thesis.

Some of the rules in the flow are not sentences. There is a hook that lints every generated CSV and quarantines the file if a credential or token made it into a data column. That one is a Python script wired into the editor event. It runs whether or not the model is having a cooperative day. It cannot be reasoned with, talked around, or forgotten when the context gets long.

That difference is the design lesson I would hand anyone building one of these:

If a rule must never be broken, make it a script. If you cannot make it a script, the human is the enforcement mechanism, and you should stop pretending otherwise.

A prompt file full of "NEVER" in capital letters feels like a control. It is a strong suggestion with good compliance statistics.

3. The Agent Reads the Ticket. The QA Reads the Room.

An agent gets a lot of context: the ticket body, the acceptance criteria, the attachments, a knowledge base of the platform built up over months. That is genuinely more than a tired human holds in their head on a Thursday afternoon.

Here is what it does not get.

It does not get the product owner saying in standup that the second half of the ticket is not in this release. It does not get the developer mentioning, in passing, that this touches a shared config and the other team is changing the same table. It does not get the memory that the last time anyone went near this module, something unrelated broke in the reports and nobody noticed for a week.

And it does not get the feeling. The one where you read an acceptance criterion, and something about it is wrong, and you cannot say what yet, so you go and ask. Half the time it is nothing. The other half it is a requirement that contradicts a rule the platform has had for three years.

Every rule in the agentic flow got there because a human ran into the problem first and wrote it down. That is the actual relationship. The flow is a very fast way to apply what we already know. It does not produce new knowledge about the product. New knowledge still comes from executing tests, talking to people, and being suspicious in the right direction.

4. Somebody Has to Own the Import

The import agent has one hard rule above all the others: never write on the first pass.

It dry runs. It prints a plan. Four to create, two to update, zero unchanged, and for the updates it shows the per-step difference so you can see what is about to be overwritten. Then it stops and asks, in plain language, for approval. Nothing goes in until somebody types the word.

That is heavy for a test case import, and it is heavy on purpose.

Those test cases carry execution history. They sit in a system that gets audited. A silent overwrite is not the kind of bug you notice on Tuesday. It is the kind you discover during an audit, when a test case someone signed off on three months ago no longer says what the sign-off referred to.

But the deeper reason has nothing to do with the tooling. Accountability does not distribute. If a wrong test case gets imported, and a real defect ships behind a green run, "the agent wrote it" is not an answer anybody accepts. It was not an answer before AI, when we copied a test case from an old ticket without reading it properly. It is not one now.

The approval prompt is the point where ownership transfers. Everything before it is a draft. Everything after it is mine.

What the Job Actually Became

Less writing. Much more reviewing. And reviewing is not the easier half.

It is harder to spot a flaw in plausible text than on a blank page. When output arrives in exactly the format you expect, in the house style you enforced, with the right number of cases and the right column headers, your guard drops. Fluent and wrong is more dangerous than obviously wrong, because obviously wrong gets fixed.

So the review has a shape now. Per batch, I read every step as though a tester who has never seen this feature will run it in six months. I check that the expected results assert something you can actually observe, not something you can hand-wave. I check the cross-system coverage is really there and not just mentioned. I check for duplicates. And I check it did not quietly hand me nine test cases when the rule says four to six, because generation is free and review is not.

None of that is glamorous. It is also the part of the job that cannot be delegated to the thing doing the work, for the same reason you do not let a developer be the only reviewer of their own pull request.

If You Are Building One of These

Six things I would tell myself before building an agentic QA pipeline.

  1. Put the gate at the write, not at the read. Let agents read anything, analyse anything, propose anything. The moment something is about to be persisted in a system other people trust, a human approves it.
  2. Make the dry run show a diff, not a summary. "Will update 2 test cases" is not reviewable. It is a number you nod at. Per-step differences are reviewable.
  3. Design for the agent being wrong, not only for it failing. An explicit UNKNOWN branch is worth more than a confident guess, and it is worth more than a crash.
  4. Anything that must never happen should be code. A linting hook beats a capitalised NEVER in a prompt file, every time.
  5. The knowledge base is why the output is good, and it rots. It is a human artifact. Somebody has to keep feeding it, and that somebody has to have actually tested the thing.
  6. Hold the quantity cap. An agent will happily hand you twenty test cases. You then own twenty test cases, forever, including the eleven that were not worth writing.

The Short Version

The flow removed the typing. It did not remove the thinking, the doubting, or the signature. What it changed is where my hours go: out of producing the first draft, and into deciding whether the draft is true.

That is a real change, and a good one. It is not replacement. A tool that needs someone to verify and own its output has not replaced that someone. It has made them faster and given them a new way to be wrong.

Somebody still has to hold the gate. AI agents did not automate that part of the job. It got heavier, because there is far more arriving at it than there used to be.

If you want the longer story of how the fleet got built in the first place, that is in From One Giant Skill to an AI-Powered QA Team.

September 2026