Testing LLM agents with reproducible test cases: why a prompt that works in a demo doesn't make a reliable agent in production

For a while we evaluated our LLM agents the way most teams start out: someone runs the agent by hand through two or three representative conversations, reads the final response, and if it "sounds right" the prompt change is considered ready for production. That manual evaluation holds up as long as the agent does one thing and the prompt barely changes. It stops holding up the moment the agent starts deciding which tool to call, with what arguments, and in what order — because at that point the final text the user sees stops being the only surface that can break.

The problem with grading only the final text

An agent that answers the user's question correctly can be arriving at that answer through a path we won't be able to trust next time. The hardest case for us to catch by hand was one where the agent, faced with a refund question, replied with the correct text — "your refund has been processed" — but in a fraction of the conversations it had reached that conclusion without calling the check_refund_status tool, inferring the status from conversation context instead of actually checking it. The final text was indistinguishable between the correct path and the made-up one. If the evaluation criterion is "does the final answer sound reasonable?", that kind of failure is invisible by design: the answer is reasonable both times.

That forced us to split what we used to treat as a single question into two: did the agent say something reasonable, and did the agent do what it needed to do to be entitled to say it. The second question can only be answered by looking at the trace of tool calls, not the final message.

What a reproducible test case actually looks like

A test case for an agent needs, at minimum, three parts: a fixed input (the message or sequence of messages the agent receives), a fixed initial state for any external system the agent can query (instead of calling the real refunds API, a test double with known data), and a set of assertions that aren't limited to the output text.

def test_refund_already_processed():
    input_messages = [
        {"role": "user", "content": "What happened to my refund for order 4821?"}
    ]
    initial_state = {
        "order_4821": {"refund_status": "processed", "amount": 45.00}
    }

    result = run_agent(input_messages, mock_backend(initial_state))

    tool_calls = result.tool_calls
    assert any(
        c.name == "check_refund_status" and c.args["order_id"] == "4821"
        for c in tool_calls
    ), "the agent answered without checking the real refund status"

    assert result.final_text is not None

The assertion on tool_calls is the part a manual review was never going to catch, because grading by hand means reading the final message, not inspecting the list of calls the agent made to get there. We add an explicit assertion that's deliberately stricter than "the answer is correct": if the agent didn't call the verification tool, the case fails even when the final text happens to be right, because what we're testing isn't whether it guessed correctly this time — it's whether the path it took is one we can trust next time with different data.

That brings in a second necessary piece: the simulated backend (mock_backend) has to return exactly the data the test case defines, not an approximation. If the test double returns real data from a staging environment that changes between runs, the case stops being reproducible: a run can fail simply because the underlying data changed, not because the agent behaved differently. A case's reproducibility depends on both the input and the state of the world the agent can query, and both need to be pinned in the test code, versioned alongside the prompt being tested.

Why ad hoc manual evaluation doesn't scale

The underlying problem with manual evaluation isn't that it's slow — though it is — it's that it doesn't accumulate. When someone tests a prompt change by hand with three conversations, those three conversations don't end up documented anywhere reproducible: the next person who touches the prompt doesn't know what cases were already tried, so they re-test similar ground, and the rare cases — the user asking about an order that doesn't exist, the one requesting someone else's refund, the one that packs two questions into a single message — depend on someone remembering to try them again every single time.

With test cases versioned in the repository, every case we ever found broken stays there as a permanent regression check. The rule that worked for us was simple: every agent bug that makes it to production gets turned, before it's closed, into a test case that reproduces that exact failure with a fixed input. That doesn't stop new bugs from showing up, but it stops the same bug from sneaking back in two prompt versions later — which was the pattern that bit us most often before we automated this: we'd fix a case, change the prompt for an unrelated reason a month later, and the original case would break again without anyone noticing until a user complained a second time.

The regression that slipped past us until we automated this

The concrete case that finally convinced us to invest in this test suite was a prompt change meant to improve the tone of the responses — make them shorter, less generic — that, as an unintended side effect, changed the order in which the agent decided to call the available tools. Before the change, the agent always called check_refund_status before check_return_policy whenever both applied. After the change, the model started reversing that order in part of the cases, which on its own didn't break anything functionally — the two tools are independent of each other — but it did break an assertion in our test suite that we hadn't realized implicitly depended on that order, because a later test assumed the result of the first call was already available in context by the time the second one was evaluated.

Reading only the final responses from those conversations, the prompt change looked like a clear improvement: the messages really were shorter and less generic. The regression only became visible when the automated test suite, running the same fixed inputs as always, turned a long-green case red. Without that suite, that order change would have shipped to production invisibly during review, and would likely have surfaced later through a production trace that's far harder to reconstruct than a test that fails at the exact moment of the change that broke it.

What we check before signing off on a new version of an agent

Before promoting a prompt or logic change for an agent to production, we confirm the test case set covers at least the main happy paths, the edge cases we've already broken in production before, and the cases where the agent has to decide not to call any tool at all. We check that every case asserts on the trace of tool calls, not just the final text, because that's the part manual review systematically lets through. We verify that each case's initial state is pinned in the test code and doesn't depend on an external environment that can change between runs. And when we find a new bug, we don't close it until a test case exists that reproduces it, because that's the only real guarantee it won't come back without anyone noticing.