Tool Calling in Production LLM Agents: What We Log Before Giving One Access to a Real API

Testing tool calling in a notebook is almost always reassuring. The model gets a function schema, picks one, the notebook prints the arguments, you eyeball them, and if something looks off you tweak the prompt and rerun the cell. The whole loop fits on one screen and in the memory of whoever is running it. The problem shows up when that same agent gets wired, in production, to an API that actually changes something: creates an order, cancels a subscription, emails a customer, deletes a record. At that point the question stops being "does the model pick the right tool most of the time?" and becomes a much less comfortable one: "if it picks wrong even once, how do we find out, and how do we undo it?" Without an answer to that second question, giving an agent real access isn't automating a task, it's delegating a decision that nobody will be able to audit afterward.

What actually changes between the notebook and production

In a notebook there is one call at a time, with arguments you built or reviewed yourself before running it. In production, a typical agent chains several tool calls within a single conversation turn, decides the order on its own, and sometimes repeats a call because it interpreted the previous one as having had no effect. Nobody is watching the screen while that happens. The only trace of what happened, if any, is whatever someone decided to persist ahead of time. And the habit most teams bring over when they move from prototype to production is saving the final message the agent showed the user, not each intermediate step that produced it. That habit is the first thing that fails.

The minimum record per tool call

The unit worth logging isn't the whole conversation or the final response: it's each tool call, individually, with its own identifier. Every row in that log needs, at minimum, a call identifier, the identifier of the session or trace it belongs to, the tool name, the exact arguments the model generated, the raw result the real API returned, the final status of that call, and a timestamp. If the model provider exposes some form of reasoning or justification tied to the tool choice, we keep that too, but never as a substitute for the real arguments and result: the reasoning explains intent, not what actually happened against the API.

def execute_tool(tool_name, arguments, call_fn, trace_id):
    call_id = str(uuid4())
    entry = {
        "call_id": call_id,
        "trace_id": trace_id,
        "tool": tool_name,
        "input": arguments,
        "timestamp": datetime.utcnow().isoformat(),
    }
    try:
        result = call_fn(**arguments)
        entry["status"] = "success"
        entry["output"] = result
    except Exception as exc:
        entry["status"] = "error"
        entry["error"] = str(exc)
        raise
    finally:
        append_log(entry)
    return result

The part of this wrapper that matters isn't the exception handling, which is obvious. It's that the log entry is written inside the finally block, before the exception propagates or the result makes its way back to the agent. If logging happened after the agent decides what to do with the result, a failure in that later step would leave the real call with no trace at all.

Capturing the state before, not just the result

Saving the result of a mutating call isn't enough to make it reversible. If an agent cancels a subscription, a log entry that just says "cancellation succeeded" doesn't say what the plan was, what the renewal date was, or what payment method was on file before the cancellation — and without that, reverting the operation means reconstructing that state by hand from whatever incomplete information survives in some other system. That's why, for any tool that mutates state, we capture a snapshot of the prior state as part of the same log entry, before the call executes:

def execute_mutating_tool(tool_name, arguments, call_fn, fetch_before_fn, trace_id):
    before = fetch_before_fn(arguments)
    entry = {"before_state": before, "tool": tool_name, "input": arguments}
    result = call_fn(**arguments)
    entry["after_result"] = result
    append_log(entry)
    return result

With that snapshot, reverting stops being a manual reconstruction and becomes applying the inverse operation using known, exact data, not assumptions about how things used to be.

Idempotency: the model retries too

An agent can call the same tool twice with the same arguments, not because it's poorly designed, but because it didn't get confirmation of the first call in time — a timeout, a slow response — and concluded it should try again. Against a read-only tool that has no consequences. Against one that charges, sends, or deletes, a duplicate call is a duplicate real-world effect. The fix isn't trusting that the model won't retry, because it will: it's requiring a deterministic idempotency key on every mutating call, derived from the session and the step, so the destination API can recognize the second attempt as the same one instead of a new one.

idempotency_key = sha256(f"{trace_id}:{step_index}:{tool_name}".encode()).hexdigest()

Human approval for the highest-risk categories

Not every tool deserves the same treatment. A search or lookup tool can run automatically, because a wrong result has no real cost beyond a bad answer. A tool that issues a refund, cancels a contract, or sends a message to a third party does have a cost, and for those categories we don't execute directly: the agent logs the proposed call, with its arguments and its justification, into an approval queue, and a human decides before the real call happens. That person's decision gets logged too, linked to the same call_id, so the full history — what the model proposed, who approved or rejected it, and what happened next — lives in one place.

What didn't work: logging only the final response

The first version of this system, on a project with an agent that managed support tickets, only saved the final message the agent sent back to the user, on the assumption that the message faithfully reflected what had happened underneath. When a ticket ended up closed with the wrong status after the agent had chained three different tool calls, there was no way to tell which of the three had caused the problem: only the final text existed, describing the outcome in general terms, not each call with its real arguments. We had to reconstruct the incident by reading the ticketing API's logs separately and lining up timestamps by hand, work that should have taken minutes and took most of an afternoon.

The second assumption that failed was thinking the model provider's own logs — the record of what prompt was sent and what completion came back — counted as an audit trail for the tools. Those logs capture the conversation with the model, not the real effect against our API: they're two different systems, and mistaking one for the other leaves a gap exactly where it matters most, which is the moment the agent stops talking and starts acting.

What we check before giving an agent access to a real API

Before wiring an agent to a tool that mutates data, we verify that every call lands in its own log row, with its own identifier, exact arguments, and raw result, independent of whatever final message the user sees; that every mutating call captures the prior state in that same record, so reverting means applying an inverse operation on real data instead of a manual reconstruction; that every mutating call carries a deterministic idempotency key that absorbs the model's retries without duplicating the effect; and that the highest-risk categories go through a human approval queue before execution, not after. When those four conditions hold, the model picking the wrong tool stops being an unexplained incident and becomes a log row that says exactly what happened and what to do to undo it.