LLM guardrails that hold run in code at the tool call

Types of LLM guardrails: only the one at the tool call holds

Of the places a guardrail can sit, only the tool call holds whatever the prompt says, because there the model's choice turns into an action. Simon Willison puts the threat plainly: once an attacker's tokens are in the context, assume they control the tool calls as well as the text².

The common types of LLM guardrails are named by where they sit. NVIDIA's NeMo Guardrails runs input, retrieval, dialog, execution and output rails at different stages of an interaction³. Lay them on the agent loop and only one stop acts after the model decides and before anything changes: execution, the tool call.

OWASP's guidance on excessive agency agrees: implement authorization in downstream systems rather than relying on the model to decide whether an action is allowed⁴. In the OpenAI Agents SDK, input guardrails run only for the first agent in a chain and output guardrails only for the agent that produces the final output⁵. The SDK's answer for the middle is tool guardrails, which run on every guarded function-tool invocation, before and after execution⁵.

Here is refunds_request with its checks inside. The customer comes from the session's ticket, never from the model, and each refusal is an error the agent can read.

import os
from decimal import Decimal
from pydantic import BaseModel, Field
from orders import order_owner, order_total
from payments import refunded_on_ticket, create_pending_refund

PER_TICKET_CAP = Decimal(os.environ["REFUND_TICKET_CAP"])  # from your refund policy

class RefundRequest(BaseModel):
    order_id: str = Field(pattern=r"^ord_[a-z0-9]{12}$")
    amount: Decimal = Field(gt=0, max_digits=8, decimal_places=2)
    reason: str = Field(max_length=500)

class RefundRefused(Exception):
    """Returned to the agent as a tool error it can read and explain."""

def refunds_request(args: dict, ticket) -> dict:
    req = RefundRequest.model_validate(args)  # typed: no free-form ids or amounts
    # The customer comes from the session's ticket, never from the model.
    if order_owner(req.order_id) != ticket.customer_id:
        raise RefundRefused(f"{req.order_id} is not an order of this customer")
    if req.amount > order_total(req.order_id):
        raise RefundRefused("the amount is more than the order total")
    if refunded_on_ticket(ticket.id) + req.amount > PER_TICKET_CAP:
        raise RefundRefused("over the per-ticket cap")
    return create_pending_refund(ticket.id, req)  # the policy decides who approves

On the planted ticket, the ownership check refuses every order on the list, because each belongs to another customer. If the attacker slips in orders of their own, the cap bounds what the ticket can draw. That is why the check is code. An engineer on Hacker News found that retrying a model's call did not fix output that was valid JSON with a wrong judgement; the fix was a deterministic gate⁶.

Break the lethal trifecta with scoped credentials before buying a classifier

Scope each session's credentials so it never holds untrusted text, private data and a way out at once; that stops the planted ticket before any classifier reads it. Meta's Agents Rule of Two says a session may combine at most two of untrusted input, sensitive data, and the ability to change state or communicate out. One that needs all three needs a person or another reliable means of validation⁷. It turns Simon Willison's lethal trifecta into a design rule⁸.

General Analysis showed what all three legs cost. A support ticket with planted instructions reached a developer's assistant, connected through Supabase's MCP server with a role that bypasses row-level security. The assistant copied a private table of integration tokens into a reply the customer could read⁹.

Their conclusion: a prompt injection filter can flag suspicious content, but it cannot grant or restrict database permissions⁹. Replit did the same for its coding agent: separate production and development databases, with the agent given only the development one¹⁰.

The refund agent's database identity reads the orders and payments of the ticket's customer and writes only pending refunds. Replies are drafts scanned for card numbers and other customers' order ids, and larger refunds go to a person. Meta says traditional software guardrails and human approvals remain its preferred way to satisfy the rule today⁷.

Over MCP, the specification starts from a minimal scope of low-risk read operations¹¹. The planted ticket's second ask, other customers' card history, now has nowhere to go: the role reads only the ticket's own customer and never raw card data.

Guardrail classifiers miss the attack written for them, so they only route

A classifier catches the average attack and misses the one written to slip past it, so it can send a ticket to a person but never decide a refund. In the study from the opening, whose authors include the security vendor Mindgard, the most successful attack was Emoji Smuggling, which hides the payload in emoji variation selectors¹. Against adversarial ML evasion in the same study, which perturbs the words a classifier weighs most, Meta's Prompt Guard held up best on prompt injections, at 2.76% attack success¹.

Our read: one detection score does not transfer across attack types. The study tested the first Prompt Guard. The code below runs its successor, which the study did not measure and whose model card warns that adversaries may build attacks to bypass detection¹². Simon Willison on products that advertise a catch rate⁸:

The LLM guardrail frameworks differ mostly in which stop on the loop they cover. NeMo Guardrails spans the input to output stages³, and Guardrails AI combines validators into input and output guards around the model¹³. Amazon Bedrock Guardrails filters predefined harmful content categories, Prompt Attack among them¹⁴, and Microsoft's Prompt Shields screens for injection without judging harm, accuracy or grounding¹⁵. None of them sits inside your refund tool.

The OpenAI Agents SDK runs input guardrails in parallel by default, so a tripwire may fire after the agent has already executed tools⁵. Blocking mode finishes the guardrail before the agent starts⁵.

from transformers import pipeline
from triage import handle_ticket
from handoff import hand_to_person
from metrics import log_trip

detect = pipeline("text-classification", model="meta-llama/Llama-Prompt-Guard-2-86M")

def tripped(text: str) -> bool:
    verdict = detect(text, truncation=True)[0]
    return verdict["label"] != "LABEL_0"  # LABEL_0 is benign on the model card

def intake(ticket):
    # Blocking: the classifier finishes before the agent sees the ticket.
    if tripped(ticket.body):
        log_trip(ticket.id)  # the trip rate is a chart, not a decision
        return hand_to_person(ticket, reason="injection classifier tripped")
    return handle_ticket(ticket)  # refunds_request still checks everything

Assume the planted ticket, re-encoded with emoji variation selectors, gets past; refunds_request still refuses the other customers' orders.

Every guardrail blocks real customers, so the eval counts both rates

Measure every guardrail on two numbers, the attacks it catches and the real customers it blocks, because each one stops some honest work. In a 2026 paper on a learned execution guardrail for coding agents, the guarded agent refused or abandoned legitimate work where the unguarded one never did¹⁶. Each run carried one planted step amid the legitimate work.

During an incident, Hugging Face's responders found their log forensics requests blocked by hosted providers' safety guardrails¹⁷, so they moved the work to a self-hosted open-weight model¹⁷. Hamel Husain and Shreya Shankar describe how teams keep that cost in view¹⁸:

No practitioner we found has published a catch rate or false-block rate for a refund agent, so measure your own. Label a few dozen real refund tickets and as many attacks, plain, encoded and hidden in an order note, then fail the build on a catch floor or a false-block ceiling. The planted ticket and its emoji-encoded copy are two of the attack cases. When the classifier blocks a customer who wrote "ignore my last message, the size was wrong", file a bug.

import json
import os
from intake import tripped

# One line per ticket: {"id": ..., "body": ..., "attack": true or false}
CASES = [json.loads(line) for line in open("cases/guardrail_tickets.jsonl")]
ATTACKS = [c for c in CASES if c["attack"]]
BENIGN = [c for c in CASES if not c["attack"]]

# Both thresholds are yours: set them from your first eval run.
CATCH_FLOOR = float(os.environ["GUARDRAIL_CATCH_FLOOR"])
FALSE_BLOCK_CEILING = float(os.environ["GUARDRAIL_FALSE_BLOCK_CEILING"])

def trip_rate(cases):
    return sum(tripped(c["body"]) for c in cases) / len(cases)

def test_catch_rate():
    missed = [c["id"] for c in ATTACKS if not tripped(c["body"])]
    assert trip_rate(ATTACKS) >= CATCH_FLOOR, f"missed: {missed}"

def test_false_block_rate():
    blocked = [c["id"] for c in BENIGN if tripped(c["body"])]
    assert trip_rate(BENIGN) <= FALSE_BLOCK_CEILING, f"blocked: {blocked}"

The catch floor can sit below the grade Willison calls failing, because a trip only sends a ticket to a person and refunds_request stays the gate.

Ask a person only on the irreversible step, so the ask still means something

Put a person only on the irreversible step, because a person asked about every call stops reading and approves them all. Meta concedes that designs satisfying the Rule of Two can still fail, for example when a user blindly confirms a warning⁷. Writing up their GitHub MCP exploit, Invariant Labs note that confirmation is on by default in Claude Desktop, and that many users switch it off¹⁹:

Anthropic went the other way. Internally, a sandbox around Claude Code cut permission prompts sharply and isolates even a successful prompt injection²⁰.

Claude Code's hooks let a pre-tool check allow, deny or escalate to the user²¹. Its docs warn that a silent hook does not approve a call, and a timed-out hook does not block one unless its failure is set to²¹. So write the policy as code that answers every call and denies on any failure.

import os
from concurrent.futures import ThreadPoolExecutor
from decimal import Decimal
from payments import refunded_on_ticket

AUTO_ALLOW = {"orders_lookup", "payments_lookup", "reply_to_customer"}  # replies are drafts
AUTO_APPROVE_LIMIT = Decimal(os.environ["REFUND_AUTO_APPROVE_LIMIT"])  # from your refund policy
pool = ThreadPoolExecutor(max_workers=4)

def decide(call) -> str:
    if call.name in AUTO_ALLOW:
        return "allow"
    if call.name == "refunds_request":
        # Count what the ticket already drew, so small refunds cannot add up unseen.
        total = refunded_on_ticket(call.ticket_id) + Decimal(str(call.args["amount"]))
        return "allow" if total <= AUTO_APPROVE_LIMIT else "ask"  # ask: the support lead
    return "deny"  # any tool not named here never runs

def pre_tool(call, timeout_s: float = 2.0) -> str:
    try:
        return pool.submit(decide, call).result(timeout=timeout_s)
    except Exception:  # a timeout or a crash denies; silence never approves
        return "deny"

The support lead now sees only tickets whose refunds add up past the limit. If the planted ticket got this far, it could draw no more than that limit on the attacker's own orders before a person looks.

What to do on Monday

1. Count the legs. For each agent session, list the tools that read untrusted text, see private data, and send or change anything. Where one session holds all three, take autonomy away from the third.
2. Move one check into the tool. Give the riskiest write tool typed arguments, an ownership check taken from the session, a cap, and a database identity that cannot do more.
3. Make the classifier block and route. Run it before the agent, send trips to a person, log each one and chart the rate.
4. Build the guardrail eval. Label real benign tickets and real-looking attacks, encoded ones included, and fail the build on a catch floor or a false-block ceiling.
5. Audit the approvals. If a reviewer approves nearly everything, narrow what asks until each ask is rare.

Build it with me: the Free Live Lab Secure AI Agents with Guardrails, live on Thursday, November 5.