LLM Policy

The llm policy category governs which LLM models an agent can use and enforces token budgets. It inspects model names and cumulative token counts reported during execution via ctx.record_llm_call().

Use it when you need to restrict agents to approved models, block deprecated or expensive models, or cap token usage per run.

Rules

Rule	Type	Default	Description
allowed_models	string[]	[] (unrestricted)	Allowlist of model name patterns. If non-empty, only these models may be used
blocked_models	string[]	[]	Blocklist of model name patterns. Checked before allowlist
max_tokens_per_call	integer (min 1)	None	Maximum cumulative tokens across all LLM calls in a run
temperature_range	object {min, max}	None	Allowed temperature range (0-2). Informational only -- not enforced
action_on_token_violation	string	"warn"	Action when max_tokens_per_call is exceeded: "warn" or "block"
require_system_prompt	boolean	false	Whether a system prompt is required. Informational only -- not enforced

How Model Matching Works

The LLM handler supports versioned variant matching. When OpenAI returns gpt-4o-mini-2024-07-18 as the actual model ID, it matches the pattern gpt-4o-mini because the full name starts with the pattern followed by a - or : separator.

Model Name	Pattern	Match?	Why
gpt-4o-mini	gpt-4o-mini	Yes	Exact match
gpt-4o-mini-2024-07-18	gpt-4o-mini	Yes	Versioned variant (dash separator)
claude-3-opus:latest	claude-3-opus	Yes	Versioned variant (colon separator)
gpt-4o	gpt-4o-mini	No	Not a prefix match
gpt-4o-mini	gpt-4o	No	gpt-4o-mini starts with gpt-4o but next char is -m, not a version separator for gpt-4o -- wait, actually gpt-4o-mini does start with gpt-4o + -. This IS a match

Versioned Variant Matching

A model name matches a pattern if: (a) they are exactly equal, or (b) the model starts with the pattern AND the next character is - or :. This means gpt-4o-mini matches the pattern gpt-4o because mini follows a dash. Design your patterns carefully -- use the full base model name to avoid unintended matches.

How It Works

Enforcement Phases

Phase	Behavior
before_workflow	Stores rules in context. Always returns ALLOW
mid_execution	Checks models_used against blocked/allowed lists (BLOCK). Checks tokens_used against limit (action_on_token_violation: WARN or BLOCK)
after_workflow	Final audit: same checks as mid_execution but returns WARN (not BLOCK) for model violations

Evaluation Order

1. Blocked models checked first. If a model matches blocked_models, return BLOCK immediately
2. Allowed models checked second. If allowed_models is non-empty and the model does not match, return BLOCK
3. Token limit checked last. If cumulative tokens exceed max_tokens_per_call, return action_on_token_violation (WARN by default, BLOCK if configured)

Context Data

Context Attribute	Type	Source
models_used	list[str]	Populated from LlmCallRecord entries in the DB (distinct models across all calls in this run)
tokens_used	int	Sum of total_tokens from all LlmCallRecord entries for this run

Example Policies

Model Allowlist Only

Restrict to approved OpenAI models:

{
  "allowed_models": ["gpt-4o-mini", "gpt-4o"],
  "blocked_models": [],
  "max_tokens_per_call": null
}

Model Blocklist Only

Block deprecated models, allow everything else:

{
  "allowed_models": [],
  "blocked_models": ["gpt-3.5-turbo", "text-davinci-003"],
  "max_tokens_per_call": null
}

Token Budget Enforcement

Cap total token usage per run:

{
  "allowed_models": [],
  "blocked_models": [],
  "max_tokens_per_call": 4000
}

Combined Restrictive Policy

Full governance: approved models only, block deprecated, enforce token budget:

{
  "allowed_models": ["gpt-4o-mini", "gpt-4o"],
  "blocked_models": ["gpt-3.5-turbo"],
  "max_tokens_per_call": 8000,
  "action_on_token_violation": "block"
}

SDK Integration

Using the Decorator

When using supported LLM frameworks (OpenAI, LangChain, Anthropic, etc.), the SDK automatically intercepts LLM calls and triggers mid_execution governance. No manual recording is needed.

import waxell_observe as waxell
from waxell_observe.errors import PolicyViolationError

waxell.init()

try:
    @waxell.observe(
        agent_name="analyst",
        enforce_policy=True,
    )
    async def run_analysis(query: str):
        # LLM calls are auto-intercepted by instrumentors
        # Mid-execution governance fires automatically after each call
        return await call_llm(query)

    result = await run_analysis("Analyze sales data")

except PolicyViolationError as e:
    print(f"LLM policy block: {e}")
    # e.g. "Blocked model 'gpt-3.5-turbo' was used"
    # e.g. "Model 'claude-3-opus' is not in allowed list: gpt-4o-mini, gpt-4o"

Using the Context Manager

try:
    async with waxell.WaxellContext(
        agent_name="analyst",
        enforce_policy=True,
    ) as ctx:
        # Auto-instrumented LLM calls trigger governance automatically
        result = await analyze_data(query)
        ctx.set_result(result)

except PolicyViolationError as e:
    print(f"LLM policy block: {e}")

Enforcement Flow

Agent starts (WaxellContext.__aenter__)
    |
    +-- before_workflow: stores LLM rules in context
    |
    +-- Agent makes LLM calls
    |   |
    |   +-- Auto-instrumented call (OpenAI, LangChain, etc.)
    |   |   -> mid_execution fires automatically:
    |   |      -> blocked_models check (BLOCK if match)
    |   |      -> allowed_models check (BLOCK if not in list)
    |   |      -> token limit check (action_on_token_violation: WARN or BLOCK)
    |   |
    |   +-- Model "gpt-3.5-turbo" used (blocked)
    |       -> BLOCK: "Blocked model 'gpt-3.5-turbo' was used"
    |
    +-- Agent completes
    |
    +-- after_workflow: final audit
        -> Same model/token checks, but returns WARN (not BLOCK)
        -> Warnings recorded in governance tab

Creating via Dashboard

1. Navigate to Governance > Policies
2. Click New Policy
3. Select category LLM
4. Configure allowed_models, blocked_models, max_tokens_per_call
5. Set scope to target specific agents (e.g., llm-governed-agent)
6. Enable

Creating via API

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://acme.waxell.dev/waxell/v1/policies/ \
  -d '{
    "name": "LLM Model Governance",
    "category": "llm",
    "rules": {
      "allowed_models": ["gpt-4o-mini", "gpt-4o"],
      "blocked_models": ["gpt-3.5-turbo"],
      "max_tokens_per_call": 8000
    },
    "scope": {
      "agents": ["analyst"]
    },
    "enabled": true
  }'

Observability

Governance Tab

LLM evaluations appear with:

Field	Example
Policy name	LLM Model Governance
Action	allow, warn, or block
Category	llm
Reason	"Models approved: gpt-4o-mini" or "Blocked model 'gpt-3.5-turbo' was used"
Metadata	{"blocked_model": "gpt-3.5-turbo"} or {"tokens_used": 5000, "limit": 4000}

After-Workflow Audit

The after_workflow phase runs a final model and token audit. Model violations at this stage return WARN (not BLOCK), since the execution has already completed. Token limit violations also return WARN.

Common Gotchas

1. temperature_range and require_system_prompt are NOT enforced. They are informational only -- the handler does not check these values. They exist in the schema for documentation purposes.

2. Versioned model names match patterns. gpt-4o-mini-2024-07-18 matches the pattern gpt-4o-mini. This is intentional -- API providers return versioned model IDs even when you request the base model name.

3. allowed_models is only checked when non-empty. An empty allowed_models list means all models are allowed. Only a non-empty list activates the allowlist.

4. blocked_models is checked BEFORE allowed_models. A model in both lists is blocked. The blocklist always takes priority.

5. mid_execution returns BLOCK for model violations. This is hardcoded -- there is no warn mode for model violations at mid_execution. The after_workflow phase returns WARN for the same violations.

6. Token limit action is configurable. By default, exceeding max_tokens_per_call returns WARN. Set action_on_token_violation: "block" to hard-stop agents that exceed token budgets. This is enforced at mid_execution — the tokens for the current call have been consumed, but subsequent calls are prevented.

7. Pattern matching is prefix-based, not glob or regex. gpt-4o matches gpt-4o-mini because gpt-4o-mini starts with gpt-4o followed by -. Use full model names in your patterns to avoid unintended matches.

8. Auto-instrumented LLM calls trigger governance automatically. When using supported frameworks (OpenAI, LangChain, Anthropic, etc.), the SDK's instrumentors intercept calls and trigger mid_execution checks. If you use record_llm_call() manually (for unsupported providers), governance evaluates at after_workflow when the agent completes.

Next Steps

• Policy & Governance -- How policy enforcement works
• Quality Policy -- Validate output quality
• Operations Policy -- Timeout enforcement
• Policy Categories & Templates -- All 26 categories