Evaluate the Runtime, End to End

This tutorial walks the public evaluation project, github.com/waxell-ai/runtime-evaluation, from an empty terminal to a signed evidence pack. The project's README lists the commands. This page adds why each step exists, what to check in the output, and where people get stuck.

Allow about forty minutes. Two image builds run in the background while you work through the middle steps.

By the end you will have:

• pushed one reviewer agent five ways: Claude Agent SDK, CrewAI and AutoGen as declarative manifests, and LangGraph and pydantic_ai as your own code, each built into its own container;
• watched every agent call a small system of record on your laptop through a domain;
• verified, offline, who each call was for (the end-user token) and what made it (the agent identity, AXID);
• blocked an action with a policy before it left the runtime;
• stood up LLM-judged evaluations and run wax test scenarios;
• exported a signed attestation and an auditor pack;
• seen four memory tiers and two graphs with per-agent grants.

your terminal ──wax push──▶ Waxell runtime ──signal──▶ governed run (any framework)
                                                            │ things__get / set_review / publish
                                                            ▼
                               Cloudflare tunnel ◀── X-Waxell-Sub-User-Token + X-Waxell-AXID
                                                            │
                                                  domain_server/ on your laptop
                                                  verifies both, logs every call

Before you start

You need	Why
Python 3.11+ and pip install 'waxell>=0.1.8'	The wax CLI. 0.1.8 is the floor; preflight.sh checks it.
Docker with Compose v2 (docker compose, not docker-compose)	Runs the domain server and the tunnel.
Port 8030 free	The domain server and its live page.
Node 18+	Only for step 7's zero-dependency verifier.
A Waxell workspace with the Runtime product	Without it, wax push answers feature_gated. Ask us to turn it on.
An Anthropic API key	Stored once as a workspace secret. Every agent and the step 9 judge use it.

git clone https://github.com/waxell-ai/runtime-evaluation
cd runtime-evaluation

Run every command below from the repository root unless a step says otherwise.

Step 0 — Connect the CLI to your workspace

pip install 'waxell>=0.1.8'
wax config init --profile eval
wax -p eval whoami
./scripts/preflight.sh eval
wax -p eval secrets set ANTHROPIC_API_KEY <your Anthropic key>

What's happening. wax config init writes a profile to ~/.waxell/config: an API URL plus an API key. Nothing goes into the repository. You create the key in the app under Settings → API Keys. The API URL is https://api.waxell.dev unless you were given another. Every command here passes -p eval. If you'd rather not type it, run wax config use eval once.

The Anthropic key becomes a workspace secret. Agents never see it. Their model calls go through Waxell's governed proxy, which adds the key on the way out. That is why no manifest in this repository contains a key, and none needs one.

Check. whoami names your workspace. Preflight is all ✓, apart from one informational line saying .env is not created yet; you create it in step 3. A CLI … is older than 0.1.8 failure means pip install -U waxell.

Step 1 — First agent: push, fire, stream

cd agents/00_hello
wax -p eval push
wax -p eval signals fire hello_run --payload '{"prompt": "Say hello to the evaluator in one sentence."}' --stream
cd ../..

What's happening. agents/00_hello/waxell.yaml is a whole agent: a framework, a model, a prompt and a signal. A signal is a named trigger; firing it starts a run. wax push registers the manifest. signals fire posts the payload, gets back a run_id, and --stream follows the run's events live.

Check. You see Registered 1/1 artifact(s), a 202 carrying a run_id, then a governance evaluation, the model call and the sentence. wax -p eval runs lists the run as success. Open it in the app to see the span tree, tokens, cost and every policy evaluation.

Push is idempotent. Pushing the same content again does nothing. Changing the content without bumping version: returns a 409 that lists what changed. This is the same shape wax init scaffolds.

Step 2 — The same thing over HTTP

export WAXELL_API_URL="$(wax -p eval axid show --format json | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwks_url"].split("/api/")[0])')"
curl -sS -X POST "$WAXELL_API_URL/api/v1/signals/hello_run/" \
  -H "X-Wax-Key: <the API key from step 0>" -H "Content-Type: application/json" \
  -d '{"prompt": "Reply with exactly: READY"}' | python3 -m json.tool

What's happening. The CLI is a thin wrapper. Any system that can send a POST can start a run: a webhook, a cron job, a queue consumer. The 202 response says where to look: events_url returns the event list, and adding /stream/ gives server-sent events. The full contract is in the signal HTTP reference.

Check. The response is {"status": "matched", "run_id": "…", "events_url": "/api/v1/runs/<id>/events/", …}. Send the same body twice with an X-Idempotency-Key header and the second response is "status": "duplicate", pointing at the first run.

Step 3 — Your system of record: domain server and tunnel

cp env.example .env                      # set EVAL_DOMAIN_SHARED_SECRET to any long random string
wax -p eval axid show --format json      # copy the API host and tenant id from jwks_url into .env
docker compose up -d --build
open http://localhost:8030/              # xdg-open on Linux
./scripts/tunnel_url.sh --write
./domains/push_domains.sh eval

In a second terminal, also from the repository root:

./scripts/tunnel_keepalive.sh eval

Back in the first terminal:

./scripts/push_all.sh eval
wax -p eval signals fire team_seed --payload '{}' --wait

What's happening. A domain is how an agent reaches your systems without holding credentials for them. domains/things.yaml is the contract: its actions (get, list, set_review, add_note, publish) become tools named things__get, things__set_review and so on. Every call is an HTTP POST from Waxell to your endpoint. Here the endpoint is domain_server/, a FastAPI and SQLite stand-in for your system of record. The contract sets per_user_auth: true, so every call also carries a signed statement of who the run acts for (step 7).

Waxell's cloud has to reach your laptop. cloudflared opens a quick tunnel, which needs no account. tunnel_url.sh --write puts the public hostname into .env, and push_domains.sh registers the contract with that URL.

The tenant id and API host in .env let the domain server find your workspace's public keys, so it can verify identity headers offline. Both are in jwks_url, which has the form https://<API host>/api/v1/tenants/<tenant id>/.well-known/….

push_all.sh pushes all seven agents. The two code agents, reviewer_langgraph and reviewer_pydantic_ai, start their image builds now, so the images are ready when you reach step 6. team_seed writes the read-only team graph that every reviewer consults.

Check.

• The live page at localhost:8030 shows server armed · verifying.
• push_domains.sh prints ==> actions: get, list, set_review, add_note, publish, then ==> per_user_auth_enabled: True, then Done..
• The agent pushes print container image building for the two code agents.
• team_seed ends with graph.remember_batch · +7 entities, +4 relationships.

The tunnel is recycled about every 25 minutes

Cloudflare recycles quick tunnels after roughly 25 minutes, which is shorter than this walkthrough. When that happens, cloudflared keeps retrying the dead hostname, and your agents report Name or service not known. The keepalive script probes the public URL every 20 seconds. When the URL stops answering, the script restarts the tunnel, writes the new hostname into .env and re-pushes the domain. Recovery takes 60 to 90 seconds. To skip the quick tunnel entirely, point EVAL_DOMAIN_BASE_URL at a public endpoint or a named tunnel of your own.

Step 4 — A reviewer that calls your system

wax -p eval signals fire review_claude --payload '{"thing_id": "thing-001"}' --stream
curl -s localhost:8030/api/calls | python3 -m json.tool | head -60

What's happening. reviewer_claude runs the real Claude Agent SDK in a container. Its prompt (agents/reviewer_prompt.md, shared by all the declarative reviewers) tells it to remember first, review, then record. You never call a memory API; the manifest declares memory slots and the runtime turns them into tools.

Check. The stream shows:

1. Recalls: memory.recall, memory.search, and graph.recall · 1 in team, which finds the owner, Grace Hopper.
2. Domain calls: things__get, things__set_review and things__add_note, with the note addressed to Grace. Each call is preceded by a governance evaluation.
3. Records: memory.remember into recent_reviews and style_rules, and a graph.remember_batch into things.

On the live page a row appears for each call, labelled by reviewer_claude with a run ↗ link. The thing-001 card fills with the review.

Step 5 — Five frameworks, one domain

./scripts/fire_all.sh eval
python3 scripts/verify.py --profile eval

What's happening. Every reviewer gets the same thing to review. Then verify.py joins each run's status and cost with the domain server's call log.

Check. It prints one row per agent. The three declarative reviewers each show three domain calls, and identity_verified 3/3: every call carried an end-user token that your server verified against your workspace's JWKS. The two code agents show isolated with zero domain calls. Their source never calls the domain, and step 6 explains why that matters. A code agent showing shared (fallback: …) means its image is still building.

Open reviewer_claude's new run. Its first event, memory.recall · 1 in recent_reviews, finds the step 4 review. Memory carries across runs with no code from you. Framework support lists exactly what each framework honours.

Step 6 — Your own code, in its own container

wax -p eval agents builds reviewer_langgraph
wax -p eval signals fire review_langgraph --payload '{"thing_id": "thing-001", "title": "Release notes 2.4", "body": "Added export to CSV, fixed timezone bug in reports, deprecated the v1 API."}' --stream

What's happening. agents/reviewer_langgraph/src/graph.py is an ordinary compiled LangGraph StateGraph, and it imports nothing from Waxell. In its manifest, source: and entrypoint: graph:app are the whole integration. On push, the runtime builds an image with your code and dependencies. On each fire, it runs that image in a container of its own. The container gets an allowlisted environment: no platform credentials, and model access only through the metered proxy.

Check. The build shows success. The run's events read Running graph:app from the agent's image, then Entrypoint resolved to a langgraph, then the proxied model call with tokens and cost, then the review JSON. If you fire before the build finishes, the run waits (Waiting for this agent's custom image…) and starts by itself.

To see your own change run, edit graph.py, bump version:, and push again.

Step 7 — Identity: who the person is, and what acted for them

wax -p eval signals fire review_autogen --payload '{"thing_id": "thing-002", "_sub_user_identity": {"sub_user_id": "u_42", "email": "ada@example.com", "roles": ["reviewer"]}}' --stream
curl -s localhost:8030/api/calls/latest | python3 -m json.tool

What's happening. _sub_user_identity says the run acts for a person. From then on, every domain call carries two signed headers. Your server verifies both offline against public JWKS, with no shared secret:

Header	Says	Verify against
X-Waxell-Sub-User-Token	who the person is (sub, email, roles, aud = your domain)	…/.well-known/waxell-sub-user-jwks.json
X-Waxell-AXID	what acted: agent, version, run, the person (act), and this exact call's call.id	…/.well-known/axid-jwks.json

The AXID is minted fresh for every call and expires after 60 seconds. Your server can store its sub on the row it changes, and later walk that row back to the person, the agent and the run.

Check. Each new row on the live page carries ✓ end user u_42 and ✓ AXID …. The header pill's verified count matches its call count; the pill counts every call so far, not just this run's. The Acting for panel decodes both headers.

Verify a header yourself, with Node and no dependencies:

cd identity/node && node --test && cd ../..
set -a; . ./.env; set +a
curl -s localhost:8030/api/calls/latest | python3 -c 'import sys,json; print(json.load(sys.stdin)["raw_headers"]["x-waxell-sub-user-token"])' > /tmp/su.txt
node identity/node/cli.mjs sub-user @/tmp/su.txt --audience things
curl -s localhost:8030/api/calls/latest | python3 -c 'import sys,json; print(json.load(sys.stdin)["raw_headers"]["x-waxell-axid"])' > /tmp/axid.txt
node identity/node/cli.mjs axid @/tmp/axid.txt --domain things --action add_note
rm -f /tmp/su.txt /tmp/axid.txt
wax -p eval axid trace axid:<hex from the call>

Both verifiers print "valid": true with the claims. axid trace lists the person, the agent and version, the run, every domain call that presented this identity, the lineage edge, and the person's activity. Run the verifiers soon after the call; the tokens expire within minutes.

Step 8 — Governance: block an action before it happens

wax -p eval policies push policies/block_publish.py --dry-run
wax -p eval policies push policies/block_publish.py
./scripts/reset.sh
wax -p eval signals fire review_autogen --payload '{"thing_id": "thing-004"}' --stream
curl -s 'localhost:8030/api/calls?limit=20' | python3 -c 'import sys,json; print([c["action"] for c in json.load(sys.stdin)["calls"]])'

What's happening. policies/block_publish.py is a policy written as code. It says the three reviewers may never call things/publish. thing-004 is a clean notice, so a careful reviewer recommends publishing it. The policy is evaluated inside the runtime, before every tool call.

Check. The stream shows [governance] eval_block_publish allowing get, set_review and add_note. When the model calls things__publish, the policy blocks it, and the run ends policy_blocked_domain. Your server's call list is ['add_note', 'set_review', 'get'], with no publish: your endpoint was never asked. The thing-004 card on the live page shows a red blocked in the runtime strip.

The policy stays on

The policy remains in force until you turn it off, and step 11 needs it off. A blocked run ends at the block, before the reviewer records its memory. policies/unblock_publish.py pushes the same policy with enabled=False.

Step 9 — Evaluations

wax -p eval eval bootstrap --pack quality --agent reviewer_claude --judge-model claude-haiku
wax -p eval test scenarios/ --list
wax -p eval test scenarios/ --tag smoke

What's happening. eval bootstrap creates three LLM-judged evaluators (helpfulness, relevance, format compliance) and three matching guardrails. The guardrails start staged: they score without acting until you review the counterfactual and re-run with --arm. --judge-model claude-haiku uses the Anthropic key you stored in step 0. The default judge, gpt-4o-mini, needs an OPENAI_API_KEY workspace secret instead.

wax test runs the scenarios in scenarios/eval_path.py. Each one fires a real signal through the API, waits for the run, and asserts on the trace.

Check. Bootstrap prints Judge: Claude Haiku — key present and creates three evaluators and three staged guardrails. The smoke scenario reports hello_answers … PASS, writes a report under /tmp/waxell-eval/, and uploads the run to the workspace's Evaluations page. Drop --tag smoke to run the reviewer scenarios too; they take a few minutes.

Step 10 — Evidence

wax -p eval compliance attest --period last_7_days -o attestation.json
wax -p eval compliance verify attestation.json
wax -p eval compliance pack --framework nist_ai_rmf --period last_7_days
wax -p eval compliance verify waxell-pack-nist_ai_rmf-last_7_days.zip

What's happening. Everything you just did is recorded. The attestation is a signed summary of the period. The pack is a zip of what an auditor asks for: policies, events, framework controls, endpoint and gateway decisions, and a redaction proof, plus a manifest carrying the same hash and signature.

Check. verify prints matching SHA-256 hashes and Signer: att_… (EdDSA), the signature checked against your workspace's attestation JWKS. Valid pack frameworks are owasp_llm, nist_ai_rmf, iso_42001, eu_ai_act, hipaa, gdpr and sox.

Step 11 — Memory: four tiers, two graphs

Every declarative reviewer declares four memory slots in its waxell.yaml:

Slot	Tier	What the reviewer keeps there
recent_reviews	episodic, 30-day list	thing, recommendation, reason — newest first
style_rules	semantic	rules it learned, found later by meaning
things	graph, read_write	thing → reviewed_by → person, thing → cites → thing
team	graph, read	who owns what — written by team_seed

The style_rules slot needs no key. By default, the runtime embeds facts with the bundled open-source BAAI/bge-small-en-v1.5 model, so your facts never leave Waxell. To use your own embedding provider, run wax -p eval provider add openai_main --from-catalog openai, or add ollama, vllm or azure_openai, and pin the provider on the slot. Each fact records the model that embedded it, so search never compares vectors from two models. Graph memory covers grants and provenance.

wax -p eval policies push policies/unblock_publish.py
wax -p eval signals fire review_claude --payload '{"thing_id": "thing-004", "_sub_user_identity": {"sub_user_id": "u_42", "email": "ada@example.com", "roles": ["reviewer"]}}' --stream
wax -p eval memory show reviewer_claude --slot recent_reviews
wax -p eval memory search reviewer_claude --slot style_rules --query "what makes a deprecation notice complete"
wax -p eval graph show things
wax -p eval graph show team

Why reviewer_claude. Semantic facts belong to the agent that learned them. reviewer_claude saved the deprecation rule while reviewing thing-001 in step 4. Thing-004 cites thing-001, so this run shows memory changing the review.

Check.

• The search finds the deprecation rule, and the review of thing-004 cites it.
• graph.recall names Grace Hopper as owner.
• The batch write records thing-004 cites thing-001 and reviewed_by Ada.
• memory show reports embeds with bundled local model · BAAI/bge-small-en-v1.5 · 384 dims.
• graph show team is unchanged, because the reviewers hold only read on it.

Change the team slot to graph_access: read_write and push, and the next run can write that graph. The grant is the control, per agent and per graph.

When something goes wrong

Symptom	Fix
wax push says feature_gated	The workspace lacks the Runtime product; ask us.
SecretNotConfiguredError / No ANTHROPIC_API_KEY	wax -p eval secrets set ANTHROPIC_API_KEY …
/api/calls empty, or Name or service not known	Tunnel recycled: keep tunnel_keepalive.sh running, or ./scripts/tunnel_url.sh --write && ./domains/push_domains.sh eval.
"identity_verified": false mentioning WAXELL_TENANT_ID	Fill the API host and tenant id in .env from wax -p eval axid show --format json, then docker compose up -d.
Code agent stays pending or ran shared (fallback: …)	The image is still building: wax -p eval agents builds <name>.
Step 11 stops at things__publish with no memory writes	Step 8's policy is still on: wax -p eval policies push policies/unblock_publish.py.
Push returns 409	Same version:, different content; bump version:.

Start over

./scripts/reset.sh                 # domain server: four seeded things, empty call log
./scripts/reset.sh eval --memory   # also every reviewer's memory and both graphs
wax -p eval signals fire team_seed --payload '{}' --wait

Clearing memory is an operator action. No agent can wipe a graph; the reviewers hold read and write grants, never delete.

Next

• Your first agent with wax init: start your own project.
• waxell.yaml reference: every manifest field.
• Domains: connect your real systems.
• Operating agents: runs, traces and day-two work.