Managing errors

The Error Explorer (Observe → Errors) is a managed feed, not a log. Every error the platform sees becomes part of one error group, and each group carries the state a team needs to actually work it.

What lands in the feed

Source	Example
Run failures	an agent raised or timed out
Policy violations	PolicyViolation: Citation count 0 below required minimum 1.
Failed tool and model calls	Tool call failed: llm.openai — AuthenticationError: 401 Incorrect API key
Degraded steps	degraded: read:HTTP 403 api/v1/mcp-gateway/red-team/runs/
Incidents	kill-switch, budget, rate-limit events

Failed tool and model calls and degraded steps used to appear only in Logs. Now they are grouped with everything else.

Grouping

Groups are keyed by agent + kind + error type + normalized message. Normalization strips the volatile parts — ids, keys, numbers, quoted values — so every Incorrect API key provided: sk-proj-… is one group, not one per key prefix, and run 8b05…failed after 120s and run 1599…failed after 95s are the same group.

Each group shows occurrences, first and last seen, the agents affected with counts, and links to sample runs and traces.

States

State	Meaning
open	new, nobody has looked at it
acknowledged	someone owns it
resolved	fixed; if the same fingerprint comes back the group becomes regressed and moves to the top
muted	hidden for N days (default 7); reopens automatically when the mute expires and it recurs
regressed	a resolved error came back — whatever fixed it did not hold

Acknowledge vs resolve. Acknowledge means "someone owns this": the group stays in the active list, keeps counting new occurrences, and nothing is sent anywhere. Resolve means "this is fixed": the group leaves the active list, error_group.resolved is raised on your webhook, and the next occurrence of the same fingerprint flips it to regressed at the top of the list. Resolve when you have shipped a fix; acknowledge when you have not.

Notes and history. Acknowledging, resolving or muting asks for a short note (why, or what you changed); acknowledging also takes an assignee. Every change is appended to the group's history, visible in the expanded card, with who, when and the note. The latest note also shows in the header.

Groups quiet for seven days resolve themselves. Use the Status filter on the explorer (active shows open + acknowledged + regressed). The groups list is paged, regressed first; load more fetches the next page, and the counters at the top cover the whole filtered set.

Configuration gaps are surfaced under Insights → Alerts like every other silent hole: open groups with neither ticketing nor a webhook, ticketing that is started but cannot file, and tickets attributed to the agent instead of the person because the platform issuer is not wired.

Webhook: one integration for every tracker

Rather than one connector per ticketing tool, the explorer sends one signed POST any system can consume — Zapier, n8n, a Lambda, or a Jira / Linear / GitHub adapter you own. Configure it at the bottom of the Error Explorer (HTTPS URL + a signing secret).

Events:

error_group.opened            a new group
error_group.regressed         a resolved group came back
error_group.resolved          resolved by a person or the 7-day sweep
error_group.threshold_crossed occurrences passed your threshold (default 25)
error_group.ticket_requested  someone clicked "file ticket"

Headers and signature:

X-Waxell-Event: error_group.regressed
X-Waxell-Delivery: <uuid>
X-Waxell-Timestamp: <unix seconds>
X-Waxell-Signature: sha256=<hex HMAC-SHA256 of "<timestamp>.<raw body>" with your secret>

Verify in a few lines (Python):

import hmac, hashlib

def verify(secret: str, headers: dict, body: bytes) -> bool:
    ts = headers["X-Waxell-Timestamp"]
    expected = "sha256=" + hmac.new(secret.encode(), f"{ts}.".encode() + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, headers["X-Waxell-Signature"])

Payload:

{
  "event": "error_group.regressed",
  "group": {
    "id": 42, "fingerprint": "…", "kind": "llm", "status": "regressed",
    "title": "AuthenticationError: llm.openai: Incorrect API key provided",
    "agent_name": "ops.sre.daily-digest", "agents": {"ops.sre.daily-digest": 12},
    "count": 12, "first_seen": "…", "last_seen": "…",
    "sample_run_ids": ["1477"], "sample_trace_ids": ["…"],
    "assignee": "", "note": "", "ticket_url": "", "ticket_ref": ""
  },
  "at": "2026-09-13T12:40:51Z"
}

Deliveries retry with backoff on network errors and 5xx; a 4xx is recorded and not retried. Every attempt is listed under Recent deliveries on the webhook card with its status, HTTP code and response excerpt, and any delivery can be replayed.

Security

• HTTPS only, and the URL is checked against private, loopback, link-local and cloud-metadata ranges when it is saved and again before every delivery (DNS can change).
• The signing secret is encrypted at rest with your workspace's encryption key and is write-only in the API (the card only shows whether one is set).
• Only workspace owners and admins can create, change or remove the webhook or replay a delivery. Every change, resolve, mute and ticket request is written to the workspace audit log.
• Deliveries are capped at 60 per hook per hour, so an error storm cannot turn into thousands of POSTs (only transitions fire, never individual occurrences).
• Events are kept for your plan's retention window; groups keep their counts and samples regardless.

Filing a ticket

File ticket on a group works one of two ways.

Through the Waxell MCP gateway (recommended)

Open Settings → Ticketing. It is one screen, in order, and every step reports what the gateway said:

1. Tracker — pick Linear, GitHub or Jira. The card shows whether the tracker is registered in your MCP gateway and, for per-user trackers, whether it is connected for you. Connect opens the tracker's authorization in a new tab.
2. Agent identity — one click creates the error-explorer agent and mints its agent-linked key on the server. The key is stored encrypted; nobody sees or pastes it. Rotate key retires the previous one.
3. Team / repository / project — chosen from a list fetched through the gateway as you (or typed, if the lookup is not allowed). Extra arguments such as labels or a priority are merged into every issue.
4. Who can file — a toggle per workspace member that grants or revokes their delegation for error-explorer → tracker. Each person on a per-user tracker also connects it for their own account.
5. Test — files a real issue titled Waxell ticketing test — safe to close with exactly the settings above, as you, and shows the disposition and the link. If the tracker answers without an issue key, the result is unconfirmed and the tracker's own text is shown, so a rejected label or a wrong team name is visible instead of a false "filed".

Each click on file ticket then sends call_external_tool to the gateway with the agent key in Authorization and the clicker's token in X-Wax-Caller-JWT, so the gateway's delegation gate decides for that person and the tracker's audit trail names them. The disposition comes back verbatim:

Disposition	What you see
allowed	the issue link on the group, ticket_filed in the audit log
unconfirmed	202 — the gateway let the call through but the tracker answered without an issue key; its text is shown, nothing is written to the group
delegation_denied	403 — that person's grant is missing or revoked; nothing is filed
auth_required	409 — the tracker isn't connected for that person yet
confirm_required	409 — a gateway policy wants a human confirmation first
error / unreachable	502 — the tracker or gateway failed; the reason is in the response

Trackers without a template go through the gateway's use_tool with the intent create an issue, which lets the gateway pick the right tool for whatever you have connected.

Through your webhook

Without a gateway configuration, the click marks the group as requested and raises error_group.ticket_requested on your webhook, so your adapter creates the issue and writes the link back with PATCH /api/v1/observability/errors/groups/<id>/ {"ticket_url": …, "ticket_ref": …}.

Two views, one filter bar

• Groups — one card per fingerprint, with the management actions.
• All events — every occurrence in time order: when, kind, agent, error type, message, links to the run, the trace and the group. Click events → on any group to see just its occurrences; load more pages back through history.

Both views share the same filters, and the filters live in the URL so a view is shareable:

Filter	What it does
Search	matches title, message, error type or agent (substring, case-insensitive)
Kind	run · policy · tool · llm · degrade · incident (multi-select)
Status	active (open + acknowledged + regressed), or one state, or all
Agent	one agent
Assignee	a person, or unassigned
Min occurrences	hide one-offs
Regressed only	only errors that came back after being resolved
Sort	last seen · occurrences · first seen (regressed always float to the top of last seen)
Time range	1 h … 30 d

Triage: the fast way through the feed

Observe → Errors → Triage turns the active groups into a deck, one card at a time, in the order worth working:

State	Meaning	Default action
recovered	every affected agent has completed a newer run since the last occurrence	resolve
regressed	it was resolved and came back	file a ticket
config	the message reads as a key, quota, permission or deployment problem	file a ticket
live	still occurring, nothing newer to say	acknowledge

Each card carries what you need to decide in one look: the affected agents with their newest run and whether they have run clean since, the sample runs with the exact error each produced, the last occurrences, and What happened — the narrator's explanation, written from those runs and cached on the group until it recurs. A recommendation is shown with its reason; a triage agent can override it by writing meta.recommendation on the group.

Scope. By default the deck walks untriaged groups — open and regressed, nobody has touched them. Acting on a card removes it from that scope, so a reload never shows you the same card twice. Switch scope to + acknowledged, regressed only, muted, resolved or all to review what you or others already handled; every card then opens with an Already handled banner — who acknowledged, resolved or muted it, when, their note, and whether it came from a triage sweep — and recommends skip unless the evidence says it has since recovered.

Keys: ← → move · R resolve · A acknowledge · M mute a week · T file a ticket · S skip · N re-narrate · / type a note. Every action is audited and lands in the group's history with your note — or, if you leave it empty, the narration's likely cause — and is tagged via: triage so a later reader can tell a sweep from a one-off click.

From the terminal

The wax CLI reads the same endpoints as the explorer, so both show the same groups, states and history.

wax errors groups                       # paged, regressed first; --status --kind --agent -q --min-count --sort --offset
wax errors show 42 --events             # one group drilled in: message, agents, sample runs, ticket, history, last occurrences
wax errors events --group 42 --follow   # every occurrence in order; --follow tails new ones
wax errors ack 42 --note "looking" --assignee me@co.com
wax errors resolve 42 --note "key rotated"
wax errors mute 42 --days 3 --note "vendor outage"
wax errors reopen 42
wax errors ticket 42                    # through the gateway as you; prints allowed / unconfirmed / auth_required / delegation_denied
wax errors triage                       # the deck in the terminal: r/a/m/t/s/q per card; --dry-run lists the queue; --no-narrate skips the LLM
wax errors triage --status active       # scope: untriaged (default) | active (+acknowledged) | acknowledged | resolved | muted | regressed | open | all

Every command takes --format json for scripts and agents.

API

GET   /api/v1/observability/errors/events/?period=24h&q=&kind=llm,tool&status=active&agent=&assignee=&group_id=&run_id=&trace_id=&before=<iso>&limit=100
GET   /api/v1/observability/errors/summary/?period=24h&status=active|open|regressed|acknowledged|resolved|muted|all&q=&kind=&assignee=&min_count=&regressed_only=1&sort=last_seen|count|first_seen
PATCH /api/v1/observability/errors/groups/<id>/     {status, mute_days, assignee, note, ticket_url, ticket_ref}
POST  /api/v1/observability/errors/groups/<id>/file-ticket/
GET   /api/v1/observability/errors/triage/?period=7d&status=untriaged&kind=&agent=&q=  the deck's queue: recovered → regressed → config → live; status = scope
GET   /api/v1/observability/errors/groups/<id>/triage/                              one card: state, agents' latest runs, sample runs, occurrences, narration, recommendation
POST  /api/v1/observability/errors/groups/<id>/narrate/     {force?}                narrate from the evidence; cached on the group until it recurs
GET/PUT/DELETE /api/v1/observability/errors/ticketing/   {tracker, defaults, upstream?, tool?, gateway_url?, gateway_key?}
GET   /api/v1/observability/errors/ticketing/setup/          everything Settings → Ticketing shows
POST  /api/v1/observability/errors/ticketing/setup/agent/    create the error-explorer identity / rotate its key (owner/admin)
PUT   /api/v1/observability/errors/ticketing/setup/members/  {email, granted} (owner/admin)
GET   /api/v1/observability/errors/ticketing/setup/options/?tracker=linear   team / repo / project choices, via the gateway as you
POST  /api/v1/observability/errors/ticketing/setup/test/     file a test issue (owner/admin)
GET | PUT | DELETE /api/v1/observability/errors/webhook/