Config Health: Presence Isn't Validity

The most expensive failures in an agent platform are the silent ones: a credential that is set but no longer valid, a secret key that was never provisioned so a feature quietly no-ops, a model id that a provider deprecated last month. Nothing crashes. Traces just get thinner, summaries stop appearing, and the cause is invisible until someone digs.

Three of these landed on the same day while dogfooding the Waxell fleet:

What was wrong	What it looked like
WAXELL_FLEET_SECRET_KEY never provisioned	The auto-provisioned admin token silently failed to store; every internal-plane agent degraded with "no token"
GROQ_API_KEY set but returning 401	The AI narrator "flashed, then nothing" — for weeks, because the UI hid the error
Fireworks llama-v3p1-70b deprecated	404 Model not found at call time; the key was fine

None of them were detectable by checking whether the variable existed. All of them were detectable by making one real call.

The doctor

The controlplane doctor runs that real call for every configured integration and reports ok / fail / skip with the cause and a concrete fix. It lives on the fleet's Mission Control page as a Doctor card, and as a management command:

scripts/prod-governance.sh observe_fleet_doctor            # human table
scripts/prod-governance.sh observe_fleet_doctor --fail-only
scripts/prod-governance.sh observe_fleet_doctor --json     # raw report

It exits non-zero when anything fails, so it works as a gate in a deploy or a scheduled check.

What it verifies

Check	The real call
Fleet secret key	Encrypt + decrypt a probe with WAXELL_FLEET_SECRET_KEY
Platform read key	Provision/resolve the scoped admin token, then GET internal/observability/health/ with it
Tenant observe key	GET v1/observe/model-costs/ with WAXELL_API_KEY
Lineage + narrator surface	WAXELL_LINEAGE_V1=1
AI narrator	Groq /models with GROQ_API_KEY — and the configured narrator model must be in the list
Slack fleet bot	auth.test with the bot token
Each LLM provider the fleet uses	The provider's model-catalog endpoint with its key (openai, anthropic, gemini, mistral, cohere, fireworks, together, openrouter, deepseek)
Every agent's pinned model	The model id must appear in its provider's live catalog — a deprecation is caught before it 404s

The catalog probe is what turns "the key is set" into "the key works": a rejected key returns no models. The per-agent model check is what turns a surprise 404 into a red line with a suggestion (kimi-k3 NOT in fireworks catalog; similar: …).

Reading a result

[OK  ] fleet secret key (WAXELL_FLEET_SECRET_KEY)  —  encrypt/decrypt round-trip OK  (1ms)
[FAIL] AI narrator (GROQ_API_KEY + model)  —  Groq key rejected (no models returned)  (412ms)
        fix: Rotate groq_api_key in the fleet secret (validate with GET /openai/v1/models first) …
[skip] provider: bedrock  —  no standalone key (task role / routes to other providers)  (0ms)
[FAIL] model: A-09 ops.sre.otlp-debugger → fireworks/llama-v3p1-70b  —  NOT in fireworks catalog; similar: llama-v3p3-70b  (203ms)
        fix: Update A-09's stack['model'] to a current fireworks model id.

skip is honest, not hidden: it means the doctor can't probe that integration (no standalone key, no catalog endpoint), so you know exactly where your blind spots are.

The habit this replaces

Before wiring any key, run the same probe the doctor runs. For an OpenAI-compatible provider that is one line:

curl -s -o /dev/null -w '%{http_code}\n' https://api.groq.com/openai/v1/models \
  -H "Authorization: Bearer $KEY"        # 200 = valid, 401 = dead

The provisioning scripts in this repo do exactly that and refuse to store a key that fails it. The doctor makes the check continuous instead of one-time — because keys get rotated, models get deprecated, and the variable stays set through all of it.