# Worked example — an agent platform that improves itself *The same 800 rows, answered twice. Once it refuses; once it clears. The difference is one `reading`, and the contrast is the whole lesson.* You run an agent platform. Runs fail. You have the traces, and you would like to know which runs are about to fail *before* they do, so a supervisor can route around whatever is degrading. The obvious move — feed the trace table to a model — does not work, and this page starts by doing it anyway, because watching it refuse is more useful than being told it would. **Everything here is runnable.** Every block tagged `hs:run` is a literal tool input against `sample:agent_traces`, a free synthetic sample: 800 labelled runs, columns `run_id, ts, agent, task, tool, duration_ms, input_tokens, failed`. No account, no upload, no quota. `$name` means "the value called `name` in the response before it". --- ## 1 · Ask the wrong question, on purpose ```json hs:run { "tool": "hs_rank_topk", "arguments": { "data": { "dataset_id": "sample:agent_traces" }, "entity_column": "run_id", "outcome_column": "failed", "subject_kind": "event", "outcome_is_desirable": false, "page": { "k": 20 } } } ``` **Result: an honest non-finding.** The engine reports too few viable arms to build a pattern from — `agent`, `task` and `tool` are three categoricals and a couple of unrelated numerics, and there is no combination in there that clears the bar. This is a **terminal success, not an error**. The run is refunded. Do **not** retry it: identical inputs return byte-identical envelopes, so the second call costs time to learn nothing. See [Refusal is a result](/docs/refusal-is-a-result). **Record it.** In the [event bus](/docs/examples/agentic-gtm#5--the-observe-event-bus) this is a `run.honest_empty`: the supervisor does not gate on failure risk this cycle, and the reason is written down. Negative knowledge is knowledge; most pipelines throw it away and rediscover it monthly. ### Why it refused, and why that is correct The features you handed it are the run's *identity*, not its *history*. Whether a run fails depends on how that agent, that task and that tool have been doing lately — and none of that is in a row about one run. The tempting fix is to compute failure rates per agent and join them in. **Do not.** A rate computed over the whole stream includes the run you are predicting, and the runs after it. That is target leakage, and it produces a beautiful model that cannot work in production, where the future is not available. The leak guard catches a column that predicts the outcome almost perfectly; it cannot catch a rate that is subtly contaminated by its own row. --- ## 2 · Ask the right question — the same 800 rows, reduced `trace@1` computes per-group prior outcome rates **over strictly earlier runs only**. Each row's cutoff is its own start minus one microsecond, so a run never sees itself, never sees the future, and never sees a tie at the same instant. The catalogue hands you the exact call: ```json hs:run { "tool": "hs_rank_topk", "arguments": { "data": { "dataset_id": "sample:agent_traces" }, "entity_column": "run_id", "outcome_column": "failed", "subject_kind": "event", "outcome_is_desirable": false, "page": { "k": 20 }, "reading": { "kind": "trace", "version": 1, "roles": { "identifier": "run_id", "time_axis": "ts", "outcome": "failed", "agent": "agent", "task": "task", "tool": "tool" } } } } ``` The reduce makes this a dataset run, so it comes back `{status: "pending", task_id}`: ```json hs:run { "tool": "hs_poll_task", "arguments": { "task_id": "$task_id" } } ``` **It clears.** Lift 4.1026, top-decile lift 4.285714, holdout `{matched: 16, confirmed: 9}`, no leakage warnings. The reduce report names the groups it bound: `["agent", "task", "tool"]`. The same 800 rows. The same outcome column. The only thing that changed is that the features are now **prior-only** — and the engine computed them, so nothing above it had to be trusted to get the cutoff right. Still free: a sample carrying a reading takes no reserve either. ### What the engine found ```json hs:run { "tool": "hs_explain_drivers", "arguments": { "ranking_ref": "$ranking_ref" } } ``` The discovered quartet includes `tool_prior_outcome_rate` — the engine's own leak-safe prior-failure feature, the exact thing you were tempted to hand-roll in §1, computed correctly. **Read the conditions as ONE joint profile**, never as independent effects. Phrase them as a single sentence: *"runs where this tool's recent failure rate is elevated AND this agent/task pairing is one of these, together, fail at N× the base rate."* The N is `pattern.lift`, out of the response. **Build the filter from `operator` and `missing_values`, never from the direction word.** On a numeric condition, `"lower"` means `operator: "<="` with `missing_values: "included"` — it holds *at* the threshold and covers rows where the value is missing. Reading "lower than t" as `< t` selects a smaller, different cohort than the one `coverage` reports, and on a threshold that lands on a common value it can select nothing at all while the envelope still looks healthy. On a categorical condition both fields are absent and `category_match` is `is_one_of` or `is_not_one_of` — take the negation literally; dropping it inverts the cohort. The sample has a mid-window tool degradation planted in it (one tool starts failing partway through the window) and a stable joint pattern (one agent/task pairing). The prior-rate features are what let the engine see the first one at all: a whole-stream rate would average the good period and the bad one into a single number that describes neither. --- ## 3 · Gate a change on the band You now have a `model_ref`. Score one run at the moment you would act on it: ```json hs:run { "tool": "hs_score_entity", "arguments": { "model_ref": "$model_ref", "entity_id": "$a_run_id_from_step_2", "row": { "agent": "planner", "task": "refactor", "tool": "browser", "duration_ms": 41000, "input_tokens": 8200 }, "subject_kind": "event" } } ``` You get a `band`, a `max_autonomy` ceiling, up to four `principal_reasons`, and a signed `verdict`. - `act` — the engine certifies this run is likely to fail. Route it: different tool, smaller task, supervisor in the loop. - `escalate` — uncertain. Put a human on it and **record what they decide**; that answer is a label, and it is the cheapest one you will ever get. - `refuse` — the engine will not vouch. Costs nothing. Do not act on a refusal as though it were an `act` with lower confidence; it is the absence of a decision. The band says the run is *likely to fail*. That it is bad is your judgement, from `verdict.outcome.polarity` — `adverse`, here. The engine does not decide good from bad. ```json hs:run { "tool": "hs_verify_verdict", "arguments": { "verdict": "$verdict", "signature": "$signature" } } ``` `status: "valid"`. Free, and credential-free over REST — so the platform team, the customer, and an auditor can all check the same decision without an account. --- ## 4 · Attest, report, and find out whether the change worked **Randomise before you act.** Split the `act` cohort by a seeded coin flip and let 10–20% of it run un-gated. Without that split the gated runs were selected by the model's own score, and any later difference is observational. The splitter and the reasoning are on [the control-arm page](/docs/control-arm). **Attest** the runs you actually intervened on: ```json { "tool": "hs_attest_action", "arguments": { "model_ref": "$model_ref", "entity_id": "$run_id", "lever_token": "$lever_token", "post_value": 0, "acted_at": "2026-09-03T10:00:00Z" } } ``` `lever_token` comes from `hs_explain_levers` on the same ranking. Only its hash is stored. **Report** the outcome for **both** arms, gated and control: ```json { "tool": "hs_report_outcome", "arguments": { "model_ref": "$model_ref", "outcomes": [ { "entity_id": "$run_id", "outcome": 0, "observed_at": "2026-09-03", "event_id": "run-88213" } ] } } ``` Append-only, and idempotent on `event_id` — send the same call twice and there is one row, which is what makes a retried collector safe. Reporting never retrains anything. **Ask whether it worked:** ```json hs:run { "tool": "hs_action_evidence", "arguments": { "model_ref": "$model_ref" } } ``` Today: `live: null`, with `live_floor: {n_min: 30, small_n_below: 100}` beside it. Nobody has reported an outcome against this model, so there is nothing to compare — and the tool says so rather than dividing two small numbers. [live: S3] --- ## 5 · Notice when the world moves The planted degradation is the point of the sample: the tool that was fine on day 1 is not fine on day 20. A model fitted before that is wrong afterwards, and nothing about its own fit statistics will tell you. On the next refresh, declare it a refresh: ```json { "tool": "hs_rank_topk", "arguments": { "data": { "dataset_id": "$next_window" }, "entity_column": "run_id", "outcome_column": "failed", "subject_kind": "event", "outcome_is_desirable": false, "refit_of": "$previous_model_ref", "reading": { "kind": "trace", "version": 1, "roles": { "identifier": "run_id", "time_axis": "ts", "outcome": "failed", "agent": "agent", "task": "task", "tool": "tool" } } } } ``` An unknown, expired or foreign `refit_of` is refused *before* the run starts, and nothing is billed. Then: ```json { "tool": "hs_drift_status", "arguments": { "model_ref": "$new_model_ref" } } ``` You get `pattern`, `pattern_diff` (conditions added or removed, thresholds moved and which way) and a `recommendation`. Labels and directions only — never a coefficient. React to the `recommendation`: on `refit`, retire the routing rules built on the old pattern and adopt the new `pattern.conditions`; on `abandon` — two consecutive honest-empties — stop gating on this outcome until the data changes. --- ## 6 · The loop, as an event table | Event | Source | What the platform does | |---|---|---| | `run.honest_empty` | §1, the raw table | do not gate this cycle; record the negative finding; do not retry the same data | | `run.completed` (cleared) | §2, the reduced table | publish the new `model_ref`; refresh routing rules from `pattern.conditions` | | `escalate` band | §3 | supervisor decides; record the answer as a label | | `evidence.updated` | §4 | keep or drop the gate; report the difference with its interval | | `verdict.drift` → `refit` | §5 | freeze the old routing rules; refit | | `verdict.drift` → `abandon` | two consecutive empties | stop gating; escalate to a human | --- ## 7 · What generalises Any labelled event stream with a binary outcome and a timestamp is this example: CI jobs (`failed`), support tickets (`escalated`), deliveries (`late`), API calls (`errored`). What you must not do is compute the history features yourself over the whole stream — that is the §1 mistake, and it is invisible until production. The constraints are the engine's, quoted from `hs_describe_capabilities` rather than typed: **Input.** tabular rows + outcome_column (binary, historically known) + entity_column + subject_kind **Problem shape.** rank any list of entities by likelihood of a yes/no outcome; optional counterfactual levers per ranked entity **Not for.** predicting a continuous number; data with no historical outcome column; pure time-series forecasting; causal guarantees. And the four guarantees the same response publishes, verbatim: - `determinism` — identical inputs return byte-identical envelopes - `provenance` — every response carries engine_version + core_hash - `leak_guard` — LIVE (engine 0.1.1): any column whose values predict the outcome almost perfectly (univariate AUC > 0.98 on datasets of ~200+ rows) is quarantined as likely target leakage, dropped from the analysis, and surfaced with a plain-English reason. On a cleared finding leak_guard lists the quarantined columns as { column, reason } entries (empty array when none were flagged); it is null ONLY when there is no finding (honest-null), not because the feature is unbuilt. The lift >= 1.5 gate does NOT catch leakage (leaky features make lift look excellent), so you must still verify the outcome column and every feature are historically known BEFORE the outcome; do not treat a null leak_guard as a clean bill of health. - `honest_empty` — below a lift >= 1.5 bar measured on every analyzed row (none held back) you get {result:"none", reasons} - never weak rankings Two bounds the engine applies are deliberately absent from this list: the minimum row count and the workable outcome-rate band. Neither is published on any wire surface, and a number that cannot be read off a response is a number this page must not print. --- *See also:* [The agentic GTM example](/docs/examples/agentic-gtm) · [The two loops](/docs/two-loops) · [Refusal is a result](/docs/refusal-is-a-result) · [The randomized control arm](/docs/control-arm) · [Status: served vs planned](/docs/status)