Hunter-Seeker Docs For agents This page as markdown

Worked example — an agent platform that improves itself

The same 800 rows, answered twice. Once it refuses; once it clears. The difference is one reading, and the contrast is the whole lesson.

You run an agent platform. Runs fail. You have the traces, and you would like to know which runs are about to fail before they do, so a supervisor can route around whatever is degrading.

The obvious move — feed the trace table to a model — does not work, and this page starts by doing it anyway, because watching it refuse is more useful than being told it would.

Everything here is runnable. Every block tagged hs:run is a literal tool input against sample:agent_traces, a free synthetic sample: 800 labelled runs, columns run_id, ts, agent, task, tool, duration_ms, input_tokens, failed. No account, no upload, no quota. $name means "the value called name in the response before it".

1 · Ask the wrong question, on purpose

{
  "tool": "hs_rank_topk",
  "arguments": {
    "data": { "dataset_id": "sample:agent_traces" },
    "entity_column": "run_id",
    "outcome_column": "failed",
    "subject_kind": "event",
    "outcome_is_desirable": false,
    "page": { "k": 20 }
  }
}

Result: an honest non-finding. The engine reports too few viable arms to build a pattern from — agent, task and tool are three categoricals and a couple of unrelated numerics, and there is no combination in there that clears the bar.

This is a terminal success, not an error. The run is refunded. Do not retry it: identical inputs return byte-identical envelopes, so the second call costs time to learn nothing. See Refusal is a result.

Record it. In the event bus this is a run.honest_empty: the supervisor does not gate on failure risk this cycle, and the reason is written down. Negative knowledge is knowledge; most pipelines throw it away and rediscover it monthly.

Why it refused, and why that is correct

The features you handed it are the run's identity, not its history. Whether a run fails depends on how that agent, that task and that tool have been doing lately — and none of that is in a row about one run.

The tempting fix is to compute failure rates per agent and join them in. Do not. A rate computed over the whole stream includes the run you are predicting, and the runs after it. That is target leakage, and it produces a beautiful model that cannot work in production, where the future is not available. The leak guard catches a column that predicts the outcome almost perfectly; it cannot catch a rate that is subtly contaminated by its own row.

2 · Ask the right question — the same 800 rows, reduced

trace@1 computes per-group prior outcome rates over strictly earlier runs only. Each row's cutoff is its own start minus one microsecond, so a run never sees itself, never sees the future, and never sees a tie at the same instant. The catalogue hands you the exact call:

{
  "tool": "hs_rank_topk",
  "arguments": {
    "data": { "dataset_id": "sample:agent_traces" },
    "entity_column": "run_id",
    "outcome_column": "failed",
    "subject_kind": "event",
    "outcome_is_desirable": false,
    "page": { "k": 20 },
    "reading": {
      "kind": "trace",
      "version": 1,
      "roles": {
        "identifier": "run_id",
        "time_axis": "ts",
        "outcome": "failed",
        "agent": "agent",
        "task": "task",
        "tool": "tool"
      }
    }
  }
}

The reduce makes this a dataset run, so it comes back {status: "pending", task_id}:

{ "tool": "hs_poll_task", "arguments": { "task_id": "$task_id" } }

It clears. Lift 4.1026, top-decile lift 4.285714, holdout {matched: 16, confirmed: 9}, no leakage warnings. The reduce report names the groups it bound: ["agent", "task", "tool"].

The same 800 rows. The same outcome column. The only thing that changed is that the features are now prior-only — and the engine computed them, so nothing above it had to be trusted to get the cutoff right.

Still free: a sample carrying a reading takes no reserve either.

What the engine found

{ "tool": "hs_explain_drivers", "arguments": { "ranking_ref": "$ranking_ref" } }

The discovered quartet includes tool_prior_outcome_rate — the engine's own leak-safe prior-failure feature, the exact thing you were tempted to hand-roll in §1, computed correctly.

Read the conditions as ONE joint profile, never as independent effects. Phrase them as a single sentence: "runs where this tool's recent failure rate is elevated AND this agent/task pairing is one of these, together, fail at N× the base rate." The N is pattern.lift, out of the response.

Build the filter from operator and missing_values, never from the direction word. On a numeric condition, "lower" means operator: "<=" with missing_values: "included" — it holds at the threshold and covers rows where the value is missing. Reading "lower than t" as < t selects a smaller, different cohort than the one coverage reports, and on a threshold that lands on a common value it can select nothing at all while the envelope still looks healthy. On a categorical condition both fields are absent and category_match is is_one_of or is_not_one_of — take the negation literally; dropping it inverts the cohort.

The sample has a mid-window tool degradation planted in it (one tool starts failing partway through the window) and a stable joint pattern (one agent/task pairing). The prior-rate features are what let the engine see the first one at all: a whole-stream rate would average the good period and the bad one into a single number that describes neither.

3 · Gate a change on the band

You now have a model_ref. Score one run at the moment you would act on it:

{
  "tool": "hs_score_entity",
  "arguments": {
    "model_ref": "$model_ref",
    "entity_id": "$a_run_id_from_step_2",
    "row": {
      "agent": "planner",
      "task": "refactor",
      "tool": "browser",
      "duration_ms": 41000,
      "input_tokens": 8200
    },
    "subject_kind": "event"
  }
}

You get a band, a max_autonomy ceiling, up to four principal_reasons, and a signed verdict.

act — the engine certifies this run is likely to fail. Route it: different tool, smaller task, supervisor in the loop.
escalate — uncertain. Put a human on it and record what they decide; that answer is a label, and it is the cheapest one you will ever get.
refuse — the engine will not vouch. Costs nothing. Do not act on a refusal as though it were an act with lower confidence; it is the absence of a decision.

The band says the run is likely to fail. That it is bad is your judgement, from verdict.outcome.polarity — adverse, here. The engine does not decide good from bad.

{ "tool": "hs_verify_verdict", "arguments": { "verdict": "$verdict", "signature": "$signature" } }

status: "valid". Free, and credential-free over REST — so the platform team, the customer, and an auditor can all check the same decision without an account.

4 · Attest, report, and find out whether the change worked

Randomise before you act. Split the act cohort by a seeded coin flip and let 10–20% of it run un-gated. Without that split the gated runs were selected by the model's own score, and any later difference is observational. The splitter and the reasoning are on the control-arm page.

Attest the runs you actually intervened on:

{ "tool": "hs_attest_action",
  "arguments": { "model_ref": "$model_ref", "entity_id": "$run_id",
                 "lever_token": "$lever_token", "post_value": 0,
                 "acted_at": "2026-09-03T10:00:00Z" } }

lever_token comes from hs_explain_levers on the same ranking. Only its hash is stored.

Report the outcome for both arms, gated and control:

{ "tool": "hs_report_outcome",
  "arguments": { "model_ref": "$model_ref",
                 "outcomes": [ { "entity_id": "$run_id", "outcome": 0,
                                 "observed_at": "2026-09-03", "event_id": "run-88213" } ] } }

Append-only, and idempotent on event_id — send the same call twice and there is one row, which is what makes a retried collector safe. Reporting never retrains anything.

Ask whether it worked:

{ "tool": "hs_action_evidence", "arguments": { "model_ref": "$model_ref" } }

Today: live: null, with live_floor: {n_min: 30, small_n_below: 100} beside it. Nobody has reported an outcome against this model, so there is nothing to compare — and the tool says so rather than dividing two small numbers. [live: S3]

5 · Notice when the world moves

The planted degradation is the point of the sample: the tool that was fine on day 1 is not fine on day 20. A model fitted before that is wrong afterwards, and nothing about its own fit statistics will tell you.

On the next refresh, declare it a refresh:

{ "tool": "hs_rank_topk",
  "arguments": { "data": { "dataset_id": "$next_window" }, "entity_column": "run_id",
                 "outcome_column": "failed", "subject_kind": "event",
                 "outcome_is_desirable": false, "refit_of": "$previous_model_ref",
                 "reading": { "kind": "trace", "version": 1, "roles": { "identifier": "run_id", "time_axis": "ts", "outcome": "failed", "agent": "agent", "task": "task", "tool": "tool" } } } }

An unknown, expired or foreign refit_of is refused before the run starts, and nothing is billed. Then:

{ "tool": "hs_drift_status", "arguments": { "model_ref": "$new_model_ref" } }

You get pattern, pattern_diff (conditions added or removed, thresholds moved and which way) and a recommendation. Labels and directions only — never a coefficient. React to the recommendation: on refit, retire the routing rules built on the old pattern and adopt the new pattern.conditions; on abandon — two consecutive honest-empties — stop gating on this outcome until the data changes.

6 · The loop, as an event table

Event	Source	What the platform does
`run.honest_empty`	§1, the raw table	do not gate this cycle; record the negative finding; do not retry the same data
`run.completed` (cleared)	§2, the reduced table	publish the new `model_ref`; refresh routing rules from `pattern.conditions`
`escalate` band	§3	supervisor decides; record the answer as a label
`evidence.updated`	§4	keep or drop the gate; report the difference with its interval
`verdict.drift` → `refit`	§5	freeze the old routing rules; refit
`verdict.drift` → `abandon`	two consecutive empties	stop gating; escalate to a human

7 · What generalises

Any labelled event stream with a binary outcome and a timestamp is this example: CI jobs (failed), support tickets (escalated), deliveries (late), API calls (errored). What you must not do is compute the history features yourself over the whole stream — that is the §1 mistake, and it is invisible until production.

The constraints are the engine's, quoted from hs_describe_capabilities rather than typed:

Input. tabular rows + outcome_column (binary, historically known) + entity_column + subject_kind

Problem shape. rank any list of entities by likelihood of a yes/no outcome; optional counterfactual levers per ranked entity

Not for. predicting a continuous number; data with no historical outcome column; pure time-series forecasting; causal guarantees.

And the four guarantees the same response publishes, verbatim:

determinism — identical inputs return byte-identical envelopes
provenance — every response carries engine_version + core_hash
leak_guard — LIVE (engine 0.1.1): any column whose values predict the outcome almost perfectly (univariate AUC > 0.98 on datasets of ~200+ rows) is quarantined as likely target leakage, dropped from the analysis, and surfaced with a plain-English reason. On a cleared finding leak_guard lists the quarantined columns as { column, reason } entries (empty array when none were flagged); it is null ONLY when there is no finding (honest-null), not because the feature is unbuilt. The lift >= 1.5 gate does NOT catch leakage (leaky features make lift look excellent), so you must still verify the outcome column and every feature are historically known BEFORE the outcome; do not treat a null leak_guard as a clean bill of health.
honest_empty — below a lift >= 1.5 bar measured on every analyzed row (none held back) you get {result:"none", reasons} - never weak rankings

Two bounds the engine applies are deliberately absent from this list: the minimum row count and the workable outcome-rate band. Neither is published on any wire surface, and a number that cannot be read off a response is a number this page must not print.