Hunter-Seeker Docs For agents This page as markdown

The randomized control arm

hs_action_evidence compares outcomes for entities you acted on against entities you did not. That comparison is only worth having if you decided who to act on by coin flip for part of the cohort. The acted group is selected by the model's own score, so acted-vs-not-acted differences are observational until you randomize. Withholding action on a random slice is not a caveat bolted onto this feature; it is the thing that turns the number into evidence.

This page is the how first, and the why underneath it.

How to run it

Five steps. Everything except step 2 you are already doing.

1 · Score the cohort. One model_ref from a cleared hs_rank_topk, one decision per row.

{ "tool": "hs_score_batch", "arguments": { "model_ref": "$model_ref", "entities": ["$rows"] } }

Keep the rows whose band is act. Those are the ones you would have acted on.

2 · Split that cohort by a seeded coin flip. This is the whole intervention.

import hashlib

def treated(entity_id: str, seed: str = "2026-Q3-outreach", holdout: float = 0.15) -> bool:
    """Deterministic, reproducible, and independent of anything the model saw.

    Hashing the id means the same entity lands the same way every time you re-run this — so a
    retry, a resumed job, or a second batch cannot quietly move somebody between arms. Do NOT
    seed on the score, the band, or anything else the model produced: that puts the model back
    in charge of the assignment and undoes the randomisation.
    """
    h = hashlib.sha256(f"{seed}:{entity_id}".encode()).digest()
    return int.from_bytes(h[:8], "big") / 2**64 >= holdout

Withhold action on 10–20% of the act cohort. Below about 10% the control arm is too small to reach the floors below in any reasonable time; above about 20% you are paying real money in forgone action for precision you probably do not need.

3 · Attest only for the treated half. An attestation is a record that you acted, so the untreated half gets none — that absence is what makes them a control.

{ "tool": "hs_attest_action",
  "arguments": { "model_ref": "$model_ref", "entity_id": "$entity_id",
                 "lever_token": "$lever_token", "dose": "$how_far_you_went" } }

4 · Report outcomes for BOTH halves. This is the step people skip, and skipping it is fatal: if you only report outcomes for entities you acted on, the control arm has no observed outcomes and hs_action_evidence has nothing to compare against.

{ "tool": "hs_report_outcome",
  "arguments": { "model_ref": "$model_ref", "entity_id": "$entity_id",
                 "outcome": true, "event_id": "$your_idempotency_key" } }

hs_report_outcome is append-only and never retrains. Pass an event_id you control: the same event_id twice is one row, so a retried job cannot double-count an outcome into the evidence.

5 · Read the evidence.

{ "tool": "hs_action_evidence", "arguments": { "model_ref": "$model_ref" } }

What you will see first, and why it is right

live: null. [live: S3]

That is the floor working. hs_action_evidence returns live: null until each cell has 30 reported outcomes, and flags small_n until 100. A difference computed across four entities per arm is not evidence — it is noise with a confidence interval drawn around it — and returning one would be worse than returning nothing, because a number gets acted on and a null does not.

Expect the floor to hold for a while. On a cohort of a few hundred with a 15% holdout, the control cell is the binding constraint, and it fills at the rate you actually run the play.

Why — the confounding, in the tool's own terms

hs_rank_topk ranks by likelihood. hs_score_entity bands by that same likelihood. If you then act on the act band and compare it to everything else, the two groups differ in the model's own estimate of their outcome likelihood before you did anything at all. Whatever difference you measure afterwards contains both the effect of acting and the effect of having been selected, and nothing in the arithmetic can separate them.

Randomising within the act band is what removes the selection: both arms were flagged, both were scored the same way, and the only thing that differs is a coin flip that no feature and no score could see.

What this defends against, and what it does not

Hunter-Seeker never grades its own homework in one specific sense: hs_model_quality (how well the model fits held-out data) and hs_action_evidence (whether acting changed outcomes) are separate tools over separate data, and are never merged. The model's fit statistics can never stand in for evidence that acting works. [test: packages/mcp/test/surfaces-parity.test.ts::B4: every surface lists exactly the tools the server registers]

It does not defend against selection bias. Nothing in the engine knows how you chose whom to act on, so nothing in the engine can correct for it. That correction is yours, and the randomised holdout above is the cheapest form of it.

If you cannot randomise

Sometimes you cannot withhold an action — a regulatory obligation, a safety case, a contract. Then the number is observational and you should say so out loud when you report it. The established remedies are:

Propensity scoring — model who got acted on, then compare like with like (Rosenbaum & Rubin, 1983).
Uplift modelling — model the difference the action makes rather than the outcome (Gutierrez & Gérardy, 2017).
Off-policy evaluation — reweight logged decisions to estimate a policy you did not run (Bottou et al., JMLR 2013).
Switchback designs — randomise over time windows rather than entities, when entity-level holdout is impossible (Bojinov, Simchi-Levi & Zhao, 2020).

All four are association under assumptions. The coin flip is the only one that is not.

See also: The two loops · Refusal is a result