# The randomized control arm `hs_action_evidence` compares outcomes for entities you acted on against entities you did not. That comparison is only worth having if you decided *who* to act on by coin flip for part of the cohort. **The acted group is selected by the model's own score, so acted-vs-not-acted differences are observational until you randomize.** Withholding action on a random slice is not a caveat bolted onto this feature; it is the thing that turns the number into evidence. This page is the how first, and the why underneath it. --- ## How to run it Five steps. Everything except step 2 you are already doing. **1 · Score the cohort.** One `model_ref` from a cleared `hs_rank_topk`, one decision per row. ```json { "tool": "hs_score_batch", "arguments": { "model_ref": "$model_ref", "entities": ["$rows"] } } ``` Keep the rows whose `band` is `act`. Those are the ones you would have acted on. **2 · Split that cohort by a seeded coin flip.** This is the whole intervention. ```python import hashlib def treated(entity_id: str, seed: str = "2026-Q3-outreach", holdout: float = 0.15) -> bool: """Deterministic, reproducible, and independent of anything the model saw. Hashing the id means the same entity lands the same way every time you re-run this — so a retry, a resumed job, or a second batch cannot quietly move somebody between arms. Do NOT seed on the score, the band, or anything else the model produced: that puts the model back in charge of the assignment and undoes the randomisation. """ h = hashlib.sha256(f"{seed}:{entity_id}".encode()).digest() return int.from_bytes(h[:8], "big") / 2**64 >= holdout ``` Withhold action on **10–20%** of the `act` cohort. Below about 10% the control arm is too small to reach the floors below in any reasonable time; above about 20% you are paying real money in forgone action for precision you probably do not need. **3 · Attest only for the treated half.** An attestation is a record that you acted, so the untreated half gets none — that absence is what makes them a control. ```json { "tool": "hs_attest_action", "arguments": { "model_ref": "$model_ref", "entity_id": "$entity_id", "lever_token": "$lever_token", "dose": "$how_far_you_went" } } ``` **4 · Report outcomes for BOTH halves.** This is the step people skip, and skipping it is fatal: if you only report outcomes for entities you acted on, the control arm has no observed outcomes and `hs_action_evidence` has nothing to compare against. ```json { "tool": "hs_report_outcome", "arguments": { "model_ref": "$model_ref", "entity_id": "$entity_id", "outcome": true, "event_id": "$your_idempotency_key" } } ``` `hs_report_outcome` is append-only and never retrains. Pass an `event_id` you control: the same `event_id` twice is one row, so a retried job cannot double-count an outcome into the evidence. **5 · Read the evidence.** ```json { "tool": "hs_action_evidence", "arguments": { "model_ref": "$model_ref" } } ``` ### What you will see first, and why it is right `live: null`. [live: S3] That is the floor working. `hs_action_evidence` returns `live: null` until each cell has **30** reported outcomes, and flags `small_n` until **100**. A difference computed across four entities per arm is not evidence — it is noise with a confidence interval drawn around it — and returning one would be worse than returning nothing, because a number gets acted on and a `null` does not. Expect the floor to hold for a while. On a cohort of a few hundred with a 15% holdout, the control cell is the binding constraint, and it fills at the rate you actually run the play. --- ## Why — the confounding, in the tool's own terms `hs_rank_topk` ranks by likelihood. `hs_score_entity` bands by that same likelihood. If you then act on the `act` band and compare it to everything else, the two groups differ in **the model's own estimate of their outcome likelihood** before you did anything at all. Whatever difference you measure afterwards contains both the effect of acting and the effect of having been selected, and nothing in the arithmetic can separate them. Randomising *within* the `act` band is what removes the selection: both arms were flagged, both were scored the same way, and the only thing that differs is a coin flip that no feature and no score could see. ### What this defends against, and what it does not Hunter-Seeker never grades its own homework in one specific sense: `hs_model_quality` (how well the model fits held-out data) and `hs_action_evidence` (whether acting changed outcomes) are separate tools over separate data, and are never merged. The model's fit statistics can never stand in for evidence that acting works. [test: packages/mcp/test/surfaces-parity.test.ts::B4: every surface lists exactly the tools the server registers] It does **not** defend against selection bias. Nothing in the engine knows how you chose whom to act on, so nothing in the engine can correct for it. That correction is yours, and the randomised holdout above is the cheapest form of it. ### If you cannot randomise Sometimes you cannot withhold an action — a regulatory obligation, a safety case, a contract. Then the number is observational and you should say so out loud when you report it. The established remedies are: - **Propensity scoring** — model who got acted on, then compare like with like ([Rosenbaum & Rubin, 1983](https://academic.oup.com/biomet/article/70/1/41/240879)). - **Uplift modelling** — model the *difference* the action makes rather than the outcome ([Gutierrez & Gérardy, 2017](https://proceedings.mlr.press/v67/gutierrez17a.html)). - **Off-policy evaluation** — reweight logged decisions to estimate a policy you did not run ([Bottou et al., JMLR 2013](https://arxiv.org/abs/1209.2355)). - **Switchback designs** — randomise over time windows rather than entities, when entity-level holdout is impossible ([Bojinov, Simchi-Levi & Zhao, 2020](https://arxiv.org/abs/2009.00148)). All four are association under assumptions. The coin flip is the only one that is not. --- *See also:* [The two loops](/docs/two-loops) · [Refusal is a result](/docs/refusal-is-a-result)