Hunter-Seeker Docs For agents This page as markdown

How it's different

Verified against vendor docs on 2026-09-03; re-verify quarterly.

Four categories of tool get compared to this one, usually because they share a buyer rather than a job. The table below is an attempt to be useful rather than flattering: every Hunter-Seeker claim carries a test or a live call, every claim about somebody else carries a link to their own documentation and the date it was checked, and every column gets a row saying where that category is genuinely better.

If you only read one line: the other four categories will all give you an answer. This one is built to refuse.

The table

Capability	Hunter-Seeker	LLM-as-judge eval tools	AI guardrails	AutoML platforms	Agent memory
Abstains below a validated lift bar	Yes — below realised lift 1.5 it returns a terminal `{result:"none"}` and refunds the run t-empty	No — a judge returns a score for every input LS	No — rails allow or block an action; they do not assess predictive signal NEMO	No — a leaderboard always has a top model H2O	N/A M0
Deterministic replay	Yes — identical inputs, byte-identical envelopes, under a pinned `core_hash` t-det	No — LLM judges are documented as inconsistent evaluators JUDGE-INCONSIST	Partly — rule rails are deterministic, LLM-based rails are not NEMO	Largely — seeded training is reproducible, but no published identity hash DR	N/A M0
Who authors the score	The engine. Nothing above it may author, alter or extrapolate a number t-invariant	An LLM, prompted LS	N/A — no score NEMO	The platform's model DR	N/A M0
Calibrated probability	Yes, with ECE reported per ranking t-quality	No LS	No NEMO	Yes DR	No M0
Holdout / out-of-time lift validation	Yes — scheme, split, train/holdout sizes and a lift curve on every cleared run t-quality	No LS	No NEMO	Yes — lift charts and validation are core DR · H2O	No M0
Column-level leak guard	Yes — post-outcome columns quarantined and named with reasons t-quality	N/A LS	No NEMO	Partly — target-leakage detection exists, surfaced as a warning DR	No M0
Point-in-time reduction of an event log	Yes — `trace@1` computes per-group priors over strictly earlier runs only t-trace	No LS	No NEMO	Manual — you build the point-in-time join yourself FEAST	No M0
Signed per-decision Verdict	Yes — a detached signature over each decision t-verdict	No LS	No NEMO	No DR	No M0
Credential-free verification by a third party	Yes — `POST /v1/verify-verdict` and the offline `hs-verify` libraries take no credential t-verify	No LS	No NEMO	No DR	No M0
Per-decision autonomy ceiling	Yes — `max_autonomy` on every decision, engine-authored t-band	No LS	Adjacent — rails gate an action, but not per-decision by model confidence NEMO	No DR	No M0
MCP-native surface	Yes — MCP is the surface, not an adapter over a REST product t-tools	Yes for some LS	Partly GR	No — SDK and UI DR	Yes M0
Primary job	Decide whether to act on one row, and refuse when the evidence is not there	Score the quality of generated text	Authorise or block an action before it happens	Build the best model it can from your table	Remember what happened across sessions
Where they're better	—	Tracing and open-ended text quality. Nothing here evaluates prose, and eval tools give you run-level tracing this does not LS	Pre-action authorisation. Rails sit in the action path and stop things; this bands a decision and leaves enforcement to you NEMO	Maturity: UI, deployment, feature engineering, model breadth. If you want the best model on a table, that is what these are for DR	Memory. Same users, a different job — mem0 remembers, this predicts; they compose M0

The five failure modes of LLM-as-judge, from the literature

This is the row that most often gets waved through, so it gets its own section with primary sources. None of these is a knock on eval tools — they are the documented properties of using a language model as an evaluator, and the people who built these tools published most of them.

Position bias — the same two answers swap winners when you swap their order. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
Verbosity bias — longer answers score higher, holding quality constant. Saito et al., Verbosity Bias in Preference Labeling by Large Language Models (2023)
Self-preference — evaluators recognise and favour their own generations. Panickssery, Bowman & Feng, LLM Evaluators Recognize and Favor Their Own Generations (2024)
Non-determinism — the same prompt to the same model does not reliably give the same answer. Atil et al., LLM Stability: A Detailed Analysis with Some Surprises (2024)
Inconsistency and rating drift — judges are inconsistent and biased across repeated ratings. Stureborg, Alikaniotis & Suhara, Large Language Models are Inconsistent and Biased Evaluators (2024)

The reason this matters for a decision rather than an eval: a judge score that moves when you reorder the options is fine for ranking model candidates offline and is not something you want underneath an action taken on a customer.

What this page is not claiming

Not that Hunter-Seeker is more accurate than an AutoML platform. It is not trying to be; a mature AutoML product will usually fit a better model on a clean table.
Not that guardrails are unnecessary. They sit in a different place in the stack — before the action — and composing the two is the sensible architecture.
Not causation. Every driver and every lever this system returns is labelled association_not_causal, and the control-arm page is about the one thing that would change that.

Marker key

Hunter-Seeker claims resolve to a test in this repository or to a live call recorded in the evidence pack. Competitor claims resolve to that vendor's own documentation, with the date checked.