# How it's different *Verified against vendor docs on 2026-09-03; re-verify quarterly.* Four categories of tool get compared to this one, usually because they share a buyer rather than a job. The table below is an attempt to be useful rather than flattering: every Hunter-Seeker claim carries a test or a live call, every claim about somebody else carries a link to **their own documentation** and the date it was checked, and every column gets a row saying where that category is genuinely better. If you only read one line: **the other four categories will all give you an answer. This one is built to refuse.** --- ## The table | Capability | Hunter-Seeker | LLM-as-judge eval tools | AI guardrails | AutoML platforms | Agent memory | |---|---|---|---|---|---| | **Abstains below a validated lift bar** | Yes — below realised lift 1.5 it returns a terminal `{result:"none"}` and refunds the run [t-empty] | No — a judge returns a score for every input [LS] | No — rails allow or block an action; they do not assess predictive signal [NEMO] | No — a leaderboard always has a top model [H2O] | N/A [M0] | | **Deterministic replay** | Yes — identical inputs, byte-identical envelopes, under a pinned `core_hash` [t-det] | No — LLM judges are documented as inconsistent evaluators [JUDGE-INCONSIST] | Partly — rule rails are deterministic, LLM-based rails are not [NEMO] | Largely — seeded training is reproducible, but no published identity hash [DR] | N/A [M0] | | **Who authors the score** | The engine. Nothing above it may author, alter or extrapolate a number [t-invariant] | An LLM, prompted [LS] | N/A — no score [NEMO] | The platform's model [DR] | N/A [M0] | | **Calibrated probability** | Yes, with ECE reported per ranking [t-quality] | No [LS] | No [NEMO] | **Yes** [DR] | No [M0] | | **Holdout / out-of-time lift validation** | Yes — scheme, split, train/holdout sizes and a lift curve on every cleared run [t-quality] | No [LS] | No [NEMO] | **Yes** — lift charts and validation are core [DR] · [H2O] | No [M0] | | **Column-level leak guard** | Yes — post-outcome columns quarantined and named with reasons [t-quality] | N/A [LS] | No [NEMO] | Partly — target-leakage detection exists, surfaced as a warning [DR] | No [M0] | | **Point-in-time reduction of an event log** | Yes — `trace@1` computes per-group priors over strictly earlier runs only [t-trace] | No [LS] | No [NEMO] | Manual — you build the point-in-time join yourself [FEAST] | No [M0] | | **Signed per-decision Verdict** | Yes — a detached signature over each decision [t-verdict] | No [LS] | No [NEMO] | No [DR] | No [M0] | | **Credential-free verification by a third party** | Yes — `POST /v1/verify-verdict` and the offline `hs-verify` libraries take no credential [t-verify] | No [LS] | No [NEMO] | No [DR] | No [M0] | | **Per-decision autonomy ceiling** | Yes — `max_autonomy` on every decision, engine-authored [t-band] | No [LS] | Adjacent — rails gate an action, but not per-decision by model confidence [NEMO] | No [DR] | No [M0] | | **MCP-native surface** | Yes — MCP is the surface, not an adapter over a REST product [t-tools] | Yes for some [LS] | Partly [GR] | No — SDK and UI [DR] | Yes [M0] | | **Primary job** | Decide whether to act on one row, and refuse when the evidence is not there | Score the quality of generated text | Authorise or block an action before it happens | Build the best model it can from your table | Remember what happened across sessions | | **Where they're better** | — | **Tracing and open-ended text quality.** Nothing here evaluates prose, and eval tools give you run-level tracing this does not [LS] | **Pre-action authorisation.** Rails sit in the action path and stop things; this bands a decision and leaves enforcement to you [NEMO] | **Maturity: UI, deployment, feature engineering, model breadth.** If you want the best model on a table, that is what these are for [DR] | **Memory. Same users, a different job** — mem0 remembers, this predicts; they compose [M0] | --- ## The five failure modes of LLM-as-judge, from the literature This is the row that most often gets waved through, so it gets its own section with primary sources. None of these is a knock on eval tools — they are the documented properties of using a language model as an evaluator, and the people who built these tools published most of them. - **Position bias** — the same two answers swap winners when you swap their order. [Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena* (2023)](https://arxiv.org/abs/2306.05685) - **Verbosity bias** — longer answers score higher, holding quality constant. [Saito et al., *Verbosity Bias in Preference Labeling by Large Language Models* (2023)](https://arxiv.org/abs/2310.10076) - **Self-preference** — evaluators recognise and favour their own generations. [Panickssery, Bowman & Feng, *LLM Evaluators Recognize and Favor Their Own Generations* (2024)](https://arxiv.org/abs/2404.13076) - **Non-determinism** — the same prompt to the same model does not reliably give the same answer. [Atil et al., *LLM Stability: A Detailed Analysis with Some Surprises* (2024)](https://arxiv.org/abs/2408.04667) - **Inconsistency and rating drift** — judges are inconsistent and biased across repeated ratings. [Stureborg, Alikaniotis & Suhara, *Large Language Models are Inconsistent and Biased Evaluators* (2024)](https://arxiv.org/abs/2405.01724) The reason this matters for a *decision* rather than an eval: a judge score that moves when you reorder the options is fine for ranking model candidates offline and is not something you want underneath an action taken on a customer. --- ## What this page is not claiming - Not that Hunter-Seeker is more accurate than an AutoML platform. It is not trying to be; a mature AutoML product will usually fit a better model on a clean table. - Not that guardrails are unnecessary. They sit in a different place in the stack — before the action — and composing the two is the sensible architecture. - Not causation. Every driver and every lever this system returns is labelled `association_not_causal`, and [the control-arm page](/docs/control-arm) is about the one thing that would change that. --- ## Marker key Hunter-Seeker claims resolve to a test in this repository or to a live call recorded in the evidence pack. Competitor claims resolve to that vendor's own documentation, with the date checked. [t-empty]: ../../packages/mcp/test/samples.test.ts "test: a REAL dataset with a reading still reserves AND still refunds on honest-empty" [t-det]: ../../packages/mcp/test/surfaces-parity.test.ts "test: B5: every surface publishes the SAME engine identity, on both axes" [t-invariant]: ../../packages/mcp/test/agent-decides-valence.test.ts "test: agent-decides-valence" [t-quality]: ../../packages/mcp/test/honest-null-survives.test.ts "test: honest-null-survives" [t-trace]: ../../packages/mcp/test/readings.test.ts "test: readings" [t-verdict]: ../../packages/mcp/test/verdict-wiring.test.ts "test: verdict-wiring" [t-verify]: ../../packages/mcp/test/verify-accepts-real-verdicts.test.ts "test: verify-accepts-real-verdicts" [t-band]: ../../packages/mcp/test/verdict-tools.test.ts "test: verdict-tools" [t-tools]: ../../packages/mcp/test/surfaces-parity.test.ts "test: B4: every surface lists exactly the tools the server registers" [LS]: https://docs.smith.langchain.com/evaluation "checked 2026-09-03 — LangSmith evaluation docs" [NEMO]: https://docs.nvidia.com/nemo/guardrails/latest/index.html "checked 2026-09-03 — NVIDIA NeMo Guardrails docs" [GR]: https://www.guardrailsai.com/docs "checked 2026-09-03 — Guardrails AI docs" [DR]: https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/lift-chart.html "checked 2026-09-03 — DataRobot lift chart docs" [H2O]: https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html "checked 2026-09-03 — H2O AutoML docs" [M0]: https://docs.mem0.ai/ "checked 2026-09-03 — mem0 docs" [FEAST]: https://docs.feast.dev/getting-started/concepts/point-in-time-joins "checked 2026-09-03 — Feast point-in-time joins" [JUDGE-INCONSIST]: https://arxiv.org/abs/2405.01724 "checked 2026-09-03 — Stureborg et al. 2024" --- *See also:* [The two loops](/docs/two-loops) · [Status: served vs planned](/docs/status) · [Benchmarks](/benchmarks)