{
  "purpose": "Source-bound paired request fulfilment TEST; not global or human-verified accuracy.",
  "method_source": "https://arxiv.org/abs/2309.15217",
  "test_cases": 80,
  "paired_trials": 160,
  "engines": [
    "rag",
    "graphrag"
  ],
  "workers": 2,
  "question_held_out": true,
  "unseen_documents": false,
  "threshold_per_engine": 0.94,
  "calibration_cases": 8,
  "no_test_directed_tuning_after_freeze": true,
  "sections": {
    "Fixed design before execution": "- 80 new exact questions, 10 profiles, eight categories per profile. Sampling\n  seed: 20261110. `build_test.py` reads raw wiki Markdown, not production answers.\n- Categories: record identity, work inventory, assignment, dated state, dated\n  plans, latest fully dated record, assignment versus dated state, absent year.\n- Exclude previously used `user_questions_v1` gold profiles and conversational\n  DEV subjects. Check exact-question overlap against earlier experiment JSONs.\n- Source paths, SHA256, literal lines and expected requirements are in\n  `test.json`. The complete gold is never sent to the answering endpoint.\n- Run each question once through RAG and GraphRAG, with the same configured\n  model, temperature, snapshot, limit and grading rules: 160 paired trials.\n- Freeze dataset, evaluator, runtime file hashes, index, knowledge snapshot and\n  model configuration before the first TEST request. No TEST-directed tuning,\n  answer regeneration or replacement of failed trials. Two concurrent workers.\n- Calibrate the separate judge on eight synthetic DEV examples: supported facts\n  and mutations of actor, work, planned/result phase and incomplete dates.\n- Success requires supported facts, all requested requirements, supporting\n  citations, appropriate response behaviour and clarity of at least 3/4.\n  Rejection of an answerable question is a failure, not successful caution.\n- Candidate and selected evidence coverage are reported separately from final\n  answer quality. The evaluator reads original source windows independently.\n- Target: at least 94% per retrieval route on this TEST. Keep all failures in\n  the denominator, including execution and judge failures. Report critical\n  substitutions and false refusals separately. Do not pool the paired routes as\n  160 independent questions.",
    "Known limitations": "Gold is AI-authored and checked against raw source bytes; it has no human review.\nThe judge uses the same configured model in a separate call, so correlated errors\nremain possible. Profiles repeat across questions, categories are structured and\nknown, and the public corpus may have been seen before. This is a question-held-out\ntest, not an unseen-document or broad population estimate. Wilson intervals are\ndescriptive IID intervals, not cluster-adjusted guarantees.\n\nConversation, transport and adversarial scope tests belong to known regressions;\ntheir success must not be added to this TEST accuracy. Real Claude browser\nverification remains a separate manual step. A backend audit cannot enforce a\nlater UI rewrite of its accepted answer.",
    "Artifacts": "`test_manifest.json` binds the frozen setup. `test_results.jsonl` preserves the\nfirst run, and `test_results.summary.json` reports its aggregate outcome. Failed\ncases can inform a later DEV cycle; a corrected rerun is reproduction, not fresh\nindependent TEST accuracy."
  }
}