jp170na/dp-zp-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Uzavretie GraphRAG evaluácie

Browse Source
This commit is contained in:
Ján Pták 2026-10-25 17:34:30 +01:00
parent 90fc7f2df0
commit d0ab5fdddf
48 changed files with 480930 additions and 64 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
README.md
View File

@ -15,8 +15,8 @@ FTS5/BM25 + embeddingy + RRF. Nad týmto baseline sa vo vetve GraphRAG
dopĺňa knowledge graph uložený v Neo4j. Aktuálne je implementovaná
grafová infraštruktúra, deterministický builder, vizualizácia grafu a
samostatné grafové vyhľadávanie. Grafová vrstva má zdrojovo overený benchmark a zmrazenú opravenú
verziu v4 s 320 novými TEST otázkami a samostatným overením generovania
odpovedí cez OpenWebUI API. Ďalším krokom je natívne zapojenie grafového
verziu v5.1 s regresiou 320 historických otázok, kontrolou všetkých vrátených
typovaných faktov a samostatným overením generovania odpovedí cez OpenWebUI API. Ďalším krokom je natívne zapojenie grafového
nástroja a hlavného agenta do OpenWebUI.
## Implementované
@ -870,20 +870,18 @@ grafové vyhľadávanie je implementované v `tools/graphrag/graph_search.py`.
GraphRAG zatiaľ nezasahuje do stabilného klasického `/rag` toku; ďalšia
etapa spojí grafové výsledky s klasickým RAG a OpenWebUI nástrojom.
Aktuálna evaluácia GraphRAG v4: DEV/regresia **722/722**, širší TEST
**320/320**, bez runtime chýb. TEST pokrýva desať kategórií, 288 kladných
a 32 záporných otázok. Nové otázky a kombinácie faktov sú oddelené od DEV;
zdrojových 72 profilov aj rodiny formulácií sú známe. Nejde o záruku
100 % správnosti na ľubovoľných otázkach. Historické v2 **67/80** a v3
**13/13** ostávajú zachované. Pôvodných 137 nesúladov má zdrojové opravy;
v4 dopĺňa kontrolu názvov oproti pôvodnému Markdownu. Generovanie odpovedí
cez OpenWebUI API sa meria zvlášť od grafového retrievalu a od natívneho
výberu nástrojov agentom.
Finálny nový TEST generovaných odpovedí cez OpenWebUI API: **281/300 (93,67 %)**.
Doplnkový audit všetkých grafových kontextových faktov: **319/320 (99,69 %)**;
strict retrieval 320/320 meria splnenie požadovaných očakávaní.
Podrobné výsledky, obmedzenia a reprodukcia sú v
[GraphRAG evaluácii](tools/graphrag/evaluation/README.md).
Aktuálna evaluácia GraphRAG **v5.1** má DEV **722/722** a historickú regresiu
**320/320**, vrátane kontroly všetkých vrátených typovaných faktov oproti raw
Markdownom. Generovanie sa meria samostatne: 39 DEV a finálna regresia 320
historických otázok vrátane týchto 39; nejde o nový nezávislý TEST.
LLM regresia dosiahla **317/320 (99,06 %)** bez neplatných JSON odpovedí.
Finálne metriky, opravy JSON, historické porovnanie a obmedzenia sú v
[finálnej správe v5.1](tools/graphrag/evaluation/results/final_v5_1_2026-10-25/REPORT.md).
Historická v4 mala grafový strict TEST 320/320, doplnkovú kontrolu kontextových
faktov 319/320 a LLM TEST 281/300 (93,67 %). Pôvodné otázky aj výsledky
v1–v4 zostávajú zachované. Aktuálne skóre nie je zárukou všeobecnej pravdivosti.
[Protokol a reprodukcia](tools/graphrag/evaluation/README.md).
## OpenWebUI
@ -1106,11 +1104,12 @@ spoľahlivého zdroja.
### 2. GraphRAG evaluácia
V4 má zmrazený benchmark a širší TEST **320/320**. [Finálna správa](tools/graphrag/evaluation/results/final_v4_2026-10-25/REPORT.md)
oddeľuje výsledky grafového retrievalu a generovania odpovedí modelom.
Historické v1–v3 sa nemenia. Pred formálnym porovnaním s klasickým RAG
potrebujeme spoločnú metodiku a ideálne otázky anotované nezávislým človekom.
Vystavené TEST otázky sú pri ďalšom ladení iba regresiou.
V5.1 má zmrazený graf a úplnú historickú regresiu **320/320**.
[Finálna správa](tools/graphrag/evaluation/results/final_v5_1_2026-10-25/REPORT.md)
oddeľuje retrieval a generovanie odpovedí; pôvodné v1–v4 sa nemenia.
Pred formálnym porovnaním s klasickým RAG potrebujeme spoločnú metodiku
a ideálne otázky anotované nezávislým človekom. Nový nezávislý TEST vyžaduje
nové otázky; opakované meranie odhalených otázok je regresia.
### 3. OpenWebUI nástroje

100
docs/graphrag_v5_2026-10-25_worklog.md Normal file
View File

@ -0,0 +1,100 @@
# GraphRAG v5.1 — dokončenie bodu 3 a história iterácie v5
## Rozsah a protokol
Ide o nové meranie opravenej implementácie na historicky odhalených otázkach.
Výsledky v1–v4, otázky ani pôvodné očakávania sa neprepisujú. Používa sa
722 DEV otázok a 320 historických grafových TEST otázok. Druhý split je vo v5
regresia, nie nový nezávislý holdout. Nezavádzajú sa nové otázky iba kvôli skóre.
Generovanie sa ladí na 39 vopred deklarovaných prípadoch: 20 pôvodných DEV
výstupov a 19 zlyhaní historického LLM TESTu. Finálna regresia generovania má
všetkých 320 historických otázok vrátane týchto 39 DEV; tento prekryv je
výslovne uložený v protokole a vo výsledkoch. Žiadna otázka sa nevynecháva
podľa úspechu vyhľadávania alebo modelu.
Záloha kódu, dát a spravovaného grafu je v
`/tmp/zp-bod3-v5-backup-20261025T155423Z/`. Konfiguračné tajomstvá sa nearchivujú.
## Opravy
- Príprava na akademický projekt je organizačná poznámka. Sama nevytvára
akademický záznam ani jeho rok. Tým sa rieši historický prípad `wide-080`.
- Označenie „Návrh na zadanie BP“ samo nedokazuje názov práce. Pri doplnkovej
kontrole DEV sa odhalili tri výstupy Martina Šarišského s takto odhadnutým
názvom. Parser teraz používa explicitné názvy/témy alebo dovolený úvodný
názvový odsek. Očakávania benchmarku sa kvôli predikciám nemenili.
- Prvá zmrazená v5 mala regresiu 319/320. `wide-275` odhalil konzultačnú
poznámku nesprávne použitú ako názov projektu na hranici chunku. V5.1
vylučuje tieto organizačné vety; prvá iterácia a overený zdrojový patch ostávajú uložené.
- Evaluator odmieta `complete=false` aj bez príznaku `truncated` a kontroluje
všetky vrátené typované fakty proti nezávislému čítaniu raw Markdownov:
osobu, dokument, URL, akademický názov/rok/typ, metadata na tom istom
dokumente, zdieľané vlastnosti a porovnania rokov. Chybné nadbytočné fakty
znižujú strict skóre. Nejde o všeobecný sémantický súd ľubovoľných viet.
- Zmrazenie zahŕňa spoločné moduly, publikačnú politiku a pravidlo prípravy;
finálna generácia kontroluje identitu kódu/promptu/dát voči dokončenému DEV.
- Pokyny pre LLM vysvetľujú väzbu názvu a tagu pri tej istej osobe aj cez
dcérsky dokument. Kategória a autor dokumentu musia mať spoločný dokument.
- JSON sa kontroluje typovanou schémou. Jeden úplný JSON blok sa môže vybrať
z Markdown ohraničenia bez prepisovania hodnôt. Skutočne neplatný JSON môže
vyvolať najviac jeden opravný požiadavok s pôvodnou otázkou a kontextom.
Gold odpovede sa neposielajú modelu; všetky pokusy aj prvotná úspešnosť sa ukladajú.
## Overenie a príkazy
Kompletná regresná sada finálnej v5.1: **525 passed, 4 skipped**. Prvý prísny
DEV mal 719/722; tri nesúlady názvu viedli k oprave parsera uvedenej vyššie.
V5 mala DEV 722/722, regresiu 319/320 a DEV generovania 38/39. V5.1 má
DEV 722/722 a regresiu 320/320 vrátane všetkých typovaných faktov; DEV
generovania zostal 38/39. Tieto priebežné výsledky sa nezatajujú.
Použité hlavné príkazy po načítaní lokálnej konfigurácie:
```bash
RUN_NEO4J_TESTS=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m pytest -q
docker compose build zp-agent-api
docker compose up -d --no-deps zp-agent-api
```
Koordinovaná obnova indexov používa autentifikovaný `POST /sync` s
`{"pull_git":false}`. Korpus zostáva 114 dokumentov/887 chunkov, aktualizovaný
graf má 446 uzlov a 877 vzťahov. Presné indexové verzie a finálne výsledky
sú identifikované vo finálnej správe v5.1 a v zázname overenia nasadenia.
Grafový protokol:
```bash
python -m tools.graphrag.evaluation.evaluate_graphrag \
--split dev --audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json \
--output-dir tools/graphrag/evaluation/results/final_v5_1_2026-10-25
python -m tools.graphrag.evaluation.freeze_graphrag \
--dev-results tools/graphrag/evaluation/results/final_v5_1_2026-10-25/graphrag_dev.json \
--audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json \
--purpose historical_regression --output tools/graphrag/evaluation/frozen_v5_1.json
python -m tools.graphrag.evaluation.evaluate_graphrag \
--split test --audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json \
--freeze-manifest tools/graphrag/evaluation/frozen_v5_1.json \
--output-dir tools/graphrag/evaluation/results/final_v5_1_2026-10-25
```
Generovanie nad rovnakým zmrazeným grafom:
```bash
python -m tools.answer_evaluation.evaluate_graph_answers \
--split dev --dev-ids tools/graphrag/evaluation/results/v5_1_preparation/generation_dev_ids.json \
--audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json \
--freeze-manifest tools/graphrag/evaluation/frozen_v5_1.json \
--output-dir tools/graphrag/evaluation/results/answers_v5_1_dev_2026-10-25
python -m tools.answer_evaluation.evaluate_graph_answers \
--split regression --dev-ids tools/graphrag/evaluation/results/v5_1_preparation/generation_dev_ids.json \
--dev-results tools/graphrag/evaluation/results/answers_v5_1_dev_2026-10-25/answers_dev.json \
--audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json \
--freeze-manifest tools/graphrag/evaluation/frozen_v5_1.json \
--output-dir tools/graphrag/evaluation/results/answers_v5_1_regression_2026-10-25
```
Príkazy vyššie dokumentujú pôvodný beh. Pri reprodukcii použite nové výstupné
adresáre a zachovajte zodpovedajúci zmrazený manifest. Nový manifest vytvárajte
pre novú verziu; existujúce finálne súbory sú nemenné. Reprodukcia nie je nový
nezávislý TEST. Aktuálny release označuje Git tag `graphrag-v5.1-frozen`.

24
tools/answer_evaluation/README.md
View File

@ -1,5 +1,29 @@
# Generovanie odpovedí nad zmrazeným grafom
Aktuálne meranie **v5.1** je historická regresia. Výsledky sú v
[spoločnej správe](../graphrag/evaluation/results/final_v5_1_2026-10-25/REPORT.md).
Používa 39 vopred vybraných DEV otázok (20 pôvodných a 19 historických zlyhaní)
a 320 finálnych regresných otázok vrátane týchto 39. Nejde o nezávislý holdout.
Generovanie overí odtlačky kódu/promptu/dát voči kompletnému DEV. Parser prijíma
platný typovaný JSON alebo jeden úplný JSON blok. Pri chybe formátu je povolený
najviac jeden opravný požiadavok bez gold; pôvodné aj opravné odpovede sa ukladajú.
Vecné zlyhanie hodnotenia neopakuje požiadavok. Report uvádza prvotnú aj konečnú
úspešnosť a počet opráv. Natívny výber OpenWebUI nástrojov zostáva bodom 4.
Reprodukcia potrebuje zodpovedajúci release checkout a graf, načítanú konfiguráciu
a nové výstupné adresáre. Najprv spustite `--split dev` s
`--dev-ids tools/graphrag/evaluation/results/v5_1_preparation/generation_dev_ids.json`,
`--audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json`
a `--freeze-manifest tools/graphrag/evaluation/frozen_v5_1.json`.
Finálny beh používa rovnaké parametre, `--split regression` a povinné
`--dev-results` smerujúce na tento dokončený DEV. Historické výsledky sa neprepisujú.
## Historický protokol v4
Nasledujúci text a pôvodný TEST 300 otázok opisujú v4; jej reprodukcia vyžaduje
pôvodný kód aj graf. Týchto 300 otázok je pri ďalšom ladení historickou regresiou.
Tento experiment meria OpenWebUI chat API po dodaní skutočného grafového
kontextu lokálnym skriptom. Neoveruje prehliadač ani automatický výber
nainštalovaných nástrojov hlavným agentom. Klasický RAG sa nemení.

41
tools/answer_evaluation/answer_format.py Normal file
View File

@ -0,0 +1,41 @@
"""Validate JSON without inventing facts or repairing values from benchmark gold."""
import json
import re
from typing import Literal
from pydantic import BaseModel, ConfigDict
class Claim(BaseModel):
model_config = ConfigDict(extra='forbid', strict=True)
person: str
title: str | None = None
year: int | None = None
start_year: int | None = None
keyword: str | None = None
category: str | None = None
document_author: str | None = None
class Answer(BaseModel):
model_config = ConfigDict(extra='forbid', strict=True)
answer: str
status: Literal['ok','no_answer','ambiguous','error']
claims: list[Claim]
source_urls: list[str]
count: int | None = None
def parse_answer(text: str) -> tuple[dict, bool]:
try:
data=json.loads(text.strip())
normalized=False
except json.JSONDecodeError:
# Exactly one complete JSON fence can be unwrapped. Never replace quotes,
# append missing braces, extract fragments or silently alter assertions.
blocks=re.findall(r'```(?:json)?\s*\n([\s\S]*?)```',text,flags=re.IGNORECASE)
if len(blocks)!=1:
raise
data=json.loads(blocks[0].strip())
normalized=True
parsed=Answer.model_validate(data)
return parsed.model_dump(exclude_none=True),normalized

92
tools/answer_evaluation/evaluate_graph_answers.py
View File

@ -7,6 +7,7 @@ from __future__ import annotations
import argparse
import concurrent.futures
import csv
import hashlib
import json
import os
@ -21,6 +22,7 @@ from tools.graphrag.evaluation.check_openwebui import grade_claims
from tools.graphrag.evaluation.reproducibility import graph_fingerprint, run_metadata, sha256_file, verify_freeze
from tools.graphrag.evaluation.source_oracle import load_source_oracle, norm
from tools.rag.evaluation.rag_runner import DEFAULT_MODEL, OPENWEBUI_URL, request_json
from .answer_format import parse_answer, Answer
HERE = Path(__file__).resolve().parent
@ -87,19 +89,32 @@ def main():
parser.add_argument('--sample', type=Path, default=BASE / 'results/final_v4_2026-10-25/openwebui_sample.json')
parser.add_argument('--workers', type=int, default=3)
parser.add_argument('--model', default=DEFAULT_MODEL)
parser.add_argument('--split', choices=['dev', 'test'], default='test')
parser.add_argument('--split', choices=['dev', 'test', 'regression'], default='test')
parser.add_argument('--questions', type=Path, default=BASE/'graphrag_questions_v4.json')
parser.add_argument('--audit', type=Path, default=BASE/'results/v4_preparation/source_audit.json')
parser.add_argument('--freeze-manifest', type=Path, default=BASE/'frozen_v4.json')
parser.add_argument('--dev-ids', type=Path)
parser.add_argument('--dev-results', type=Path, help='Required completed DEV before final regression.')
args = parser.parse_args()
if args.output_dir.exists():
parser.error('Refusing to overwrite an answer-level TEST.')
if not os.getenv('OPENWEBUI_API_KEY'):
parser.error('OPENWEBUI_API_KEY missing.')
dataset = BASE / 'graphrag_questions_v4.json'; audit = BASE / 'results/v4_preparation/source_audit.json'
frozen = BASE / 'frozen_v4.json'; manifest = json.loads(frozen.read_text())
if args.workers < 1:
parser.error('--workers must be positive.')
if args.split=='regression' and args.dev_results is None:
parser.error('Final regression requires completed --dev-results.')
dataset=args.questions; audit=args.audit; frozen=args.freeze_manifest
manifest = json.loads(frozen.read_text())
questions = json.loads(dataset.read_text()); oracle = load_source_oracle()
sample = json.loads(args.sample.read_text()); exposed = set(sample['selected_ids'])
test = [q for q in questions if q['split'] == 'test' and q['id'] not in exposed]
assert len(exposed) == 20 and len(test) == 300
selected = test if args.split == 'test' else [q for q in questions if q['id'] in exposed]
development=set(json.loads(args.dev_ids.read_text())) if args.dev_ids else exposed
selected = ([q for q in questions if q['split']=='test'] if args.split=='regression' else
test if args.split=='test' else [q for q in questions if q['id'] in development])
if args.split=='regression' and manifest.get('evaluation_kind')!='historical_regression':
parser.error('Regression requires an explicitly declared historical regression freeze.')
prompt_path = HERE / 'graph_answer_prompt.txt'; prompt = prompt_path.read_text()
args.output_dir.mkdir(parents=True)
metadata = run_metadata(dataset); settings = get_neo4j_settings(); contexts = {}
@ -108,16 +123,31 @@ def main():
# Generate every context before making any external request.
for case in selected:
contexts[case['id']] = graph_context(search_graph_with_session(session, case['question'], **manifest['parameters']))
code_files = [Path(__file__), prompt_path, Path('tools/rag/evaluation/rag_runner.py')]
code_files = [Path(__file__), prompt_path, HERE/'answer_format.py', Path('tools/rag/evaluation/rag_runner.py')]
hashes = {str(p): sha256_file(p) for p in code_files}
declaration = {'method': 'Fresh LLM-generation holdout over previously measured, frozen graph queries.',
if args.split=='regression':
dev=json.loads(args.dev_results.read_text())
if (dev['configuration']['selected_split']!='dev'
or set(dev['configuration']['selected_ids'])!=development
or dev['summary']['questions']!=len(development)
or dev['configuration']['generation_files']!=hashes
or dev['run_metadata']['fingerprints']!=metadata['fingerprints']):
raise RuntimeError('Generation code, prompt, data or complete DEV differs; rerun DEV before final freeze.')
declaration = {'method': ('Previously exposed LLM questions; regression, not independent TEST.'
if manifest.get('evaluation_kind')=='historical_regression' else
'Fresh LLM-generation holdout over previously measured, frozen graph queries.'),
'graph_freeze_sha256': sha256_file(frozen), 'audit_sha256': sha256_file(audit),
'generation_files': hashes, 'model': args.model, 'temperature': 0, 'workers': args.workers,
'dev_ids': sorted(exposed), 'test_ids': [q['id'] for q in test],
'dev_count': 20, 'test_count': 300, 'target_accuracy': 0.92,
'selected_split': args.split, 'selected_ids': [q['id'] for q in selected],
'policy': 'Evaluate all 300; retain every failure; do not tune on this TEST.',
'test_categories': dict(Counter(q['category'] for q in test)),
'dev_ids': sorted(development), 'test_ids': [q['id'] for q in test],
'dev_count': len(development), 'test_count': len(test), 'target_accuracy': 0.92,
'selected_split': args.split, 'selected_ids': [q['id'] for q in selected], 'selected_count':len(selected),
'selected_dev_overlap': len(development & {q['id'] for q in selected}),
'format_repair_limit':1, 'repair_policy':'Only JSON/schema failures; same question/context, no gold; retain all attempts.',
'dev_results_sha256':sha256_file(args.dev_results) if args.dev_results else None,
'policy': ('Evaluate every declared regression case, including DEV overlap; retain all failures and stop tuning after this freeze.'
if manifest.get('evaluation_kind')=='historical_regression' else
'Evaluate all 300; retain every failure; do not tune on this TEST.'),
'selected_categories': dict(Counter(q['category'] for q in selected)),
'limitations': ['Known corpus/profiles and templates; model outputs for these 300 questions were not previously observed.',
'Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.',
'Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose.']}
@ -127,18 +157,34 @@ def main():
started = time.perf_counter(); context = contexts[case['id']]
row = {'id': case['id'], 'category': case['category'], 'question': case['question'],
'graph_context': context, 'strict': False}
messages=[{'role':'system','content':prompt},{'role':'user','content':json.dumps(
{'question':case['question'],'graph_context':context},ensure_ascii=False)}]
row['attempts']=[]
try:
for attempt in range(2):
response = request_json(OPENWEBUI_URL, method='POST', timeout=90, max_attempts=2,
headers={'Authorization': f'Bearer {os.environ["OPENWEBUI_API_KEY"]}', 'Content-Type': 'application/json'},
payload={'model': args.model, 'temperature': 0, 'stream': False,
'response_format': {'type': 'json_object'}, 'messages': [
{'role': 'system', 'content': prompt}, {'role': 'user', 'content': json.dumps(
{'question': case['question'], 'graph_context': context}, ensure_ascii=False)}]})
'response_format': {'type': 'json_object'}, 'messages': messages})
row['response'] = response
text = response['choices'][0]['message']['content'].strip()
if text.startswith('```'):
text = text.split('\n', 1)[1].rsplit('```', 1)[0]
answer = json.loads(text); row['answer'] = answer; row.update(grade_answer(case, answer, oracle))
record={'response':response}
row['attempts'].append(record)
try:
answer,normalized=parse_answer(text)
except (ValueError, TypeError) as exc:
record['format_error']=type(exc).__name__
if attempt==1:
raise
messages += [{'role':'assistant','content':text}, {'role':'user','content':
'Predchádzajúci výstup nie je platný JSON podľa schémy. Vráť len opravený jediný JSON objekt '
'bez zmeny vecných údajov; používaj iba pôvodný kontext. Schéma: '+json.dumps(Answer.model_json_schema(),ensure_ascii=False)}]
continue
record['format_normalized']=normalized
grade=grade_answer(case,answer,oracle)
record['strict']=grade['strict']
row['answer']=answer; row.update(grade)
break
except Exception as error:
row['error'] = f'{type(error).__name__}: {str(error)[:500]}'
row['latency_seconds'] = time.perf_counter() - started
@ -151,7 +197,10 @@ def main():
output['questions'] = [results[q['id']] for q in selected if q['id'] in results]
output['summary'] = {'questions': len(results), 'correct': sum(r['strict'] for r in results.values()),
'accuracy': sum(r['strict'] for r in results.values()) / len(results),
'errors': sum(bool(r.get('error')) for r in results.values())}
'errors': sum(bool(r.get('error')) for r in results.values()),
'first_pass_correct':sum(bool(r.get('attempts') and r['attempts'][0].get('strict')) for r in results.values()),
'format_repair_attempted':sum(len(r.get('attempts',[]))>1 for r in results.values()),
'normalized_first_response':sum(bool(r.get('attempts') and r['attempts'][0].get('format_normalized')) for r in results.values())}
(args.output_dir / f'answers_{args.split}.json').write_text(json.dumps(output, ensure_ascii=False, indent=2) + '\n')
print(f'{len(results)}/{len(selected)} {row["id"]}: {"OK" if row["strict"] else "FAIL"} {row["latency_seconds"]:.1f}s', flush=True)
with create_neo4j_driver(settings) as driver, driver.session(database=settings.database) as session:
@ -161,6 +210,13 @@ def main():
'accuracy': sum(r['strict'] for r in rows) / len(rows)} for category in sorted({q['category'] for q in selected})
for rows in [[r for r in output['questions'] if r['category'] == category]]}
(args.output_dir / f'answers_{args.split}.json').write_text(json.dumps(output, ensure_ascii=False, indent=2) + '\n')
fields=['id','category','strict','error','latency_seconds','attempts','first_pass_strict']
with (args.output_dir/f'answers_{args.split}.csv').open('w',newline='',encoding='utf-8') as handle:
writer=csv.DictWriter(handle,fieldnames=fields);writer.writeheader()
for row in output['questions']:
writer.writerow({**{k:row.get(k) for k in fields if k not in {'attempts','first_pass_strict'}},
'attempts':len(row.get('attempts',[])),
'first_pass_strict':bool(row.get('attempts') and row['attempts'][0].get('strict'))})
print(json.dumps(output['summary']))

4
tools/answer_evaluation/graph_answer_prompt.txt
View File

@ -7,3 +7,7 @@ Title znamená výhradne názov akademickej práce alebo projektu; meno profilu
Pri výbere kandidátov a počte uvádzaj len zodpovedajúcich kandidátov. Použi vypočítaný count, nikdy počet uvedených zdrojov. Pri porovnaní ľudí daj každému samostatnú claim.
Source_urls musí obsahovať konkrétne relevantné URL zo zdrojov; neodhaduj ich a neupravuj. Cituj ich aj v answer. Uveď všetky zdroje potrebné na doloženie uvedených faktov.
Ak status kontextu je no_answer, oznám, že požadované údaje sa nenašli, status no_answer, claims prázdne. Nevymýšľaj žiadny názov ani pozitívny fakt.
Kľúčové slovo sa v tomto nástroji viaže na osobu cez jej vlastné aj dcérske dokumenty. Pri otázke na názov a kľúčové slovo spoj title, year a keyword do jednej claim pre tú istú person, aj keď majú odlišné document_path. Nevyžaduj rovnaký dokument pre tag a názov. Rovnaký dokument je povinný pri spoločnej kategórii a autorovi dokumentu.
Pred odoslaním pri otázke na názov a tag over, že claim s názvom obsahuje súčasne person, title, year aj keyword. Nerozdeľ túto požadovanú väzbu na samostatnú claim s názvom a samostatnú claim s tagom.
Pri rokoch porovnaj len akademické záznamy dodané v kontexte, nie poznámky o príprave. Pri počte nemiešaj nevyhovujúcich kandidátov do claims.
Vráť iba jediný platný JSON objekt bez úvodu, vysvetlení mimo JSON a bez Markdown ohraničenia. Reťazce ohraničuj ASCII dvojitými úvodzovkami. Vo vnútri answer názvy neobaľuj ďalšími ASCII dvojitými úvodzovkami; ak ich použiješ, musia byť escapované. Roky a count sú celé čísla. Nevypisuj čiarku za poslednou položkou.

39
tools/graphrag/evaluation/README.md
View File

@ -1,6 +1,43 @@
# GraphRAG evaluácia
Aktuálna zmrazená verzia je **v4**. [Finálna správa](results/final_v4_2026-10-25/REPORT.md) obsahuje protokol, výsledky a obmedzenia.
Aktuálna verzia je **v5.1**, tag `graphrag-v5.1-frozen`.
[Finálna správa](results/final_v5_1_2026-10-25/REPORT.md) a
[strojový súhrn](results/final_v5_1_2026-10-25/summary.json) oddeľujú
vyhľadávanie, zdrojovú kontrolu všetkých vrátených typovaných faktov a generovanie.
- Graf DEV: **722/722**.
- Graf historická regresia: **320/320**, vrátane kontroly nadbytočných faktov.
- Generovanie: 39 vopred deklarovaných DEV a finálna regresia všetkých 320
historických otázok. Finálny výsledok: **317/320 (99,06 %)**, bez chýb JSON
a bez opravného požiadavku. Všetky tri zlyhania zostávajú započítané.
- Kompletné automatizované testy: **525 passed, 4 skipped**.
Všetky otázky sú už známe. V5.1 je regresia, nie nový nezávislý TEST; finálna
regresia generovania obsahuje aj 39 DEV otázok. Pôvodné gold ani výsledky sa
nemenili. Prvá iterácia v5 (319/320) zostáva uložená a má overený zdrojový patch.
Reprodukcia v5.1 vyžaduje zodpovedajúci release checkout, raw wiki, JSON vstupy
aj graf podľa manifestu. Používajte vždy nový výstupný adresár:
```bash
python -m tools.graphrag.evaluation.evaluate_graphrag \
--split test \
--audit tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json \
--freeze-manifest tools/graphrag/evaluation/frozen_v5_1.json \
--output-dir /tmp/graphrag-v5-1-reproduction
```
Nové zmrazenie po vývoji používa kompletný DEV a `--purpose historical_regression`.
Odtlačky zahŕňajú spoločné moduly aj efektívnu publikačnú politiku. Každý reálny
DEV výsledok musí prejsť nezávislou kontrolou všetkých vrátených typovaných faktov.
Pri ďalšom nezávislom experimente treba nové, vopred zdrojovo overené otázky.
## Historický protokol v4
Nasledujúce merania a príkazy opisujú pôvodnú v4. Na reprodukciu použite jej
pôvodný checkout/graf; aktuálny kód nezodpovedá odtlačkom frozen_v4.
Historická zmrazená verzia je **v4**. [Finálna správa](results/final_v4_2026-10-25/REPORT.md) obsahuje protokol, výsledky a obmedzenia.
| Meranie grafového vyhľadávania | Správne | Úspešnosť |
|---|---:|---:|

3
tools/graphrag/evaluation/check_openwebui.py
View File

@ -19,6 +19,7 @@ from ..graph_search import search_graph_with_session
from .benchmark_v2 import BASE, academics, supporting_documents
from .reproducibility import graph_fingerprint, run_metadata, verify_freeze
from .source_oracle import load_source_oracle, norm
from .source_validation import claim_issues
SYSTEM = '''Odpovedaj po slovensky výhradne podľa dodaného výsledku grafového vyhľadávania.
Nedoplňuj vedomosti ani údaje z otázky bez potvrdenia v kontexte. Pri no_answer oznám,
@ -59,6 +60,8 @@ def grade_claims(case: dict, answer: dict, oracle: dict) -> dict:
if claim.get(k) is not None) for r in records)
if not valid:
unsupported.append(claim)
elif claim_issues(claim, oracle):
unsupported.append(claim)
urls = answer.get('source_urls', [])
sources_ok = isinstance(urls, list) and all(any(a['source_url'] in urls for a in proof['any_of'])
for proof in graph.get('expected_evidence_documents', []))

6
tools/graphrag/evaluation/evaluate_graphrag.py
View File

@ -25,6 +25,7 @@ from .graphrag_evaluator import (
load_questions,
)
from .reproducibility import graph_fingerprint, run_metadata, sha256_file, verify_freeze
from .source_oracle import load_source_oracle
EVALUATION_DIR = Path(
@ -380,6 +381,7 @@ def main() -> None:
all_questions = load_questions(
args.questions
)
source_oracle = load_source_oracle()
ids = {
value.strip()
@ -532,6 +534,7 @@ def main() -> None:
latency
),
error=error,
source_oracle=source_oracle,
)
rows.append(
@ -584,6 +587,9 @@ def main() -> None:
'evaluation_schema_version': 4,
'run_metadata': metadata,
"configuration": {
'evaluation_kind': (json.loads(args.freeze_manifest.read_text()).get('evaluation_kind', 'independent_holdout')
if args.freeze_manifest else 'development_regression'),
'published_only': True,
"questions_file": str(
args.questions
),

22
tools/graphrag/evaluation/freeze_graphrag.py
View File

@ -9,7 +9,10 @@ from pathlib import Path
from .reproducibility import content_fingerprints, sha256_file
def make_freeze(dev: dict, questions: list[dict], fingerprints: dict, audit: dict) -> dict:
def make_freeze(dev: dict, questions: list[dict], fingerprints: dict, audit: dict,
*, purpose: str = 'independent_holdout') -> dict:
if purpose not in {'independent_holdout','historical_regression'}:
raise ValueError('Unknown evaluation purpose.')
expected_ids = {q['id'] for q in questions if q['split'] == 'dev'}
actual_ids = [row['id'] for row in dev['questions']]
if dev['configuration']['split'] != 'dev' or set(actual_ids) != expected_ids or len(actual_ids) != len(expected_ids):
@ -18,6 +21,10 @@ def make_freeze(dev: dict, questions: list[dict], fingerprints: dict, audit: dic
raise RuntimeError('Freeze requires DEV without runtime errors.')
if dev['run_metadata']['fingerprints'] != fingerprints:
raise RuntimeError('Code or data changed after DEV; rerun DEV before freezing.')
if fingerprints.get('retrieval_policy') and any(
row.get('source_checked') is not True or row.get('all_facts_supported') is not True
for row in dev['questions']):
raise RuntimeError('Every DEV result must pass independent validation of all returned facts.')
if audit['dataset_sha256'] != fingerprints['dataset_sha256'] or audit['inputs'] != fingerprints['inputs']:
raise RuntimeError('Source audit does not match the benchmark and inputs.')
if len(audit['questions']) != len(questions) or {r['id'] for r in audit['questions']} != {q['id'] for q in questions}:
@ -33,7 +40,12 @@ def make_freeze(dev: dict, questions: list[dict], fingerprints: dict, audit: dic
for profile in question.get('graph', {}).get('source_profiles', [])}
for split in ('dev', 'test')}
policy = audit.get('partition_policy', {})
if policy.get('method') == 'disjoint logical query groups':
if purpose == 'historical_regression':
partition = {'method': 'Previously exposed questions; regression, not independent TEST',
'dev_profile_count': len(profiles['dev']),
'test_profile_count': len(profiles['test']),
'source_profile_overlap': len(profiles['dev'] & profiles['test'])}
elif policy.get('method') == 'disjoint logical query groups':
# This mode explicitly reports known profiles. It must never be
# presented as the stronger, historic unseen-profile experiment.
from .benchmark_v4 import logical_group
@ -65,7 +77,7 @@ def make_freeze(dev: dict, questions: list[dict], fingerprints: dict, audit: dic
partition = {'method': 'disjoint source profiles',
'dev_profile_count': len(profiles['dev']),
'test_profile_count': len(profiles['test']), 'overlap': 0}
return {'schema_version': 2 if schema3 else 1,
return {'schema_version': 2 if schema3 else 1, 'evaluation_kind':purpose,
'partition': partition, 'frozen_at_utc': datetime.now(timezone.utc).isoformat(),
'fingerprints': fingerprints, 'graph': dev['run_metadata']['graph'],
'parameters': {key: dev['configuration'][key] for key in ['limit', 'max_entities']},
@ -79,11 +91,13 @@ def main() -> None:
parser.add_argument('--questions', type=Path, default=Path(__file__).parent / 'graphrag_questions_v4.json')
parser.add_argument('--audit', type=Path, required=True)
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--purpose', choices=['independent_holdout','historical_regression'],
default='independent_holdout')
args = parser.parse_args()
if args.output.exists():
parser.error('Freeze manifest already exists; choose a new version/output path.')
manifest = make_freeze(json.loads(args.dev_results.read_text()), json.loads(args.questions.read_text()),
content_fingerprints(args.questions), json.loads(args.audit.read_text()))
content_fingerprints(args.questions), json.loads(args.audit.read_text()), purpose=args.purpose)
manifest.update({'dev_results_sha256': sha256_file(args.dev_results),
'audit_sha256': sha256_file(args.audit)})
args.output.parent.mkdir(parents=True, exist_ok=True)

233
tools/graphrag/evaluation/frozen_v5.json Normal file
View File

@ -0,0 +1,233 @@
{
"schema_version": 2,
"evaluation_kind": "historical_regression",
"partition": {
"method": "Previously exposed questions; regression, not independent TEST",
"dev_profile_count": 72,
"test_profile_count": 72,
"source_profile_overlap": 72
},
"frozen_at_utc": "2026-10-25T16:08:00.798503+00:00",
"fingerprints": {
"implementation": {
"tools/graphrag/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/build_graphrag.py": "256292101d6d7d1946e926e2717ac45465b169f4091c51ecfed4f1a779dfd700",
"tools/graphrag/evaluation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/evaluation/audit_benchmark.py": "6619c818a52fc284e87ed6c48a503a54a3c9dd95447463ed78f075b7c58bad80",
"tools/graphrag/evaluation/benchmark_v2.py": "129dd774faa414f06733e1ee6333e7b411b1c75b5601b895c405778df478a563",
"tools/graphrag/evaluation/benchmark_v3.py": "30bae6c605ed26f4b49973acec799087c4e2b97a883ccedcafb965dd909c4c4d",
"tools/graphrag/evaluation/benchmark_v4.py": "70a8b08f7c2b2cb3e2bdc77be9efedd23ea24ac564ec1c1eb224a9f6388b3a67",
"tools/graphrag/evaluation/check_openwebui.py": "d6b5d9947679aeabf4ec9599a7ad70b5ee917dd7c6ad747ddbf54579d8de475b",
"tools/graphrag/evaluation/evaluate_graphrag.py": "89c364d15b70c9cb55df39d5970b4d0d79b681ba70eed7530a4e3fe64167812e",
"tools/graphrag/evaluation/evidence_contract.py": "78e24264c176e163e1782fef1634b6c45c7f31bd4692726d02a30ab45a06648e",
"tools/graphrag/evaluation/freeze_graphrag.py": "793c6d04a13c3217347283c609f647cd0d69b020e44bbde458bdc7ebbab8ec7e",
"tools/graphrag/evaluation/graphrag_evaluator.py": "7f99bf0f97974058650b619ea49a8ee32284cb4b324f1ffab9baf54f5c4495f4",
"tools/graphrag/evaluation/reproducibility.py": "f6dde1a2a6f68abd0e916dfb38d5eb756a7f8621a66b7c11e43e7580b347134a",
"tools/graphrag/evaluation/source_oracle.py": "bfe8375556d64975433ff098e8d227aecbe59afd4e4fbe5745054524e913f3da",
"tools/graphrag/evaluation/source_validation.py": "042a3f99b18c028d88d1ecb6bf2430b8979be3d7414c19ee506bd245082bff46",
"tools/graphrag/graph_db.py": "c4b1a2edb765856b807398e4c243a8d17753529a5d2dfdb51791e28bc8de17f7",
"tools/graphrag/graph_search.py": "8cc3668b4ad7beabf849e6978ea1d5defc818b142ee8b157b19c319998fb7541",
"tools/graphrag/graphrag.py": "128f888a0c76d06457af8c1dd1cf56fc1955e25b976f367613f0829f7e990ee6",
"tools/graphrag/neo4j_smoke.py": "fb5244c72c29ef2acf96d1e7f54335ff168eed4119db1692025eb1b99194b7bb",
"tools/graphrag/publication.py": "fba1fa0d0bed325fd87f8d3593baae9271de1766b772d7214c7f63fd3f1c37b6",
"tools/graphrag/tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/tests/test_evaluation_pipeline_v2.py": "63d1b3e80ae3c7b563374fd7ab4aa5ffad3c614eafbd15142c577abd7f756bf9",
"tools/graphrag/tests/test_evaluation_v5.py": "0475ebd4dc28d3e940ab499a7bc4ef713dc4c679d25b650161b78d1886f3bc0b",
"tools/graphrag/tests/test_evaluation_wide.py": "1b28bb74fd6b9457c869186b88606f1f7863d573aa3ab5226162e878f236d99e",
"tools/graphrag/tests/test_evaluator_connected_v2.py": "29d7c9838ed1ff37ea3276df6feb1b6142158c43a0525712f490b648181882de",
"tools/graphrag/tests/test_graph_publication.py": "96e53c1ae28e36c7a4290e1e60c8b398bf095ceaf883e819e1a477e2672b11c0",
"tools/graphrag/tests/test_graph_rebuild_atomic.py": "fe37d4fff0b1f0fb1d565f0a96152b9b4e336837bc78b5977cf3b4879d7fbaf3",
"tools/graphrag/tests/test_graph_search.py": "eb58661aef9dafe6bf7188fcdc2d79cef6652d57c64d1b81ae81a52e8e0204d7",
"tools/graphrag/tests/test_graph_search_entity_v2.py": "0ef1a8ebd513c5a4507eb38fcf5ae6137c69e9901b5a2975ba513be5902ac693",
"tools/graphrag/tests/test_graph_search_neo4j.py": "fec2bf72de8f752f17c9164fee5754bdb2e153828855e47efbf0000dbe82cf3b",
"tools/graphrag/tests/test_graph_search_regressions.py": "9db7779aeb8ed9e8c0f92c1657011a81080aff3c928649e811358e3009a18c48",
"tools/graphrag/tests/test_graphrag.py": "9ee8629d6b528dd78c2ddfc25cee95e17613f45b454fc0a38d4c591862637a9f",
"tools/graphrag/tests/test_graphrag_evaluator.py": "1c6d08aeec9ae56ac4123239a5aed12a0081d66cc3f34b2047d95764579d7298",
"tools/graphrag/tests/test_graphrag_freeze.py": "c4b760baf4c7ebf13b16bd4cd3c51874f5512eef3b4c61470dc4d428b0303d9f",
"tools/graphrag/tests/test_graphrag_research_v2.py": "711008a91ab8a58ae0854e7f90329bcd42e5189fde2e2001589e97b0c22c5198",
"tools/graphrag/tests/test_graphrag_root_title.py": "085651fe6e6081521a5dde9554abfee93456250108cf15978df25378d886a46c",
"tools/graphrag/tests/test_source_oracle_v2.py": "0afdbe235e185ffe9f38aa667f90f5a6c20504fe29e9ac572642b39a8eb7faca",
"requirements.txt": "26a6d097727eba03c1b6c71cd02b06373ef01ce03e567dd2bc4357ae330e5a1a",
"tools/publication.py": "f99d445073390a733afc8d849b9fcd4b4b0890bb54d136db67d4015b78436e1a",
"tools/retrieval_contract.py": "b8fa272c3babf924a3ce8b937fe7ae437cd959ba9f7667193ac67c18d2c5b58d",
"tools/index_state.py": "c33920a63d04009f11ef23ca9e9eee9bf86453b7d0586d027bf3b39156afea0d",
"tools/rag/common.py": "8497c46d5c6c27a7c76df1568360ad5e9d00d5e5a7ef186e758960cc79d81d1c",
"tools/sync_indexes.py": "5b909b4394c24e5ce22848594b16d9aea45fb06418f5cd92bb6aa6381a0ccf10"
},
"dataset_sha256": "9a3b873882bbf29b905d79c9448c3c287fda30323f93a6533b88dbd59b19f558",
"inputs": {
"data/documents.json": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"data/chunks.json": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9"
},
"source_corpus": {
"sha256": "440da12a7a991824e923f70cada7c616c3fb3b0f12767ec7d75039a6a3231b10",
"document_count": 114,
"documents": {
"pages/interns/bogdan_paul_chis/README.md": "3b4b946eb3c10e9f9a4092e6cffa01c5080e7579aded2f6ed3120c41ed62b647",
"pages/interns/cesar_gutierrez/README.md": "b0be60576eed28d4b1920e9acf198684519373dfc13171567247c853b3ed699e",
"pages/interns/oliver_pejic/README.md": "0470f3c8465be8f02204b08e4c75ef47f68c7e4ad1038faa5641cdf1811f33d5",
"pages/interns/sevval_bulburu/README.md": "ff2a9d66253802f1af0fdfd74832e40fae41b6df23c5e61dc266f6cd0cc16c4d",
"pages/interns/yussef_ressaissi/README.md": "f997a65699da0db3222ef7a3c37652d05dfc3419ad4ec74683bafc2c897bb987",
"pages/students/2016/darius_lindvai/README.md": "fcaa47bb9708a6439eec5c4b7f7b4ed6c7359e423e1d940e1aed9f44957e6d62",
"pages/students/2016/darius_lindvai/dp2021/README.md": "be3d0fea2a6d592b225eecabd9b19445c2fce62b8289c07ab6af93835db12970",
"pages/students/2016/darius_lindvai/timovy_projekt/resers/README.md": "352162de1b1e192228fa5d26506c028ac47e567e90df53b1bbaa6ad736ec4766",
"pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md": "b4a81ff457624b089d618e4479afb1e3d748f6724f72104ef5518e535ae02fe5",
"pages/students/2016/dominik_nagy/README.md": "1c4037a158d84c45fa705a0c38e47142a9d46e226f0c138bc2d56d7c38f6d74d",
"pages/students/2016/dominik_nagy/timovy_projekt/README.md": "b81a4dab7316f7b846dcf36515f6aa9d44035c86f46f0e07b7a43308bcd31458",
"pages/students/2016/ediz_morochovic/README.md": "045e5eb9a3df2b84140808fbca0c3163266ad0efa144bc42d9d5c597881c6ac6",
"pages/students/2016/jakub_maruniak/README.md": "6734c41a95b9d0c1e3d226b393ad109ee4a7b82ebfffa73df2d7e0a743fefa14",
"pages/students/2016/jakub_maruniak/dp2021/README.md": "9cd884aec70705cdb2aaa0f514b513f16de263c37b749ced7f92916cd2f91c39",
"pages/students/2016/jakub_maruniak/dp2021/annotation/README.md": "debae0c50aea1accef0840912b9b1f37b2138f856002bb50e91d7ea2dbbe7963",
"pages/students/2016/jakub_maruniak/dp2021/train_experiments/README.md": "01dff38b37043414fa1db204394ddbe7aade00e14d02d495572b65c85d09b194",
"pages/students/2016/jakub_maruniak/timovy_projekt/README.md": "04114b1f3d84664aad5bdba5a0b614c5a6f23392be3c31fd9123d63db1be0a35",
"pages/students/2016/jan_holp/README.md": "80d6a8492766e5bfa61673a372e5f059d30c8e335de39842866c51485960e4d3",
"pages/students/2016/jan_holp/dp2021/README.md": "a62acb74009376766c7e2f9856ee01d21f84593eb9b3ce85cf2e6032aa412476",
"pages/students/2016/jan_holp/dp2021/zdrojove_subory/README.md": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"pages/students/2016/jan_holp/timovy_projekt/README.md": "79eebdfd3af7e8d5c279077e285a580731b1cf8cc9b4454f6596796e9915f70c",
"pages/students/2016/lukas_pokryvka/README.md": "f279456c5c277ff1ee0dd89a3464ed9f9f67b44bc185a5da9811ef5cfb675002",
"pages/students/2016/lukas_pokryvka/dp2021/README.md": "39b61376ce108e4a930f926ab2d6a997207457121bccfcf617129a3b210a15cf",
"pages/students/2016/lukas_pokryvka/timovy_projekt/README.md": "4de4b6c8cfed17b6c60090f82df1d55a66be70a79d3013850689b2baebdd801b",
"pages/students/2016/mark_feher/README.md": "bec37c0319fe0869a0377d771db23d090dbda93c27328f5697cdaeb3bbccfe9b",
"pages/students/2016/maros_harahus/README.md": "0da150e91918632c9e8cd1443f04da403595a372517e254928ce9f76c9d5033d",
"pages/students/2016/maros_harahus/timovy_projekt/README.md": "75af19f7c68f4b646fa1dbae75f3f3dfc52633a9f9da1400d21808bb20be0d83",
"pages/students/2016/patrik_pavlisin/README.md": "dc1c648d7cd9b0f0e021c57f9b98189e37c100871439e7e304390715913f2763",
"pages/students/2016/patrik_pavlisin/dp21/README.md": "ac1eb0fd0360ad34aa21b7c8da5bb010a6edf10be5721f28dedc3986cd1d8c54",
"pages/students/2016/patrik_pavlisin/dp22/README.md": "f2a8c619a812b99abab618ae388eb672407950e1f33fb8d7da7a528c4de81d50",
"pages/students/2016/patrik_pavlisin/tp20/README.md": "ae9c771f27e7e093bc29c6888c8d0b27ed61fb41519fd0d7e7f4f6cbb4626edc",
"pages/students/2016/tomas_kucharik/README.md": "6a1a215433eb1857660177b4cb2ecec03c6a86743d3a04cd6d5c9d9682c9e63b",
"pages/students/2017/dmytro_ushatenko/README.md": "65f4ac177d492162b08ca5da3d9eb048bd088a3c117a28a662113b7e4bc3a4ec",
"pages/students/2017/martin_jancura/README.md": "6a0a18d5095dcfb12abfdd72c9cfd8025d714fb5e29a7858cd2b285e4cb986b9",
"pages/students/2017/stanislav_matsunych/README.md": "90ffa11eff4a4360b0252dbc53a116c3a5777b3d5dd86ce5c89c02bdd2051c73",
"pages/students/2017/than_trung_thanh/README.md": "2150519b8580e7337520db4de64d2d1c82d2634380537d019f41c506dbf2979d",
"pages/students/2018/NaN1name_buday/README.md": "23c8c58090bdc954347e36f141bea91f43c8d5b1d777e346864a2ebe84c0bcce",
"pages/students/2018/adrian_remias/README.md": "e5bfe1d5d4e626beaa636a02d34ec58cf2cff9b0e57833905af35b80bf1465a2",
"pages/students/2018/david_ilas/README.md": "96c45ff9c5a821ddcb1a2fc86d487bd8f4729d1b76887914b0f5fbf99af45af0",
"pages/students/2018/david_omasta/README.md": "4164051870e1e1cf8949ed18a74196e9ea789dfbe4fee24bdeb736ca42848327",
"pages/students/2018/martin_wencel/README.md": "0e6a7a981689f22304f8baa5de97d4315c0695c94c9277de7851943eb012253b",
"pages/students/2018/oleh_bilykh/README.md": "4072c62dd8ee753a0c8fa43fdfbbdf566e394c2df13e6a79d08a302e332710b1",
"pages/students/2018/ondrej_megela/README.md": "969ff2e1a436b8f957e7168cf5eb678196b99aeda17701c502b016441f644065",
"pages/students/2018/samuel_sirotnik/README.md": "cc230ede67b6e8632511c92a5ed6d1e9deb42e70819280222bdfd369ee77fc70",
"pages/students/2019/alina_vitko/README.md": "9b071fe2aff470c275ae027be236fe836f741f774cabb9cbbb55304a8d5102cd",
"pages/students/2019/artem_yatsenko/README.md": "328b101e46c51c959e792cf7ca226942f81d2ab8fe9648ad6d0ecd07016429e2",
"pages/students/2019/david_stromp/README.md": "e46ca8691d5ec894128f6f6f2ed33e2c7bd141f80719bb0449b20f10f30cca09",
"pages/students/2019/dmytro_mural/README.md": "e881a47d847c3ef8d680844113fdbfbb87553b28982bf9eef4936f6d30923e93",
"pages/students/2019/filip_tomas/README.md": "75a1fe7b921e4bff0fdf9e2a957b4adad4b4a57aa7ebdffbb73dd82c02123f94",
"pages/students/2019/kristian_sopkovic/README.md": "3286960fb30f584df35455f9b6dfae81ca4827a1d711d2bcd6e48c3f150c9279",
"pages/students/2019/michal_stromko/README.md": "3ae1dd9b6638a6a249a94be3f3846286555666aefa225bb3a2c7288b35900ce7",
"pages/students/2019/michal_stromko/vp2021/README.md": "feb2e093fdd6461239e79b1e1f96e65a1c8cd96e8a2b70535ae556a9c198046a",
"pages/students/2019/michal_stromko/vp2021/tutorials/README.md": "40e3fa448c7329ef863e705435e23fe6b4788de440710f066fe0eed92d132bad",
"pages/students/2019/michal_stromko/vp2021/tutorials/img/README.md": "e8d3ac160b17dfdee2c3e996bf6a03af7a6f16f9140bf5cf7ee343a82dd1865f",
"pages/students/2019/patrik_pokrivcak/README.md": "3e3957ad0f7aca59d6eb64c38a3f5e21d8fd953200825d0d0e5cacabfbc35a7d",
"pages/students/2019/samuel_horani/README.md": "86e6959ccc612b069687b57996fa3f61660075d6a0d65bfec64f423d39a0a999",
"pages/students/2020/david_kostilnik/README.md": "2927524b75da59a64120fc364a0fc374986fa4d364d54e52c36800762c203264",
"pages/students/2020/jakub_kristian_lukas/README.md": "f139d9b1ebf5a3f45612f3df5703ecdd935b90701763c27c7c529044c0b0455a",
"pages/students/2020/kamil_tomcufcik/README.md": "9d865adf538bc6d7992ee4f4882713447da5c936fca17c43ca0f576072537048",
"pages/students/2020/matej_kobylan/README.md": "3d7b289e17b8f25678a07e9ee0e6741688485a0ccb1e50940a6a6acc6312a547",
"pages/students/2020/matus_suchanic/README.md": "770ca49538cca22df35f8e7ba736ee97483e4fe5788c221496553592b4e78dc6",
"pages/students/2020/pavol_hudak/README.md": "33d06fccc6451e5f19b27b0cad001a34e81635b4ea294a01dec7771c823abc40",
"pages/students/2020/vladyslav_krupko/README.md": "4c28e6d38651bb2cafcbf4a18e35210cd369aef2caff6c5b0aa52fac38f0fe7d",
"pages/students/2021/artur_hyrenko/README.md": "0568261f4ade4de5472edcfeb1ffed9abae18dc67e4050f4ce2ad088839a2bb3",
"pages/students/2021/eduard_matovka/README.md": "de2a7ae38ec87110e347956b37d0605a298ed2847f972961cc9b977da8221d43",
"pages/students/2021/manohar_gowdru_shridharu/README.md": "c00ceb620ed7087ead5d9448d476bbb98d273d1d9e7bf1bc4355c367b26c7a6b",
"pages/students/2021/martin_sarissky/README.md": "8e97b45a253745faa55f63c844e0b5dfe680aeaf6eb01fb8f9d06ac6e8ee8bcf",
"pages/students/2021/matej_novotny/README.md": "b083a7cadde532e3428dc9a57bc61456fcc5905bda91e011b96033d42bf5d9cb",
"pages/students/2021/matej_scislak/README.md": "b9a32cce8605e044a4f2828ff2d91a2995a62ef3a841e62f0a225c97221f0cc8",
"pages/students/2021/nikita_bodnar/README.md": "2016c0dd923c0f8e561d5cea42ac81da3610fdcc54a4527f56d078e34e53ce90",
"pages/students/2021/oleksandr_hryshchenko/README.md": "4b073992b715b6c75b79fd2c8f850a69f940966e3a61a82c625c32c945a93036",
"pages/students/2021/vladimir_ferko/README.md": "68e7d311896eda2171e5eec1ec3a8da90d326ed91acaace8058f68664a50d271",
"pages/students/2022/andrii_pervashov/README.md": "319dcaa1cc2e29dd7e3a710bab26cb3ac6944ab71b0b97e1de076b1199122334",
"pages/students/2022/daniil_huzenko/README.md": "390782fbff6220a9462060ce1a62959f2a2b88f42d06e2c51d1f406b75d73f2c",
"pages/students/2022/jakub_schwarc/README.md": "f541a8852c2b0f6f9d7182e01a34c4e3beee9bfa55427a06b331638b11eb8be6",
"pages/students/2022/jan_malinovsky/README.md": "8af021bfb131fa005a7aec3e2f2f12fc17fa5dc14e2675311adbbfc445a4bcab",
"pages/students/2022/jan_ptak/README.md": "131cf053764fd5423541ebe672c3b4f5da875cfb8e4b821f1335ddb967282282",
"pages/students/2022/oleh_poiasnik/README.md": "c2f7dc3f61f8b4a5b43ded1e3ae4c7d3e7f874439ee6b17dd9c8d45dabc25bc8",
"pages/students/2022/samuel_vasil/README.md": "67c807049147aacd04f3050f4ff0233a8b3f0a239ab173b430e13dfe113ad0a5",
"pages/students/2022/serhii_yemets/README.md": "ffbeac893947acafe855d576322a8826f9d4f87ac344cb31f8ceeefa244b9133",
"pages/students/2022/tetiana_mohorian/README.md": "c9efdc632bf1ca89a0bfde0dc00be7aaac8a85da3d92e2ce8506236c73e93cfb",
"pages/students/2022/valerii_kutsenko/README.md": "dc764c609ad3c1df59cfea997d4db82a591b1b1e0764453e319d841115507e4f",
"pages/students/2022/vladyslav_yanchenko/README.md": "4d1ce216e3b526ee437ddbc658e51d7bb83530087e43628c096a569a3854a47f",
"pages/students/2022/yevhenii_leonov/README.md": "b90e41db0cec2451027020a9102d32a24d777e8a38b604ccca87b36cd6b3a292",
"pages/students/2023/denis_le_thanh/README.md": "2cb83f942d86a02712b0c58778e46dc2211346d1a6eba606471dfa832245db61",
"pages/students/2023/maksym_zatirka/README.md": "7a7d2cbe276c352dacfbc6ba6de5f15303d30ecf34845ff8da8d085f3d67c3dd",
"pages/students/2023/oleksandr_dorybohov/README.md": "c24d6f941de0dafb4459cea8259cf147821cafc86ba65be5b528f419f734bc18",
"pages/students/2023/ronald_zalacko/README.md": "724b0f8e6e641236dba9bddbd77f4546e09514076c7dfb151807aaf4b882c96f",
"pages/students/2023/rostyslav_rodzhuk/README.md": "de356dd7427c8a0716c165a0f88ae47b691a86ee7a388f60e2d2e9306298415c",
"pages/students/2023/samuel_dzurina/README.md": "4818191d044b91219c796a35010c95da5cf2093e050d494f30b219b91e13fa0e",
"pages/students/2023/simona_bobrovcanova/README.md": "e473a1911dae10ab40e2cf6696a8be616a58c22aaf09af5953fbac475791a35e",
"pages/students/2023/yevhenii_medushivskyi/README.md": "5710ebf91c893a12fe6d19f5372311938d1e7c7bbdf203f2dc8ed2fe7c4f5a24",
"pages/topics/akopisat/README.md": "c7245e1fbfa4160618502c0baf8cc6bc47cd9a82b8a85bc1c645314829a7b76a",
"pages/topics/bert/README.md": "35e507f7ec2ad1d3526e2a15f94ea085ee25f680a02daf6df62fd429bd9c25c9",
"pages/topics/chatbot/README.md": "6f2f83ffb80109be47f9750cca9d3036ec3645abef9f2b590bd6825780c7c48a",
"pages/topics/faq/README.md": "15cfbd9c2a1649c80a7a58922e2b66dadab5c86a5ef33ef16c82bebd4d2dbf06",
"pages/topics/hatespeech/README.md": "364df94e70dd740c63544e2586ba7b957780f714fc0c941d1b04b02233e6cbc5",
"pages/topics/legal/README.md": "003bb9bfe1e0ca21652091c2d11496312eb04e0689fe459e14c2d5a738e8eeea",
"pages/topics/morphology/README.md": "e242cc5ff1a4cc98e227566c0ead999bbc95937b7c20be62cb2afb773d0d35c0",
"pages/topics/named-entity/README.md": "66b2e25f70211d432d596e377d552b3b8f0c525b14c31e3bc9c956c3aeadca84",
"pages/topics/named-entity/navod/README.md": "1ff87079a98411de0414754c60241e08d2214ddcd63ddf974e4340cce2ce5b02",
"pages/topics/nlp/README.md": "6f90abf9dc3f0ff33475283c13b4a08561abf631425ec2daacb076155a9e02c2",
"pages/topics/otvorene/README.md": "cd5544925553fe3840c0ea6f8ef60ab756e26ab6757bf810025c948d8d4f98fc",
"pages/topics/prodigy/README.md": "5e4f691f2754c2ec4afbaab2f41e6db77feb5b322dd9344465eb40c35f116499",
"pages/topics/python/README.md": "2ef073adba672142c39e9a59fa2e00a98ab52bda54233d46fbe6940ee23f9fcc",
"pages/topics/question/README.md": "408078a2a67cea2699e12359e85ab641136231e6104c2afdf874a705879525df",
"pages/topics/question/navod/README.md": "061e737c69184b2bd244316ca7ba841f7186fe01ee80ee993742a733dcad3ecd",
"pages/topics/question/nezodpovedatelne/README.md": "b6703d9690f4599b973efb27595ce157773ae0fc0d55cf2c0a09d9f50e8e4c92",
"pages/topics/question/validacie/README.md": "3216a7127abdfc6da50ed116f44ff2d6785e7da4af70d23aceda774e745ad562",
"pages/topics/resources/README.md": "09405397387f00cbcc5680f7d655f2d861e4d92e5df792a62d9dfd04f3534f48",
"pages/topics/seq2seq/README.md": "969d15c03a63a3f8d5edc9017fe5a0fb7311fdf4de4a3473a08266241f92c1e2",
"pages/topics/spacy/README.md": "f504e5db77cb8694f08a57fb8636e522d59a0e669216f4bc8273bc3c5f375f87",
"pages/topics/steel/README.md": "b4068cef2ed197711467ac47630d5b2064fdc8828f898f7af99664e94d833330",
"pages/topics/translation/README.md": "a7d4289e72199b307dc5ad8fc69cda39c1a33c27a6c7447af70aa9c3cc6b8635"
}
},
"retrieval_policy": {
"published_only": true,
"publication_policy": "explicit-false-hidden-v1",
"academic_preparation_is_work": false
}
},
"graph": {
"sha256": "f5f85ce72994baa1c4df62c0423fd6f3a349ef7de2174a65184a6dc4628e1cdb",
"nodes": 446,
"relationships": 877
},
"parameters": {
"limit": 40,
"max_entities": 8
},
"dev_summary": {
"questions": 722,
"source_checked_count": 722,
"all_facts_supported_count": 722,
"strict_count": 722,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.041665,
"latency_median_seconds": 0.040584,
"latency_p95_seconds": 0.050737
},
"source_audit_summary": {
"dev": {
"questions": 722,
"source_consistent": 722,
"issues": {}
},
"test": {
"questions": 320,
"source_consistent": 320,
"issues": {}
}
},
"policy": "Report all questions including negatives. No exclusions or holdout tuning.",
"dev_results_sha256": "5a35b798860cac4376e6b323956eb4e1437469062deec99687c953b147d238f0",
"audit_sha256": "091e2ee00a88e9e6a459e91fb54efabd23eec5eb4dd70570aa313c3e51e93410"
}

233
tools/graphrag/evaluation/frozen_v5_1.json Normal file
View File

@ -0,0 +1,233 @@
{
"schema_version": 2,
"evaluation_kind": "historical_regression",
"partition": {
"method": "Previously exposed questions; regression, not independent TEST",
"dev_profile_count": 72,
"test_profile_count": 72,
"source_profile_overlap": 72
},
"frozen_at_utc": "2026-10-25T16:14:12.663856+00:00",
"fingerprints": {
"implementation": {
"tools/graphrag/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/build_graphrag.py": "256292101d6d7d1946e926e2717ac45465b169f4091c51ecfed4f1a779dfd700",
"tools/graphrag/evaluation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/evaluation/audit_benchmark.py": "6619c818a52fc284e87ed6c48a503a54a3c9dd95447463ed78f075b7c58bad80",
"tools/graphrag/evaluation/benchmark_v2.py": "129dd774faa414f06733e1ee6333e7b411b1c75b5601b895c405778df478a563",
"tools/graphrag/evaluation/benchmark_v3.py": "30bae6c605ed26f4b49973acec799087c4e2b97a883ccedcafb965dd909c4c4d",
"tools/graphrag/evaluation/benchmark_v4.py": "70a8b08f7c2b2cb3e2bdc77be9efedd23ea24ac564ec1c1eb224a9f6388b3a67",
"tools/graphrag/evaluation/check_openwebui.py": "d6b5d9947679aeabf4ec9599a7ad70b5ee917dd7c6ad747ddbf54579d8de475b",
"tools/graphrag/evaluation/evaluate_graphrag.py": "89c364d15b70c9cb55df39d5970b4d0d79b681ba70eed7530a4e3fe64167812e",
"tools/graphrag/evaluation/evidence_contract.py": "78e24264c176e163e1782fef1634b6c45c7f31bd4692726d02a30ab45a06648e",
"tools/graphrag/evaluation/freeze_graphrag.py": "793c6d04a13c3217347283c609f647cd0d69b020e44bbde458bdc7ebbab8ec7e",
"tools/graphrag/evaluation/graphrag_evaluator.py": "7f99bf0f97974058650b619ea49a8ee32284cb4b324f1ffab9baf54f5c4495f4",
"tools/graphrag/evaluation/reproducibility.py": "f6dde1a2a6f68abd0e916dfb38d5eb756a7f8621a66b7c11e43e7580b347134a",
"tools/graphrag/evaluation/source_oracle.py": "25aba0ebd6af1c94ec42e4d5044bad55ae0964c89ed3f06aa63a0959ab2d7ddb",
"tools/graphrag/evaluation/source_validation.py": "9671b8978623888dbed8b8b2d1b6b01c1ce73cb0a512b03919e48873990f15ed",
"tools/graphrag/graph_db.py": "c4b1a2edb765856b807398e4c243a8d17753529a5d2dfdb51791e28bc8de17f7",
"tools/graphrag/graph_search.py": "8cc3668b4ad7beabf849e6978ea1d5defc818b142ee8b157b19c319998fb7541",
"tools/graphrag/graphrag.py": "3a5ae3bfd12ef171d650548cb26c5708b472424a11cd8cb9ed64e21c9ab5bbd2",
"tools/graphrag/neo4j_smoke.py": "fb5244c72c29ef2acf96d1e7f54335ff168eed4119db1692025eb1b99194b7bb",
"tools/graphrag/publication.py": "fba1fa0d0bed325fd87f8d3593baae9271de1766b772d7214c7f63fd3f1c37b6",
"tools/graphrag/tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/tests/test_evaluation_pipeline_v2.py": "63d1b3e80ae3c7b563374fd7ab4aa5ffad3c614eafbd15142c577abd7f756bf9",
"tools/graphrag/tests/test_evaluation_v5.py": "9387e9d1418ebab87530986b0e29dac31cf33d762777a29a31cff24cc72e552a",
"tools/graphrag/tests/test_evaluation_wide.py": "1b28bb74fd6b9457c869186b88606f1f7863d573aa3ab5226162e878f236d99e",
"tools/graphrag/tests/test_evaluator_connected_v2.py": "29d7c9838ed1ff37ea3276df6feb1b6142158c43a0525712f490b648181882de",
"tools/graphrag/tests/test_graph_publication.py": "96e53c1ae28e36c7a4290e1e60c8b398bf095ceaf883e819e1a477e2672b11c0",
"tools/graphrag/tests/test_graph_rebuild_atomic.py": "fe37d4fff0b1f0fb1d565f0a96152b9b4e336837bc78b5977cf3b4879d7fbaf3",
"tools/graphrag/tests/test_graph_search.py": "eb58661aef9dafe6bf7188fcdc2d79cef6652d57c64d1b81ae81a52e8e0204d7",
"tools/graphrag/tests/test_graph_search_entity_v2.py": "0ef1a8ebd513c5a4507eb38fcf5ae6137c69e9901b5a2975ba513be5902ac693",
"tools/graphrag/tests/test_graph_search_neo4j.py": "fec2bf72de8f752f17c9164fee5754bdb2e153828855e47efbf0000dbe82cf3b",
"tools/graphrag/tests/test_graph_search_regressions.py": "9db7779aeb8ed9e8c0f92c1657011a81080aff3c928649e811358e3009a18c48",
"tools/graphrag/tests/test_graphrag.py": "9ee8629d6b528dd78c2ddfc25cee95e17613f45b454fc0a38d4c591862637a9f",
"tools/graphrag/tests/test_graphrag_evaluator.py": "1c6d08aeec9ae56ac4123239a5aed12a0081d66cc3f34b2047d95764579d7298",
"tools/graphrag/tests/test_graphrag_freeze.py": "c4b760baf4c7ebf13b16bd4cd3c51874f5512eef3b4c61470dc4d428b0303d9f",
"tools/graphrag/tests/test_graphrag_research_v2.py": "711008a91ab8a58ae0854e7f90329bcd42e5189fde2e2001589e97b0c22c5198",
"tools/graphrag/tests/test_graphrag_root_title.py": "085651fe6e6081521a5dde9554abfee93456250108cf15978df25378d886a46c",
"tools/graphrag/tests/test_source_oracle_v2.py": "0afdbe235e185ffe9f38aa667f90f5a6c20504fe29e9ac572642b39a8eb7faca",
"requirements.txt": "26a6d097727eba03c1b6c71cd02b06373ef01ce03e567dd2bc4357ae330e5a1a",
"tools/publication.py": "f99d445073390a733afc8d849b9fcd4b4b0890bb54d136db67d4015b78436e1a",
"tools/retrieval_contract.py": "b8fa272c3babf924a3ce8b937fe7ae437cd959ba9f7667193ac67c18d2c5b58d",
"tools/index_state.py": "c33920a63d04009f11ef23ca9e9eee9bf86453b7d0586d027bf3b39156afea0d",
"tools/rag/common.py": "8497c46d5c6c27a7c76df1568360ad5e9d00d5e5a7ef186e758960cc79d81d1c",
"tools/sync_indexes.py": "5b909b4394c24e5ce22848594b16d9aea45fb06418f5cd92bb6aa6381a0ccf10"
},
"dataset_sha256": "9a3b873882bbf29b905d79c9448c3c287fda30323f93a6533b88dbd59b19f558",
"inputs": {
"data/documents.json": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"data/chunks.json": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9"
},
"source_corpus": {
"sha256": "440da12a7a991824e923f70cada7c616c3fb3b0f12767ec7d75039a6a3231b10",
"document_count": 114,
"documents": {
"pages/interns/bogdan_paul_chis/README.md": "3b4b946eb3c10e9f9a4092e6cffa01c5080e7579aded2f6ed3120c41ed62b647",
"pages/interns/cesar_gutierrez/README.md": "b0be60576eed28d4b1920e9acf198684519373dfc13171567247c853b3ed699e",
"pages/interns/oliver_pejic/README.md": "0470f3c8465be8f02204b08e4c75ef47f68c7e4ad1038faa5641cdf1811f33d5",
"pages/interns/sevval_bulburu/README.md": "ff2a9d66253802f1af0fdfd74832e40fae41b6df23c5e61dc266f6cd0cc16c4d",
"pages/interns/yussef_ressaissi/README.md": "f997a65699da0db3222ef7a3c37652d05dfc3419ad4ec74683bafc2c897bb987",
"pages/students/2016/darius_lindvai/README.md": "fcaa47bb9708a6439eec5c4b7f7b4ed6c7359e423e1d940e1aed9f44957e6d62",
"pages/students/2016/darius_lindvai/dp2021/README.md": "be3d0fea2a6d592b225eecabd9b19445c2fce62b8289c07ab6af93835db12970",
"pages/students/2016/darius_lindvai/timovy_projekt/resers/README.md": "352162de1b1e192228fa5d26506c028ac47e567e90df53b1bbaa6ad736ec4766",
"pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md": "b4a81ff457624b089d618e4479afb1e3d748f6724f72104ef5518e535ae02fe5",
"pages/students/2016/dominik_nagy/README.md": "1c4037a158d84c45fa705a0c38e47142a9d46e226f0c138bc2d56d7c38f6d74d",
"pages/students/2016/dominik_nagy/timovy_projekt/README.md": "b81a4dab7316f7b846dcf36515f6aa9d44035c86f46f0e07b7a43308bcd31458",
"pages/students/2016/ediz_morochovic/README.md": "045e5eb9a3df2b84140808fbca0c3163266ad0efa144bc42d9d5c597881c6ac6",
"pages/students/2016/jakub_maruniak/README.md": "6734c41a95b9d0c1e3d226b393ad109ee4a7b82ebfffa73df2d7e0a743fefa14",
"pages/students/2016/jakub_maruniak/dp2021/README.md": "9cd884aec70705cdb2aaa0f514b513f16de263c37b749ced7f92916cd2f91c39",
"pages/students/2016/jakub_maruniak/dp2021/annotation/README.md": "debae0c50aea1accef0840912b9b1f37b2138f856002bb50e91d7ea2dbbe7963",
"pages/students/2016/jakub_maruniak/dp2021/train_experiments/README.md": "01dff38b37043414fa1db204394ddbe7aade00e14d02d495572b65c85d09b194",
"pages/students/2016/jakub_maruniak/timovy_projekt/README.md": "04114b1f3d84664aad5bdba5a0b614c5a6f23392be3c31fd9123d63db1be0a35",
"pages/students/2016/jan_holp/README.md": "80d6a8492766e5bfa61673a372e5f059d30c8e335de39842866c51485960e4d3",
"pages/students/2016/jan_holp/dp2021/README.md": "a62acb74009376766c7e2f9856ee01d21f84593eb9b3ce85cf2e6032aa412476",
"pages/students/2016/jan_holp/dp2021/zdrojove_subory/README.md": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"pages/students/2016/jan_holp/timovy_projekt/README.md": "79eebdfd3af7e8d5c279077e285a580731b1cf8cc9b4454f6596796e9915f70c",
"pages/students/2016/lukas_pokryvka/README.md": "f279456c5c277ff1ee0dd89a3464ed9f9f67b44bc185a5da9811ef5cfb675002",
"pages/students/2016/lukas_pokryvka/dp2021/README.md": "39b61376ce108e4a930f926ab2d6a997207457121bccfcf617129a3b210a15cf",
"pages/students/2016/lukas_pokryvka/timovy_projekt/README.md": "4de4b6c8cfed17b6c60090f82df1d55a66be70a79d3013850689b2baebdd801b",
"pages/students/2016/mark_feher/README.md": "bec37c0319fe0869a0377d771db23d090dbda93c27328f5697cdaeb3bbccfe9b",
"pages/students/2016/maros_harahus/README.md": "0da150e91918632c9e8cd1443f04da403595a372517e254928ce9f76c9d5033d",
"pages/students/2016/maros_harahus/timovy_projekt/README.md": "75af19f7c68f4b646fa1dbae75f3f3dfc52633a9f9da1400d21808bb20be0d83",
"pages/students/2016/patrik_pavlisin/README.md": "dc1c648d7cd9b0f0e021c57f9b98189e37c100871439e7e304390715913f2763",
"pages/students/2016/patrik_pavlisin/dp21/README.md": "ac1eb0fd0360ad34aa21b7c8da5bb010a6edf10be5721f28dedc3986cd1d8c54",
"pages/students/2016/patrik_pavlisin/dp22/README.md": "f2a8c619a812b99abab618ae388eb672407950e1f33fb8d7da7a528c4de81d50",
"pages/students/2016/patrik_pavlisin/tp20/README.md": "ae9c771f27e7e093bc29c6888c8d0b27ed61fb41519fd0d7e7f4f6cbb4626edc",
"pages/students/2016/tomas_kucharik/README.md": "6a1a215433eb1857660177b4cb2ecec03c6a86743d3a04cd6d5c9d9682c9e63b",
"pages/students/2017/dmytro_ushatenko/README.md": "65f4ac177d492162b08ca5da3d9eb048bd088a3c117a28a662113b7e4bc3a4ec",
"pages/students/2017/martin_jancura/README.md": "6a0a18d5095dcfb12abfdd72c9cfd8025d714fb5e29a7858cd2b285e4cb986b9",
"pages/students/2017/stanislav_matsunych/README.md": "90ffa11eff4a4360b0252dbc53a116c3a5777b3d5dd86ce5c89c02bdd2051c73",
"pages/students/2017/than_trung_thanh/README.md": "2150519b8580e7337520db4de64d2d1c82d2634380537d019f41c506dbf2979d",
"pages/students/2018/NaN1name_buday/README.md": "23c8c58090bdc954347e36f141bea91f43c8d5b1d777e346864a2ebe84c0bcce",
"pages/students/2018/adrian_remias/README.md": "e5bfe1d5d4e626beaa636a02d34ec58cf2cff9b0e57833905af35b80bf1465a2",
"pages/students/2018/david_ilas/README.md": "96c45ff9c5a821ddcb1a2fc86d487bd8f4729d1b76887914b0f5fbf99af45af0",
"pages/students/2018/david_omasta/README.md": "4164051870e1e1cf8949ed18a74196e9ea789dfbe4fee24bdeb736ca42848327",
"pages/students/2018/martin_wencel/README.md": "0e6a7a981689f22304f8baa5de97d4315c0695c94c9277de7851943eb012253b",
"pages/students/2018/oleh_bilykh/README.md": "4072c62dd8ee753a0c8fa43fdfbbdf566e394c2df13e6a79d08a302e332710b1",
"pages/students/2018/ondrej_megela/README.md": "969ff2e1a436b8f957e7168cf5eb678196b99aeda17701c502b016441f644065",
"pages/students/2018/samuel_sirotnik/README.md": "cc230ede67b6e8632511c92a5ed6d1e9deb42e70819280222bdfd369ee77fc70",
"pages/students/2019/alina_vitko/README.md": "9b071fe2aff470c275ae027be236fe836f741f774cabb9cbbb55304a8d5102cd",
"pages/students/2019/artem_yatsenko/README.md": "328b101e46c51c959e792cf7ca226942f81d2ab8fe9648ad6d0ecd07016429e2",
"pages/students/2019/david_stromp/README.md": "e46ca8691d5ec894128f6f6f2ed33e2c7bd141f80719bb0449b20f10f30cca09",
"pages/students/2019/dmytro_mural/README.md": "e881a47d847c3ef8d680844113fdbfbb87553b28982bf9eef4936f6d30923e93",
"pages/students/2019/filip_tomas/README.md": "75a1fe7b921e4bff0fdf9e2a957b4adad4b4a57aa7ebdffbb73dd82c02123f94",
"pages/students/2019/kristian_sopkovic/README.md": "3286960fb30f584df35455f9b6dfae81ca4827a1d711d2bcd6e48c3f150c9279",
"pages/students/2019/michal_stromko/README.md": "3ae1dd9b6638a6a249a94be3f3846286555666aefa225bb3a2c7288b35900ce7",
"pages/students/2019/michal_stromko/vp2021/README.md": "feb2e093fdd6461239e79b1e1f96e65a1c8cd96e8a2b70535ae556a9c198046a",
"pages/students/2019/michal_stromko/vp2021/tutorials/README.md": "40e3fa448c7329ef863e705435e23fe6b4788de440710f066fe0eed92d132bad",
"pages/students/2019/michal_stromko/vp2021/tutorials/img/README.md": "e8d3ac160b17dfdee2c3e996bf6a03af7a6f16f9140bf5cf7ee343a82dd1865f",
"pages/students/2019/patrik_pokrivcak/README.md": "3e3957ad0f7aca59d6eb64c38a3f5e21d8fd953200825d0d0e5cacabfbc35a7d",
"pages/students/2019/samuel_horani/README.md": "86e6959ccc612b069687b57996fa3f61660075d6a0d65bfec64f423d39a0a999",
"pages/students/2020/david_kostilnik/README.md": "2927524b75da59a64120fc364a0fc374986fa4d364d54e52c36800762c203264",
"pages/students/2020/jakub_kristian_lukas/README.md": "f139d9b1ebf5a3f45612f3df5703ecdd935b90701763c27c7c529044c0b0455a",
"pages/students/2020/kamil_tomcufcik/README.md": "9d865adf538bc6d7992ee4f4882713447da5c936fca17c43ca0f576072537048",
"pages/students/2020/matej_kobylan/README.md": "3d7b289e17b8f25678a07e9ee0e6741688485a0ccb1e50940a6a6acc6312a547",
"pages/students/2020/matus_suchanic/README.md": "770ca49538cca22df35f8e7ba736ee97483e4fe5788c221496553592b4e78dc6",
"pages/students/2020/pavol_hudak/README.md": "33d06fccc6451e5f19b27b0cad001a34e81635b4ea294a01dec7771c823abc40",
"pages/students/2020/vladyslav_krupko/README.md": "4c28e6d38651bb2cafcbf4a18e35210cd369aef2caff6c5b0aa52fac38f0fe7d",
"pages/students/2021/artur_hyrenko/README.md": "0568261f4ade4de5472edcfeb1ffed9abae18dc67e4050f4ce2ad088839a2bb3",
"pages/students/2021/eduard_matovka/README.md": "de2a7ae38ec87110e347956b37d0605a298ed2847f972961cc9b977da8221d43",
"pages/students/2021/manohar_gowdru_shridharu/README.md": "c00ceb620ed7087ead5d9448d476bbb98d273d1d9e7bf1bc4355c367b26c7a6b",
"pages/students/2021/martin_sarissky/README.md": "8e97b45a253745faa55f63c844e0b5dfe680aeaf6eb01fb8f9d06ac6e8ee8bcf",
"pages/students/2021/matej_novotny/README.md": "b083a7cadde532e3428dc9a57bc61456fcc5905bda91e011b96033d42bf5d9cb",
"pages/students/2021/matej_scislak/README.md": "b9a32cce8605e044a4f2828ff2d91a2995a62ef3a841e62f0a225c97221f0cc8",
"pages/students/2021/nikita_bodnar/README.md": "2016c0dd923c0f8e561d5cea42ac81da3610fdcc54a4527f56d078e34e53ce90",
"pages/students/2021/oleksandr_hryshchenko/README.md": "4b073992b715b6c75b79fd2c8f850a69f940966e3a61a82c625c32c945a93036",
"pages/students/2021/vladimir_ferko/README.md": "68e7d311896eda2171e5eec1ec3a8da90d326ed91acaace8058f68664a50d271",
"pages/students/2022/andrii_pervashov/README.md": "319dcaa1cc2e29dd7e3a710bab26cb3ac6944ab71b0b97e1de076b1199122334",
"pages/students/2022/daniil_huzenko/README.md": "390782fbff6220a9462060ce1a62959f2a2b88f42d06e2c51d1f406b75d73f2c",
"pages/students/2022/jakub_schwarc/README.md": "f541a8852c2b0f6f9d7182e01a34c4e3beee9bfa55427a06b331638b11eb8be6",
"pages/students/2022/jan_malinovsky/README.md": "8af021bfb131fa005a7aec3e2f2f12fc17fa5dc14e2675311adbbfc445a4bcab",
"pages/students/2022/jan_ptak/README.md": "131cf053764fd5423541ebe672c3b4f5da875cfb8e4b821f1335ddb967282282",
"pages/students/2022/oleh_poiasnik/README.md": "c2f7dc3f61f8b4a5b43ded1e3ae4c7d3e7f874439ee6b17dd9c8d45dabc25bc8",
"pages/students/2022/samuel_vasil/README.md": "67c807049147aacd04f3050f4ff0233a8b3f0a239ab173b430e13dfe113ad0a5",
"pages/students/2022/serhii_yemets/README.md": "ffbeac893947acafe855d576322a8826f9d4f87ac344cb31f8ceeefa244b9133",
"pages/students/2022/tetiana_mohorian/README.md": "c9efdc632bf1ca89a0bfde0dc00be7aaac8a85da3d92e2ce8506236c73e93cfb",
"pages/students/2022/valerii_kutsenko/README.md": "dc764c609ad3c1df59cfea997d4db82a591b1b1e0764453e319d841115507e4f",
"pages/students/2022/vladyslav_yanchenko/README.md": "4d1ce216e3b526ee437ddbc658e51d7bb83530087e43628c096a569a3854a47f",
"pages/students/2022/yevhenii_leonov/README.md": "b90e41db0cec2451027020a9102d32a24d777e8a38b604ccca87b36cd6b3a292",
"pages/students/2023/denis_le_thanh/README.md": "2cb83f942d86a02712b0c58778e46dc2211346d1a6eba606471dfa832245db61",
"pages/students/2023/maksym_zatirka/README.md": "7a7d2cbe276c352dacfbc6ba6de5f15303d30ecf34845ff8da8d085f3d67c3dd",
"pages/students/2023/oleksandr_dorybohov/README.md": "c24d6f941de0dafb4459cea8259cf147821cafc86ba65be5b528f419f734bc18",
"pages/students/2023/ronald_zalacko/README.md": "724b0f8e6e641236dba9bddbd77f4546e09514076c7dfb151807aaf4b882c96f",
"pages/students/2023/rostyslav_rodzhuk/README.md": "de356dd7427c8a0716c165a0f88ae47b691a86ee7a388f60e2d2e9306298415c",
"pages/students/2023/samuel_dzurina/README.md": "4818191d044b91219c796a35010c95da5cf2093e050d494f30b219b91e13fa0e",
"pages/students/2023/simona_bobrovcanova/README.md": "e473a1911dae10ab40e2cf6696a8be616a58c22aaf09af5953fbac475791a35e",
"pages/students/2023/yevhenii_medushivskyi/README.md": "5710ebf91c893a12fe6d19f5372311938d1e7c7bbdf203f2dc8ed2fe7c4f5a24",
"pages/topics/akopisat/README.md": "c7245e1fbfa4160618502c0baf8cc6bc47cd9a82b8a85bc1c645314829a7b76a",
"pages/topics/bert/README.md": "35e507f7ec2ad1d3526e2a15f94ea085ee25f680a02daf6df62fd429bd9c25c9",
"pages/topics/chatbot/README.md": "6f2f83ffb80109be47f9750cca9d3036ec3645abef9f2b590bd6825780c7c48a",
"pages/topics/faq/README.md": "15cfbd9c2a1649c80a7a58922e2b66dadab5c86a5ef33ef16c82bebd4d2dbf06",
"pages/topics/hatespeech/README.md": "364df94e70dd740c63544e2586ba7b957780f714fc0c941d1b04b02233e6cbc5",
"pages/topics/legal/README.md": "003bb9bfe1e0ca21652091c2d11496312eb04e0689fe459e14c2d5a738e8eeea",
"pages/topics/morphology/README.md": "e242cc5ff1a4cc98e227566c0ead999bbc95937b7c20be62cb2afb773d0d35c0",
"pages/topics/named-entity/README.md": "66b2e25f70211d432d596e377d552b3b8f0c525b14c31e3bc9c956c3aeadca84",
"pages/topics/named-entity/navod/README.md": "1ff87079a98411de0414754c60241e08d2214ddcd63ddf974e4340cce2ce5b02",
"pages/topics/nlp/README.md": "6f90abf9dc3f0ff33475283c13b4a08561abf631425ec2daacb076155a9e02c2",
"pages/topics/otvorene/README.md": "cd5544925553fe3840c0ea6f8ef60ab756e26ab6757bf810025c948d8d4f98fc",
"pages/topics/prodigy/README.md": "5e4f691f2754c2ec4afbaab2f41e6db77feb5b322dd9344465eb40c35f116499",
"pages/topics/python/README.md": "2ef073adba672142c39e9a59fa2e00a98ab52bda54233d46fbe6940ee23f9fcc",
"pages/topics/question/README.md": "408078a2a67cea2699e12359e85ab641136231e6104c2afdf874a705879525df",
"pages/topics/question/navod/README.md": "061e737c69184b2bd244316ca7ba841f7186fe01ee80ee993742a733dcad3ecd",
"pages/topics/question/nezodpovedatelne/README.md": "b6703d9690f4599b973efb27595ce157773ae0fc0d55cf2c0a09d9f50e8e4c92",
"pages/topics/question/validacie/README.md": "3216a7127abdfc6da50ed116f44ff2d6785e7da4af70d23aceda774e745ad562",
"pages/topics/resources/README.md": "09405397387f00cbcc5680f7d655f2d861e4d92e5df792a62d9dfd04f3534f48",
"pages/topics/seq2seq/README.md": "969d15c03a63a3f8d5edc9017fe5a0fb7311fdf4de4a3473a08266241f92c1e2",
"pages/topics/spacy/README.md": "f504e5db77cb8694f08a57fb8636e522d59a0e669216f4bc8273bc3c5f375f87",
"pages/topics/steel/README.md": "b4068cef2ed197711467ac47630d5b2064fdc8828f898f7af99664e94d833330",
"pages/topics/translation/README.md": "a7d4289e72199b307dc5ad8fc69cda39c1a33c27a6c7447af70aa9c3cc6b8635"
}
},
"retrieval_policy": {
"published_only": true,
"publication_policy": "explicit-false-hidden-v1",
"academic_preparation_is_work": false
}
},
"graph": {
"sha256": "34d3ff672efe834d2b7d07e2f1cd1c043db011d0db37fb07cc7616bced0234a0",
"nodes": 446,
"relationships": 877
},
"parameters": {
"limit": 40,
"max_entities": 8
},
"dev_summary": {
"questions": 722,
"source_checked_count": 722,
"all_facts_supported_count": 722,
"strict_count": 722,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.042052,
"latency_median_seconds": 0.040355,
"latency_p95_seconds": 0.052196
},
"source_audit_summary": {
"dev": {
"questions": 722,
"source_consistent": 722,
"issues": {}
},
"test": {
"questions": 320,
"source_consistent": 320,
"issues": {}
}
},
"policy": "Report all questions including negatives. No exclusions or holdout tuning.",
"dev_results_sha256": "b940a1a42a09d25fc008f34948ee3c8b274e9fab8fe0e692e04e1bfa472f2a25",
"audit_sha256": "81708ca63f8772673cd836fe7a1a1938fbe0e713588efcb07ed3fd4280f27829"
}

14
tools/graphrag/evaluation/graphrag_evaluator.py
View File

@ -10,6 +10,7 @@ from pathlib import Path
from typing import Any
from .evidence_contract import evaluate_contract
from .source_validation import default_oracle, validate_returned_facts
VALID_SPLITS = (
@ -866,6 +867,7 @@ def evaluate_result(
*,
latency_seconds: float,
error: str | None = None,
source_oracle: dict | None = None,
) -> dict[str, Any]:
graph = question.get(
"graph",
@ -1087,11 +1089,18 @@ def evaluate_result(
and normalized_set(declared_values) == actual_values
and len(declared_values) == actual_count
)
complete = not result.get("truncated", False)
complete = not result.get("truncated", False) and result.get('complete', True) is True
# Real source-grounded benchmarks always run the independent check. Small
# synthetic unit fixtures without source_evidence can inject their own oracle.
if source_oracle is None and question.get('source_evidence'):
source_oracle = default_oracle()
source_check = (validate_returned_facts(result, source_oracle) if source_oracle is not None else
{'source_checked':False,'all_facts_supported':None,'unsupported_facts':[]})
contract = evaluate_contract(question, result)
strict = all(
(
error is None,
source_check['all_facts_supported'] is not False,
contract['answerability_ok'],
contract['bindings_ok'],
contract['paths_ok'],
@ -1115,6 +1124,7 @@ def evaluate_result(
return {
**contract,
**source_check,
"count_ok": count_ok,
"expected_count": expected_count,
"actual_count": aggregation.get("count"),
@ -1356,6 +1366,8 @@ def aggregate_rows(
"questions": len(
rows
),
'source_checked_count':sum(row.get('source_checked') is True for row in rows),
'all_facts_supported_count':sum(row.get('all_facts_supported') is True for row in rows),
"strict_count": (
strict_count
),

8
tools/graphrag/evaluation/reproducibility.py
View File

@ -9,6 +9,7 @@ import subprocess
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from tools.publication import DEFAULT_PUBLISHED_ONLY, PUBLICATION_POLICY_VERSION
ROOT = Path(__file__).resolve().parents[3]
@ -33,12 +34,17 @@ def source_corpus_fingerprint(root: Path = ROOT) -> dict[str, Any]:
def content_fingerprints(questions: Path, root: Path = ROOT) -> dict[str, Any]:
files = sorted((root / 'tools/graphrag').rglob('*.py'))
# requirements.txt and tools/ are present both in the checkout and image.
files.append(root / 'requirements.txt')
files.extend(root / name for name in [
'requirements.txt', 'tools/publication.py', 'tools/retrieval_contract.py',
'tools/index_state.py', 'tools/rag/common.py', 'tools/sync_indexes.py'])
return {
'implementation': {str(path.relative_to(root)): sha256_file(path) for path in files},
'dataset_sha256': sha256_file(questions),
'inputs': {name: sha256_file(root / name) for name in ['data/documents.json', 'data/chunks.json']},
'source_corpus': source_corpus_fingerprint(root),
'retrieval_policy': {'published_only': DEFAULT_PUBLISHED_ONLY,
'publication_policy': PUBLICATION_POLICY_VERSION,
'academic_preparation_is_work': False},
}

40
tools/graphrag/evaluation/results/answers_v5_1_dev_2026-10-25/answers_dev.csv Normal file
View File

@ -0,0 +1,40 @@
id,category,strict,error,latency_seconds,attempts,first_pass_strict
wide-001,shared_keyword_pair,True,,7.432791330997134,1,True
wide-002,shared_keyword_pair,True,,9.3359067220008,1,True
wide-041,same_academic_year_pair,True,,13.052469432994258,1,True
wide-042,same_academic_year_pair,True,,7.94166571200185,1,True
wide-080,same_academic_year_pair,True,,8.928297609003494,1,True
wide-081,same_start_year_pair,True,,8.421164927000063,1,True
wide-082,same_start_year_pair,True,,8.470174660003977,1,True
wide-104,same_start_year_pair,True,,9.088728798000375,1,True
wide-113,person_title_year,True,,7.292930210009217,1,True
wide-114,person_title_year,True,,8.822794015999534,1,True
wide-119,person_title_year,True,,5.405082925994066,1,True
wide-137,person_keyword_title,True,,9.147465852001915,1,True
wide-138,person_keyword_title,True,,6.318841665008222,1,True
wide-139,person_keyword_title,True,,10.469266973988852,1,True
wide-141,person_keyword_title,True,,6.316172414997709,1,True
wide-142,person_keyword_title,False,,10.959449250003672,1,False
wide-143,person_keyword_title,True,,8.508081905994914,1,True
wide-145,person_keyword_title,True,,13.52746086299885,1,True
wide-155,person_keyword_title,True,,6.8860766970028635,1,True
wide-159,person_keyword_title,True,,10.890385270991828,1,True
wide-161,person_keyword_title,True,,6.801796521991491,1,True
wide-163,person_keyword_title,True,,10.777118418001919,1,True
wide-169,document_author_category_multi_hop,True,,4.879302582994569,1,True
wide-170,document_author_category_multi_hop,True,,8.241740879006102,1,True
wide-178,document_author_category_multi_hop,True,,5.5909052140050335,1,True
wide-181,document_author_category_multi_hop,True,,8.192060016008327,1,True
wide-193,person_multiple_categories,True,,4.390313146999688,1,True
wide-194,person_multiple_categories,True,,6.805427877989132,1,True
wide-209,person_multiple_categories,True,,5.125700485994457,1,True
wide-217,candidate_set_aggregation,True,,9.28586104400165,1,True
wide-218,candidate_set_aggregation,True,,7.510667574999388,1,True
wide-232,candidate_set_aggregation,True,,10.86943264301226,1,True
wide-236,candidate_set_aggregation,True,,7.126413513993612,1,True
wide-239,candidate_set_aggregation,True,,11.048067500989418,1,True
wide-249,conjunctive_disambiguation,True,,4.892320775994449,1,True
wide-250,conjunctive_disambiguation,True,,8.952342251010123,1,True
wide-283,conjunctive_disambiguation,True,,7.288101250000182,1,True
wide-289,negative_lookup,True,,5.516414188998169,1,True
wide-290,negative_lookup,True,,4.058507394001936,1,True
1 id category strict error latency_seconds attempts first_pass_strict
2 wide-001 shared_keyword_pair True 7.432791330997134 1 True
3 wide-002 shared_keyword_pair True 9.3359067220008 1 True
4 wide-041 same_academic_year_pair True 13.052469432994258 1 True
5 wide-042 same_academic_year_pair True 7.94166571200185 1 True
6 wide-080 same_academic_year_pair True 8.928297609003494 1 True
7 wide-081 same_start_year_pair True 8.421164927000063 1 True
8 wide-082 same_start_year_pair True 8.470174660003977 1 True
9 wide-104 same_start_year_pair True 9.088728798000375 1 True
10 wide-113 person_title_year True 7.292930210009217 1 True
11 wide-114 person_title_year True 8.822794015999534 1 True
12 wide-119 person_title_year True 5.405082925994066 1 True
13 wide-137 person_keyword_title True 9.147465852001915 1 True
14 wide-138 person_keyword_title True 6.318841665008222 1 True
15 wide-139 person_keyword_title True 10.469266973988852 1 True
16 wide-141 person_keyword_title True 6.316172414997709 1 True
17 wide-142 person_keyword_title False 10.959449250003672 1 False
18 wide-143 person_keyword_title True 8.508081905994914 1 True
19 wide-145 person_keyword_title True 13.52746086299885 1 True
20 wide-155 person_keyword_title True 6.8860766970028635 1 True
21 wide-159 person_keyword_title True 10.890385270991828 1 True
22 wide-161 person_keyword_title True 6.801796521991491 1 True
23 wide-163 person_keyword_title True 10.777118418001919 1 True
24 wide-169 document_author_category_multi_hop True 4.879302582994569 1 True
25 wide-170 document_author_category_multi_hop True 8.241740879006102 1 True
26 wide-178 document_author_category_multi_hop True 5.5909052140050335 1 True
27 wide-181 document_author_category_multi_hop True 8.192060016008327 1 True
28 wide-193 person_multiple_categories True 4.390313146999688 1 True
29 wide-194 person_multiple_categories True 6.805427877989132 1 True
30 wide-209 person_multiple_categories True 5.125700485994457 1 True
31 wide-217 candidate_set_aggregation True 9.28586104400165 1 True
32 wide-218 candidate_set_aggregation True 7.510667574999388 1 True
33 wide-232 candidate_set_aggregation True 10.86943264301226 1 True
34 wide-236 candidate_set_aggregation True 7.126413513993612 1 True
35 wide-239 candidate_set_aggregation True 11.048067500989418 1 True
36 wide-249 conjunctive_disambiguation True 4.892320775994449 1 True
37 wide-250 conjunctive_disambiguation True 8.952342251010123 1 True
38 wide-283 conjunctive_disambiguation True 7.288101250000182 1 True
39 wide-289 negative_lookup True 5.516414188998169 1 True
40 wide-290 negative_lookup True 4.058507394001936 1 True

6068
tools/graphrag/evaluation/results/answers_v5_1_dev_2026-10-25/answers_dev.json Normal file
View File

File diff suppressed because it is too large Load Diff

425
tools/graphrag/evaluation/results/answers_v5_1_dev_2026-10-25/frozen_answers.json Normal file
View File

@ -0,0 +1,425 @@
{
"method": "Previously exposed LLM questions; regression, not independent TEST.",
"graph_freeze_sha256": "ca1d2d71f60a8e65b82d727ad14556aabf2e03f2acdce1755c3e2a7a9a3149d3",
"audit_sha256": "81708ca63f8772673cd836fe7a1a1938fbe0e713588efcb07ed3fd4280f27829",
"generation_files": {
"/home/janko/DP/zp-agent/tools/answer_evaluation/evaluate_graph_answers.py": "3b58e210fcd934a5488575e1b0ebb46a2e94ce72ee12f1e1a666c726c66d68af",
"/home/janko/DP/zp-agent/tools/answer_evaluation/graph_answer_prompt.txt": "64d2bd817657f390c306ea01bf2dad41f70cd6993e06b2e3b2f4713bae0d249c",
"/home/janko/DP/zp-agent/tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/rag/evaluation/rag_runner.py": "a011b29e2b6aa47df9350e74fc358f05780a355170a819dbb1cfd16d2971493b"
},
"model": "model120-fast",
"temperature": 0,
"workers": 3,
"dev_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"test_ids": [
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
],
"dev_count": 39,
"test_count": 300,
"target_accuracy": 0.92,
"selected_split": "dev",
"selected_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"selected_count": 39,
"selected_dev_overlap": 39,
"format_repair_limit": 1,
"repair_policy": "Only JSON/schema failures; same question/context, no gold; retain all attempts.",
"dev_results_sha256": null,
"policy": "Evaluate every declared regression case, including DEV overlap; retain all failures and stop tuning after this freeze.",
"selected_categories": {
"shared_keyword_pair": 2,
"same_academic_year_pair": 3,
"same_start_year_pair": 3,
"person_title_year": 3,
"person_keyword_title": 11,
"document_author_category_multi_hop": 4,
"person_multiple_categories": 3,
"candidate_set_aggregation": 5,
"conjunctive_disambiguation": 3,
"negative_lookup": 2
},
"limitations": [
"Known corpus/profiles and templates; model outputs for these 300 questions were not previously observed.",
"Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.",
"Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose."
]
}

321
tools/graphrag/evaluation/results/answers_v5_1_regression_2026-10-25/answers_regression.csv Normal file
View File

@ -0,0 +1,321 @@
id,category,strict,error,latency_seconds,attempts,first_pass_strict
wide-001,shared_keyword_pair,True,,12.681157764003729,1,True
wide-002,shared_keyword_pair,True,,7.440348913005437,1,True
wide-003,shared_keyword_pair,True,,6.8628689809993375,1,True
wide-004,shared_keyword_pair,True,,7.915003027010243,1,True
wide-005,shared_keyword_pair,True,,11.64334254800633,1,True
wide-006,shared_keyword_pair,True,,9.263334328003111,1,True
wide-007,shared_keyword_pair,True,,10.080044332993566,1,True
wide-008,shared_keyword_pair,True,,9.102414163004141,1,True
wide-009,shared_keyword_pair,True,,8.990864911000244,1,True
wide-010,shared_keyword_pair,True,,10.98848542298947,1,True
wide-011,shared_keyword_pair,True,,9.469930938998004,1,True
wide-012,shared_keyword_pair,True,,11.050285377990804,1,True
wide-013,shared_keyword_pair,True,,8.279340490000322,1,True
wide-014,shared_keyword_pair,True,,10.769162045005942,1,True
wide-015,shared_keyword_pair,True,,10.199135344999377,1,True
wide-016,shared_keyword_pair,True,,12.438964514003601,1,True
wide-017,shared_keyword_pair,True,,10.442358953005169,1,True
wide-018,shared_keyword_pair,True,,10.590256554991356,1,True
wide-019,shared_keyword_pair,True,,9.393680679990212,1,True
wide-020,shared_keyword_pair,True,,10.050156186000095,1,True
wide-021,shared_keyword_pair,True,,10.283129505012766,1,True
wide-022,shared_keyword_pair,True,,9.261356490998878,1,True
wide-023,shared_keyword_pair,True,,11.352632169000572,1,True
wide-024,shared_keyword_pair,True,,7.952293816997553,1,True
wide-025,shared_keyword_pair,True,,11.707699684004183,1,True
wide-026,shared_keyword_pair,True,,7.244572512994637,1,True
wide-027,shared_keyword_pair,True,,11.86879978099023,1,True
wide-028,shared_keyword_pair,True,,6.436980010010302,1,True
wide-029,shared_keyword_pair,True,,13.4112125190004,1,True
wide-030,shared_keyword_pair,True,,6.320292321004672,1,True
wide-031,shared_keyword_pair,True,,13.030589675996453,1,True
wide-032,shared_keyword_pair,True,,8.23478609599988,1,True
wide-033,shared_keyword_pair,True,,11.427597190995584,1,True
wide-034,shared_keyword_pair,True,,7.945492083003046,1,True
wide-035,shared_keyword_pair,True,,10.91667144499661,1,True
wide-036,shared_keyword_pair,True,,7.704312329005916,1,True
wide-037,shared_keyword_pair,True,,9.866947228001663,1,True
wide-038,shared_keyword_pair,True,,9.0944732450007,1,True
wide-039,shared_keyword_pair,True,,9.955326469003921,1,True
wide-040,shared_keyword_pair,True,,10.126550430999487,1,True
wide-041,same_academic_year_pair,True,,8.00595778299612,1,True
wide-042,same_academic_year_pair,True,,10.224936337006511,1,True
wide-043,same_academic_year_pair,True,,7.973652732995106,1,True
wide-044,same_academic_year_pair,True,,11.643997175007826,1,True
wide-045,same_academic_year_pair,True,,7.812763041001745,1,True
wide-046,same_academic_year_pair,True,,11.01834614200925,1,True
wide-047,same_academic_year_pair,True,,9.787194718010142,1,True
wide-048,same_academic_year_pair,True,,11.489561196009163,1,True
wide-049,same_academic_year_pair,True,,10.164007376006339,1,True
wide-050,same_academic_year_pair,True,,9.938125018001301,1,True
wide-051,same_academic_year_pair,True,,9.78176777700719,1,True
wide-052,same_academic_year_pair,True,,9.160806122003123,1,True
wide-053,same_academic_year_pair,True,,9.269089786990662,1,True
wide-054,same_academic_year_pair,True,,9.431611267005792,1,True
wide-055,same_academic_year_pair,True,,9.327960707989405,1,True
wide-056,same_academic_year_pair,True,,9.804446037000162,1,True
wide-057,same_academic_year_pair,True,,8.234563728998182,1,True
wide-058,same_academic_year_pair,True,,9.821240933990339,1,True
wide-059,same_academic_year_pair,True,,6.477854401004151,1,True
wide-060,same_academic_year_pair,True,,10.462991334003164,1,True
wide-061,same_academic_year_pair,True,,6.2554830580047565,1,True
wide-062,same_academic_year_pair,True,,10.774600299002486,1,True
wide-063,same_academic_year_pair,True,,6.288768262995291,1,True
wide-064,same_academic_year_pair,True,,11.15352754900232,1,True
wide-065,same_academic_year_pair,True,,5.480506381005398,1,True
wide-066,same_academic_year_pair,True,,10.729390800988767,1,True
wide-067,same_academic_year_pair,True,,7.499470337992534,1,True
wide-068,same_academic_year_pair,True,,11.094555151008535,1,True
wide-069,same_academic_year_pair,True,,7.680634538002778,1,True
wide-070,same_academic_year_pair,True,,10.041368676989805,1,True
wide-071,same_academic_year_pair,True,,9.15008538699476,1,True
wide-072,same_academic_year_pair,True,,9.846425030991668,1,True
wide-073,same_academic_year_pair,True,,8.89766675599094,1,True
wide-074,same_academic_year_pair,True,,8.42804765701294,1,True
wide-075,same_academic_year_pair,True,,8.904354819998844,1,True
wide-076,same_academic_year_pair,True,,8.653263764994335,1,True
wide-077,same_academic_year_pair,True,,8.350220588996308,1,True
wide-078,same_academic_year_pair,True,,9.132759956002701,1,True
wide-079,same_academic_year_pair,True,,8.293541837992962,1,True
wide-080,same_academic_year_pair,True,,10.156925554998452,1,True
wide-081,same_start_year_pair,True,,7.918382621995988,1,True
wide-082,same_start_year_pair,True,,10.321019872004399,1,True
wide-083,same_start_year_pair,True,,7.002511290003895,1,True
wide-084,same_start_year_pair,True,,10.437800674000755,1,True
wide-085,same_start_year_pair,True,,6.816185226009111,1,True
wide-086,same_start_year_pair,True,,10.71233119200042,1,True
wide-087,same_start_year_pair,True,,6.4720146839972585,1,True
wide-088,same_start_year_pair,True,,10.198497953009792,1,True
wide-089,same_start_year_pair,True,,6.79203049800708,1,True
wide-090,same_start_year_pair,True,,10.413568222007598,1,True
wide-091,same_start_year_pair,True,,7.510395374993095,1,True
wide-092,same_start_year_pair,True,,10.110175805006293,1,True
wide-093,same_start_year_pair,True,,7.382416222986649,1,True
wide-094,same_start_year_pair,True,,9.637755793010001,1,True
wide-095,same_start_year_pair,True,,7.721356611000374,1,True
wide-096,same_start_year_pair,True,,9.783646965996013,1,True
wide-097,same_start_year_pair,True,,8.19954388300539,1,True
wide-098,same_start_year_pair,True,,9.142149596998934,1,True
wide-099,same_start_year_pair,True,,7.995075654995162,1,True
wide-100,same_start_year_pair,True,,9.23304197700054,1,True
wide-101,same_start_year_pair,True,,8.445941337995464,1,True
wide-102,same_start_year_pair,True,,9.29434829299862,1,True
wide-103,same_start_year_pair,True,,8.262059757005773,1,True
wide-104,same_start_year_pair,True,,8.660338338988367,1,True
wide-105,same_start_year_pair,True,,8.136423168994952,1,True
wide-106,same_start_year_pair,True,,8.826232392006204,1,True
wide-107,same_start_year_pair,True,,8.023831066006096,1,True
wide-108,same_start_year_pair,True,,9.481848634008202,1,True
wide-109,same_start_year_pair,True,,7.782530160999158,1,True
wide-110,same_start_year_pair,True,,10.06139503599843,1,True
wide-111,same_start_year_pair,True,,7.194188486013445,1,True
wide-112,same_start_year_pair,True,,9.618882115988526,1,True
wide-113,person_title_year,True,,6.316762108996045,1,True
wide-114,person_title_year,True,,8.973571240989259,1,True
wide-115,person_title_year,True,,5.77858125900093,1,True
wide-116,person_title_year,True,,8.171107051006402,1,True
wide-117,person_title_year,True,,5.923224907994154,1,True
wide-118,person_title_year,True,,8.248448436992476,1,True
wide-119,person_title_year,True,,6.6704190939926775,1,True
wide-120,person_title_year,True,,7.390637057993445,1,True
wide-121,person_title_year,True,,6.828236892994028,1,True
wide-122,person_title_year,True,,7.436154732989962,1,True
wide-123,person_title_year,True,,7.201046439004131,1,True
wide-124,person_title_year,True,,7.452169609998236,1,True
wide-125,person_title_year,True,,7.024459202992148,1,True
wide-126,person_title_year,True,,7.134706636992632,1,True
wide-127,person_title_year,True,,6.701897156002815,1,True
wide-128,person_title_year,True,,7.7986834320036,1,True
wide-129,person_title_year,False,,7.78502085400396,1,False
wide-130,person_title_year,True,,7.579418053996051,1,True
wide-131,person_title_year,True,,7.29737891399418,1,True
wide-132,person_title_year,True,,6.5710127700003795,1,True
wide-133,person_title_year,True,,7.670121143004508,1,True
wide-134,person_title_year,True,,6.268109227006789,1,True
wide-135,person_title_year,True,,7.80891695599712,1,True
wide-136,person_title_year,True,,6.535969970005681,1,True
wide-137,person_keyword_title,True,,8.499110547010787,1,True
wide-138,person_keyword_title,True,,7.348516454003402,1,True
wide-139,person_keyword_title,True,,8.490620700002182,1,True
wide-140,person_keyword_title,True,,7.142724603007082,1,True
wide-141,person_keyword_title,True,,7.750912368996069,1,True
wide-142,person_keyword_title,False,,9.134571484988555,1,False
wide-143,person_keyword_title,True,,9.924621155005298,1,True
wide-144,person_keyword_title,True,,9.80673293399741,1,True
wide-145,person_keyword_title,False,,9.125109145010356,1,False
wide-146,person_keyword_title,True,,7.816306863998761,1,True
wide-147,person_keyword_title,True,,9.222393247997388,1,True
wide-148,person_keyword_title,True,,6.4315986309957225,1,True
wide-149,person_keyword_title,True,,8.975530234994949,1,True
wide-150,person_keyword_title,True,,6.5634262770036,1,True
wide-151,person_keyword_title,True,,8.981235069994,1,True
wide-152,person_keyword_title,True,,6.80172690100153,1,True
wide-153,person_keyword_title,True,,8.104526370007079,1,True
wide-154,person_keyword_title,True,,7.002095560004818,1,True
wide-155,person_keyword_title,True,,8.536377580996486,1,True
wide-156,person_keyword_title,True,,7.33249492100731,1,True
wide-157,person_keyword_title,True,,8.53190982498927,1,True
wide-158,person_keyword_title,True,,6.987181315998896,1,True
wide-159,person_keyword_title,True,,9.206439533008961,1,True
wide-160,person_keyword_title,True,,7.2076548890036065,1,True
wide-161,person_keyword_title,True,,9.755865868995897,1,True
wide-162,person_keyword_title,True,,6.598434728002758,1,True
wide-163,person_keyword_title,True,,9.665300851003849,1,True
wide-164,person_keyword_title,True,,6.409714323002845,1,True
wide-165,person_keyword_title,True,,9.189696266010287,1,True
wide-166,person_keyword_title,True,,6.362806601988268,1,True
wide-167,person_keyword_title,True,,9.12334693899902,1,True
wide-168,person_keyword_title,True,,6.939789495998411,1,True
wide-169,document_author_category_multi_hop,True,,8.274773390003247,1,True
wide-170,document_author_category_multi_hop,True,,5.761120869006845,1,True
wide-171,document_author_category_multi_hop,True,,7.484641218994511,1,True
wide-172,document_author_category_multi_hop,True,,5.7914811780065065,1,True
wide-173,document_author_category_multi_hop,True,,7.744038484990597,1,True
wide-174,document_author_category_multi_hop,True,,5.857030038998346,1,True
wide-175,document_author_category_multi_hop,True,,8.176481470989529,1,True
wide-176,document_author_category_multi_hop,True,,7.8557383849984035,1,True
wide-177,document_author_category_multi_hop,True,,7.886023836006643,1,True
wide-178,document_author_category_multi_hop,True,,7.453686430002563,1,True
wide-179,document_author_category_multi_hop,True,,5.2365985850046854,1,True
wide-180,document_author_category_multi_hop,True,,7.151028209991637,1,True
wide-181,document_author_category_multi_hop,True,,5.116186750004999,1,True
wide-182,document_author_category_multi_hop,True,,7.575539704004768,1,True
wide-183,document_author_category_multi_hop,True,,5.5021829689940205,1,True
wide-184,document_author_category_multi_hop,True,,7.557064304011874,1,True
wide-185,document_author_category_multi_hop,True,,5.2788128620013595,1,True
wide-186,document_author_category_multi_hop,True,,7.775028784992173,1,True
wide-187,document_author_category_multi_hop,True,,5.468235724998522,1,True
wide-188,document_author_category_multi_hop,True,,7.458582812003442,1,True
wide-189,document_author_category_multi_hop,True,,5.423984528999426,1,True
wide-190,document_author_category_multi_hop,True,,6.802945887990063,1,True
wide-191,document_author_category_multi_hop,True,,5.720807204997982,1,True
wide-192,document_author_category_multi_hop,True,,6.970476242990117,1,True
wide-193,person_multiple_categories,True,,5.315446266002255,1,True
wide-194,person_multiple_categories,True,,6.611033188004512,1,True
wide-195,person_multiple_categories,True,,4.514709312992636,1,True
wide-196,person_multiple_categories,True,,6.597614583995892,1,True
wide-197,person_multiple_categories,True,,4.054451817995869,1,True
wide-198,person_multiple_categories,True,,6.207983192012762,1,True
wide-199,person_multiple_categories,True,,4.590510104011628,1,True
wide-200,person_multiple_categories,True,,6.553350798989413,1,True
wide-201,person_multiple_categories,True,,4.605876973000704,1,True
wide-202,person_multiple_categories,True,,5.585335472002043,1,True
wide-203,person_multiple_categories,True,,4.492845022992697,1,True
wide-204,person_multiple_categories,True,,6.641766903994721,1,True
wide-205,person_multiple_categories,True,,5.62207466000109,1,True
wide-206,person_multiple_categories,True,,6.369739195986767,1,True
wide-207,person_multiple_categories,True,,4.450628942999174,1,True
wide-208,person_multiple_categories,True,,5.57498624100117,1,True
wide-209,person_multiple_categories,True,,5.34890157400514,1,True
wide-210,person_multiple_categories,True,,6.2012351820012555,1,True
wide-211,person_multiple_categories,True,,5.767825509014074,1,True
wide-212,person_multiple_categories,True,,5.083963163997396,1,True
wide-213,person_multiple_categories,True,,5.3684891809971305,1,True
wide-214,person_multiple_categories,True,,4.774883028992917,1,True
wide-215,person_multiple_categories,True,,5.859735618010745,1,True
wide-216,person_multiple_categories,True,,5.045369378000032,1,True
wide-217,candidate_set_aggregation,True,,8.18277867299912,1,True
wide-218,candidate_set_aggregation,True,,7.061751248998917,1,True
wide-219,candidate_set_aggregation,True,,9.888737866000156,1,True
wide-220,candidate_set_aggregation,True,,6.534448446996976,1,True
wide-221,candidate_set_aggregation,True,,10.02607991900004,1,True
wide-222,candidate_set_aggregation,True,,6.639642160007497,1,True
wide-223,candidate_set_aggregation,True,,10.05624425499991,1,True
wide-224,candidate_set_aggregation,True,,6.380346581994672,1,True
wide-225,candidate_set_aggregation,True,,9.750748607999412,1,True
wide-226,candidate_set_aggregation,True,,6.927533381007379,1,True
wide-227,candidate_set_aggregation,True,,9.749435025994899,1,True
wide-228,candidate_set_aggregation,True,,7.000018507009372,1,True
wide-229,candidate_set_aggregation,True,,9.393544332007878,1,True
wide-230,candidate_set_aggregation,True,,7.075158320993069,1,True
wide-231,candidate_set_aggregation,True,,9.68925523001235,1,True
wide-232,candidate_set_aggregation,True,,7.470996176998597,1,True
wide-233,candidate_set_aggregation,True,,9.546211912995204,1,True
wide-234,candidate_set_aggregation,True,,7.121646208994207,1,True
wide-235,candidate_set_aggregation,True,,9.106454324006336,1,True
wide-236,candidate_set_aggregation,True,,7.33974453000701,1,True
wide-237,candidate_set_aggregation,True,,9.92777099799423,1,True
wide-238,candidate_set_aggregation,True,,7.6532076459989185,1,True
wide-239,candidate_set_aggregation,True,,9.878883134995704,1,True
wide-240,candidate_set_aggregation,True,,6.794067700000596,1,True
wide-241,candidate_set_aggregation,True,,9.484167262999108,1,True
wide-242,candidate_set_aggregation,True,,6.586464612002601,1,True
wide-243,candidate_set_aggregation,True,,9.679727968003135,1,True
wide-244,candidate_set_aggregation,True,,6.853099594998639,1,True
wide-245,candidate_set_aggregation,True,,9.588607818004675,1,True
wide-246,candidate_set_aggregation,True,,6.615246096000192,1,True
wide-247,candidate_set_aggregation,True,,9.338731531999656,1,True
wide-248,candidate_set_aggregation,True,,6.656504976010183,1,True
wide-249,conjunctive_disambiguation,True,,8.06964380700083,1,True
wide-250,conjunctive_disambiguation,True,,5.727493058991968,1,True
wide-251,conjunctive_disambiguation,True,,7.403562118997797,1,True
wide-252,conjunctive_disambiguation,True,,6.389556170004653,1,True
wide-253,conjunctive_disambiguation,True,,7.190451546994154,1,True
wide-254,conjunctive_disambiguation,True,,5.571824425001978,1,True
wide-255,conjunctive_disambiguation,True,,6.363713938990259,1,True
wide-256,conjunctive_disambiguation,True,,5.9705328209965955,1,True
wide-257,conjunctive_disambiguation,True,,6.814727099001175,1,True
wide-258,conjunctive_disambiguation,True,,6.060012061992893,1,True
wide-259,conjunctive_disambiguation,True,,6.477906736996374,1,True
wide-260,conjunctive_disambiguation,True,,6.108023989989306,1,True
wide-261,conjunctive_disambiguation,True,,6.89838379000139,1,True
wide-262,conjunctive_disambiguation,True,,6.639250446998631,1,True
wide-263,conjunctive_disambiguation,True,,7.647332847991493,1,True
wide-264,conjunctive_disambiguation,True,,6.806039610994048,1,True
wide-265,conjunctive_disambiguation,True,,7.480212347989436,1,True
wide-266,conjunctive_disambiguation,True,,5.939997524998034,1,True
wide-267,conjunctive_disambiguation,True,,7.483691226007068,1,True
wide-268,conjunctive_disambiguation,True,,5.128309691994218,1,True
wide-269,conjunctive_disambiguation,True,,7.950514240990742,1,True
wide-270,conjunctive_disambiguation,True,,3.838128978997702,1,True
wide-271,conjunctive_disambiguation,True,,7.203258510999149,1,True
wide-272,conjunctive_disambiguation,True,,5.274275976000354,1,True
wide-273,conjunctive_disambiguation,True,,7.145530178997433,1,True
wide-274,conjunctive_disambiguation,True,,5.765552001001197,1,True
wide-275,conjunctive_disambiguation,True,,7.404864780008211,1,True
wide-276,conjunctive_disambiguation,True,,6.352571280993288,1,True
wide-277,conjunctive_disambiguation,True,,7.504300720000174,1,True
wide-278,conjunctive_disambiguation,True,,6.016011193001759,1,True
wide-279,conjunctive_disambiguation,True,,7.295396494999295,1,True
wide-280,conjunctive_disambiguation,True,,5.834808207000606,1,True
wide-281,conjunctive_disambiguation,True,,7.8035292659915285,1,True
wide-282,conjunctive_disambiguation,True,,6.399204970992287,1,True
wide-283,conjunctive_disambiguation,True,,9.671940870990511,1,True
wide-284,conjunctive_disambiguation,True,,5.51546201200108,1,True
wide-285,conjunctive_disambiguation,True,,10.154824249999365,1,True
wide-286,conjunctive_disambiguation,True,,4.581112927000504,1,True
wide-287,conjunctive_disambiguation,True,,9.536005122994538,1,True
wide-288,conjunctive_disambiguation,True,,5.152376526006265,1,True
wide-289,negative_lookup,True,,5.77029365299677,1,True
wide-290,negative_lookup,True,,4.0657596189994365,1,True
wide-291,negative_lookup,True,,5.455527030004305,1,True
wide-292,negative_lookup,True,,3.0121111600019503,1,True
wide-293,negative_lookup,True,,2.800542648998089,1,True
wide-294,negative_lookup,True,,2.3347310710087186,1,True
wide-295,negative_lookup,True,,3.137155468997662,1,True
wide-296,negative_lookup,True,,4.624817319010617,1,True
wide-297,negative_lookup,True,,2.6547789269970963,1,True
wide-298,negative_lookup,True,,3.314983567004674,1,True
wide-299,negative_lookup,True,,4.695426902995678,1,True
wide-300,negative_lookup,True,,4.431371640996076,1,True
wide-301,negative_lookup,True,,6.186935618010466,1,True
wide-302,negative_lookup,True,,2.142409003005014,1,True
wide-303,negative_lookup,True,,3.43618969200179,1,True
wide-304,negative_lookup,True,,5.18562207800278,1,True
wide-305,negative_lookup,True,,4.38843695500691,1,True
wide-306,negative_lookup,True,,5.134904260004987,1,True
wide-307,negative_lookup,True,,2.6632863889972214,1,True
wide-308,negative_lookup,True,,3.1229624890111154,1,True
wide-309,negative_lookup,True,,4.524710755009437,1,True
wide-310,negative_lookup,True,,3.0614984529966023,1,True
wide-311,negative_lookup,True,,3.266464348009322,1,True
wide-312,negative_lookup,True,,2.7247217559925048,1,True
wide-313,negative_lookup,True,,4.237170866996166,1,True
wide-314,negative_lookup,True,,2.5491485459933756,1,True
wide-315,negative_lookup,True,,3.5208440470014466,1,True
wide-316,negative_lookup,True,,4.705917885992676,1,True
wide-317,negative_lookup,True,,4.300244467012817,1,True
wide-318,negative_lookup,True,,4.2143690299999435,1,True
wide-319,negative_lookup,True,,4.686885363000329,1,True
wide-320,negative_lookup,True,,3.806223741004942,1,True
1 id category strict error latency_seconds attempts first_pass_strict
2 wide-001 shared_keyword_pair True 12.681157764003729 1 True
3 wide-002 shared_keyword_pair True 7.440348913005437 1 True
4 wide-003 shared_keyword_pair True 6.8628689809993375 1 True
5 wide-004 shared_keyword_pair True 7.915003027010243 1 True
6 wide-005 shared_keyword_pair True 11.64334254800633 1 True
7 wide-006 shared_keyword_pair True 9.263334328003111 1 True
8 wide-007 shared_keyword_pair True 10.080044332993566 1 True
9 wide-008 shared_keyword_pair True 9.102414163004141 1 True
10 wide-009 shared_keyword_pair True 8.990864911000244 1 True
11 wide-010 shared_keyword_pair True 10.98848542298947 1 True
12 wide-011 shared_keyword_pair True 9.469930938998004 1 True
13 wide-012 shared_keyword_pair True 11.050285377990804 1 True
14 wide-013 shared_keyword_pair True 8.279340490000322 1 True
15 wide-014 shared_keyword_pair True 10.769162045005942 1 True
16 wide-015 shared_keyword_pair True 10.199135344999377 1 True
17 wide-016 shared_keyword_pair True 12.438964514003601 1 True
18 wide-017 shared_keyword_pair True 10.442358953005169 1 True
19 wide-018 shared_keyword_pair True 10.590256554991356 1 True
20 wide-019 shared_keyword_pair True 9.393680679990212 1 True
21 wide-020 shared_keyword_pair True 10.050156186000095 1 True
22 wide-021 shared_keyword_pair True 10.283129505012766 1 True
23 wide-022 shared_keyword_pair True 9.261356490998878 1 True
24 wide-023 shared_keyword_pair True 11.352632169000572 1 True
25 wide-024 shared_keyword_pair True 7.952293816997553 1 True
26 wide-025 shared_keyword_pair True 11.707699684004183 1 True
27 wide-026 shared_keyword_pair True 7.244572512994637 1 True
28 wide-027 shared_keyword_pair True 11.86879978099023 1 True
29 wide-028 shared_keyword_pair True 6.436980010010302 1 True
30 wide-029 shared_keyword_pair True 13.4112125190004 1 True
31 wide-030 shared_keyword_pair True 6.320292321004672 1 True
32 wide-031 shared_keyword_pair True 13.030589675996453 1 True
33 wide-032 shared_keyword_pair True 8.23478609599988 1 True
34 wide-033 shared_keyword_pair True 11.427597190995584 1 True
35 wide-034 shared_keyword_pair True 7.945492083003046 1 True
36 wide-035 shared_keyword_pair True 10.91667144499661 1 True
37 wide-036 shared_keyword_pair True 7.704312329005916 1 True
38 wide-037 shared_keyword_pair True 9.866947228001663 1 True
39 wide-038 shared_keyword_pair True 9.0944732450007 1 True
40 wide-039 shared_keyword_pair True 9.955326469003921 1 True
41 wide-040 shared_keyword_pair True 10.126550430999487 1 True
42 wide-041 same_academic_year_pair True 8.00595778299612 1 True
43 wide-042 same_academic_year_pair True 10.224936337006511 1 True
44 wide-043 same_academic_year_pair True 7.973652732995106 1 True
45 wide-044 same_academic_year_pair True 11.643997175007826 1 True
46 wide-045 same_academic_year_pair True 7.812763041001745 1 True
47 wide-046 same_academic_year_pair True 11.01834614200925 1 True
48 wide-047 same_academic_year_pair True 9.787194718010142 1 True
49 wide-048 same_academic_year_pair True 11.489561196009163 1 True
50 wide-049 same_academic_year_pair True 10.164007376006339 1 True
51 wide-050 same_academic_year_pair True 9.938125018001301 1 True
52 wide-051 same_academic_year_pair True 9.78176777700719 1 True
53 wide-052 same_academic_year_pair True 9.160806122003123 1 True
54 wide-053 same_academic_year_pair True 9.269089786990662 1 True
55 wide-054 same_academic_year_pair True 9.431611267005792 1 True
56 wide-055 same_academic_year_pair True 9.327960707989405 1 True
57 wide-056 same_academic_year_pair True 9.804446037000162 1 True
58 wide-057 same_academic_year_pair True 8.234563728998182 1 True
59 wide-058 same_academic_year_pair True 9.821240933990339 1 True
60 wide-059 same_academic_year_pair True 6.477854401004151 1 True
61 wide-060 same_academic_year_pair True 10.462991334003164 1 True
62 wide-061 same_academic_year_pair True 6.2554830580047565 1 True
63 wide-062 same_academic_year_pair True 10.774600299002486 1 True
64 wide-063 same_academic_year_pair True 6.288768262995291 1 True
65 wide-064 same_academic_year_pair True 11.15352754900232 1 True
66 wide-065 same_academic_year_pair True 5.480506381005398 1 True
67 wide-066 same_academic_year_pair True 10.729390800988767 1 True
68 wide-067 same_academic_year_pair True 7.499470337992534 1 True
69 wide-068 same_academic_year_pair True 11.094555151008535 1 True
70 wide-069 same_academic_year_pair True 7.680634538002778 1 True
71 wide-070 same_academic_year_pair True 10.041368676989805 1 True
72 wide-071 same_academic_year_pair True 9.15008538699476 1 True
73 wide-072 same_academic_year_pair True 9.846425030991668 1 True
74 wide-073 same_academic_year_pair True 8.89766675599094 1 True
75 wide-074 same_academic_year_pair True 8.42804765701294 1 True
76 wide-075 same_academic_year_pair True 8.904354819998844 1 True
77 wide-076 same_academic_year_pair True 8.653263764994335 1 True
78 wide-077 same_academic_year_pair True 8.350220588996308 1 True
79 wide-078 same_academic_year_pair True 9.132759956002701 1 True
80 wide-079 same_academic_year_pair True 8.293541837992962 1 True
81 wide-080 same_academic_year_pair True 10.156925554998452 1 True
82 wide-081 same_start_year_pair True 7.918382621995988 1 True
83 wide-082 same_start_year_pair True 10.321019872004399 1 True
84 wide-083 same_start_year_pair True 7.002511290003895 1 True
85 wide-084 same_start_year_pair True 10.437800674000755 1 True
86 wide-085 same_start_year_pair True 6.816185226009111 1 True
87 wide-086 same_start_year_pair True 10.71233119200042 1 True
88 wide-087 same_start_year_pair True 6.4720146839972585 1 True
89 wide-088 same_start_year_pair True 10.198497953009792 1 True
90 wide-089 same_start_year_pair True 6.79203049800708 1 True
91 wide-090 same_start_year_pair True 10.413568222007598 1 True
92 wide-091 same_start_year_pair True 7.510395374993095 1 True
93 wide-092 same_start_year_pair True 10.110175805006293 1 True
94 wide-093 same_start_year_pair True 7.382416222986649 1 True
95 wide-094 same_start_year_pair True 9.637755793010001 1 True
96 wide-095 same_start_year_pair True 7.721356611000374 1 True
97 wide-096 same_start_year_pair True 9.783646965996013 1 True
98 wide-097 same_start_year_pair True 8.19954388300539 1 True
99 wide-098 same_start_year_pair True 9.142149596998934 1 True
100 wide-099 same_start_year_pair True 7.995075654995162 1 True
101 wide-100 same_start_year_pair True 9.23304197700054 1 True
102 wide-101 same_start_year_pair True 8.445941337995464 1 True
103 wide-102 same_start_year_pair True 9.29434829299862 1 True
104 wide-103 same_start_year_pair True 8.262059757005773 1 True
105 wide-104 same_start_year_pair True 8.660338338988367 1 True
106 wide-105 same_start_year_pair True 8.136423168994952 1 True
107 wide-106 same_start_year_pair True 8.826232392006204 1 True
108 wide-107 same_start_year_pair True 8.023831066006096 1 True
109 wide-108 same_start_year_pair True 9.481848634008202 1 True
110 wide-109 same_start_year_pair True 7.782530160999158 1 True
111 wide-110 same_start_year_pair True 10.06139503599843 1 True
112 wide-111 same_start_year_pair True 7.194188486013445 1 True
113 wide-112 same_start_year_pair True 9.618882115988526 1 True
114 wide-113 person_title_year True 6.316762108996045 1 True
115 wide-114 person_title_year True 8.973571240989259 1 True
116 wide-115 person_title_year True 5.77858125900093 1 True
117 wide-116 person_title_year True 8.171107051006402 1 True
118 wide-117 person_title_year True 5.923224907994154 1 True
119 wide-118 person_title_year True 8.248448436992476 1 True
120 wide-119 person_title_year True 6.6704190939926775 1 True
121 wide-120 person_title_year True 7.390637057993445 1 True
122 wide-121 person_title_year True 6.828236892994028 1 True
123 wide-122 person_title_year True 7.436154732989962 1 True
124 wide-123 person_title_year True 7.201046439004131 1 True
125 wide-124 person_title_year True 7.452169609998236 1 True
126 wide-125 person_title_year True 7.024459202992148 1 True
127 wide-126 person_title_year True 7.134706636992632 1 True
128 wide-127 person_title_year True 6.701897156002815 1 True
129 wide-128 person_title_year True 7.7986834320036 1 True
130 wide-129 person_title_year False 7.78502085400396 1 False
131 wide-130 person_title_year True 7.579418053996051 1 True
132 wide-131 person_title_year True 7.29737891399418 1 True
133 wide-132 person_title_year True 6.5710127700003795 1 True
134 wide-133 person_title_year True 7.670121143004508 1 True
135 wide-134 person_title_year True 6.268109227006789 1 True
136 wide-135 person_title_year True 7.80891695599712 1 True
137 wide-136 person_title_year True 6.535969970005681 1 True
138 wide-137 person_keyword_title True 8.499110547010787 1 True
139 wide-138 person_keyword_title True 7.348516454003402 1 True
140 wide-139 person_keyword_title True 8.490620700002182 1 True
141 wide-140 person_keyword_title True 7.142724603007082 1 True
142 wide-141 person_keyword_title True 7.750912368996069 1 True
143 wide-142 person_keyword_title False 9.134571484988555 1 False
144 wide-143 person_keyword_title True 9.924621155005298 1 True
145 wide-144 person_keyword_title True 9.80673293399741 1 True
146 wide-145 person_keyword_title False 9.125109145010356 1 False
147 wide-146 person_keyword_title True 7.816306863998761 1 True
148 wide-147 person_keyword_title True 9.222393247997388 1 True
149 wide-148 person_keyword_title True 6.4315986309957225 1 True
150 wide-149 person_keyword_title True 8.975530234994949 1 True
151 wide-150 person_keyword_title True 6.5634262770036 1 True
152 wide-151 person_keyword_title True 8.981235069994 1 True
153 wide-152 person_keyword_title True 6.80172690100153 1 True
154 wide-153 person_keyword_title True 8.104526370007079 1 True
155 wide-154 person_keyword_title True 7.002095560004818 1 True
156 wide-155 person_keyword_title True 8.536377580996486 1 True
157 wide-156 person_keyword_title True 7.33249492100731 1 True
158 wide-157 person_keyword_title True 8.53190982498927 1 True
159 wide-158 person_keyword_title True 6.987181315998896 1 True
160 wide-159 person_keyword_title True 9.206439533008961 1 True
161 wide-160 person_keyword_title True 7.2076548890036065 1 True
162 wide-161 person_keyword_title True 9.755865868995897 1 True
163 wide-162 person_keyword_title True 6.598434728002758 1 True
164 wide-163 person_keyword_title True 9.665300851003849 1 True
165 wide-164 person_keyword_title True 6.409714323002845 1 True
166 wide-165 person_keyword_title True 9.189696266010287 1 True
167 wide-166 person_keyword_title True 6.362806601988268 1 True
168 wide-167 person_keyword_title True 9.12334693899902 1 True
169 wide-168 person_keyword_title True 6.939789495998411 1 True
170 wide-169 document_author_category_multi_hop True 8.274773390003247 1 True
171 wide-170 document_author_category_multi_hop True 5.761120869006845 1 True
172 wide-171 document_author_category_multi_hop True 7.484641218994511 1 True
173 wide-172 document_author_category_multi_hop True 5.7914811780065065 1 True
174 wide-173 document_author_category_multi_hop True 7.744038484990597 1 True
175 wide-174 document_author_category_multi_hop True 5.857030038998346 1 True
176 wide-175 document_author_category_multi_hop True 8.176481470989529 1 True
177 wide-176 document_author_category_multi_hop True 7.8557383849984035 1 True
178 wide-177 document_author_category_multi_hop True 7.886023836006643 1 True
179 wide-178 document_author_category_multi_hop True 7.453686430002563 1 True
180 wide-179 document_author_category_multi_hop True 5.2365985850046854 1 True
181 wide-180 document_author_category_multi_hop True 7.151028209991637 1 True
182 wide-181 document_author_category_multi_hop True 5.116186750004999 1 True
183 wide-182 document_author_category_multi_hop True 7.575539704004768 1 True
184 wide-183 document_author_category_multi_hop True 5.5021829689940205 1 True
185 wide-184 document_author_category_multi_hop True 7.557064304011874 1 True
186 wide-185 document_author_category_multi_hop True 5.2788128620013595 1 True
187 wide-186 document_author_category_multi_hop True 7.775028784992173 1 True
188 wide-187 document_author_category_multi_hop True 5.468235724998522 1 True
189 wide-188 document_author_category_multi_hop True 7.458582812003442 1 True
190 wide-189 document_author_category_multi_hop True 5.423984528999426 1 True
191 wide-190 document_author_category_multi_hop True 6.802945887990063 1 True
192 wide-191 document_author_category_multi_hop True 5.720807204997982 1 True
193 wide-192 document_author_category_multi_hop True 6.970476242990117 1 True
194 wide-193 person_multiple_categories True 5.315446266002255 1 True
195 wide-194 person_multiple_categories True 6.611033188004512 1 True
196 wide-195 person_multiple_categories True 4.514709312992636 1 True
197 wide-196 person_multiple_categories True 6.597614583995892 1 True
198 wide-197 person_multiple_categories True 4.054451817995869 1 True
199 wide-198 person_multiple_categories True 6.207983192012762 1 True
200 wide-199 person_multiple_categories True 4.590510104011628 1 True
201 wide-200 person_multiple_categories True 6.553350798989413 1 True
202 wide-201 person_multiple_categories True 4.605876973000704 1 True
203 wide-202 person_multiple_categories True 5.585335472002043 1 True
204 wide-203 person_multiple_categories True 4.492845022992697 1 True
205 wide-204 person_multiple_categories True 6.641766903994721 1 True
206 wide-205 person_multiple_categories True 5.62207466000109 1 True
207 wide-206 person_multiple_categories True 6.369739195986767 1 True
208 wide-207 person_multiple_categories True 4.450628942999174 1 True
209 wide-208 person_multiple_categories True 5.57498624100117 1 True
210 wide-209 person_multiple_categories True 5.34890157400514 1 True
211 wide-210 person_multiple_categories True 6.2012351820012555 1 True
212 wide-211 person_multiple_categories True 5.767825509014074 1 True
213 wide-212 person_multiple_categories True 5.083963163997396 1 True
214 wide-213 person_multiple_categories True 5.3684891809971305 1 True
215 wide-214 person_multiple_categories True 4.774883028992917 1 True
216 wide-215 person_multiple_categories True 5.859735618010745 1 True
217 wide-216 person_multiple_categories True 5.045369378000032 1 True
218 wide-217 candidate_set_aggregation True 8.18277867299912 1 True
219 wide-218 candidate_set_aggregation True 7.061751248998917 1 True
220 wide-219 candidate_set_aggregation True 9.888737866000156 1 True
221 wide-220 candidate_set_aggregation True 6.534448446996976 1 True
222 wide-221 candidate_set_aggregation True 10.02607991900004 1 True
223 wide-222 candidate_set_aggregation True 6.639642160007497 1 True
224 wide-223 candidate_set_aggregation True 10.05624425499991 1 True
225 wide-224 candidate_set_aggregation True 6.380346581994672 1 True
226 wide-225 candidate_set_aggregation True 9.750748607999412 1 True
227 wide-226 candidate_set_aggregation True 6.927533381007379 1 True
228 wide-227 candidate_set_aggregation True 9.749435025994899 1 True
229 wide-228 candidate_set_aggregation True 7.000018507009372 1 True
230 wide-229 candidate_set_aggregation True 9.393544332007878 1 True
231 wide-230 candidate_set_aggregation True 7.075158320993069 1 True
232 wide-231 candidate_set_aggregation True 9.68925523001235 1 True
233 wide-232 candidate_set_aggregation True 7.470996176998597 1 True
234 wide-233 candidate_set_aggregation True 9.546211912995204 1 True
235 wide-234 candidate_set_aggregation True 7.121646208994207 1 True
236 wide-235 candidate_set_aggregation True 9.106454324006336 1 True
237 wide-236 candidate_set_aggregation True 7.33974453000701 1 True
238 wide-237 candidate_set_aggregation True 9.92777099799423 1 True
239 wide-238 candidate_set_aggregation True 7.6532076459989185 1 True
240 wide-239 candidate_set_aggregation True 9.878883134995704 1 True
241 wide-240 candidate_set_aggregation True 6.794067700000596 1 True
242 wide-241 candidate_set_aggregation True 9.484167262999108 1 True
243 wide-242 candidate_set_aggregation True 6.586464612002601 1 True
244 wide-243 candidate_set_aggregation True 9.679727968003135 1 True
245 wide-244 candidate_set_aggregation True 6.853099594998639 1 True
246 wide-245 candidate_set_aggregation True 9.588607818004675 1 True
247 wide-246 candidate_set_aggregation True 6.615246096000192 1 True
248 wide-247 candidate_set_aggregation True 9.338731531999656 1 True
249 wide-248 candidate_set_aggregation True 6.656504976010183 1 True
250 wide-249 conjunctive_disambiguation True 8.06964380700083 1 True
251 wide-250 conjunctive_disambiguation True 5.727493058991968 1 True
252 wide-251 conjunctive_disambiguation True 7.403562118997797 1 True
253 wide-252 conjunctive_disambiguation True 6.389556170004653 1 True
254 wide-253 conjunctive_disambiguation True 7.190451546994154 1 True
255 wide-254 conjunctive_disambiguation True 5.571824425001978 1 True
256 wide-255 conjunctive_disambiguation True 6.363713938990259 1 True
257 wide-256 conjunctive_disambiguation True 5.9705328209965955 1 True
258 wide-257 conjunctive_disambiguation True 6.814727099001175 1 True
259 wide-258 conjunctive_disambiguation True 6.060012061992893 1 True
260 wide-259 conjunctive_disambiguation True 6.477906736996374 1 True
261 wide-260 conjunctive_disambiguation True 6.108023989989306 1 True
262 wide-261 conjunctive_disambiguation True 6.89838379000139 1 True
263 wide-262 conjunctive_disambiguation True 6.639250446998631 1 True
264 wide-263 conjunctive_disambiguation True 7.647332847991493 1 True
265 wide-264 conjunctive_disambiguation True 6.806039610994048 1 True
266 wide-265 conjunctive_disambiguation True 7.480212347989436 1 True
267 wide-266 conjunctive_disambiguation True 5.939997524998034 1 True
268 wide-267 conjunctive_disambiguation True 7.483691226007068 1 True
269 wide-268 conjunctive_disambiguation True 5.128309691994218 1 True
270 wide-269 conjunctive_disambiguation True 7.950514240990742 1 True
271 wide-270 conjunctive_disambiguation True 3.838128978997702 1 True
272 wide-271 conjunctive_disambiguation True 7.203258510999149 1 True
273 wide-272 conjunctive_disambiguation True 5.274275976000354 1 True
274 wide-273 conjunctive_disambiguation True 7.145530178997433 1 True
275 wide-274 conjunctive_disambiguation True 5.765552001001197 1 True
276 wide-275 conjunctive_disambiguation True 7.404864780008211 1 True
277 wide-276 conjunctive_disambiguation True 6.352571280993288 1 True
278 wide-277 conjunctive_disambiguation True 7.504300720000174 1 True
279 wide-278 conjunctive_disambiguation True 6.016011193001759 1 True
280 wide-279 conjunctive_disambiguation True 7.295396494999295 1 True
281 wide-280 conjunctive_disambiguation True 5.834808207000606 1 True
282 wide-281 conjunctive_disambiguation True 7.8035292659915285 1 True
283 wide-282 conjunctive_disambiguation True 6.399204970992287 1 True
284 wide-283 conjunctive_disambiguation True 9.671940870990511 1 True
285 wide-284 conjunctive_disambiguation True 5.51546201200108 1 True
286 wide-285 conjunctive_disambiguation True 10.154824249999365 1 True
287 wide-286 conjunctive_disambiguation True 4.581112927000504 1 True
288 wide-287 conjunctive_disambiguation True 9.536005122994538 1 True
289 wide-288 conjunctive_disambiguation True 5.152376526006265 1 True
290 wide-289 negative_lookup True 5.77029365299677 1 True
291 wide-290 negative_lookup True 4.0657596189994365 1 True
292 wide-291 negative_lookup True 5.455527030004305 1 True
293 wide-292 negative_lookup True 3.0121111600019503 1 True
294 wide-293 negative_lookup True 2.800542648998089 1 True
295 wide-294 negative_lookup True 2.3347310710087186 1 True
296 wide-295 negative_lookup True 3.137155468997662 1 True
297 wide-296 negative_lookup True 4.624817319010617 1 True
298 wide-297 negative_lookup True 2.6547789269970963 1 True
299 wide-298 negative_lookup True 3.314983567004674 1 True
300 wide-299 negative_lookup True 4.695426902995678 1 True
301 wide-300 negative_lookup True 4.431371640996076 1 True
302 wide-301 negative_lookup True 6.186935618010466 1 True
303 wide-302 negative_lookup True 2.142409003005014 1 True
304 wide-303 negative_lookup True 3.43618969200179 1 True
305 wide-304 negative_lookup True 5.18562207800278 1 True
306 wide-305 negative_lookup True 4.38843695500691 1 True
307 wide-306 negative_lookup True 5.134904260004987 1 True
308 wide-307 negative_lookup True 2.6632863889972214 1 True
309 wide-308 negative_lookup True 3.1229624890111154 1 True
310 wide-309 negative_lookup True 4.524710755009437 1 True
311 wide-310 negative_lookup True 3.0614984529966023 1 True
312 wide-311 negative_lookup True 3.266464348009322 1 True
313 wide-312 negative_lookup True 2.7247217559925048 1 True
314 wide-313 negative_lookup True 4.237170866996166 1 True
315 wide-314 negative_lookup True 2.5491485459933756 1 True
316 wide-315 negative_lookup True 3.5208440470014466 1 True
317 wide-316 negative_lookup True 4.705917885992676 1 True
318 wide-317 negative_lookup True 4.300244467012817 1 True
319 wide-318 negative_lookup True 4.2143690299999435 1 True
320 wide-319 negative_lookup True 4.686885363000329 1 True
321 wide-320 negative_lookup True 3.806223741004942 1 True

45140
tools/graphrag/evaluation/results/answers_v5_1_regression_2026-10-25/answers_regression.json Normal file
View File

File diff suppressed because it is too large Load Diff

411
tools/graphrag/evaluation/results/answers_v5_1_regression_2026-10-25/failures.json Normal file
View File

@ -0,0 +1,411 @@
[
{
"id": "wide-129",
"category": "person_title_year",
"question": "Uveď názov diplomového projektu osoby Maroš Harahus a rok práce pre rok 2020.",
"graph_context": {
"status": "ok",
"reason": null,
"facts": [
{
"person": "Maroš Harahus",
"title": "Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair)",
"year": 2020,
"document_path": "pages/students/2016/maros_harahus/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
}
],
"sources": [
{
"document_path": "pages/students/2016/maros_harahus/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
}
],
"aggregation": null
},
"strict": false,
"attempts": [
{
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n\"answer\": \"Maroš Harahus v roku 2020 vypracoval diplomový projekt s názvom Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair). Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\",\n\"status\": \"ok\",\n\"claims\": [\n{\n\"person\": \"Maroš Harahus\",\n\"title\": \"Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair)\",\n\"year\": 2020,\n\"keyword\": \"diplomový projekt\"\n}\n],\n\"source_urls\": [\n\"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\"\n],\n\"count\": 1\n}"
}
}
],
"created": 1792945390,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 207,
"prompt_tokens": 1060,
"total_tokens": 1267,
"prompt_tokens_details": {
"cached_tokens": 741
}
},
"id": "chatcmpl-g1NApedgMI4Zy7dBR2SrY5S62SKWblvZ",
"timings": {
"cache_n": 741,
"prompt_n": 319,
"prompt_ms": 526.979,
"prompt_per_token_ms": 1.651971786833856,
"prompt_per_second": 605.3372145759129,
"predicted_n": 207,
"predicted_ms": 4996.334,
"predicted_per_token_ms": 24.13687922705314,
"predicted_per_second": 41.430376752234736
}
},
"format_normalized": false,
"strict": false
}
],
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n\"answer\": \"Maroš Harahus v roku 2020 vypracoval diplomový projekt s názvom Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair). Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\",\n\"status\": \"ok\",\n\"claims\": [\n{\n\"person\": \"Maroš Harahus\",\n\"title\": \"Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair)\",\n\"year\": 2020,\n\"keyword\": \"diplomový projekt\"\n}\n],\n\"source_urls\": [\n\"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\"\n],\n\"count\": 1\n}"
}
}
],
"created": 1792945390,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 207,
"prompt_tokens": 1060,
"total_tokens": 1267,
"prompt_tokens_details": {
"cached_tokens": 741
}
},
"id": "chatcmpl-g1NApedgMI4Zy7dBR2SrY5S62SKWblvZ",
"timings": {
"cache_n": 741,
"prompt_n": 319,
"prompt_ms": 526.979,
"prompt_per_token_ms": 1.651971786833856,
"prompt_per_second": 605.3372145759129,
"predicted_n": 207,
"predicted_ms": 4996.334,
"predicted_per_token_ms": 24.13687922705314,
"predicted_per_second": 41.430376752234736
}
},
"answer": {
"answer": "Maroš Harahus v roku 2020 vypracoval diplomový projekt s názvom Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair). Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
"status": "ok",
"claims": [
{
"person": "Maroš Harahus",
"title": "Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair)",
"year": 2020,
"keyword": "diplomový projekt"
}
],
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
],
"count": 1
},
"bindings_ok": true,
"unsupported_claims": [
{
"person": "Maroš Harahus",
"title": "Doplnenie podpory morfologického značkovania slovenského jazyka do nlp frameworku (spacy alebo flair)",
"year": 2020,
"keyword": "diplomový projekt"
}
],
"sources_ok": true,
"status_ok": true,
"count_ok": true,
"set_ok": true,
"text_values_ok": true,
"missing_text_values": [],
"contradictory_thesis_label": false,
"latency_seconds": 7.78502085400396
},
{
"id": "wide-142",
"category": "person_keyword_title",
"question": "Uveď názov práce osoby Dárius Lindvai v roku 2019 s kľúčovým slovom lstm.",
"graph_context": {
"status": "ok",
"reason": null,
"facts": [
{
"person": "Dárius Lindvai",
"title": "rešerš",
"year": 2019,
"document_path": "pages/students/2016/darius_lindvai/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
},
{
"person": "Dárius Lindvai",
"keyword": "lstm",
"document_path": "pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
}
],
"sources": [
{
"document_path": "pages/students/2016/darius_lindvai/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
},
{
"document_path": "pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
}
],
"aggregation": null
},
"strict": false,
"attempts": [
{
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\"answer\": \"V roku 2019 má Dárius Lindvai prácu s názvom \\\"rešerš\\\", ale v dodaných grafových faktoch nie je pre túto prácu (dokument pages/students/2016/darius_lindvai/README.md) uvedený kľúčové slovo \\\"lstm\\\". Kľúčové slovo \\\"lstm\\\" je priradené k inému dokumentu (pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md), pre ktorý nie je v roku 2019 uvedený názov práce. Zdroje: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai, https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\", \"status\": \"no_answer\", \"claims\": [], \"source_urls\": [\"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\", \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"], \"count\": 0}"
}
}
],
"created": 1792945424,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 271,
"prompt_tokens": 1179,
"total_tokens": 1450,
"prompt_tokens_details": {
"cached_tokens": 757
}
},
"id": "chatcmpl-GN6RM6r1ZLYbSY2fIyiP4vJ01UYBpkFB",
"timings": {
"cache_n": 757,
"prompt_n": 422,
"prompt_ms": 564.135,
"prompt_per_token_ms": 1.3368127962085308,
"prompt_per_second": 748.0478963368697,
"predicted_n": 271,
"predicted_ms": 6580.341,
"predicted_per_token_ms": 24.28170110701107,
"predicted_per_second": 41.18327606426475
}
},
"format_normalized": false,
"strict": false
}
],
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\"answer\": \"V roku 2019 má Dárius Lindvai prácu s názvom \\\"rešerš\\\", ale v dodaných grafových faktoch nie je pre túto prácu (dokument pages/students/2016/darius_lindvai/README.md) uvedený kľúčové slovo \\\"lstm\\\". Kľúčové slovo \\\"lstm\\\" je priradené k inému dokumentu (pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md), pre ktorý nie je v roku 2019 uvedený názov práce. Zdroje: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai, https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\", \"status\": \"no_answer\", \"claims\": [], \"source_urls\": [\"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\", \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"], \"count\": 0}"
}
}
],
"created": 1792945424,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 271,
"prompt_tokens": 1179,
"total_tokens": 1450,
"prompt_tokens_details": {
"cached_tokens": 757
}
},
"id": "chatcmpl-GN6RM6r1ZLYbSY2fIyiP4vJ01UYBpkFB",
"timings": {
"cache_n": 757,
"prompt_n": 422,
"prompt_ms": 564.135,
"prompt_per_token_ms": 1.3368127962085308,
"prompt_per_second": 748.0478963368697,
"predicted_n": 271,
"predicted_ms": 6580.341,
"predicted_per_token_ms": 24.28170110701107,
"predicted_per_second": 41.18327606426475
}
},
"answer": {
"answer": "V roku 2019 má Dárius Lindvai prácu s názvom \"rešerš\", ale v dodaných grafových faktoch nie je pre túto prácu (dokument pages/students/2016/darius_lindvai/README.md) uvedený kľúčové slovo \"lstm\". Kľúčové slovo \"lstm\" je priradené k inému dokumentu (pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md), pre ktorý nie je v roku 2019 uvedený názov práce. Zdroje: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai, https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial",
"status": "no_answer",
"claims": [],
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
],
"count": 0
},
"bindings_ok": false,
"unsupported_claims": [],
"sources_ok": true,
"status_ok": false,
"count_ok": true,
"set_ok": true,
"text_values_ok": true,
"missing_text_values": [],
"contradictory_thesis_label": false,
"latency_seconds": 9.134571484988555
},
{
"id": "wide-145",
"category": "person_keyword_title",
"question": "Uveď názov práce osoby Dárius Lindvai v roku 2019 s kľúčovým slovom python.",
"graph_context": {
"status": "ok",
"reason": null,
"facts": [
{
"person": "Dárius Lindvai",
"title": "rešerš",
"year": 2019,
"document_path": "pages/students/2016/darius_lindvai/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
},
{
"person": "Dárius Lindvai",
"keyword": "python",
"document_path": "pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
}
],
"sources": [
{
"document_path": "pages/students/2016/darius_lindvai/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
},
{
"document_path": "pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
},
{
"document_path": "pages/topics/python/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/topics/python"
}
],
"aggregation": null
},
"strict": false,
"attempts": [
{
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n\"answer\": \"Nenašli sa žiadne fakty o práci Dárius Lindvai z roku 2019 s kľúčovým slovom python. Zdrojové údaje uvádzajú pre Dárius Lindvai v roku 2019 prácu s názvom \\\"rešerš\\\" (https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai), ale kľúčové slovo \\\"python\\\" je spojené s iným dokumentom bez uvedeného roku a názvu práce (https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial).\",\n\"status\": \"no_answer\",\n\"claims\": [],\n\"source_urls\": [\n\"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\",\n\"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"\n],\n\"count\": 0\n}"
}
}
],
"created": 1792945433,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 235,
"prompt_tokens": 1208,
"total_tokens": 1443,
"prompt_tokens_details": {
"cached_tokens": 741
}
},
"id": "chatcmpl-wedgIgmgk8EORiFxPHOLuxXJnrOP8YqX",
"timings": {
"cache_n": 741,
"prompt_n": 467,
"prompt_ms": 621.672,
"prompt_per_token_ms": 1.331203426124197,
"prompt_per_second": 751.1999897051821,
"predicted_n": 235,
"predicted_ms": 5720.465,
"predicted_per_token_ms": 24.34240425531915,
"predicted_per_second": 41.080576491596396
}
},
"format_normalized": false,
"strict": false
}
],
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n\"answer\": \"Nenašli sa žiadne fakty o práci Dárius Lindvai z roku 2019 s kľúčovým slovom python. Zdrojové údaje uvádzajú pre Dárius Lindvai v roku 2019 prácu s názvom \\\"rešerš\\\" (https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai), ale kľúčové slovo \\\"python\\\" je spojené s iným dokumentom bez uvedeného roku a názvu práce (https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial).\",\n\"status\": \"no_answer\",\n\"claims\": [],\n\"source_urls\": [\n\"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\",\n\"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"\n],\n\"count\": 0\n}"
}
}
],
"created": 1792945433,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 235,
"prompt_tokens": 1208,
"total_tokens": 1443,
"prompt_tokens_details": {
"cached_tokens": 741
}
},
"id": "chatcmpl-wedgIgmgk8EORiFxPHOLuxXJnrOP8YqX",
"timings": {
"cache_n": 741,
"prompt_n": 467,
"prompt_ms": 621.672,
"prompt_per_token_ms": 1.331203426124197,
"prompt_per_second": 751.1999897051821,
"predicted_n": 235,
"predicted_ms": 5720.465,
"predicted_per_token_ms": 24.34240425531915,
"predicted_per_second": 41.080576491596396
}
},
"answer": {
"answer": "Nenašli sa žiadne fakty o práci Dárius Lindvai z roku 2019 s kľúčovým slovom python. Zdrojové údaje uvádzajú pre Dárius Lindvai v roku 2019 prácu s názvom \"rešerš\" (https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai), ale kľúčové slovo \"python\" je spojené s iným dokumentom bez uvedeného roku a názvu práce (https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial).",
"status": "no_answer",
"claims": [],
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
],
"count": 0
},
"bindings_ok": false,
"unsupported_claims": [],
"sources_ok": true,
"status_ok": false,
"count_ok": true,
"set_ok": true,
"text_values_ok": true,
"missing_text_values": [],
"contradictory_thesis_label": false,
"latency_seconds": 9.125109145010356
}
]

706
tools/graphrag/evaluation/results/answers_v5_1_regression_2026-10-25/frozen_answers.json Normal file
View File

@ -0,0 +1,706 @@
{
"method": "Previously exposed LLM questions; regression, not independent TEST.",
"graph_freeze_sha256": "ca1d2d71f60a8e65b82d727ad14556aabf2e03f2acdce1755c3e2a7a9a3149d3",
"audit_sha256": "81708ca63f8772673cd836fe7a1a1938fbe0e713588efcb07ed3fd4280f27829",
"generation_files": {
"/home/janko/DP/zp-agent/tools/answer_evaluation/evaluate_graph_answers.py": "3b58e210fcd934a5488575e1b0ebb46a2e94ce72ee12f1e1a666c726c66d68af",
"/home/janko/DP/zp-agent/tools/answer_evaluation/graph_answer_prompt.txt": "64d2bd817657f390c306ea01bf2dad41f70cd6993e06b2e3b2f4713bae0d249c",
"/home/janko/DP/zp-agent/tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/rag/evaluation/rag_runner.py": "a011b29e2b6aa47df9350e74fc358f05780a355170a819dbb1cfd16d2971493b"
},
"model": "model120-fast",
"temperature": 0,
"workers": 3,
"dev_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"test_ids": [
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
],
"dev_count": 39,
"test_count": 300,
"target_accuracy": 0.92,
"selected_split": "regression",
"selected_ids": [
"wide-001",
"wide-002",
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-041",
"wide-042",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-081",
"wide-082",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-113",
"wide-114",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-137",
"wide-138",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-169",
"wide-170",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-193",
"wide-194",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-217",
"wide-218",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-249",
"wide-250",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-289",
"wide-290",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
],
"selected_count": 320,
"selected_dev_overlap": 39,
"format_repair_limit": 1,
"repair_policy": "Only JSON/schema failures; same question/context, no gold; retain all attempts.",
"dev_results_sha256": "cd9b206a5a89602441e5d3b47c9de2febfd8f34ef4ec32cf20f567079761e619",
"policy": "Evaluate every declared regression case, including DEV overlap; retain all failures and stop tuning after this freeze.",
"selected_categories": {
"shared_keyword_pair": 40,
"same_academic_year_pair": 40,
"same_start_year_pair": 32,
"person_title_year": 24,
"person_keyword_title": 32,
"document_author_category_multi_hop": 24,
"person_multiple_categories": 24,
"candidate_set_aggregation": 32,
"conjunctive_disambiguation": 40,
"negative_lookup": 32
},
"limitations": [
"Known corpus/profiles and templates; model outputs for these 300 questions were not previously observed.",
"Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.",
"Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose."
]
}

40
tools/graphrag/evaluation/results/answers_v5_dev_2026-10-25/answers_dev.csv Normal file
View File

@ -0,0 +1,40 @@
id,category,strict,error,latency_seconds,attempts,first_pass_strict
wide-001,shared_keyword_pair,True,,13.102745643991511,1,True
wide-002,shared_keyword_pair,True,,8.686387695997837,1,True
wide-041,same_academic_year_pair,True,,7.175007558995276,1,True
wide-042,same_academic_year_pair,True,,7.3540544569987105,1,True
wide-080,same_academic_year_pair,True,,9.493409784001415,1,True
wide-081,same_start_year_pair,True,,7.360842023990699,1,True
wide-082,same_start_year_pair,True,,9.634056855007657,1,True
wide-104,same_start_year_pair,True,,7.5196549139946,1,True
wide-113,person_title_year,True,,8.4268134659942,1,True
wide-114,person_title_year,True,,6.6590490480011795,1,True
wide-119,person_title_year,True,,8.043334904010408,1,True
wide-137,person_keyword_title,True,,7.18094556601136,1,True
wide-138,person_keyword_title,True,,8.415209711994976,1,True
wide-139,person_keyword_title,True,,7.5367714180029,1,True
wide-141,person_keyword_title,True,,8.302764993000892,1,True
wide-142,person_keyword_title,True,,9.15925429300114,1,True
wide-143,person_keyword_title,True,,11.15336542800651,1,True
wide-145,person_keyword_title,False,,10.935504723005579,1,False
wide-155,person_keyword_title,True,,9.731682198005728,1,True
wide-159,person_keyword_title,True,,8.722476019000169,1,True
wide-161,person_keyword_title,True,,8.795275380005478,1,True
wide-163,person_keyword_title,True,,8.375112182999146,1,True
wide-169,document_author_category_multi_hop,True,,7.371920378995128,1,True
wide-170,document_author_category_multi_hop,True,,6.7976903180097,1,True
wide-178,document_author_category_multi_hop,True,,6.5963496360054705,1,True
wide-181,document_author_category_multi_hop,True,,6.7874484959902475,1,True
wide-193,person_multiple_categories,True,,5.843567528994754,1,True
wide-194,person_multiple_categories,True,,5.834254229994258,1,True
wide-209,person_multiple_categories,True,,6.06473808499868,1,True
wide-217,candidate_set_aggregation,True,,8.168832301991642,1,True
wide-218,candidate_set_aggregation,True,,8.408147340000141,1,True
wide-232,candidate_set_aggregation,True,,9.107497649005381,1,True
wide-236,candidate_set_aggregation,True,,8.041833495008177,1,True
wide-239,candidate_set_aggregation,True,,8.753410860008444,1,True
wide-249,conjunctive_disambiguation,True,,6.555928866990143,1,True
wide-250,conjunctive_disambiguation,True,,7.249271184002282,1,True
wide-283,conjunctive_disambiguation,True,,6.916102559000137,1,True
wide-289,negative_lookup,True,,4.995769290995668,1,True
wide-290,negative_lookup,True,,4.3209586270095315,1,True
1 id category strict error latency_seconds attempts first_pass_strict
2 wide-001 shared_keyword_pair True 13.102745643991511 1 True
3 wide-002 shared_keyword_pair True 8.686387695997837 1 True
4 wide-041 same_academic_year_pair True 7.175007558995276 1 True
5 wide-042 same_academic_year_pair True 7.3540544569987105 1 True
6 wide-080 same_academic_year_pair True 9.493409784001415 1 True
7 wide-081 same_start_year_pair True 7.360842023990699 1 True
8 wide-082 same_start_year_pair True 9.634056855007657 1 True
9 wide-104 same_start_year_pair True 7.5196549139946 1 True
10 wide-113 person_title_year True 8.4268134659942 1 True
11 wide-114 person_title_year True 6.6590490480011795 1 True
12 wide-119 person_title_year True 8.043334904010408 1 True
13 wide-137 person_keyword_title True 7.18094556601136 1 True
14 wide-138 person_keyword_title True 8.415209711994976 1 True
15 wide-139 person_keyword_title True 7.5367714180029 1 True
16 wide-141 person_keyword_title True 8.302764993000892 1 True
17 wide-142 person_keyword_title True 9.15925429300114 1 True
18 wide-143 person_keyword_title True 11.15336542800651 1 True
19 wide-145 person_keyword_title False 10.935504723005579 1 False
20 wide-155 person_keyword_title True 9.731682198005728 1 True
21 wide-159 person_keyword_title True 8.722476019000169 1 True
22 wide-161 person_keyword_title True 8.795275380005478 1 True
23 wide-163 person_keyword_title True 8.375112182999146 1 True
24 wide-169 document_author_category_multi_hop True 7.371920378995128 1 True
25 wide-170 document_author_category_multi_hop True 6.7976903180097 1 True
26 wide-178 document_author_category_multi_hop True 6.5963496360054705 1 True
27 wide-181 document_author_category_multi_hop True 6.7874484959902475 1 True
28 wide-193 person_multiple_categories True 5.843567528994754 1 True
29 wide-194 person_multiple_categories True 5.834254229994258 1 True
30 wide-209 person_multiple_categories True 6.06473808499868 1 True
31 wide-217 candidate_set_aggregation True 8.168832301991642 1 True
32 wide-218 candidate_set_aggregation True 8.408147340000141 1 True
33 wide-232 candidate_set_aggregation True 9.107497649005381 1 True
34 wide-236 candidate_set_aggregation True 8.041833495008177 1 True
35 wide-239 candidate_set_aggregation True 8.753410860008444 1 True
36 wide-249 conjunctive_disambiguation True 6.555928866990143 1 True
37 wide-250 conjunctive_disambiguation True 7.249271184002282 1 True
38 wide-283 conjunctive_disambiguation True 6.916102559000137 1 True
39 wide-289 negative_lookup True 4.995769290995668 1 True
40 wide-290 negative_lookup True 4.3209586270095315 1 True

6079
tools/graphrag/evaluation/results/answers_v5_dev_2026-10-25/answers_dev.json Normal file
View File

File diff suppressed because it is too large Load Diff

425
tools/graphrag/evaluation/results/answers_v5_dev_2026-10-25/frozen_answers.json Normal file
View File

@ -0,0 +1,425 @@
{
"method": "Previously exposed LLM questions; regression, not independent TEST.",
"graph_freeze_sha256": "0dc97fdba37557d76392538eb122e2eafd5f58480fee7de347e1f5ca251823d4",
"audit_sha256": "091e2ee00a88e9e6a459e91fb54efabd23eec5eb4dd70570aa313c3e51e93410",
"generation_files": {
"/home/janko/DP/zp-agent/tools/answer_evaluation/evaluate_graph_answers.py": "3b58e210fcd934a5488575e1b0ebb46a2e94ce72ee12f1e1a666c726c66d68af",
"/home/janko/DP/zp-agent/tools/answer_evaluation/graph_answer_prompt.txt": "801dcd48be8014b65667ced52810b75b43f9b7c525fdbfcbd08fd8f668673016",
"/home/janko/DP/zp-agent/tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/rag/evaluation/rag_runner.py": "a011b29e2b6aa47df9350e74fc358f05780a355170a819dbb1cfd16d2971493b"
},
"model": "model120-fast",
"temperature": 0,
"workers": 3,
"dev_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"test_ids": [
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
],
"dev_count": 39,
"test_count": 300,
"target_accuracy": 0.92,
"selected_split": "dev",
"selected_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"selected_count": 39,
"selected_dev_overlap": 39,
"format_repair_limit": 1,
"repair_policy": "Only JSON/schema failures; same question/context, no gold; retain all attempts.",
"dev_results_sha256": null,
"policy": "Evaluate every declared regression case, including DEV overlap; retain all failures and stop tuning after this freeze.",
"selected_categories": {
"shared_keyword_pair": 2,
"same_academic_year_pair": 3,
"same_start_year_pair": 3,
"person_title_year": 3,
"person_keyword_title": 11,
"document_author_category_multi_hop": 4,
"person_multiple_categories": 3,
"candidate_set_aggregation": 5,
"conjunctive_disambiguation": 3,
"negative_lookup": 2
},
"limitations": [
"Known corpus/profiles and templates; model outputs for these 300 questions were not previously observed.",
"Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.",
"Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose."
]
}

123
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/REPORT.md Normal file
View File

@ -0,0 +1,123 @@
# GraphRAG v5.1 — uzavretie bodu 3, 25. 10. 2026
Finálna regresia grafového vyhľadávania má **320/320**, vrátane zdrojovej
kontroly všetkých vrátených typovaných faktov. DEV má **722/722**.
Finálne generovanie dosiahlo **317/320 — 99.06 %**.
Cieľ ≥92 % je **splnený**.
Ide o **historickú regresiu**, nie o nový nezávislý TEST. Žiadne nové otázky
neboli potrebné pre kontrolu opráv; pôvodné otázky, gold ani výsledky v1–v4
sa nemenili. Zistenia sa nesmú vydávať za univerzálnu pravdivosť odpovedí.
## Protokol a dátové rozdelenie
Používa sa pôvodný v4 benchmark: 722 DEV a 320 historických TEST otázok,
v desiatich kategóriách, z toho 288 kladných a 32 záporných. Všetky otázky,
osoby aj šablóny boli odhalené pred touto verziou. Zdrojom je raw Markdown
repozitára `jp170na/zpwiki`, commit `3e892a1d49e9bce118cd03478ce15e01c2bebb4e`.
DEV generovania má 39 vopred deklarovaných otázok: pôvodných 20 DEV a 19
historických zlyhaní modelu. Finálna regresia zahŕňa všetkých 320 otázok
vrátane týchto 39; prekryv je uvedený v `frozen_answers.json`. Výber sa nemenil
podľa úspechu aktuálneho finálneho behu a všetky zlyhania ostávajú v menovateli.
## Opravy a história iterácií
1. Organizačná príprava na projekt nevytvára akademický záznam ani rok práce.
`wide-080` teraz vracia iba zdrojovo doložené spoločné akademické roky.
2. Označenie návrhu zadania bez explicitného názvu nepreukazuje názov práce.
Prvý prísny DEV mal 719/722 pre tri nadbytočné názvy pri Martinovi Šarišskom;
nasledovala oprava parsera bez zmeny benchmarkových očakávaní.
3. Prvá zmrazená v5 mala regresiu 319/320. Pri `wide-275` parser považoval vetu
o odporúčanom čase konzultácie za názov projektu na hranici chunku.
V5.1 také organizačné vety odmieta. V5 a jej 38/39 DEV generovania ostávajú
archivované; overený patch umožňuje rekonštrukciu prvej iterácie.
4. Evaluator kontroluje `complete=false` aj nezávisle od `truncated`, všetky
vrátené typované fakty, vlastnosť–hodnotu pri správnej práci/osobe/dokumente,
druh práce, metadata, URL aj jednotlivé a spoločné roky.
5. Prompt vysvetľuje, že tag aj akademický názov môžu pochádzať z odlišných
dokumentov tej istej osoby. Požadovaná spojená väzba sa uvádza v jednej claim.
6. JSON sa kontroluje typovanou schémou. Jediný celý JSON blok sa môže vybrať
z Markdown ohraničenia bez prepisovania hodnôt. Pri chybe formátu je povolený
najviac jeden opravný požiadavok, s rovnakou otázkou a kontextom, bez gold.
Vecná chyba hodnotenia nespúšťa opravný pokus.
Po finálnom zmrazení v5.1 sa kód, prompt ani gold podľa finálneho merania
nemenili. Správa otvorene uvádza zostávajúce zlyhania.
## Výsledky
| Meranie | Správne | Strict úspešnosť |
|---|---:|---:|
| Graf DEV | 722/722 | 100,00 % |
| Graf historická regresia | 320/320 | 100,00 % |
| Zdrojovo podporené typované fakty, regresné výstupy | 320/320 | 100,00 % |
| LLM DEV | 38/39 | 97.44 % |
| LLM finálna regresia bez opravného požiadavku | 317/320 | 99.06 % |
| LLM finálna regresia po deklarovanom spracovaní | 317/320 | 99.06 % |
Bez opravného požiadavku znamená po konzervatívnom parsovaní, prípadne po
vybratí jediného úplného JSON bloku. Takto normalizovaných prvých výstupov:
**0**. Opravný požiadavok na formát sa použil
pri **0** otázkach. Finálne chyby spracovania alebo
transportu: **0**. Pôvodné odpovede a všetky opravné pokusy sú uložené.
| Kategória LLM regresie | Správne | Strict úspešnosť |
|---|---:|---:|
| Výber a počet kandidátov | 32/32 | 100.00 % |
| Tag a rok pri tej istej osobe | 40/40 | 100.00 % |
| Osoba, kategória a autor dokumentu | 24/24 | 100.00 % |
| Nenájdené údaje/osoby | 32/32 | 100.00 % |
| Názov a kľúčové slovo | 30/32 | 93.75 % |
| Úplná množina kategórií | 24/24 | 100.00 % |
| Názov a rok práce | 23/24 | 95.83 % |
| Spoločné roky práce | 40/40 | 100.00 % |
| Spoločný začiatok štúdia | 32/32 | 100.00 % |
| Spoločné kľúčové slovo | 40/40 | 100.00 % |
Zostávajúce zlyhania: `{'unsupported_claim': 1, 'answerability': 2}`. Úplné odpovede a kontrolné príznaky sú
v `../answers_v5_1_regression_2026-10-25/answers_regression.json`, tabuľka v CSV
a zoznam zlyhaní v `failures.json`. Prípadné vecne správne, ale neúplne
štruktúrované odpovede strácajú strict bod; nejde o percento nepravdivých viet.
## Porovnanie s historickou v4
V4 grafový strict TEST mal 320/320, ale doplnková kontrola všetkých kontextových
faktov 319/320. V5.1 má 320/320 aj v prísnejšej kontrole typovaných faktov.
V4 generovanie malo 281/300 (93,67 %). Na rovnakých 300 otázkach má aktuálna
historická regresia **297/300 — 99.00 %**.
V5.1 navyše uvádza celkový 320-otázkový výsledok vyššie. Ide o opisné
porovnanie: zmenili sa kritériá, prompt, spracovanie JSON aj rozdelenie DEV.
## Zmrazenie, reprodukcia a nasadenie
Grafový manifest je `../../frozen_v5_1.json`; označenie release je
`graphrag-v5.1-frozen`. Zahŕňa odtlačky grafového kódu, spoločných modulov,
requirements, publikačnej politiky, raw wiki, vstupných JSON a obsahu grafu.
Manifest generovania viaže prompt, formátovanie, HTTP klienta, dokončený DEV
a presný zoznam všetkých vyhodnotených ID. Pri meraní bol pracovný kód ešte
pred release commitom; obsahové odtlačky určujú meraný kód, na checkout použite
release tag, nie iba rodičovský `git_head` uložený v metadátach.
Aktuálny graf má 446 uzlov/877 vzťahov; vstupy majú 114 dokumentov/887 chunkov.
Publikačná politika zahŕňa chýbajúce `published` a vylučuje explicitné false.
Koordinovaná synchronizácia a oddelené readiness stavy zostávajú zachované.
Automatizovaná sada: **525 passed, 4 skipped**. Overenie nasadeného obrazu
a opravených prípadov je v `deployment_check.json`; presné verzie hostiteľského
a kontajnerového prostredia aj identifikátor obrazu sú v `runtime_versions.json`.
Reprodukcia používa vždy nové adresáre. Existujúce finálne výsledky a manifesty
sa neprepisujú. Presné príkazy sú v `docs/graphrag_v5_2026-10-25_worklog.md`.
## Obmedzenia a ďalší blok
Zdrojové overenie je nezávislé od grafového buildera, ale pravidlové; nenahrádza
nezávislú ľudskú anotáciu všetkých otázok. Hodnotenie kontroluje typované tvrdenia,
väzby, zdroje a povinné hodnoty v texte, nie sémantiku každej možnej vety.
Výsledky nad známym korpusom a šablónami nemerajú všeobecnú generalizáciu.
Modelové váhy riadi externý poskytovateľ; reprodukcia nemusí vrátiť rovnaké texty.
Tento release uzatvára dohodnuté podbody 3.1–3.6 pre uvedený regresný protokol.
Nasledujú samostatne dohodnuté prierezové úpravy vrátane odovzdania všetkých
spoločných polí kontraktu do LLM kontextu. Natívny výber a inštalácia nástrojov
OpenWebUI hlavným agentom patrí do bodu 4 a tento experiment ho nehodnotí.

40
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/deployment_check.json Normal file
View File

@ -0,0 +1,40 @@
{
"checked_at_utc": "2026-10-25T16:31:09.537130+00:00",
"health": {
"ready": true,
"tools": {
"rag": true,
"neo4j": true,
"graphrag": true
}
},
"index_version": {
"version": "7f2794f292294d898d08b3f179f9ebc8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-25T16:13:38.893054+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
},
"frozen_implementation_inputs_and_graph_match": true,
"graph": {
"sha256": "34d3ff672efe834d2b7d07e2f1cd1c043db011d0db37fb07cc7616bced0234a0",
"nodes": 446,
"relationships": 877
},
"regression_cases": {
"wide-080": {
"strict": true,
"shared_years": [
2022
]
},
"wide-275": {
"strict": true,
"organizational_sentence_is_not_title": true
}
},
"rag_live_query": "ok"
}

723
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/graphrag_dev.csv Normal file
View File

File diff suppressed because one or more lines are too long

137765
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/graphrag_dev.json Normal file
View File

File diff suppressed because it is too large Load Diff

321
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/graphrag_test.csv Normal file
View File

File diff suppressed because one or more lines are too long

63522
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/graphrag_test.json Normal file
View File

File diff suppressed because it is too large Load Diff

31
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/runtime_versions.json Normal file
View File

@ -0,0 +1,31 @@
{
"evaluation_python": "3.10.12",
"evaluation_packages": {
"neo4j": "6.3.1",
"pydantic": "2.13.4",
"numpy": "2.2.6",
"fastapi": "0.136.3",
"starlette": "1.6.0",
"torch": "2.11.0+cpu",
"sentence-transformers": "5.7.0",
"python-frontmatter": "1.3.0",
"tiktoken": "0.13.0",
"requests": "2.34.2",
"transformers": "5.15.0"
},
"deployment_python": "3.11.17",
"deployment_packages": {
"neo4j": "6.3.1",
"pydantic": "2.13.4",
"numpy": "2.2.6",
"fastapi": "0.136.3",
"starlette": "1.9.0",
"torch": "2.11.0+cpu",
"sentence-transformers": "5.7.0",
"python-frontmatter": "1.3.0",
"tiktoken": "0.14.0",
"requests": "2.34.2",
"transformers": "5.20.2"
},
"deployment_image": "sha256:558684d06d25184201e5720f66f7e8cc2d5f584e50adc0ef7f347a21c822b632"
}

178
tools/graphrag/evaluation/results/final_v5_1_2026-10-25/summary.json Normal file
View File

@ -0,0 +1,178 @@
{
"completed_at_utc": "2026-10-25T16:30:27.413476+00:00",
"release": "v5.1",
"release_tag": "graphrag-v5.1-frozen",
"evaluation_kind": "historical_regression",
"graph_dev": {
"questions": 722,
"source_checked_count": 722,
"all_facts_supported_count": 722,
"strict_count": 722,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.042052,
"latency_median_seconds": 0.040355,
"latency_p95_seconds": 0.052196
},
"graph_regression": {
"questions": 320,
"source_checked_count": 320,
"all_facts_supported_count": 320,
"strict_count": 320,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.042475,
"latency_median_seconds": 0.042369,
"latency_p95_seconds": 0.049721
},
"generation_dev": {
"questions": 39,
"correct": 38,
"accuracy": 0.9743589743589743,
"errors": 0,
"first_pass_correct": 38,
"format_repair_attempted": 0,
"normalized_first_response": 0
},
"generation_regression": {
"questions": 320,
"correct": 317,
"accuracy": 0.990625,
"errors": 0,
"first_pass_correct": 317,
"format_repair_attempted": 0,
"normalized_first_response": 0
},
"generation_regression_by_category": {
"candidate_set_aggregation": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"conjunctive_disambiguation": {
"questions": 40,
"correct": 40,
"accuracy": 1.0
},
"document_author_category_multi_hop": {
"questions": 24,
"correct": 24,
"accuracy": 1.0
},
"negative_lookup": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"person_keyword_title": {
"questions": 32,
"correct": 30,
"accuracy": 0.9375
},
"person_multiple_categories": {
"questions": 24,
"correct": 24,
"accuracy": 1.0
},
"person_title_year": {
"questions": 24,
"correct": 23,
"accuracy": 0.9583333333333334
},
"same_academic_year_pair": {
"questions": 40,
"correct": 40,
"accuracy": 1.0
},
"same_start_year_pair": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"shared_keyword_pair": {
"questions": 40,
"correct": 40,
"accuracy": 1.0
}
},
"generation_failure_reasons": {
"unsupported_claim": 1,
"answerability": 2
},
"comparison_v4_test_300": {
"questions": 300,
"v4_correct": 281,
"v5_1_correct": 297
},
"actual_models": {
"Qwen3.5-122B-A10B-Q4_K_M": 320
},
"all_attempt_token_usage": {
"completion_tokens": 57707,
"prompt_tokens": 362926,
"total_tokens": 420633
},
"target": 0.92,
"target_met": true,
"automated_tests": {
"passed": 525,
"skipped": 4,
"failed": 0
},
"development_overlap_in_final_generation": 39,
"attempt_history": [
{
"name": "strict DEV before assignment-title fix",
"correct": 719,
"questions": 722
},
{
"name": "v5 graph regression",
"correct": 319,
"questions": 320
},
{
"name": "v5 generation DEV",
"correct": 38,
"questions": 39
}
],
"point_3_status": {
"3.1": "Completed for v5.1 historical regression with declared limitations",
"3.2": "Completed for v5.1 historical regression with declared limitations",
"3.3": "Completed for v5.1 historical regression with declared limitations",
"3.4": "Completed for v5.1 historical regression with declared limitations",
"3.5": "Completed for v5.1 historical regression with declared limitations",
"3.6": "Completed for v5.1 historical regression with declared limitations"
},
"limitations": [
"Known questions, profiles, templates and corpus. This is not a new independent holdout.",
"39 development questions occur in the final 320-case generation regression.",
"Strict criteria and JSON handling changed relative to v4; comparisons are descriptive, not a controlled causal experiment.",
"Independent rule-based raw-source oracle, not independent human annotation of the full dataset.",
"Checks typed factual claims, bindings, citations and required prose values; not a semantic judge of every possible sentence.",
"OpenWebUI API receives locally supplied graph context; native tool routing remains point 4.",
"Shared-context contract forwarding and further cross-cutting work are deferred as requested.",
"External model weights are controlled by the provider; repeated requests may differ."
]
}

24
tools/graphrag/evaluation/results/final_v5_2026-10-25/REPORT.md Normal file
View File

@ -0,0 +1,24 @@
# Prvá iterácia v5 — zachovaná diagnostická regresia
Tento adresár zachováva prvý zmrazený beh sprísneného hodnotenia. **Finálnou
verziou je nasledujúca v5.1**, nie tento diagnostický výsledok. Všetky otázky
boli historicky odhalené; nejde o nový nezávislý TEST.
- DEV: 722/722; všetky vrátené typované fakty prešli zdrojovou kontrolou.
- Regresia pôvodného TESTu: 319/320; požadované fakty boli nájdené vo všetkých
otázkach, ale `wide-275` navyše obsahoval nepravdivý názov projektu.
- Konkrétny chybný názov bol „Odporúčaný čas konzultácie je piatok.“ Parser
ho prevzal z organizačnej vety na hranici chunku. Nasledujúca verzia ho
odmieta podľa všeobecného pravidla pre organizačné poznámky.
- DEV generovania: 38/39, bez neplatných JSON odpovedí; `wide-145` mal správne
fakty rozdelené do dvoch claims namiesto požadovanej spojenej väzby.
Výsledky sa neprepisujú. V5.1 je ďalšia vývojová iterácia nad rovnakou
historickou regresiou. Po finálnom zmrazení v5.1 sa podľa jej výsledkov neladí.
Odtlačky prvej iterácie sú v `../../frozen_v5.json`. Na rekonštrukciu jej kódu
z finálneho release checkoutu slúži
[`candidate1_from_final.patch`](../v5_preparation/candidate1_from_final.patch).
Rekonštruované súbory boli porovnané so SHA v pôvodnom manifeste; osobitne
sa overil aj odtlačok pôvodného promptu pre LLM. Patch používajte v izolovanom
checkoute; na reprodukciu grafu treba jeho zodpovedajúcu obnovu.

723
tools/graphrag/evaluation/results/final_v5_2026-10-25/graphrag_dev.csv Normal file
View File

File diff suppressed because one or more lines are too long

137772
tools/graphrag/evaluation/results/final_v5_2026-10-25/graphrag_dev.json Normal file
View File

File diff suppressed because it is too large Load Diff

321
tools/graphrag/evaluation/results/final_v5_2026-10-25/graphrag_test.csv Normal file
View File

File diff suppressed because one or more lines are too long

63529
tools/graphrag/evaluation/results/final_v5_2026-10-25/graphrag_test.json Normal file
View File

File diff suppressed because it is too large Load Diff

41
tools/graphrag/evaluation/results/v5_1_preparation/generation_dev_ids.json Normal file
View File

@ -0,0 +1,41 @@
[
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
]

62
tools/graphrag/evaluation/results/v5_1_preparation/protocol.json Normal file
View File

@ -0,0 +1,62 @@
{
"evaluation_kind": "historical_regression",
"dataset_sha256": "9a3b873882bbf29b905d79c9448c3c287fda30323f93a6533b88dbd59b19f558",
"graph_dev": 722,
"graph_regression": 320,
"llm_dev_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"llm_dev_count": 39,
"llm_final_regression": 320,
"llm_final_dev_overlap": 39,
"target_strict_accuracy": 0.92,
"format_repair_limit": 1,
"policy": "Do not tune after final freeze. Retain all failures. Same question and context for one format-only retry; never send gold to LLM.",
"academic_preparation_policy": "Preparatory/planning headings do not establish an academic work or year.",
"limitations": [
"Known corpus, persons, templates and questions. Repeated measurement is regression, not independent TEST.",
"Native OpenWebUI tool routing is point 4."
],
"previous_iteration": {
"graph": "v5: 319/320; consultation sentence misclassified as title",
"llm_dev": "38/39; no JSON errors"
}
}

7442
tools/graphrag/evaluation/results/v5_1_preparation/source_audit.json Normal file
View File

File diff suppressed because it is too large Load Diff

74
tools/graphrag/evaluation/results/v5_preparation/candidate1_from_final.patch Normal file
View File

@ -0,0 +1,74 @@
--- a/tools/graphrag/evaluation/source_oracle.py
+++ b/tools/graphrag/evaluation/source_oracle.py
@@ -91,7 +91,6 @@
'pozriet si', 'prejst si', 'konzultacie', 'projekt odlozeny',
'napady na', 'virtualne stretnutie', 'chceme vytvorit',
'praca suvisi', 'pisomna praca', 'spolupraca', 'suvisiaca praca',
- 'odporucany cas konzult', 'odporucany termin konzult', 'cas konzult', 'termin konzult',
])
--- a/tools/graphrag/evaluation/source_validation.py
+++ b/tools/graphrag/evaluation/source_validation.py
@@ -78,10 +78,6 @@
else:
years={oracle['people'][n].get('start_year') for n in people}
same=len(years)==1 and None not in years
- expected={norm(n):oracle['people'][n].get('start_year') for n in people}
- actual={norm(n):year for n,year in row.get('start_years',{}).items()}
- if actual != expected:
- issues.append('incorrect_individual_start_years')
if row.get('same_start_year') != same or row.get('shared_start_year') != (next(iter(years)) if same else None):
issues.append('incorrect_shared_start_year')
else:
--- a/tools/graphrag/graphrag.py
+++ b/tools/graphrag/graphrag.py
@@ -90,10 +90,6 @@
)
NON_TITLE_PREFIXES = (
- "odporucany cas konzult",
- "odporucany termin konzult",
- "cas konzult",
- "termin konzult",
"pozriet si",
"prejst si",
"konzultacie",
--- a/tools/graphrag/tests/test_evaluation_v5.py
+++ b/tools/graphrag/tests/test_evaluation_v5.py
@@ -19,10 +19,6 @@
def test_assignment_proposal_does_not_establish_an_explicit_title():
assert extract_work_title(['Návrh na zadanie BP:\n\nDialógový systém pre otázky\n\n1. Vypracujte prehľad.']) is None
assert extract_work_title(['Názov práce:\n\nDialógový systém pre otázky']) == 'Dialógový systém pre otázky'
-
-
-def test_consultation_sentence_at_chunk_boundary_is_not_a_project_title():
- assert extract_work_title(['Odporúčaný čas konzultácie je piatok.\n\nMožné backendy:\n\n- OpenNMT']) is None
@pytest.fixture
@@ -88,15 +84,6 @@
assert not validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
-def test_comparison_checks_individual_years_as_well_as_shared_year(evidence):
- oracle,_,_=evidence
- row={'fact_type':'start_year_comparison','people':['Alena Nová'],'shared_start_year':2020,
- 'same_start_year':True,'start_years':{'Alena Nová':1999}}
- assert not validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
- row['start_years']['Alena Nová']=2020
- assert validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
-
-
def test_freeze_includes_shared_modules_and_effective_publication_policy():
questions=Path('tools/graphrag/evaluation/graphrag_questions_v4.json')
fp=content_fingerprints(questions)
--- a/tools/answer_evaluation/graph_answer_prompt.txt
+++ b/tools/answer_evaluation/graph_answer_prompt.txt
@@ -8,6 +8,5 @@
Source_urls musí obsahovať konkrétne relevantné URL zo zdrojov; neodhaduj ich a neupravuj. Cituj ich aj v answer. Uveď všetky zdroje potrebné na doloženie uvedených faktov.
Ak status kontextu je no_answer, oznám, že požadované údaje sa nenašli, status no_answer, claims prázdne. Nevymýšľaj žiadny názov ani pozitívny fakt.
Kľúčové slovo sa v tomto nástroji viaže na osobu cez jej vlastné aj dcérske dokumenty. Pri otázke na názov a kľúčové slovo spoj title, year a keyword do jednej claim pre tú istú person, aj keď majú odlišné document_path. Nevyžaduj rovnaký dokument pre tag a názov. Rovnaký dokument je povinný pri spoločnej kategórii a autorovi dokumentu.
-Pred odoslaním pri otázke na názov a tag over, že claim s názvom obsahuje súčasne person, title, year aj keyword. Nerozdeľ túto požadovanú väzbu na samostatnú claim s názvom a samostatnú claim s tagom.
Pri rokoch porovnaj len akademické záznamy dodané v kontexte, nie poznámky o príprave. Pri počte nemiešaj nevyhovujúcich kandidátov do claims.
Vráť iba jediný platný JSON objekt bez úvodu, vysvetlení mimo JSON a bez Markdown ohraničenia. Reťazce ohraničuj ASCII dvojitými úvodzovkami. Vo vnútri answer názvy neobaľuj ďalšími ASCII dvojitými úvodzovkami; ak ich použiješ, musia byť escapované. Roky a count sú celé čísla. Nevypisuj čiarku za poslednou položkou.

41
tools/graphrag/evaluation/results/v5_preparation/generation_dev_ids.json Normal file
View File

@ -0,0 +1,41 @@
[
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
]

58
tools/graphrag/evaluation/results/v5_preparation/protocol.json Normal file
View File

@ -0,0 +1,58 @@
{
"evaluation_kind": "historical_regression",
"dataset_sha256": "9a3b873882bbf29b905d79c9448c3c287fda30323f93a6533b88dbd59b19f558",
"graph_dev": 722,
"graph_regression": 320,
"llm_dev_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-119",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"llm_dev_count": 39,
"llm_final_regression": 320,
"llm_final_dev_overlap": 39,
"target_strict_accuracy": 0.92,
"format_repair_limit": 1,
"policy": "Do not tune after final freeze. Retain all failures. Same question and context for one format-only retry; never send gold to LLM.",
"academic_preparation_policy": "Preparatory/planning headings do not establish an academic work or year.",
"limitations": [
"Known corpus, persons, templates and questions. Repeated measurement is regression, not independent TEST.",
"Native OpenWebUI tool routing is point 4."
]
}

7442
tools/graphrag/evaluation/results/v5_preparation/source_audit.json Normal file
View File

File diff suppressed because it is too large Load Diff

7
tools/graphrag/evaluation/source_oracle.py
View File

@ -8,6 +8,7 @@ payload, chunk JSON or retrieval result is used to construct this oracle.
from __future__ import annotations
import hashlib
import os
import json
import re
import unicodedata
@ -17,7 +18,7 @@ from typing import Any
import frontmatter
ROOT = Path(__file__).resolve().parents[3]
DEFAULT_WIKI = ROOT.parent / 'zpwiki'
DEFAULT_WIKI = Path(os.getenv('ZPWIKI_ROOT', str(ROOT.parent / 'zpwiki')))
def norm(value: Any) -> str:
@ -58,6 +59,8 @@ def category_type(category: str) -> tuple[str | None, int | None]:
def academic_type(heading: str) -> str | None:
text = norm(heading)
if text.startswith(('priprava na ', 'priprava k ')):
return None
for prefix, kind in [('bakalarska praca', 'bachelor_thesis'),
('bakalarsky projekt', 'bachelor_project'),
('diplomova praca', 'diploma_thesis'),
@ -88,6 +91,7 @@ def acceptable_title(value: str, *, explicit: bool = False) -> bool:
'pozriet si', 'prejst si', 'konzultacie', 'projekt odlozeny',
'napady na', 'virtualne stretnutie', 'chceme vytvorit',
'praca suvisi', 'pisomna praca', 'spolupraca', 'suvisiaca praca',
'odporucany cas konzult', 'odporucany termin konzult', 'cas konzult', 'termin konzult',
])
@ -212,6 +216,7 @@ def load_source_oracle(wiki_root: Path = DEFAULT_WIKI) -> dict:
if raw_lines and raw_lines[0].strip() == '---':
body_offset = next((index + 1 for index in range(1, len(raw_lines)) if raw_lines[index].strip() == '---'), 0)
documents[path] = {'path': path, 'title': str(page.get('title') or '').strip(),
'published': page.get('published') is not False,
'tags': values(taxonomy.get('tag')), 'categories': values(taxonomy.get('category')),
'author': str(taxonomy.get('author') or page.get('author') or '').strip() or None,
'body_lines': raw_lines[body_offset:], 'body_offset': body_offset,

93
tools/graphrag/evaluation/source_validation.py Normal file
View File

@ -0,0 +1,93 @@
"""Check every returned factual claim against the independent raw-source oracle.
Gold bindings define required coverage; this check also rejects false extras.
It does not use graph construction, query predictions or expected gold values.
"""
from __future__ import annotations
from functools import lru_cache
from .source_oracle import load_source_oracle, norm
@lru_cache(maxsize=1)
def default_oracle():
return load_source_oracle()
def claim_issues(claim: dict, oracle: dict, *, published_only: bool = True) -> list[str]:
names = {norm(name): name for name in oracle['people']}
name = names.get(norm(claim.get('person')))
if name is None:
return ['unknown_person']
person = oracle['people'][name]
issues = []
if claim.get('start_year') is not None and claim['start_year'] != person.get('start_year'):
issues.append('unsupported_start_year')
path = claim.get('document_path')
documents = [p for p in person.get('documents', []) if path is None or p == path]
if path and not documents:
issues.append('document_belongs_to_other_person')
documents = [p for p in documents if not published_only or oracle['documents'][p].get('published', True)]
if path and not documents:
issues.append('missing_or_unpublished_source')
if claim.get('source_url') and not any(oracle['documents'][p]['source_url'] == claim['source_url'] for p in documents):
issues.append('incorrect_source_url')
metadata = {k: claim[k] for k in ['keyword','category','document_author'] if claim.get(k) is not None}
def supported(p, k, value):
d=oracle['documents'][p]
values = d['tags'] if k=='keyword' else d['categories'] if k=='category' else [d['author']]
return norm(value) in {norm(v) for v in values}
if metadata and not any(all(supported(p,k,v) for k,v in metadata.items()) for p in documents):
issues.append('unsupported_metadata_on_document')
academic = {k: claim[k] for k in ['title','year','academic_type','academic_kind'] if claim.get(k) is not None}
if academic and not any(
all(norm(record.get(k)) == norm(v) for k,v in academic.items())
and (not path or record['document_path']==path)
for record in person.get('academics', [])
):
issues.append('unsupported_academic_record')
return issues
def validate_returned_facts(result: dict, oracle: dict) -> dict:
failures = []
published_only = result.get('constraints', {}).get('published_only', True)
for index, row in enumerate(result.get('facts', [])):
kind=row.get('fact_type') if isinstance(row, dict) else None
claims=[]; issues=[]
if kind in {'person_thesis','person_work'}:
claims=[{**row,'academic_kind':'Thesis' if kind=='person_thesis' else 'Work',
'academic_type':row.get('thesis_type') if kind=='person_thesis' else row.get('work_type')}]
elif kind in {'person_keyword','person_category','person_document_author','person_start_year','person_academic_year'}:
claims=[row]
elif kind in {'shared_keyword','shared_category','shared_document_author'}:
field={'shared_keyword':'keyword','shared_category':'category','shared_document_author':'document_author'}[kind]
claims=[{'person':row.get(side+'_person'),field:row.get(field),
'document_path':row.get(side+'_document_path'),'source_url':row.get(side+'_source_url')}
for side in ['left','right']]
elif kind in {'academic_year_comparison','start_year_comparison'}:
names={norm(n):n for n in oracle['people']}
people=[names.get(norm(n)) for n in row.get('people', [])]
if not people or any(n is None for n in people):
issues.append('invalid_comparison_people')
elif kind=='academic_year_comparison':
sets=[{r['year'] for r in oracle['people'][n]['academics'] if r.get('year') is not None} for n in people]
years=sorted(set.intersection(*sets))
if row.get('shared_years')!=years or row.get('same_academic_year') != bool(years):
issues.append('incorrect_shared_academic_years')
else:
years={oracle['people'][n].get('start_year') for n in people}
same=len(years)==1 and None not in years
expected={norm(n):oracle['people'][n].get('start_year') for n in people}
actual={norm(n):year for n,year in row.get('start_years',{}).items()}
if actual != expected:
issues.append('incorrect_individual_start_years')
if row.get('same_start_year') != same or row.get('shared_start_year') != (next(iter(years)) if same else None):
issues.append('incorrect_shared_start_year')
else:
issues.append('unsupported_fact_type')
for claim in claims:
issues.extend(claim_issues(claim,oracle,published_only=published_only))
if issues:
failures.append({'fact_index':index,'fact_type':kind,'issues':sorted(set(issues))})
return {'source_checked':True,'all_facts_supported':not failures,'unsupported_facts':failures}

26
tools/graphrag/graphrag.py
View File

@ -55,15 +55,6 @@ TITLE_LABELS = {
"navrh na nazov diplomovej prace",
}
WEAK_TITLE_LABELS = {
"navrh na zadanie",
"navrh na zadanie bp",
"navrh na zadanie dp",
"navrh na zadanie prace",
"navrh na zadanie bakalarskej prace",
"navrh na zadanie diplomovej prace",
}
FALLBACK_TITLE_LABELS = {
"bachelor_thesis": {
"navrh na nazov bp",
@ -99,6 +90,10 @@ TITLE_BLOCK_BOUNDARY_PREFIXES = (
)
NON_TITLE_PREFIXES = (
"odporucany cas konzult",
"odporucany termin konzult",
"cas konzult",
"termin konzult",
"pozriet si",
"prejst si",
"konzultacie",
@ -328,6 +323,11 @@ def find_work_heading(
) -> WorkHeading | None:
normalized = normalize_text(heading)
# Preparation/planning notes are not an academic work record. In
# particular, a year in such a heading must not become the work's year.
if normalized.startswith(('priprava na ', 'priprava k ')):
return None
for (
work_type,
phrase,
@ -702,14 +702,6 @@ def extract_work_title(
return title
title = extract_labeled_title(texts, {"tema", "navrh na temu", "pisomna praca"})
if title:
return title
title = extract_labeled_title(
texts,
WEAK_TITLE_LABELS,
)
if title:
return title

121
tools/graphrag/tests/test_evaluation_v5.py Normal file
View File

@ -0,0 +1,121 @@
import json
from pathlib import Path
import pytest
from tools.graphrag.graphrag import find_work_heading, extract_work_title
from tools.graphrag.evaluation.source_oracle import academic_type
from tools.graphrag.evaluation.source_validation import validate_returned_facts, claim_issues
from tools.graphrag.evaluation.graphrag_evaluator import evaluate_result
from tools.graphrag.evaluation.reproducibility import content_fingerprints
from tools.answer_evaluation.answer_format import parse_answer
@pytest.mark.parametrize('heading',['Príprava na Diplomový projekt 2 2021','Príprava k bakalárskej práci 2024'])
def test_preparation_is_not_academic_work(heading):
assert find_work_heading(heading) is None and academic_type(heading) is None
def test_assignment_proposal_does_not_establish_an_explicit_title():
assert extract_work_title(['Návrh na zadanie BP:\n\nDialógový systém pre otázky\n\n1. Vypracujte prehľad.']) is None
assert extract_work_title(['Názov práce:\n\nDialógový systém pre otázky']) == 'Dialógový systém pre otázky'
def test_consultation_sentence_at_chunk_boundary_is_not_a_project_title():
assert extract_work_title(['Odporúčaný čas konzultácie je piatok.\n\nMožné backendy:\n\n- OpenNMT']) is None
@pytest.fixture
def evidence():
oracle={'people':{'Alena Nová':{'start_year':2020,'documents':['a.md'],
'academics':[{'year':2024,'title':'Práca A','academic_kind':'Thesis',
'academic_type':'bachelor_thesis','document_path':'a.md'}]}},
'documents':{'a.md':{'source_url':'https://example.test/a','tags':['RAG'],
'categories':['bp2024'],'author':'Autor'}}}
fact={'fact_type':'person_thesis','person':'Alena Nová','year':2024,'title':'Práca A',
'thesis_id':'thesis-a','thesis_type':'bachelor_thesis',
'document_path':'a.md','source_url':'https://example.test/a'}
question={'graph':{'schema_version':3,'expected_bindings':[{'person':'Alena Nová','year':2024}]}}
return oracle,fact,question
def test_explicit_incomplete_result_fails_even_without_truncation(evidence):
oracle,fact,q=evidence
result={'status':'ok','facts':[fact],'complete':False,'truncated':False}
assert not evaluate_result(q,result,latency_seconds=0,source_oracle=oracle)['strict']
def test_wrong_extra_year_fails_even_when_required_fact_is_present(evidence):
oracle,fact,q=evidence
result={'status':'ok','facts':[fact],'complete':True}
assert evaluate_result(q,result,latency_seconds=0,source_oracle=oracle)['strict']
result['facts'].append({**fact,'year':1999,'thesis_id':'unrelated-wrong-extra'})
grade=evaluate_result(q,result,latency_seconds=0,source_oracle=oracle)
assert grade['bindings_ok'] and not grade['strict']
assert grade['source_checked'] and not grade['all_facts_supported']
@pytest.mark.parametrize('change',[{'title':'Vymyslený názov'},{'thesis_type':'doctoral_thesis'},
{'document_path':'other.md'},{'source_url':'https://wrong.test'}])
def test_every_extra_claim_has_correct_source_and_properties(evidence,change):
oracle,fact,_=evidence
assert not validate_returned_facts({'facts':[{**fact,**change}]},oracle)['all_facts_supported']
def test_shared_metadata_validated_for_both_people(evidence):
oracle,_,_=evidence
row={'fact_type':'shared_keyword','left_person':'Alena Nová','right_person':'Unknown',
'keyword':'RAG','left_document_path':'a.md','right_document_path':'a.md'}
assert not validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
def test_source_checker_rejects_unknown_fact_type(evidence):
oracle,_,_=evidence
assert not validate_returned_facts({'facts':[{'year':1999}]},oracle)['all_facts_supported']
def test_metadata_from_different_documents_cannot_form_one_document_claim(evidence):
oracle,_,_=evidence
oracle['people']['Alena Nová']['documents'].append('child.md')
oracle['documents']['child.md']={'categories':[],'tags':[],'author':'Iný autor','source_url':'https://example.test/child'}
assert claim_issues({'person':'Alena Nová','category':'bp2024','document_author':'Iný autor'},oracle)
def test_shared_comparison_years_cannot_contain_unbacked_extra_year(evidence):
oracle,_,_=evidence
row={'fact_type':'academic_year_comparison','people':['Alena Nová'],
'shared_years':[2024,1999],'same_academic_year':True}
assert not validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
def test_comparison_checks_individual_years_as_well_as_shared_year(evidence):
oracle,_,_=evidence
row={'fact_type':'start_year_comparison','people':['Alena Nová'],'shared_start_year':2020,
'same_start_year':True,'start_years':{'Alena Nová':1999}}
assert not validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
row['start_years']['Alena Nová']=2020
assert validate_returned_facts({'facts':[row]},oracle)['all_facts_supported']
def test_freeze_includes_shared_modules_and_effective_publication_policy():
questions=Path('tools/graphrag/evaluation/graphrag_questions_v4.json')
fp=content_fingerprints(questions)
assert {'tools/publication.py','tools/retrieval_contract.py','tools/index_state.py'} <= fp['implementation'].keys()
assert fp['retrieval_policy']['published_only'] is True
assert fp['retrieval_policy']['academic_preparation_is_work'] is False
def test_only_one_complete_json_fence_is_unwrapped():
answer={'answer':'Správny text','status':'ok','claims':[], 'source_urls':[],'count':None}
text='Sprievodný text\n```json\n'+json.dumps(answer)+'\n```'
parsed,normalized=parse_answer(text)
assert parsed['answer']=='Správny text' and normalized
with pytest.raises(ValueError):
parse_answer(text+'\n```json\n{}\n```')
@pytest.mark.parametrize('text',['{"answer":"zlá "citácia""}',
'{"answer":"test","status":"ok","claims":[],"source_urls":[],"count":"2"}'])
def test_malformed_or_wrongly_typed_json_is_not_silently_repaired(text):
with pytest.raises(ValueError):
parse_answer(text)