jp170na/dp-zp-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Dokončenie GraphRAG evaluácie

Browse Source
This commit is contained in:
Ján Pták 2026-10-26 01:41:35 +01:00
parent 9389a54dcd
commit 427bbc3222
57 changed files with 512558 additions and 118 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
README.md
View File

@ -15,7 +15,7 @@ FTS5/BM25 + embeddingy + RRF. Nad týmto baseline sa vo vetve GraphRAG
dopĺňa knowledge graph uložený v Neo4j. Aktuálne je implementovaná
grafová infraštruktúra, deterministický builder, vizualizácia grafu a
samostatné grafové vyhľadávanie. Grafová vrstva má zdrojovo overený benchmark a zmrazenú opravenú
verziu v6 s regresiou 320 historických otázok, kontrolou všetkých vrátených
verziu v7 s novým TESTom 80 kombinácií a regresiou 320 historických otázok, kontrolou všetkých vrátených
typovaných faktov a samostatným overením generovania odpovedí cez OpenWebUI API. Ďalším krokom je natívne zapojenie grafového
nástroja a hlavného agenta do OpenWebUI.
@ -870,17 +870,22 @@ grafové vyhľadávanie je implementované v `tools/graphrag/graph_search.py`.
GraphRAG zatiaľ nezasahuje do stabilného klasického `/rag` toku; ďalšia
etapa spojí grafové výsledky s klasickým RAG a OpenWebUI nástrojom.
Aktuálna evaluácia GraphRAG **v6** má DEV **722/722** a historickú regresiu
**320/320**, vrátane kontroly všetkých vrátených typovaných faktov oproti raw
Markdownom. Generovanie sa meria samostatne: 42 DEV a finálna regresia 320
historických otázok vrátane týchto 42; nejde o nový nezávislý TEST.
Strict výsledok generovania je **316/320 (98,75 %)**. Celá sada vrátane
živých testov má **565 úspešných testov**, bez preskočených prípadov.
V6 prenáša úplnosť a verziu dát do kontextu, kontroluje všetky citované URL
oproti podkladom konkrétneho dotazu a spája názov/rok s tagom tej istej osoby.
Priame čítanie zdrojov opravilo 15 očakávaní; v1–v5.1 ostávajú zachované.
Finálne metriky, prvotné odpovede aj opravné pokusy a obmedzenia sú v
[finálnej správe v6](tools/graphrag/evaluation/results/final_v6_2026-10-25/REPORT.md).
Aktuálna evaluácia GraphRAG **v7** má DEV **1 046/1 046** a nový TEST
**80/80** pre grafové vyhľadávanie aj finálne odpovede. Modelový DEV má
**46/46**; pri novom TESTe prešlo na prvý pokus **77/80 (96,25 %)**,
po jednej povolenej oprave **80/80 (100 %)**. Prirodzený text sa skladá
z overených štruktúrovaných tvrdení. Sú to nové kombinácie nad známym
korpusom a šablónami, nie neznáme osoby ani ľudsky anotovaný benchmark.
Samostatná historická regresia má **319/320 (99,69 %)**; nepodložená
odpoveď v zostávajúcom prípade bola odmietnutá pred doručením.
Všetkých **591 testov** vrátane živých prešlo. V7 opravuje ignorovanie rokov
mimo 20xx, kontrolu rozporných tvrdení o úplnosti, počtov, povinných tvrdení
a citácií vrátane referenčného Markdownu a HTML. Začiatok štúdia vyžaduje
explicitný zdroj, neodvodzuje sa z adresára; opravilo sa 10 očakávaní.
Všetky v1–v6 merania ostávajú zachované.
[Finálna správa v7](tools/graphrag/evaluation/results/final_v7_2026-10-26/REPORT.md)
uvádza presný protokol, opravy, regresiu aj obmedzenia.
Historická v4 mala grafový strict TEST 320/320, doplnkovú kontrolu kontextových
faktov 319/320 a LLM TEST 281/300 (93,67 %). Pôvodné otázky aj výsledky
@ -1108,9 +1113,10 @@ spoľahlivého zdroja.
### 2. GraphRAG evaluácia
V6 má zmrazený graf a úplnú historickú regresiu **320/320**.
[Finálna správa](tools/graphrag/evaluation/results/final_v6_2026-10-25/REPORT.md)
oddeľuje retrieval a generovanie odpovedí; pôvodné v1–v5.1 sa nemenia.
V7 má zmrazený graf, DEV **1 046/1 046** a nový TEST **80/80**.
[Finálna správa](tools/graphrag/evaluation/results/final_v7_2026-10-26/REPORT.md)
oddeľuje nový TEST od historickej regresie a vyhľadávanie od modelu
s kontrolami a rendererom; pôvodné v1–v6 sa nemenia.
Pred formálnym porovnaním s klasickým RAG potrebujeme spoločnú metodiku
a ideálne otázky anotované nezávislým človekom. Nový nezávislý TEST vyžaduje
nové otázky; opakované meranie odhalených otázok je regresia.

133
docs/graphrag_v7_2026-10-26_worklog.md Normal file
View File

@ -0,0 +1,133 @@
# Uzavretie bodu 3 — GraphRAG v7, 26. 10. 2026
V7 opravuje konkrétne chyby z posledného auditu. Historické otázky,
manifesty, tagy a výsledky v1–v6 zostávajú zachované. Záloha projektu,
indexových súborov a grafu je v `/tmp/zp-v7-backup-20261026/`.
## Opravy
1. Dotaz rešpektuje ľubovoľný štvormiestny rok vrátane 1899, 1999, 2100
a 9999. Nepodporovaný zápis po „roku“, napríklad 10000 alebo 20xx,
sa odmieta cez `unsupported_year_constraint`. Podmienka sa nepremení
na vyhľadávanie bez časového obmedzenia.
2. Upozornenie na neúplnosť už nepovoľuje kladné tvrdenie o úplnom zozname
v inej vete. Kontrola rozlišuje aj pravdivé popretie „Nie je to úplný
zoznam“. Testy obsahujú obe možnosti aj skloňované „všetkých študentov“.
3. Model vyberá štruktúrované `claims`, `source_urls`, `status`, `count`;
`answer` ponecháva prázdne. Text pre používateľa zostaví
`render_graph_answer()` z overených hodnôt, bez gold a bez nových faktov.
Pôvodná HTTP odpoveď, parsované tvrdenia a vykreslená odpoveď ostávajú
osobitne uložené. Voľná próza modelu sa nepoužíva ako doručený text.
4. Pri kontrole starého alebo neočakávaného voľného textu sa odmietajú
nepodložené číselné údaje. URL a doslovné hodnoty overených titulov sa
pred týmto porovnaním odlíšia od nových tvrdení. Prípad s dopísaným
rokom 2099 tak neprejde. Samotné porovnávanie čísel nie je úplný
sémantický analyzátor slovenčiny; doručený grafový text je preto
deterministický a obsahuje iba overené polia.
5. `count` musí mať podklad v úplnej agregácii. Pozitívna faktová odpoveď
musí obsahovať tvrdenia a citácie; kontrolujú sa zdroje jednotlivých
uvedených atribútov. Prázdna úplná agregácia s počtom 0 je platná.
6. Citácie sa kontrolujú v bežnom texte, inline a referenčnom Markdowne
aj HTML `href`/`src`, vrátane relatívnych URL, HTML entít a vnorených
zátvoriek v platnej URL. Povolené sú iba URL dodané pre konkrétny dotaz.
7. Ročník adresára sa prestal používať ako začiatok štúdia. Builder aj
nezávislý oracle vyžadujú explicitnú vetu v texte. Graf uvádza
`start_year_basis=explicit_text|missing` a
`start_year_confidence=documented|unknown`. Oracle vedie `cohort_year`
oddelene; táto hodnota sa nevydáva za začiatok štúdia. Pri chýbajúcom
údaji nástroj vráti `missing_documented_start_year`.
8. Hodnotenie mena povoľuje obmedzené gramatické prípony v prirodzenom
texte, ale štruktúrovanú identitu naďalej porovnáva presne. „Martina
NaN1“ sa nezamieňa za iného Martina. Pri deterministickom texte sú
mená aj tak v pôvodnom tvare. Parser JSON neprepisuje neplatné hodnoty;
jeden spoločný opravný požiadavok pre formát/kontext ostáva limitom.
9. Evaluačné CLI vyžadujú explicitný dataset, audit a manifest. Odpovedný
runner vyžaduje aj zoznam DEV; historická regresia navyše vlastné ID.
Už nevyberajú v4 ako tichý predvolený benchmark. Hashované cesty kódu
generovania sú relatívne voči projektu, aby sa dali overiť aj inde.
## Benchmark a oddelenie meraní
Všetkých 1 042 odhalených v6 otázok prešlo do DEV. Pridali sa štyri
potrebné regresné otázky na roky mimo 20xx. DEV má **1 046 otázok**.
Odstránenie odhadu štúdia opravilo 10 očakávaní; údaje, ktoré dokument
neuvádza, sa očakávajú ako nenájdené. ID ani text pôvodných otázok sa
nevymazali. Zmeny sú v `v7_preparation/source_corrections.json`.
Nový **TEST má 80 otázok**, 72 kladných a 8 záporných, vo všetkých
desiatich kategóriách. Seed, kvóty a výber boli uložené pred vyhľadávaním.
Žiadna logická query skupina sa neprekrýva s odhalenými otázkami ani DEV.
Ide o nové kombinácie nad známou wiki, osobami a šablónami. Nie je to
experiment s neznámymi osobami, novým korpusom alebo ľudskou anotáciou.
Počet otázok v kategóriách je obmedzený dostupnými novými skupinami:
spojenie autor–kategória má dve nové skupiny, výpis kategórií sedem.
Kvóty sa neurčovali podľa úspechu retrievera. Reprezentatívnu vzorku
20 otázok (dve z každej kategórie) skontrolovala AI priamo oproti
Markdownom pred meraním. `source_review_packet.json` obsahuje pôvodné
výňatky a dôkazy; `source_review.csv` závery. **Človek ju neanotoval.**
Odstránenie odhadu z adresára teda rieši neistotu z auditu; nenahrádza
nezávislú ľudskú anotáciu celého korpusu.
Graf sa zmrazí po kompletnom DEV. Generovanie má vopred deklarovaných
46 DEV otázok: predchádzajúcich 42 a štyri nové rokové regresie.
Finálny nový TEST sa spustí raz po úspešnom DEV. Výsledky sa nepoužijú
na ladenie tejto verzie. Všetkých 320 pôvodných historických otázok sa
meria oddelene ako regresia; jej prekryv s modelovým DEV je priznaný.
## Zmrazenie a nasadenie
`frozen_v7.json` viaže implementáciu, raw Markdown, dataset, JSON vstupy,
graf, publikačnú politiku a parametre. Deklarácia `frozen_answers.json`
navyše fixuje model, prompt, renderer a presné ID merania. Kompletný
`frozen_release_v7.json` pokrýva všetky runtime moduly a merané artefakty.
Docker sa overuje cez `/health`, reálne `/rag`, grafové pozitívne aj
negatívne dotazy, novú rokovú regresiu a chýbajúci začiatok štúdia.
JSON, SQLite a Neo4j majú spoločnú generáciu po `/sync` s `pull_git=false`.
Publikačná politika ostáva: iba explicitné `published=false` sa vylúči.
## Príkazy
Konfiguráciu načítajte z prostredia bez jej vypisovania. Pri reprodukcii
vždy používajte nové výstupné cesty; existujúce otázky a merania neprepisujte.
```bash
# Jednorazové vyhlásenie benchmarku; hotový dataset neregenerovať.
python -m tools.graphrag.evaluation.benchmark_v7
RUN_NEO4J_TESTS=1 RUN_LIVE_TESTS=1 RUN_LIVE_RAG_E2E=1 \
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest -q
python -m tools.graphrag.evaluation.evaluate_graphrag \
--questions tools/graphrag/evaluation/graphrag_questions_v7.json --split dev \
--audit tools/graphrag/evaluation/results/v7_preparation/source_audit.json \
--output-dir NOVY_GRAF_DEV
python -m tools.graphrag.evaluation.freeze_graphrag \
--questions tools/graphrag/evaluation/graphrag_questions_v7.json \
--dev-results NOVY_GRAF_DEV/graphrag_dev.json \
--audit tools/graphrag/evaluation/results/v7_preparation/source_audit.json \
--purpose independent_holdout --output NOVY_MANIFEST.json
python -m tools.graphrag.evaluation.evaluate_graphrag \
--questions tools/graphrag/evaluation/graphrag_questions_v7.json --split test \
--audit tools/graphrag/evaluation/results/v7_preparation/source_audit.json \
--freeze-manifest NOVY_MANIFEST.json --output-dir NOVY_GRAF_DEV
python -m tools.answer_evaluation.evaluate_graph_answers \
--questions tools/graphrag/evaluation/graphrag_questions_v7.json \
--audit tools/graphrag/evaluation/results/v7_preparation/source_audit.json \
--freeze-manifest NOVY_MANIFEST.json \
--dev-ids tools/graphrag/evaluation/results/v7_preparation/generation_dev_ids.json \
--split dev --output-dir NOVY_LLM_DEV
```
Pre nový TEST použite tie isté parametre, `--split test`,
`--dev-results NOVY_LLM_DEV/answers_dev.json` a nový výstupný adresár.
Pri historickej regresii použite `--split regression` a navyše
`--regression-ids tools/graphrag/evaluation/results/v7_preparation/regression_ids.json`.
Opakovaný beh už odhaleného TESTu je reprodukcia/regresia, nie ďalší
nezávislý experiment. Po ladení podľa TESTu treba nový vopred deklarovaný TEST.
Kompletné výsledky a zostávajúce metodické obmedzenia sú v
[správe v7](../tools/graphrag/evaluation/results/final_v7_2026-10-26/REPORT.md).
Ďalší blok je natívna integrácia nástrojov a hlavného agenta v OpenWebUI.

4
docs/vystupny_kontrakt.md
View File

@ -1,5 +1,9 @@
# Spoločný výstupný kontrakt RAG a GraphRAG
V7 rozširuje kontrolu formátov citácií a rozporov v úplnosti. Grafové
generovanie má validované štruktúrované tvrdenia a deterministický renderer.
[Aktuálny protokol](graphrag_v7_2026-10-26_worklog.md).
V6 prenáša polia úplnosti a verzie aj do kontextu LLM a kontroluje ich pri
generovaní. Kontrola citácií sa vzťahuje na zoznam aj odkazy v texte odpovede.
[Implementácia, meranie a obmedzenia](graphrag_v6_2026-10-25_worklog.md).

113
test/test_bod3_closure.py Normal file
View File

@ -0,0 +1,113 @@
import copy
import json
import pytest
from tools.answer_evaluation.context_policy import answer_context_issues, citation_check, coverage_issues, prose_issues, text_contains_value
from tools.answer_evaluation.checked_generation import generate_checked_answer
from tools.answer_evaluation.render_answer import render_graph_answer
from tools.graphrag.graph_search import extract_query_constraints
from tools.graphrag.graphrag import extract_start_year
@pytest.fixture
def context():
return {'status': 'ok', 'complete': True, 'truncated': False, 'aggregation': None,
'sources': [{'source_url': 'https://example.test/students/2018/eva'}],
'facts': [{'person': 'Eva Nová', 'year': 2026, 'title': 'Práca o 3 modeloch',
'source_url': 'https://example.test/students/2018/eva'}], 'answer_candidates': []}
@pytest.fixture
def answer(context):
return {'answer': '', 'status': 'ok', 'claims': [{'person': 'Eva Nová', 'year': 2026, 'title': 'Práca o 3 modeloch'}],
'source_urls': [context['sources'][0]['source_url']], 'count': None}
@pytest.mark.parametrize('year', [1899, 1999, 2099, 2100, 9999])
def test_year_constraint_is_never_silently_dropped(year):
assert extract_query_constraints(f'Aká práca je v roku {year}?')['years'] == [year]
@pytest.mark.parametrize('text', ['Toto je úplný zoznam. Výpis je neúplný.',
'Uvádzam všetkých študentov. Výpis je neúplný.',
'Úplný zoznam, ale dôkazy sú orezané.'])
def test_caveat_does_not_license_contradictory_exhaustive_claim(text):
assert 'unsupported_exhaustive_statement' in coverage_issues({'answer': text, 'status': 'ok'}, {'complete': False})
@pytest.mark.parametrize('text', ['Výpis je neúplný.', 'Toto nie je úplný zoznam.', 'Nie je to úplný zoznam.'])
def test_truthful_scope_notice_is_not_rejected(text):
assert coverage_issues({'answer': text, 'status': 'ok'}, {'complete': False}) == []
@pytest.mark.parametrize('text', ['[zdroj][s]\n\n[s]: /vymysleny', '<a href="/vymysleny">zdroj</a>',
'<img src="//evil.test/x">', '[s]: javascript:alert(1)',
'<a href="https:&#47;&#47;evil.test/x">x</a>'])
def test_all_citation_syntaxes_are_checked(text):
assert not citation_check({'answer': text, 'source_urls': []}, {'https://example.test/a'})['citations_ok']
def test_legitimate_balanced_parentheses_and_reference_html_links():
url = 'https://example.test/a_(b)'
text = f'[x]({url}) a {url}.\n[s]: <{url}>\n<a href="{url}">x</a>'
assert citation_check({'answer': text, 'source_urls': [url]}, {url})['citations_ok']
def test_count_requires_explicit_complete_aggregation(context, answer):
answer['count'] = 98765
assert 'unsupported_total_count' in answer_context_issues(answer, context, check_text=False)
@pytest.mark.parametrize('field,issue', [('claims', 'missing_required_claims'), ('source_urls', 'missing_required_citations')])
def test_positive_answer_needs_facts_and_citations(context, answer, field, issue):
answer[field] = []
assert issue in answer_context_issues(answer, context, check_text=False)
def test_every_claim_field_needs_its_source(context, answer):
context['facts'].append({'person': 'Eva Nová', 'keyword': 'python', 'source_url': 'https://example.test/child'})
context['sources'].append({'source_url': 'https://example.test/child'})
answer['claims'][0]['keyword'] = 'python'
assert 'missing_claim_source' in answer_context_issues(answer, context, check_text=False)
answer['source_urls'].append('https://example.test/child')
assert answer_context_issues(answer, context, check_text=False) == []
def test_wrong_extra_prose_year_is_rejected_but_citation_path_year_is_not(context, answer):
answer['answer'] = 'Eva Nová má prácu z roku 2026. ' + answer['source_urls'][0]
assert prose_issues(answer) == []
answer['answer'] += ' Obhajoba bola v roku 2099.'
assert prose_issues(answer) == ['unsupported_prose_number']
def test_renderer_uses_only_validated_fields_and_preserves_model_reply(context, answer):
original = copy.deepcopy(answer)
def complete(_): return {'choices': [{'message': {'content': json.dumps(answer)}}]}
result = generate_checked_answer('Názov a rok?', context, 'Pokyny', complete)
parsed_original = {key: value for key, value in original.items() if value is not None}
assert result['accepted_for_delivery'] and result['model_answer'] == parsed_original
assert result['attempts'][0]['answer'] == parsed_original
assert result['attempts'][0]['response']['choices'][0]['message']['content'] == json.dumps(original)
assert '2026' in result['answer']['answer'] and 'Práca o 3 modeloch' in result['answer']['answer']
assert answer == original
def test_zero_count_is_valid_without_claims_or_sources():
context = {'status': 'ok', 'facts': [], 'sources': [], 'complete': True,
'aggregation': {'count': 0, 'complete': True}}
answer = {'answer': '', 'status': 'ok', 'claims': [], 'source_urls': [], 'count': 0}
rendered = render_graph_answer(answer, context)
assert answer_context_issues(rendered, context) == []
def test_directory_cohort_does_not_become_study_start():
assert extract_start_year('pages/students/2016/eva/README.md', [{'text': 'Diplomová práca 2026'}]) is None
assert extract_start_year('pages/students/2016/eva/README.md', [{'text': 'Rok začiatku štúdia: 2017'}]) == 2017
assert extract_start_year('pages/students/2016/eva/README.md', [{'text': 'Beginning of the study: 2018'}]) == 2018
def test_person_inflection_keeps_distinct_names_distinct():
assert text_contains_value('Adriána Remiáša a Martina NaN1', 'Martin NaN1', person=True)
assert not text_contains_value('Martin Jančura', 'Martin NaN1', person=True)
assert not text_contains_value('Martin Malinovský', 'Martin Malý', person=True)

25
tools/answer_evaluation/README.md
View File

@ -1,6 +1,29 @@
# Generovanie odpovedí nad zmrazeným grafom
Aktuálne meranie **v6** je historická regresia 320 známych otázok vrátane
Aktuálne meranie **v7** používa 46 známych DEV otázok, nový query-group TEST
80 otázok a oddelenú historickú regresiu 320 otázok.
Model vracia štruktúrované tvrdenia, `answer` je prázdne. Aplikácia overí
fakty, väzby, počty a citácie a vykreslí prirodzený text deterministicky.
Voľná próza modelu sa ako finálna odpoveď nedoručuje. Pôvodné HTTP odpovede,
parsované aj vykreslené odpovede a každý opravný pokus sú uložené.
Nový TEST má prvý pokus **77/80 (96,25 %)** a finálny výsledok **80/80**;
jednu opravu potrebovali tri otázky. Historická regresia má 319/320,
s jedným odmietnutým nepodloženým akademickým názvom. Nejde o meranie voľnej jazykovej
formulácie ani neznámeho korpusu. Modelový DEV má 46/46.
[Správa v7](../graphrag/evaluation/results/final_v7_2026-10-26/REPORT.md).
Všetky evaluačné vstupy treba zvoliť explicitne. Nový TEST používa
`--split test`; známa regresia `--split regression --regression-ids ...`.
Oba potrebujú dokončený `--dev-results` s rovnakými hashmi a modelom.
Príkazy sú v [pracovnom zázname v7](../../docs/graphrag_v7_2026-10-26_worklog.md).
Budúci OpenWebUI agent musí dodržať `accepted_for_delivery` a odovzdať
overený text; nekontrolované prepísanie rendererovej odpovede by tieto
záruky obišlo. Natívny výber nástrojov ostáva ďalším blokom.
## Historická v6
Meranie **v6** je historická regresia 320 známych otázok vrátane
42 DEV otázok. [Výsledná správa](../graphrag/evaluation/results/final_v6_2026-10-25/REPORT.md)
uvádza samostatne prvý pokus, výsledok po jednom povolenom opravnom požiadavku,
odmietnuté odpovede a JSON chyby. Žiadna chyba sa nevynecháva z menovateľa.

2
tools/answer_evaluation/check_context_scenarios.py
View File

@ -67,7 +67,7 @@ def main():
args = parser.parse_args()
if args.output.exists():
parser.error('Refusing to overwrite results.')
files = [Path(__file__), HERE/'graph_answer_prompt.txt', HERE/'context_policy.py', HERE/'checked_generation.py', HERE/'answer_format.py']
files = [Path(__file__), HERE/'graph_answer_prompt.txt', HERE/'context_policy.py', HERE/'checked_generation.py', HERE/'answer_format.py', HERE/'render_answer.py']
hashes = {str(p): sha256_file(p) for p in files}
declared = scenarios()
args.output.parent.mkdir(parents=True, exist_ok=True)

21
tools/answer_evaluation/check_deployment.py
View File

@ -19,24 +19,28 @@ from tools.rag.common import DB_FILE
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--questions', type=Path, required=True)
parser.add_argument('--audit', type=Path, required=True)
parser.add_argument('--freeze-manifest', type=Path, required=True)
parser.add_argument('--release-manifest', type=Path)
args = parser.parse_args()
if args.output.exists(): parser.error('Refusing to overwrite deployment verification.')
base = Path('tools/graphrag/evaluation')
release_path = base / 'frozen_release_v6.json'
release_path = args.release_manifest
release_match = None
if release_path.exists():
if release_path is not None:
release = json.loads(release_path.read_text())
for path, digest in release['runtime_files'].items():
assert sha256_file(Path(path)) == digest, f'Release code mismatch: {path}'
assert sha256_file(base/'frozen_v6.json') == release['graph_freeze_sha256']
assert sha256_file(args.freeze_manifest) == release['graph_freeze_sha256']
release_match = True
freeze = json.loads((base/'frozen_v6.json').read_text())
freeze = json.loads(args.freeze_manifest.read_text())
health = request_json('http://localhost:8000/health')
assert health['ready'] and all(v['ready'] for v in health['tools'].values())
with create_neo4j_driver() as driver, driver.session() as session:
graph = graph_fingerprint(session)
verify_freeze(freeze, run_metadata(base/'graphrag_questions_v6.json'), graph,
audit=base/'results/v6_preparation/source_audit.json', **freeze['parameters'])
verify_freeze(freeze, run_metadata(args.questions), graph,
audit=args.audit, **freeze['parameters'])
current = read_version(Path('data'))
assert current == sqlite_version(DB_FILE) == graph_version(session) == health['index_version']
documents = session.run('MATCH (d:Document {graph_scope:"zpwiki"}) RETURN d.path AS path, d.published AS published').data()
@ -50,7 +54,10 @@ def main():
for id, question, expected in [
('known_title', 'Aký je názov práce osoby Ján Holp a rok práce?', 'ok'),
('generic_label_not_title', 'Uveď názov práce osoby Dárius Lindvai v roku 2019 s kľúčovým slovom python.', 'no_answer'),
('unknown_person', 'Akú prácu má osoba Jozef Neznámy Overovací?', 'no_answer')]:
('unknown_person', 'Akú prácu má osoba Jozef Neznámy Overovací?', 'no_answer'),
('outside_century', 'Aký je názov práce osoby Ján Holp v roku 1999?', 'no_answer'),
('unsupported_year', 'Aký je názov práce osoby Ján Holp v roku 10000?', 'no_answer'),
('unknown_start', 'Aký je rok začiatku štúdia osoby Tomáš Kuchárik?', 'no_answer')]:
result = search_graph(question)
assert result['status'] == expected and result['index_version'] == current
cases[id] = {'status': result['status'], 'reason': result['reason'], 'complete': result['complete']}

14
tools/answer_evaluation/checked_generation.py
View File

@ -2,6 +2,7 @@
import json
from .answer_format import parse_answer, Answer
from .context_policy import answer_context_issues
from .render_answer import render_graph_answer
def generate_checked_answer(question: str, context: dict, prompt: str, complete) -> dict:
@ -29,10 +30,19 @@ def generate_checked_answer(question: str, context: dict, prompt: str, complete)
return result
feedback='Vráť jediný platný JSON objekt podľa schémy, bez zmeny faktov z pôvodného kontextu. Schéma: '+json.dumps(Answer.model_json_schema(),ensure_ascii=False)
else:
issues=answer_context_issues(answer,context)
issues=answer_context_issues(answer,context,check_text=False)
record['context_issues']=issues
result['answer']=answer
if not issues:
rendered = render_graph_answer(answer, context)
delivery_issues = answer_context_issues(rendered, context)
if delivery_issues:
issues = delivery_issues
record['context_issues'] = issues
else:
record['rendered_answer'] = rendered
result['model_answer'] = answer
result['answer'] = rendered
result['accepted_for_delivery']=True
return result
if number==1:
@ -41,6 +51,6 @@ def generate_checked_answer(question: str, context: dict, prompt: str, complete)
feedback=('Odpoveď porušila pravidlá dodaného kontextu: '+', '.join(issues)+'. '
'Oprav jediný JSON objekt výhradne z pôvodných podkladov. Použi iba source_urls zo sources. '
'Typ práce nie je keyword. answer_candidates dokladajú spojenie požadovaného názvu alebo roku a tagu cez tú istú osobu. '
'Pri neúplnom výstupe uveď: Výpis je neúplný.')
'Answer ponechaj prázdny; text aj upozornenie zostaví aplikácia.')
messages += [{'role':'assistant','content':text},{'role':'user','content':feedback}]
return result

116
tools/answer_evaluation/context_policy.py
View File

@ -2,6 +2,8 @@
from __future__ import annotations
import re
from html import unescape
from html.parser import HTMLParser
import unicodedata
@ -10,12 +12,40 @@ def normalized(value) -> str:
return ' '.join(re.findall(r'[a-z0-9]+',''.join(c for c in text if not unicodedata.combining(c)).casefold()))
class _HTMLLinks(HTMLParser):
def __init__(self):
super().__init__(convert_charrefs=True)
self.links = set()
def handle_starttag(self, tag, attrs):
self.links.update(value for key, value in attrs if key in {'href', 'src'} and value)
def text_urls(text: str) -> set[str]:
urls = re.findall(r'(?:[a-zA-Z][a-zA-Z0-9+.-]*://|www\.)[^\s<>"\x27`“”„«»‘’\]\)]+', text)
# Relative Markdown targets are citations too, and must be supplied by
# the tool just like absolute links. Never resolve/guess another URL.
urls += re.findall(r'\[[^\]]*\]\(\s*<?([^\s)>]+)>?', text)
return {url.rstrip('.,;:!?') for url in urls}
"""Inspect absolute, inline, reference-style Markdown and HTML links."""
text = unescape(text)
urls = set()
for url in re.findall(r'(?:[a-zA-Z][a-zA-Z0-9+.-]*://|www\.)[^\s<>"\x27`“”„«»‘’\]]+', text):
url = url.rstrip('.,;:!?')
while url.endswith(')') and url.count(')') > url.count('('):
url = url[:-1]
urls.add(url)
for match in re.finditer(r'\]\(\s*', text):
tail = text[match.end():]
if tail.startswith('<'):
urls.add(tail[1:].split('>', 1)[0]); continue
level = 0; end = 0
for end, char in enumerate(tail):
if char.isspace() or (char == ')' and level == 0): break
if char == '(': level += 1
if char == ')': level -= 1
else:
end = len(tail)
if tail[:end]: urls.add(tail[:end])
for match in re.finditer(r'^\s{0,3}\[[^]\n]+\]:\s*(?:<([^>]+)>|(\S+))', text, flags=re.M):
urls.add(match[1] or match[2])
parser = _HTMLLinks(); parser.feed(text)
return urls | parser.links
def citation_check(answer: dict, allowed: set[str]) -> dict:
@ -54,32 +84,74 @@ def joined_candidates(facts: list[dict], intent: dict) -> list[dict]:
def coverage_issues(answer: dict, context: dict) -> list[str]:
incomplete=context.get('truncated') is True or context.get('complete') is False
unknown=context.get('complete') is None
text=normalized(answer.get('answer',''))
caveat=any(phrase in text for phrase in ['vysledok je neuplny','vysledky su neuplne','vypis je neuplny',
'zoznam je neuplny','dokazy su orezane','vysledok je orezany','vysledky su orezane',
'nemusi byt uplny','nemusi byt uplna','nie je uplny','nie su uplne','nemozno zarucit uplnost'])
issues=[]
if incomplete and answer.get('status')=='ok' and not caveat:
incomplete = context.get('truncated') is True or context.get('complete') is False
unknown = context.get('complete') is None
clauses = [normalized(part) for part in re.split(r'[.!?;\n]+', answer.get('answer', ''))]
caveat = any(re.search(r'(?:vypis|zoznam|vysled\w*|dokazy) (?:je |su |nie je |nie su )?(?:neupln\w*|orezan\w*)'
r'|nie (?:je|su) (?:to )?upln\w*|nemozno zarucit uplnost|nemusi byt upln', clause)
for clause in clauses)
issues = []
if incomplete and answer.get('status') == 'ok' and not caveat:
issues.append('missing_incompleteness_notice')
exhaustive=any(phrase in text for phrase in ['uplny zoznam','kompletny zoznam','vsetci studenti','vsetky osoby','vsetky prace'])
if (incomplete or unknown) and exhaustive and not caveat:
if incomplete or unknown:
for clause in clauses:
for match in re.finditer(r'(?:upln\w*|kompletn\w*) (?:zoznam|vypis|prehlad)'
r'|vset(?:ci|ky|kych) (?:student\w*|osob\w*|prac\w*)', clause):
prefix = clause[:match.start()]
# Negation must qualify this particular assertion. A caveat in
# another sentence never licenses a positive exhaustive claim.
if not re.search(r'(?:nie (?:je|su)(?: to)?|nejde o|neuvadzam|'
r'nemozno (?:zarucit|uviest)|nemozem (?:zarucit|poskytnut))\s*$', prefix):
issues.append('unsupported_exhaustive_statement')
return issues
return list(dict.fromkeys(issues))
def answer_context_issues(answer: dict, context: dict) -> list[str]:
def prose_issues(answer: dict) -> list[str]:
"""Conservative numeric consistency check; final prose is rendered from claims."""
text = str(answer.get('answer', ''))
for url in sorted(text_urls(text), key=len, reverse=True):
text = text.replace(url, '')
allowed_numbers = set()
for claim in answer.get('claims', []):
for key, value in claim.items():
if value is None: continue
if key in {'year', 'start_year'}: allowed_numbers.add(str(value))
else: text = re.sub(re.escape(str(value)), '', text, flags=re.I)
if answer.get('count') is not None: allowed_numbers.add(str(answer['count']))
unsupported = set(re.findall(r'\b\d+\b', text)) - allowed_numbers
return ['unsupported_prose_number'] if unsupported else []
def text_contains_value(text: str, value: str, *, person: bool = False) -> bool:
value = normalized(value); text = normalized(text)
if not person: return value in text
suffix = r'(?:a|u|ovi|om|e|ou|y|ovej|oveho|ym|emu)?'
pattern = r'\b' + r'\s+'.join(re.escape(token) + (suffix if token.isalpha() else '')
for token in value.split()) + r'\b'
return bool(re.search(pattern, text))
def answer_context_issues(answer: dict, context: dict, *, check_text: bool = True) -> list[str]:
issues=[]
if not citation_check(answer,allowed_urls(context))['citations_ok']:
issues.append('unsupported_citation')
facts=context.get('facts',[])
aggregation=context.get('aggregation') or {}
if answer.get('count') is not None and aggregation:
if answer.get('count') is not None:
if aggregation.get('complete') is not True:
issues.append('unsupported_total_count')
elif answer['count']!=aggregation.get('count'):
issues.append('incorrect_context_count')
claims = answer.get('claims', [])
cited = set(answer.get('source_urls', []))
if answer.get('status') == 'ok' and facts and not claims:
issues.append('missing_required_claims')
if claims and not cited:
issues.append('missing_required_citations')
if answer.get('status') == 'no_answer' and claims:
issues.append('claims_on_no_answer')
if check_text and not str(answer.get('answer', '')).strip():
issues.append('empty_answer_text')
for claim in answer.get('claims',[]):
person=normalized(claim.get('person'))
rows=[f for f in facts if normalized(f.get('person'))==person]
@ -88,6 +160,12 @@ def answer_context_issues(answer: dict, context: dict) -> list[str]:
for key in ['keyword','category','document_author','start_year']:
if claim.get(key) is not None and not any(normalized(row.get(key))==normalized(claim[key]) for row in rows):
issues.append('unsupported_'+key)
for key in ['title', 'year', 'start_year', 'keyword', 'category', 'document_author']:
if claim.get(key) is None: continue
evidence = {row.get('source_url') for row in rows
if normalized(row.get(key)) == normalized(claim[key]) and row.get('source_url')}
if evidence and not evidence & cited:
issues.append('missing_claim_source')
academic={k:claim[k] for k in ['title','year'] if claim.get(k) is not None}
if academic and not any(all(normalized(row.get(k))==normalized(v) for k,v in academic.items()) for row in rows):
issues.append('unsupported_academic_binding')
@ -105,5 +183,7 @@ def answer_context_issues(answer: dict, context: dict) -> list[str]:
issues.append('split_or_missing_person_join')
if context.get('status') in {'no_answer','ambiguous','error'} and answer.get('status')=='ok':
issues.append('unsupported_positive_status')
issues.extend(prose_issues(answer))
if check_text or str(answer.get('answer', '')).strip():
issues.extend(coverage_issues(answer,context))
return list(dict.fromkeys(issues))

75
tools/answer_evaluation/evaluate_graph_answers.py
View File

@ -24,7 +24,7 @@ from tools.graphrag.evaluation.reproducibility import graph_fingerprint, run_met
from tools.graphrag.evaluation.source_oracle import load_source_oracle, norm
from tools.rag.evaluation.rag_runner import DEFAULT_MODEL, OPENWEBUI_URL, request_json
from .checked_generation import generate_checked_answer
from .context_policy import joined_candidates, citation_check, allowed_urls, answer_context_issues
from .context_policy import joined_candidates, citation_check, allowed_urls, answer_context_issues, prose_issues, text_contains_value
HERE = Path(__file__).resolve().parent
@ -86,7 +86,10 @@ def grade_answer(case: dict, answer: dict, oracle: dict, *, context: dict | None
policy_issues=answer_context_issues(normalized,context) if context is not None else []
grade['context_issues']=policy_issues
text = norm(answer.get('answer', ''))
missing = [v for v in case.get('expected_answer_contains', []) if norm(v) not in text]
people = {str(b['person']) for b in case['graph']['expected_bindings'] if b.get('person')}
missing = [v for v in case.get('expected_answer_contains', [])
if not text_contains_value(answer.get('answer', ''), str(v), person=str(v) in people)]
policy_issues.extend(prose_issues(normalized))
# The initial sample exposed a real contradiction in otherwise correct
# category claims. Check it explicitly rather than silently accepting prose.
wrong_thesis_label = 'dizertac' in text and any(str(b.get('category', '')).startswith('dp')
@ -101,14 +104,14 @@ def grade_answer(case: dict, answer: dict, oracle: dict, *, context: dict | None
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output-dir', type=Path, required=True)
parser.add_argument('--sample', type=Path, default=BASE / 'results/final_v4_2026-10-25/openwebui_sample.json')
parser.add_argument('--workers', type=int, default=3)
parser.add_argument('--model', default=DEFAULT_MODEL)
parser.add_argument('--split', choices=['dev', 'test', 'regression'], default='test')
parser.add_argument('--questions', type=Path, default=BASE/'graphrag_questions_v4.json')
parser.add_argument('--audit', type=Path, default=BASE/'results/v4_preparation/source_audit.json')
parser.add_argument('--freeze-manifest', type=Path, default=BASE/'frozen_v4.json')
parser.add_argument('--dev-ids', type=Path)
parser.add_argument('--questions', type=Path, required=True)
parser.add_argument('--audit', type=Path, required=True)
parser.add_argument('--freeze-manifest', type=Path, required=True)
parser.add_argument('--dev-ids', type=Path, required=True)
parser.add_argument('--regression-ids', type=Path)
parser.add_argument('--dev-results', type=Path, help='Required completed DEV before final regression.')
args = parser.parse_args()
if args.output_dir.exists():
@ -117,19 +120,26 @@ def main():
parser.error('OPENWEBUI_API_KEY missing.')
if args.workers < 1:
parser.error('--workers must be positive.')
if args.split=='regression' and args.dev_results is None:
if args.split in {'regression', 'test'} and args.dev_results is None:
parser.error('Final regression requires completed --dev-results.')
dataset=args.questions; audit=args.audit; frozen=args.freeze_manifest
manifest = json.loads(frozen.read_text())
questions = json.loads(dataset.read_text()); oracle = load_source_oracle()
sample = json.loads(args.sample.read_text()); exposed = set(sample['selected_ids'])
test = [q for q in questions if q['split'] == 'test' and q['id'] not in exposed]
assert len(exposed) == 20 and len(test) == 300
development=set(json.loads(args.dev_ids.read_text())) if args.dev_ids else exposed
selected = ([q for q in questions if q['split']=='test'] if args.split=='regression' else
test if args.split=='test' else [q for q in questions if q['id'] in development])
if args.split=='regression' and manifest.get('evaluation_kind')!='historical_regression':
parser.error('Regression requires an explicitly declared historical regression freeze.')
development=set(json.loads(args.dev_ids.read_text()))
test = [q for q in questions if q['split'] == 'test']
if args.split == 'regression':
if args.regression_ids is None: parser.error('Regression needs explicit --regression-ids.')
regression_ids = set(json.loads(args.regression_ids.read_text()))
selected = [q for q in questions if q['id'] in regression_ids]
if len(selected) != len(regression_ids): parser.error('Unknown/duplicate regression IDs.')
else:
selected = test if args.split == 'test' else [q for q in questions if q['id'] in development]
if not selected: parser.error('Empty selected split.')
if args.split == 'test' and (manifest.get('evaluation_kind') != 'independent_holdout'
or development & {q['id'] for q in selected}):
parser.error('Fresh TEST requires independent freeze and zero DEV overlap.')
if not development <= {q['id'] for q in questions if q['split'] == 'dev'}:
parser.error('Generation DEV must be declared inside the dataset DEV split.')
prompt_path = HERE / 'graph_answer_prompt.txt'; prompt = prompt_path.read_text()
args.output_dir.mkdir(parents=True)
metadata = run_metadata(dataset); settings = get_neo4j_settings(); contexts = {}
@ -141,9 +151,9 @@ def main():
result=search_graph_with_session(session,case['question'],**manifest['parameters'])
result['index_version']=metadata['index_version']
contexts[case['id']]=graph_context(result)
code_files = [Path(__file__), prompt_path, HERE/'answer_format.py',HERE/'context_policy.py',HERE/'checked_generation.py', Path('tools/rag/evaluation/rag_runner.py')]
hashes = {str(p): sha256_file(p) for p in code_files}
if args.split=='regression':
code_files = [Path(__file__), prompt_path, HERE/'answer_format.py',HERE/'context_policy.py',HERE/'checked_generation.py', HERE/'render_answer.py', Path('tools/rag/evaluation/rag_runner.py')]
hashes = {str(p.resolve().relative_to(HERE.parents[1])): sha256_file(p) for p in code_files}
if args.split in {'regression', 'test'}:
dev=json.loads(args.dev_results.read_text())
if (dev['configuration']['selected_split']!='dev'
or set(dev['configuration']['selected_ids'])!=development
@ -156,25 +166,29 @@ def main():
or dev['configuration']['model']!=args.model
or dev['run_metadata'].get('index_version')!=metadata['index_version']):
raise RuntimeError('Generation code, prompt, data or complete DEV differs; rerun DEV before final freeze.')
declaration = {'method': ('Previously exposed LLM questions; regression, not independent TEST.'
if manifest.get('evaluation_kind')=='historical_regression' else
'Fresh LLM-generation holdout over previously measured, frozen graph queries.'),
declaration = {'method': ('Known-question development selection.' if args.split=='dev' else
'Previously exposed LLM questions; regression, not independent TEST.'
if args.split=='regression' else
'Fresh query-group holdout over known sources and templates; zero DEV query-group overlap.'),
'graph_freeze_sha256': sha256_file(frozen), 'audit_sha256': sha256_file(audit),
'generation_files': hashes, 'model': args.model, 'temperature': 0, 'workers': args.workers,
'dev_ids': sorted(development), 'test_ids': [q['id'] for q in test],
'dev_count': len(development), 'test_count': len(test), 'target_accuracy': 0.92,
'evaluation_kind': 'historical_regression' if args.split=='regression' else 'independent_query_group_holdout' if args.split=='test' else 'development',
'rendering': 'Verified structured claims rendered deterministically; raw model outputs preserved.',
'selected_split': args.split, 'selected_ids': [q['id'] for q in selected], 'selected_count':len(selected),
'selected_dev_overlap': len(development & {q['id'] for q in selected}),
'corrective_request_limit':1,'format_repair_limit':1,
'repair_policy':'At most one total corrective request for format, citations or retrieval-context violations. Same question/context, no gold; retain all attempts.',
'dev_results_sha256':sha256_file(args.dev_results) if args.dev_results else None,
'policy': ('Evaluate every declared regression case, including DEV overlap; retain all failures and stop tuning after this freeze.'
if manifest.get('evaluation_kind')=='historical_regression' else
'Evaluate all 300; retain every failure; do not tune on this TEST.'),
if args.split=='regression' else
'Evaluate every declared fresh TEST case once; retain every failure; do not tune on this TEST.'),
'selected_categories': dict(Counter(q['category'] for q in selected)),
'limitations': [('All questions and previous model outputs are known; this is historical regression.'
if manifest.get('evaluation_kind')=='historical_regression' else
'Known corpus/profiles and templates; model outputs for these 300 questions were not previously observed.'),
'limitations': [('Known development questions; not a holdout.' if args.split=='dev' else
'All questions and previous model outputs are known; this is historical regression.'
if args.split=='regression' else
'Known corpus/profiles and templates; these query groups were not previously evaluated.'),
'Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.',
'Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose.']}
(args.output_dir / 'frozen_answers.json').write_text(json.dumps(declaration, ensure_ascii=False, indent=2) + '\n')
@ -193,8 +207,9 @@ def main():
# Grading occurs after all generation/correction decisions, never inside
# the generator. Gold cannot influence whether another request is sent.
for attempt in row['attempts']:
attempt['strict']=(grade_answer(case,attempt['answer'],oracle,context=context)['strict']
if attempt.get('answer') else False)
candidate = attempt.get('rendered_answer', attempt.get('answer'))
attempt['strict']=(grade_answer(case,candidate,oracle,context=context)['strict']
if candidate else False)
if row.get('answer'):
row.update(grade_answer(case,row['answer'],oracle,context=context))
row['strict']=bool(row['strict'] and row['accepted_for_delivery'])
@ -219,7 +234,7 @@ def main():
with create_neo4j_driver(settings) as driver, driver.session(database=settings.database) as session:
verify_freeze(manifest, run_metadata(dataset), graph_fingerprint(session), audit=audit, **manifest['parameters'])
assert graph_version(session)==metadata['index_version'],'Index version changed during generation.'
assert hashes == {str(p): sha256_file(p) for p in code_files}, 'Generation code changed during TEST.'
assert hashes == {str(p.resolve().relative_to(HERE.parents[1])): sha256_file(p) for p in code_files}, 'Generation code changed during TEST.'
output['by_category'] = {category: {'questions': len(rows), 'correct': sum(r['strict'] for r in rows),
'accuracy': sum(r['strict'] for r in rows) / len(rows)} for category in sorted({q['category'] for q in selected})
for rows in [[r for r in output['questions'] if r['category'] == category]]}

27
tools/answer_evaluation/graph_answer_prompt.txt
View File

@ -1,16 +1,11 @@
Odpovedaj po slovensky iba z dodaných overených grafových faktov. Žiadne ďalšie vedomosti.
Vráť JSON: {"answer": "stručná prirodzená odpoveď so zdrojmi", "status": "ok alebo no_answer", "claims": [], "source_urls": [], "count": null}.
Každá claim obsahuje person a iba relevantné pravdivé polia: title, year, start_year, keyword, category, document_author. Nepovinné polia vynechaj; nevypĺňaj ich null ani odhadom.
Claims musia pokryť všetky požadované fakty. Podmienky patria tej istej osobe a názov aj rok tej istej práci. Metadata category a document_author spájaj len pri rovnakom document_path.
Presne zachovaj mená, názvy, tagy a kódy kategórií. Mená uvádzaj v pôvodnom tvare aj v texte answer. V answer uveď všetky požadované hodnoty; nezamieňaj začiatok štúdia s rokom práce.
Title znamená výhradne názov akademickej práce alebo projektu; meno profilu nikdy nie je názov práce. Category je iba skutočný kód taxonomy.category, napríklad dp2026. Študent nie je kategória. Typ akademickej práce nie je kategória. Kódy kategórií nerozpisuj ani z nich neodhaduj rok práce. bp = bakalárska práca, dp = diplomová práca, phd = dizertačná práca; diplomová práca nie je dizertačná.
Pri výbere kandidátov a počte uvádzaj len zodpovedajúcich kandidátov. Použi vypočítaný count, nikdy počet uvedených zdrojov. Pri porovnaní ľudí daj každému samostatnú claim.
Source_urls musí obsahovať konkrétne relevantné URL zo zdrojov; neodhaduj ich a neupravuj. Cituj ich aj v answer. Uveď všetky zdroje potrebné na doloženie uvedených faktov.
Ak status kontextu je no_answer, oznám, že požadované údaje sa nenašli, status no_answer, claims prázdne. Nevymýšľaj žiadny názov ani pozitívny fakt.
Kľúčové slovo sa v tomto nástroji viaže na osobu cez jej vlastné aj dcérske dokumenty. Pri otázke na názov a kľúčové slovo spoj title, year a keyword do jednej claim pre tú istú person, aj keď majú odlišné document_path. Nevyžaduj rovnaký dokument pre tag a názov. Rovnaký dokument je povinný pri spoločnej kategórii a autorovi dokumentu.
Pred odoslaním pri otázke na názov a tag over, že claim s názvom obsahuje súčasne person, title, year aj keyword. Nerozdeľ túto požadovanú väzbu na samostatnú claim s názvom a samostatnú claim s tagom.
Pri rokoch porovnaj len akademické záznamy dodané v kontexte, nie poznámky o príprave. Pri počte nemiešaj nevyhovujúcich kandidátov do claims.
Vráť iba jediný platný JSON objekt bez úvodu, vysvetlení mimo JSON a bez Markdown ohraničenia. Reťazce ohraničuj ASCII dvojitými úvodzovkami. Vo vnútri answer názvy neobaľuj ďalšími ASCII dvojitými úvodzovkami; ak ich použiješ, musia byť escapované. Roky a count sú celé čísla. Nevypisuj čiarku za poslednou položkou.
Rešpektuj complete, truncated, truncation_reasons a index_version. Ak truncated=true alebo complete=false a uvádzaš výsledky, do answer vlož vetu „Výpis je neúplný.“ Neoznačuj výpis za úplný zoznam ani za všetky existujúce práce. Pri complete=null úplnosť nemožno zaručiť. Ak aggregation.complete=true, počet môže byť presný aj pri orezaných dôkazoch; vtedy uveď „Počet je úplný, dôkazy sú orezané.“ Inak z počtu zobrazených položiek neodvodzuj celkový počet.
Všetky URL v answer aj source_urls musia byť presne zo sources. Typ práce (napríklad diplomový projekt) nie je kľúčové slovo; keyword uveď iba ak ho dokladá fakt keyword pre tú istú person.
answer_candidates sú doložené spojenia požadovaného názvu alebo roku a tagu cez tú istú osobu, nie nové odhady. Ak je taký kandidát poskytnutý a status kontextu je ok, odpovedz podľa neho a neodmietaj ho len preto, že tag pochádza z dcérskeho dokumentu. V claims ponechaj jeho person, year, keyword a prípadne title spolu v jednom objekte a cituj všetky jeho podklady. Aj pri samotnej kombinácii tag + rok práce sa tag viaže na osobu, nemusí byť v dokumente obsahujúcom rok práce.
Vyber iba pravdivé štruktúrované tvrdenia z dodaného graph_context. Nepoužívaj vlastné vedomosti, zdrojový oracle ani údaje z otázky ako dôkaz.
Vráť jediný platný JSON objekt: {"answer":"","status":"ok","claims":[],"source_urls":[],"count":null}. Answer MUSÍ zostať prázdny reťazec. Prirodzený text, citácie a upozornenie na úplnosť zostaví aplikácia z overených polí. Nepíš voľný text ani Markdown mimo JSON. Všetky reťazce ohraničuj ASCII dvojitými úvodzovkami; úvodzovky v hodnotách escapuj.
Každá claim obsahuje person a len požadované polia, ktoré sú priamo doložené faktami: title, year, start_year, keyword, category, document_author. Nepovinné polia vynechaj. Roky a count sú celé čísla. Presne zachovaj mená, názvy, tagy a kódy kategórií. Nevytváraj nové osoby, hodnoty ani typy tvrdení.
Title je názov akademickej práce/projektu, nikdy názov profilovej stránky alebo generický odkaz na rešerš. Year je rok tej istej práce ako title. Start_year je explicitne doložený začiatok štúdia; rok adresára nie je dôkaz. Keyword je skutočný tag osoby, nikdy typ práce ako diplomový projekt. Category je presný taxonomy.category kód. Document_author je autor dokumentu, nie automaticky študent.
Všetky podmienky jednej claim patria tej istej osobe. Title a year patria rovnakému akademickému záznamu. Category a document_author musia byť doložené tým istým document_path.
Tag patrí osobe cez jej profil aj dcérske dokumenty. Pre kombináciu názvu/roku a tagu nevyžaduj rovnaký dokument. Answer_candidates sú už doložené spojenia cez tú istú osobu; zachovaj ich person, year, keyword a prípadne title spolu v jednej claim. Neodmietaj toto spojenie len pre odlišné document_path.
Claims musia pokryť všetky relevantné výsledky dotazu a požadované väzby. Pri porovnaní ľudí uveď každého v samostatnej claim. Pri výbere kandidátov vynechaj nevyhovujúce osoby. Pri názve práce uveď aj rok, ak je vo faktoch známy.
Source_urls musí obsahovať konkrétne relevantné source_url zo sources pre všetky tvrdenia. Nevymýšľaj ani neupravuj URL. Ak názov/rok dokladá profil a tag dcérsky dokument, cituj oba potrebné zdroje. Nepoužívaj URL iných osôb ako náhradnú citáciu.
Count môže byť číslo iba pri aggregation.complete=true a musí sa rovnať aggregation.count. Inak count=null, aj pri no_answer. Počet nie je počet zobrazených zdrojov. Presný agregovaný počet môže byť úplný aj pri orezanom výpise dôkazov.
Pri status kontextu no_answer vráť status=no_answer, claims=[], count=null. Pri ambiguous alebo error zachovaj tento status a prázdne claims. Nevymýšľaj pozitívnu odpoveď. Nulová úplná agregácia so status=ok je platný výsledok s count=0.
Complete, truncated, truncation_reasons a index_version nemenia pravdivosť faktov. Význam a obmedzenia výsledku oznámi aplikácia. Answer nechaj prázdne aj pri neúplnosti, nenájdení a chybe.

29
tools/answer_evaluation/render_answer.py Normal file
View File

@ -0,0 +1,29 @@
"""Render only validated structured facts; never infer facts or read gold."""
from copy import deepcopy
def render_graph_answer(answer: dict, context: dict) -> dict:
out = deepcopy(answer)
lines = []
if out['status'] == 'ok':
labels = {'title': 'názov práce', 'year': 'rok práce', 'start_year': 'začiatok štúdia',
'keyword': 'kľúčové slovo', 'category': 'kategória', 'document_author': 'autor dokumentu'}
for claim in out['claims']:
values = [f'{label}: {claim[key]}' for key, label in labels.items() if claim.get(key) is not None]
lines.append(claim['person'] + (' — ' + '; '.join(values) if values else '') + '.')
if out.get('count') is not None:
lines.append(f'Počet vyhovujúcich osôb: {out["count"]}.')
if context.get('complete') is False or context.get('truncated'):
lines.append('Výpis je neúplný.')
elif context.get('complete') is None:
lines.append('Úplnosť vyhľadania nemožno zaručiť.')
elif out['status'] == 'ambiguous':
lines.append('Požiadavka je nejednoznačná; údaje nemožno spoľahlivo určiť.')
elif out['status'] == 'error':
lines.append('Vyhľadanie zlyhalo; odpoveď nemožno pripraviť.')
else:
lines.append('Požadované údaje sa v dostupných podkladoch nenašli.')
if out['source_urls']:
lines.append('Zdroje: ' + ', '.join(out['source_urls']))
out['answer'] = '\n'.join(lines)
return out

18
tools/graphrag/evaluation/README.md
View File

@ -1,6 +1,22 @@
# GraphRAG evaluácia
Aktuálna verzia je **v6**, tag `graphrag-v6-frozen`.
Aktuálna verzia je **v7**, tag `graphrag-v7-frozen`.
[Správa v7](results/final_v7_2026-10-26/REPORT.md): graf DEV 1 046/1 046,
nový query-group TEST 80/80, model DEV 46/46, finálne odpovede TEST 80/80
(prvý pokus 77/80). Všetkých 591 testov prešlo. Stará regresia má 319/320;
jeden nepodložený modelový názov sa nedoručil. TEST sa neprekrýva so žiadnou odhalenou query skupinou;
korpus, ľudia a šablóny sú známe. Dve kategórie majú len 2 a 7 nových skupín.
CLI vyžaduje explicitné `--questions`, `--audit`, `--freeze-manifest`
(pri TESTe) a nový výstupný adresár. Použite `graphrag_questions_v7.json`,
`results/v7_preparation/source_audit.json`, `frozen_v7.json`.
Generátor `benchmark_v7` vytvára otázky jednorazovo z raw zdrojov, bez
predikcií. Odstránil sa odhad začiatku štúdia z adresára, nie nedokumentované
otázky z merania. [Príkazy a postup](../../../docs/graphrag_v7_2026-10-26_worklog.md).
## Historická v6
Historická verzia je **v6**, tag `graphrag-v6-frozen`.
[Finálna správa v6](results/final_v6_2026-10-25/REPORT.md) oddeľuje grafový
DEV 722/722, regresiu 320/320 a generované odpovede modelu.
Generovanie má strict **316/320 (98,75 %)**; všetkých **565 testov** prešlo.

8
tools/graphrag/evaluation/benchmark_v2.py
View File

@ -99,7 +99,8 @@ def binding_source_evidence(binding: dict, oracle: dict) -> list[dict]:
'year_basis': row['year_basis']})
if 'start_year' in binding:
evidence.append({'document_path': person['profile'], 'sha256': oracle['documents'][person['profile']]['sha256'],
'field': 'start_year', 'value': binding['start_year']})
'field': 'start_year', 'value': binding['start_year'],
'basis': person.get('start_year_basis', 'explicit_text'), 'confidence': 'documented'})
return evidence
@ -378,6 +379,9 @@ def validate_case(case: dict, oracle: dict) -> list[str]:
if constraints.get('unknown_person'):
if norm(constraints['unknown_person']) in {norm(name) for name in oracle['people']}:
issues.append('negative_person_actually_exists')
elif constraints.get('missing_start_year'):
if names and all(oracle['people'][name]['start_year'] is not None for name in names):
issues.append('negative_start_year_actually_documented')
elif constraints.get('shared_academic_year'):
year_sets = [{r['year'] for r in academics(oracle, name) if r['year'] is not None} for name in names]
if year_sets and set.intersection(*year_sets):
@ -408,7 +412,7 @@ def validate_case(case: dict, oracle: dict) -> list[str]:
if names and set(graph['expected_entity_set']) != set(oracle['people'][names[0]]['categories']):
issues.append('incomplete_category_set')
if graph.get('expected_count') is not None:
years = re.findall(r'\b20\d{2}\b', case['question'])
years = re.findall(r'\b\d{4}\b', case['question'])
if not years:
issues.append('count_without_supported_year_constraint')
else:

2
tools/graphrag/evaluation/benchmark_v4.py
View File

@ -33,7 +33,7 @@ def logical_group(case: dict) -> str:
names = sorted(norm(row['value']) for row in graph['start_entities'] if row['type'] == 'person')
bindings = graph.get('expected_bindings', [])
keywords = sorted({norm(row['keyword']) for row in bindings if row.get('keyword')})
years = sorted({int(value) for value in re.findall(r'\b20\d{2}\b', case['question'])})
years = sorted({int(value) for value in re.findall(r'\b\d{4}\b', case['question'])})
# A shared-year query has no requested year; its complete answer is derived.
selectors = {'keywords': keywords, 'requested_years': years}
if category == 'document_author_category_multi_hop':

91
tools/graphrag/evaluation/benchmark_v7.py Normal file
View File

@ -0,0 +1,91 @@
"""Declare a fresh query-group holdout using raw sources only, before retrieval.
All 1,042 previously exposed questions become DEV. No profile is new; this is
a new-combination holdout over a known corpus, not an unseen-person experiment.
"""
import copy
import json
import random
from collections import Counter
from .benchmark_v2 import BASE, audit_dataset, make_case, validate_case, write_json, binding_source_evidence
from .benchmark_v4 import build_pool, logical_group, refresh_titles
from .source_oracle import load_source_oracle, norm
QUOTAS = {'shared_keyword_pair': 10, 'same_academic_year_pair': 10,
'same_start_year_pair': 10, 'person_title_year': 10, 'person_keyword_title': 10,
'document_author_category_multi_hop': 2, 'person_multiple_categories': 7,
'candidate_set_aggregation': 7, 'conjunctive_disambiguation': 6, 'negative_lookup': 8}
SEED = 20261026
def main():
output = BASE/'graphrag_questions_v7.json'; artifacts = BASE/'results/v7_preparation'
if output.exists(): raise RuntimeError('Refusing to overwrite declared questions.')
oracle = load_source_oracle()
old = json.loads((BASE/'graphrag_questions_v6.json').read_text())
dev = []; changes = []
exposed = {logical_group(q) for q in old}
for original in old:
case = copy.deepcopy(original); case['split'] = 'dev'
names = [r['value'] for r in case['graph']['start_entities'] if r['type'] == 'person']
if any(b.get('start_year') is not None and oracle['people'][b['person']]['start_year'] is None
for b in case['graph']['expected_bindings']):
case = make_case(case['id'], 'dev', case['category'], case['question'], names, oracle, [],
no_answer=True, reason='The original Markdown does not explicitly document every requested study-start year.',
negative_constraints={'missing_start_year': True})
changes.append({'id': case['id'], 'old_bindings': original['graph']['expected_bindings'],
'new_status': 'no_answer', 'reason': 'Directory cohort is not a documented study-start year.'})
elif case['should_answer']:
case['source_evidence'] = [e for b in case['graph']['expected_bindings'] for e in binding_source_evidence(b, oracle)]
case['graph']['partition_group'] = logical_group(case); dev.append(case)
# Fixed audit regressions; these are explicitly DEV, not extra easy TEST negatives.
for year in [1999, 1899, 2100, 9999]:
case = make_case(f'v7-dev-year-{year}', 'dev', 'negative_lookup',
f'Aký je názov práce osoby Ján Holp v roku {year}?', ['Ján Holp'], oracle, [],
no_answer=True, negative_constraints={'year': year})
case['graph']['partition_group'] = logical_group(case); dev.append(case)
exposed.update(logical_group(q) for q in dev)
texts = {norm(q['question']) for q in dev}; rng = random.Random(SEED)
holdout = []; capacity = {}
for category, pool in build_pool(oracle).items():
unique = {}
for candidate in pool:
case = refresh_titles(candidate, oracle); group = logical_group(case)
if group not in exposed and norm(case['question']) not in texts:
unique[group] = case
capacity[category] = len(unique)
candidates = [unique[key] for key in sorted(unique)]; rng.shuffle(candidates)
quota = QUOTAS[category]
if len(candidates) < quota: raise RuntimeError(f'Insufficient source capacity: {category}')
for case in candidates[:quota]:
case['id'] = f'fresh7-{len(holdout)+1:03d}'; case['split'] = 'test'
case['graph']['partition_group'] = logical_group(case); holdout.append(case)
questions = dev + holdout
for case in questions:
issues = validate_case(case, oracle)
if issues: raise RuntimeError(f'{case["id"]}: {issues}')
write_json(output, questions)
audit = audit_dataset(questions, oracle, output)
audit['partition_policy'] = {'method': 'disjoint logical query groups',
'shared_source_profiles': True, 'seed': SEED, 'previously_exposed_groups': sorted(exposed),
'limitations': ['Known corpus, persons and template families; only the query/fact combinations are fresh.',
'Two remaining document-author/category groups and seven category-list groups constrain category coverage.',
'AI source annotation, not human annotation. No tuning on this TEST.']}
write_json(artifacts/'source_audit.json', audit)
write_json(artifacts/'source_corrections.json', {'questions': changes,
'policy': 'Only explicit study-start text is a fact. Cohort remains separate metadata; missing means unknown.'})
write_json(artifacts/'regression_ids.json', [q['id'] for q in old if q['split'] == 'test'])
ids = set(json.loads((BASE/'results/v6_preparation/generation_dev_ids.json').read_text()))
ids.update(f'v7-dev-year-{year}' for year in [1999, 1899, 2100, 9999])
write_json(artifacts/'generation_dev_ids.json', sorted(ids))
write_json(artifacts/'protocol.json', {'seed': SEED, 'dev_count': len(dev), 'fresh_test_count': len(holdout),
'test_categories': dict(Counter(q['category'] for q in holdout)), 'test_positives': sum(q['should_answer'] for q in holdout),
'test_negatives': sum(not q['should_answer'] for q in holdout), 'available_groups': capacity,
'old_gold_corrections': len(changes), 'regression_count': 320, 'llm_dev_count': len(ids),
'target': .92, 'policy': 'Freeze after complete DEV; run fresh TEST once, retain every failure. Regression remains separate.',
'annotation': 'Independent raw-source oracle plus AI source review; not human review.'})
print(json.dumps({'dev': len(dev), 'test': len(holdout), 'source_corrections': len(changes), 'quotas': QUOTAS}))
if __name__ == '__main__': main()

2
tools/graphrag/evaluation/check_openwebui.py
View File

@ -88,7 +88,7 @@ def main():
from tools.graphrag.build_graphrag import graph_version
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--questions', type=Path, default=BASE / 'graphrag_questions_v4.json')
parser.add_argument('--questions', type=Path, required=True)
parser.add_argument('--freeze-manifest', type=Path, required=True)
parser.add_argument('--audit', type=Path, required=True)
parser.add_argument('--output', type=Path, required=True)

4
tools/graphrag/evaluation/evaluate_graphrag.py
View File

@ -54,13 +54,13 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--questions",
type=Path,
default=QUESTIONS_PATH,
required=True,
)
parser.add_argument(
"--output-dir",
type=Path,
default=RESULTS_DIR / 'development_v4',
required=True,
)
parser.add_argument(

2
tools/graphrag/evaluation/freeze_graphrag.py
View File

@ -88,7 +88,7 @@ def make_freeze(dev: dict, questions: list[dict], fingerprints: dict, audit: dic
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--dev-results', type=Path, required=True)
parser.add_argument('--questions', type=Path, default=Path(__file__).parent / 'graphrag_questions_v4.json')
parser.add_argument('--questions', type=Path, required=True)
parser.add_argument('--audit', type=Path, required=True)
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--purpose', choices=['independent_holdout','historical_regression'],

137
tools/graphrag/evaluation/frozen_release_v7.json Normal file
View File

@ -0,0 +1,137 @@
{
"release": "graphrag-v7-frozen",
"evaluation_kind": "fresh_query_group_holdout_plus_historical_regression",
"graph_freeze_sha256": "9e97794200520d0f2288e452c9e186c0291a54c63d2c1382f693a9753e4081cf",
"runtime_files": {
"app/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"app/main.py": "ff1adeff8b770047139fe3f632ad837517c95ec04649849c4315d6f892f9eef3",
"app/routes.py": "dfb82171fea33d33b6e48cb238d2c19a29254b47a9a87308f4864cdbac53fa7b",
"app/security.py": "24f437aff6642477049f02fdccfb3429c15e610d044bbb66128903771510055f",
"requirements.txt": "26a6d097727eba03c1b6c71cd02b06373ef01ce03e567dd2bc4357ae330e5a1a",
"tools/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/answer_evaluation/__init__.py": "795be75bf60e4aa9209e5b820207e52e8420c41a4dc2e37d0b6018cd2934a51a",
"tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/answer_evaluation/check_context_scenarios.py": "99dbae2747183622e0823cd8f6536f4ff38ef56d0e41dd7855096d47af3b1c37",
"tools/answer_evaluation/check_deployment.py": "14618b102c0ade954dc73cbc5036a628166f0eb703d435b679d1a000d045ab5a",
"tools/answer_evaluation/check_rag_runtime.py": "1e1b8bd48ac4d668ec057fee162ef30d34ae20ae026361bc8cc296bc455d4067",
"tools/answer_evaluation/checked_generation.py": "cb821a65327424c19cbd996190b14e6aa30d9b909493f11716552f71249a0529",
"tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47",
"tools/answer_evaluation/evaluate_graph_answers.py": "d81504e8fe51598c6bd1b0d482f61b9b6f51f87ba3f38b03ae1bc584675a3b76",
"tools/answer_evaluation/graph_answer_prompt.txt": "fc9f2f8d52e70a8d67b9f4814b3f15e6be64bf88afe53ce4db4f037bde00e064",
"tools/answer_evaluation/rag_delivery.py": "b83f93d446f44fdd7d41c12eeeaae69c0b02908076624f4eb3f4f72b9b88dbc8",
"tools/answer_evaluation/regrade_citations.py": "330f437aec12927fd8c99b0aa601373f6ab26cb111971a6fb06098931d1e4aab",
"tools/answer_evaluation/render_answer.py": "609c534a37899b3febcea32d983da4b806c287de8f2ddf9bc0675e3280859002",
"tools/graphrag/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/build_graphrag.py": "256292101d6d7d1946e926e2717ac45465b169f4091c51ecfed4f1a779dfd700",
"tools/graphrag/evaluation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/evaluation/audit_benchmark.py": "6619c818a52fc284e87ed6c48a503a54a3c9dd95447463ed78f075b7c58bad80",
"tools/graphrag/evaluation/benchmark_v2.py": "62a0b6af852d5ae8060c06db3f983ef4b797d74443f9df25a6b8bd77bf133330",
"tools/graphrag/evaluation/benchmark_v3.py": "30bae6c605ed26f4b49973acec799087c4e2b97a883ccedcafb965dd909c4c4d",
"tools/graphrag/evaluation/benchmark_v4.py": "7a731fe649d3d5170045b8bbcabf31ad527e5d26a934d434a4be3a4c94dc5c38",
"tools/graphrag/evaluation/benchmark_v6.py": "f955e600d17358fafcb33bab772aef09d787452f6b33352e299fdd42669aaf34",
"tools/graphrag/evaluation/benchmark_v7.py": "9477cd6e9a6dd0558837cf044535de75a8f101b17115e28e57a82d96d1aa01cb",
"tools/graphrag/evaluation/check_openwebui.py": "863801048231d474ba82930cf6bf6853b4410f8295b3f97fd3d5934852431fbe",
"tools/graphrag/evaluation/evaluate_graphrag.py": "22f48e9ac6dfae92d79ad648ca173547c6a37f98aa8199f7c37595f1b0b186f9",
"tools/graphrag/evaluation/evidence_contract.py": "78e24264c176e163e1782fef1634b6c45c7f31bd4692726d02a30ab45a06648e",
"tools/graphrag/evaluation/freeze_graphrag.py": "d8764d1aeb72208de55604785acff880002da1f818d8a8ad9477cf39e1f29efb",
"tools/graphrag/evaluation/graphrag_evaluator.py": "7f99bf0f97974058650b619ea49a8ee32284cb4b324f1ffab9baf54f5c4495f4",
"tools/graphrag/evaluation/reproducibility.py": "0e2dab6ffb17691f05f9d81dbec12ffc0f2e4e0b1a2114a75b66a5165cf7b4d2",
"tools/graphrag/evaluation/source_oracle.py": "aed0309435fd21bc274fde7905a0720f68f359b3c30bb0b912e3371ebf0d8072",
"tools/graphrag/evaluation/source_validation.py": "9671b8978623888dbed8b8b2d1b6b01c1ce73cb0a512b03919e48873990f15ed",
"tools/graphrag/graph_db.py": "c4b1a2edb765856b807398e4c243a8d17753529a5d2dfdb51791e28bc8de17f7",
"tools/graphrag/graph_search.py": "c41cbfbac0d5daa1d0b51f5f4b798a15688bbd2e0498d2cbe93c6584a99f6b7d",
"tools/graphrag/graphrag.py": "dbc75403aa038e4021b5a70d1f17c62784b478252a94bac8db682bf38dcb39b3",
"tools/graphrag/neo4j_smoke.py": "fb5244c72c29ef2acf96d1e7f54335ff168eed4119db1692025eb1b99194b7bb",
"tools/graphrag/publication.py": "fba1fa0d0bed325fd87f8d3593baae9271de1766b772d7214c7f63fd3f1c37b6",
"tools/graphrag/tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/tests/test_evaluation_pipeline_v2.py": "63d1b3e80ae3c7b563374fd7ab4aa5ffad3c614eafbd15142c577abd7f756bf9",
"tools/graphrag/tests/test_evaluation_v5.py": "9387e9d1418ebab87530986b0e29dac31cf33d762777a29a31cff24cc72e552a",
"tools/graphrag/tests/test_evaluation_wide.py": "1b28bb74fd6b9457c869186b88606f1f7863d573aa3ab5226162e878f236d99e",
"tools/graphrag/tests/test_evaluator_connected_v2.py": "29d7c9838ed1ff37ea3276df6feb1b6142158c43a0525712f490b648181882de",
"tools/graphrag/tests/test_graph_publication.py": "96e53c1ae28e36c7a4290e1e60c8b398bf095ceaf883e819e1a477e2672b11c0",
"tools/graphrag/tests/test_graph_rebuild_atomic.py": "fe37d4fff0b1f0fb1d565f0a96152b9b4e336837bc78b5977cf3b4879d7fbaf3",
"tools/graphrag/tests/test_graph_search.py": "eb58661aef9dafe6bf7188fcdc2d79cef6652d57c64d1b81ae81a52e8e0204d7",
"tools/graphrag/tests/test_graph_search_entity_v2.py": "0ef1a8ebd513c5a4507eb38fcf5ae6137c69e9901b5a2975ba513be5902ac693",
"tools/graphrag/tests/test_graph_search_neo4j.py": "fec2bf72de8f752f17c9164fee5754bdb2e153828855e47efbf0000dbe82cf3b",
"tools/graphrag/tests/test_graph_search_regressions.py": "9db7779aeb8ed9e8c0f92c1657011a81080aff3c928649e811358e3009a18c48",
"tools/graphrag/tests/test_graphrag.py": "9ee8629d6b528dd78c2ddfc25cee95e17613f45b454fc0a38d4c591862637a9f",
"tools/graphrag/tests/test_graphrag_evaluator.py": "1c6d08aeec9ae56ac4123239a5aed12a0081d66cc3f34b2047d95764579d7298",
"tools/graphrag/tests/test_graphrag_freeze.py": "c4b760baf4c7ebf13b16bd4cd3c51874f5512eef3b4c61470dc4d428b0303d9f",
"tools/graphrag/tests/test_graphrag_research_v2.py": "711008a91ab8a58ae0854e7f90329bcd42e5189fde2e2001589e97b0c22c5198",
"tools/graphrag/tests/test_graphrag_root_title.py": "085651fe6e6081521a5dde9554abfee93456250108cf15978df25378d886a46c",
"tools/graphrag/tests/test_source_oracle_v2.py": "0afdbe235e185ffe9f38aa667f90f5a6c20504fe29e9ac572642b39a8eb7faca",
"tools/graphrag/tests/test_title_labels.py": "2983ead68ad6e2e346d24af0fc96dc6771c163e9c2a351af426a1e0383998e6f",
"tools/index_state.py": "c33920a63d04009f11ef23ca9e9eee9bf86453b7d0586d027bf3b39156afea0d",
"tools/publication.py": "f99d445073390a733afc8d849b9fcd4b4b0890bb54d136db67d4015b78436e1a",
"tools/rag/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/rag/build_chunks.py": "fe060e25b138abcb689a6eb1b1e993d7017a4b3c5dd63e7cd275e1279da6053b",
"tools/rag/build_sqlite_index.py": "107c0987e9ec4086ff41f9e7ae30195a67be34616875635e705dc79236e79416",
"tools/rag/common.py": "8497c46d5c6c27a7c76df1568360ad5e9d00d5e5a7ef186e758960cc79d81d1c",
"tools/rag/embedding_utils.py": "a2ea0b5be1a1782efff1c85c0eabfedba043e0e8bf7eb8b6617eab60eb4559c3",
"tools/rag/evaluation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/rag/evaluation/evaluate_rag_answers.py": "945c2da655537fb37fe82a08e10dc3ed66a2d90980921f61f61efb171a4a26a2",
"tools/rag/evaluation/evaluate_retrieval.py": "aba865751ab74d9c2b962548b1e476508dbe6cf7bf811bda79e25f853061cbd7",
"tools/rag/evaluation/metrics.py": "40b7d16f7729599c1d2fa2370f7cb40dfa04a968cdadef3c4fae83a168202c28",
"tools/rag/evaluation/rag_answer_data.py": "9a369e9ced254cf723a5d3006877959c6f144f4c770e382835af776f7f569762",
"tools/rag/evaluation/rag_answer_state.py": "da4ab19f22378899d5c13e93e803a978518f8edcdd62e4990a06aae12d8eb469",
"tools/rag/evaluation/rag_metrics.py": "3e95922df90dd1b757a550040a2d9bd4f857e17d179f60170d7cc881a08f94b6",
"tools/rag/evaluation/rag_runner.py": "031abf7d5b1a026eb199aa2a579bfb8baaa01175279d0018a364bfe50b76c3c1",
"tools/rag/evaluation/retrieval_runner.py": "fe3dd05459f825d72252b7a0841f28004fc81537096a7de487807e4d5f0c4c45",
"tools/rag/rag_document_expansion.py": "d565b9794e30bc0f387c791a0557d5d03f2c1fa2fe5db53e49a9b19b1158f48e",
"tools/rag/rag_query_evidence.py": "90fced0fecd8ef976a1537506d2e7e8e9d3ba22a6371131f978fb91df5cc0b19",
"tools/rag/rag_supplemental_expansion.py": "a593579e543149633e3146c3886536a9196aff43f4df6f3d7ade94ca16dd243a",
"tools/rag/rag_utils.py": "4f898cf0facc2ae9f2ea28a36f48d4ed37bd7fc716e097b319bcf4d36a4a4f03",
"tools/rag/rebuild_index.py": "6a4d86a551d1141c403a0f6570c568896af37faeaa8c67cac170d2e6a393ca06",
"tools/rag/scan_zpwiki.py": "ef770ce4edb6352ff0690af5783766c95062a0e56f638cca6e3d929973728273",
"tools/rag/search_core.py": "624c8ddfe3edf78d06014579b747a59b2e33c0191a7cf90cdfd624b52ed5e96b",
"tools/rag/search_db.py": "cc63aeaf2854e87b7448c3978959569b097074cdb16a9b2c16b2c37a4677fdbb",
"tools/rag/search_utils.py": "69597bf5785f7852b1131a0648d9d2c65b6babba9ad279581f08b40d185ee646",
"tools/rag/tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/rag/tests/conftest.py": "0c94402d5ca971233d9802ded81cdb39ae7dfddda5eb9126e2035f7599a8f1c7",
"tools/rag/tests/test_chunking.py": "94904a20cfed0512d508017eaadbb43ef45ca74a5e96631241f6ca481222c498",
"tools/rag/tests/test_common.py": "86f0aecb79b7a3764112c833ea3b732073c073a5d570ef8c99900fb2a3f04430",
"tools/rag/tests/test_database.py": "31fc3553d76127c2d7e2c113b9053e537a72bb92f57a9c24cad6a579a1ffc589",
"tools/rag/tests/test_e5_1_remaining.py": "a8a52d5204e7376f33bb1aef96aad20990126a0b92266596b7cf8dd0dcbea4bc",
"tools/rag/tests/test_evaluate_rag_answers.py": "2143f3c9efbacf7b9119eeb9ae69e6cd7ea3f428ff5076d8e9973d0943675296",
"tools/rag/tests/test_live_data.py": "3c5e04d4a29ec8360d89881be980dacacccf0e12594fe51a22ec23c32c98b3e3",
"tools/rag/tests/test_rag.py": "13e97c88bf4afe45a8d633b269d110890f0f646887cee861bbf55ada0be7ad24",
"tools/rag/tests/test_rag_document_expansion.py": "9adf39c7a9b683b1d40c36fce576272e71869b7c1236c371c1b077820874eb7e",
"tools/rag/tests/test_rag_metrics_morphology.py": "070ee314d326e4c7218e7c19959a94076a2b241c4f19b9ecd9ca1fb3cc2639d4",
"tools/rag/tests/test_rag_metrics_semantic.py": "3642394d2636edd7420589348fb75bf1d214b539983b7e620008d626deffabf0",
"tools/rag/tests/test_rag_query_evidence.py": "c3fb5115306171010759b0cfd0cd05d1439d6a57de049efea02330ae7c553bfc",
"tools/rag/tests/test_rag_query_evidence_topn.py": "ebf01ad57a52f030b908b7fb0312d8d32cf9f60ea2953c5a3fb8849c1f2dceaf",
"tools/rag/tests/test_rag_runner.py": "1813086999d1e9826b42c2ac7978ea77d29e1c4b12f5e606f1fba572829e24ea",
"tools/rag/tests/test_rag_runner_routing.py": "c9454e5dddfdb7afc013567811efec15545daf20733b7167f78f2dd9ff0380d6",
"tools/rag/tests/test_rag_runner_tool_prompt.py": "073403907717d8d388c28ea92d1d1f9426edab2e8bd320c8f6329bcbff082e6c",
"tools/rag/tests/test_rag_supplemental_expansion.py": "65449054e2d94396def0dc8d505a9573306adbf184820ddd6c9b5ea92e165f8b",
"tools/rag/tests/test_rebuild_index.py": "aa0688b81e4b699ce99e97a4b8c1a6d4a49c5c153caa02301ebd5a5d8be764a4",
"tools/rag/tests/test_scan_zpwiki.py": "324510ca504dff11ecf39b28aff26b44a358e14084b3b015c0c410690d8aa129",
"tools/rag/tests/test_search.py": "82c96d35719d2233e4636b6f25d89b22f004848d8bae46a9705ee384d4abe9a8",
"tools/retrieval_contract.py": "b8fa272c3babf924a3ce8b937fe7ae437cd959ba9f7667193ac67c18d2c5b58d",
"tools/sync_indexes.py": "5b909b4394c24e5ce22848594b16d9aea45fb06418f5cd92bb6aa6381a0ccf10",
"tools/tool_health.py": "8ad37345889ad58a737edac5960b5b6c5002af2397d60248f434e03fb4eddd89"
},
"deployment_configuration": {
"Dockerfile": "fc7aefe4a99222daedc321acf8359773dc24eff82be04e688507e135956a92a6",
"docker-compose.yml": "e33b65c58f155ac057b01be272bc323fab3ff7eb4f51c9e4ca1b53904cc17af4"
},
"measured_artifacts": {
"tools/graphrag/evaluation/graphrag_questions_v7.json": "0f78e9749de369d752b6e1079f174cf3f6b390f4718a672119dfd43fa5d9dfac",
"tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_dev.json": "d1a5d7b6d5aff3c0002f36a60cebc07627a79ec6d57de119dd587e8ab70143b0",
"tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_test.json": "07d3d73c1b7ab0805549d2d3ebea0c02d1c00eacf3d6092911e34e9787cd9612",
"tools/graphrag/evaluation/results/final_v7_2026-10-26/historical_graph_summary.json": "f0386fde85ed1779d1262264614d4d93faf90cca9a3671cd7591ff6b57759a9b",
"tools/graphrag/evaluation/results/final_v7_2026-10-26/summary.json": "cdc7809eb1033bdf27569ea80deb53b3e82cc2beb0141e64d6251756737a396c",
"tools/graphrag/evaluation/results/v7_preparation/source_audit.json": "7825d4f3e4e535eb5f49e975dfef363b9b813f1de5c97e2387e5c2f3e578e6a5",
"tools/graphrag/evaluation/results/v7_preparation/protocol.json": "7ae343bba36fe25e8603f00325fe2dec1f5753616db9ef9c54f74ccffc1a7d83",
"tools/graphrag/evaluation/results/v7_preparation/source_review.csv": "7ee21230e5dda7be9dc847b8ada6111d14d38f7b075a9c64d38f3ff69950b5c0",
"tools/graphrag/evaluation/results/answers_v7_dev_2026-10-26/answers_dev.json": "4af9b52c250c837707c0690d892ec833347b3d3ee71d338ec631f6450eef8045",
"tools/graphrag/evaluation/results/answers_v7_test_2026-10-26/answers_test.json": "19d5e6ef484495a0acc1402b2d8f6ba91b3b63b8229d1118cd75d763082c148a",
"tools/graphrag/evaluation/results/answers_v7_regression_2026-10-26/answers_regression.json": "4bad6d54955b7645d861ea1ae2b412625b1ea032297caa57984f3b0cf21e3a25"
},
"policy": "Runtime code, prompt and gold remained unchanged throughout the final TEST and historical regression. All failures retained.",
"secrets_excluded": [
".env",
"credentials",
"private keys"
]
}

252
tools/graphrag/evaluation/frozen_v7.json Normal file
View File

@ -0,0 +1,252 @@
{
"schema_version": 2,
"evaluation_kind": "independent_holdout",
"partition": {
"method": "disjoint logical query groups",
"logical_group_overlap": 0,
"previously_exposed_test_groups": 0,
"dev_group_count": 901,
"test_group_count": 80,
"dev_profile_count": 72,
"test_profile_count": 55,
"source_profile_overlap": 55,
"limitations": [
"Known corpus, persons and template families; only the query/fact combinations are fresh.",
"Two remaining document-author/category groups and seven category-list groups constrain category coverage.",
"AI source annotation, not human annotation. No tuning on this TEST."
]
},
"frozen_at_utc": "2026-10-26T00:14:20.132176+00:00",
"fingerprints": {
"implementation": {
"tools/graphrag/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/build_graphrag.py": "256292101d6d7d1946e926e2717ac45465b169f4091c51ecfed4f1a779dfd700",
"tools/graphrag/evaluation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/evaluation/audit_benchmark.py": "6619c818a52fc284e87ed6c48a503a54a3c9dd95447463ed78f075b7c58bad80",
"tools/graphrag/evaluation/benchmark_v2.py": "62a0b6af852d5ae8060c06db3f983ef4b797d74443f9df25a6b8bd77bf133330",
"tools/graphrag/evaluation/benchmark_v3.py": "30bae6c605ed26f4b49973acec799087c4e2b97a883ccedcafb965dd909c4c4d",
"tools/graphrag/evaluation/benchmark_v4.py": "7a731fe649d3d5170045b8bbcabf31ad527e5d26a934d434a4be3a4c94dc5c38",
"tools/graphrag/evaluation/benchmark_v6.py": "f955e600d17358fafcb33bab772aef09d787452f6b33352e299fdd42669aaf34",
"tools/graphrag/evaluation/benchmark_v7.py": "9477cd6e9a6dd0558837cf044535de75a8f101b17115e28e57a82d96d1aa01cb",
"tools/graphrag/evaluation/check_openwebui.py": "863801048231d474ba82930cf6bf6853b4410f8295b3f97fd3d5934852431fbe",
"tools/graphrag/evaluation/evaluate_graphrag.py": "22f48e9ac6dfae92d79ad648ca173547c6a37f98aa8199f7c37595f1b0b186f9",
"tools/graphrag/evaluation/evidence_contract.py": "78e24264c176e163e1782fef1634b6c45c7f31bd4692726d02a30ab45a06648e",
"tools/graphrag/evaluation/freeze_graphrag.py": "d8764d1aeb72208de55604785acff880002da1f818d8a8ad9477cf39e1f29efb",
"tools/graphrag/evaluation/graphrag_evaluator.py": "7f99bf0f97974058650b619ea49a8ee32284cb4b324f1ffab9baf54f5c4495f4",
"tools/graphrag/evaluation/reproducibility.py": "0e2dab6ffb17691f05f9d81dbec12ffc0f2e4e0b1a2114a75b66a5165cf7b4d2",
"tools/graphrag/evaluation/source_oracle.py": "aed0309435fd21bc274fde7905a0720f68f359b3c30bb0b912e3371ebf0d8072",
"tools/graphrag/evaluation/source_validation.py": "9671b8978623888dbed8b8b2d1b6b01c1ce73cb0a512b03919e48873990f15ed",
"tools/graphrag/graph_db.py": "c4b1a2edb765856b807398e4c243a8d17753529a5d2dfdb51791e28bc8de17f7",
"tools/graphrag/graph_search.py": "c41cbfbac0d5daa1d0b51f5f4b798a15688bbd2e0498d2cbe93c6584a99f6b7d",
"tools/graphrag/graphrag.py": "dbc75403aa038e4021b5a70d1f17c62784b478252a94bac8db682bf38dcb39b3",
"tools/graphrag/neo4j_smoke.py": "fb5244c72c29ef2acf96d1e7f54335ff168eed4119db1692025eb1b99194b7bb",
"tools/graphrag/publication.py": "fba1fa0d0bed325fd87f8d3593baae9271de1766b772d7214c7f63fd3f1c37b6",
"tools/graphrag/tests/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tools/graphrag/tests/test_evaluation_pipeline_v2.py": "63d1b3e80ae3c7b563374fd7ab4aa5ffad3c614eafbd15142c577abd7f756bf9",
"tools/graphrag/tests/test_evaluation_v5.py": "9387e9d1418ebab87530986b0e29dac31cf33d762777a29a31cff24cc72e552a",
"tools/graphrag/tests/test_evaluation_wide.py": "1b28bb74fd6b9457c869186b88606f1f7863d573aa3ab5226162e878f236d99e",
"tools/graphrag/tests/test_evaluator_connected_v2.py": "29d7c9838ed1ff37ea3276df6feb1b6142158c43a0525712f490b648181882de",
"tools/graphrag/tests/test_graph_publication.py": "96e53c1ae28e36c7a4290e1e60c8b398bf095ceaf883e819e1a477e2672b11c0",
"tools/graphrag/tests/test_graph_rebuild_atomic.py": "fe37d4fff0b1f0fb1d565f0a96152b9b4e336837bc78b5977cf3b4879d7fbaf3",
"tools/graphrag/tests/test_graph_search.py": "eb58661aef9dafe6bf7188fcdc2d79cef6652d57c64d1b81ae81a52e8e0204d7",
"tools/graphrag/tests/test_graph_search_entity_v2.py": "0ef1a8ebd513c5a4507eb38fcf5ae6137c69e9901b5a2975ba513be5902ac693",
"tools/graphrag/tests/test_graph_search_neo4j.py": "fec2bf72de8f752f17c9164fee5754bdb2e153828855e47efbf0000dbe82cf3b",
"tools/graphrag/tests/test_graph_search_regressions.py": "9db7779aeb8ed9e8c0f92c1657011a81080aff3c928649e811358e3009a18c48",
"tools/graphrag/tests/test_graphrag.py": "9ee8629d6b528dd78c2ddfc25cee95e17613f45b454fc0a38d4c591862637a9f",
"tools/graphrag/tests/test_graphrag_evaluator.py": "1c6d08aeec9ae56ac4123239a5aed12a0081d66cc3f34b2047d95764579d7298",
"tools/graphrag/tests/test_graphrag_freeze.py": "c4b760baf4c7ebf13b16bd4cd3c51874f5512eef3b4c61470dc4d428b0303d9f",
"tools/graphrag/tests/test_graphrag_research_v2.py": "711008a91ab8a58ae0854e7f90329bcd42e5189fde2e2001589e97b0c22c5198",
"tools/graphrag/tests/test_graphrag_root_title.py": "085651fe6e6081521a5dde9554abfee93456250108cf15978df25378d886a46c",
"tools/graphrag/tests/test_source_oracle_v2.py": "0afdbe235e185ffe9f38aa667f90f5a6c20504fe29e9ac572642b39a8eb7faca",
"tools/graphrag/tests/test_title_labels.py": "2983ead68ad6e2e346d24af0fc96dc6771c163e9c2a351af426a1e0383998e6f",
"requirements.txt": "26a6d097727eba03c1b6c71cd02b06373ef01ce03e567dd2bc4357ae330e5a1a",
"tools/publication.py": "f99d445073390a733afc8d849b9fcd4b4b0890bb54d136db67d4015b78436e1a",
"tools/retrieval_contract.py": "b8fa272c3babf924a3ce8b937fe7ae437cd959ba9f7667193ac67c18d2c5b58d",
"tools/index_state.py": "c33920a63d04009f11ef23ca9e9eee9bf86453b7d0586d027bf3b39156afea0d",
"tools/rag/common.py": "8497c46d5c6c27a7c76df1568360ad5e9d00d5e5a7ef186e758960cc79d81d1c",
"tools/sync_indexes.py": "5b909b4394c24e5ce22848594b16d9aea45fb06418f5cd92bb6aa6381a0ccf10",
"tools/rag/rag_utils.py": "4f898cf0facc2ae9f2ea28a36f48d4ed37bd7fc716e097b319bcf4d36a4a4f03",
"tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47",
"tools/answer_evaluation/checked_generation.py": "cb821a65327424c19cbd996190b14e6aa30d9b909493f11716552f71249a0529",
"tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/answer_evaluation/render_answer.py": "609c534a37899b3febcea32d983da4b806c287de8f2ddf9bc0675e3280859002",
"tools/rag/evaluation/rag_metrics.py": "3e95922df90dd1b757a550040a2d9bd4f857e17d179f60170d7cc881a08f94b6",
"tools/rag/evaluation/rag_runner.py": "031abf7d5b1a026eb199aa2a579bfb8baaa01175279d0018a364bfe50b76c3c1"
},
"dataset_sha256": "0f78e9749de369d752b6e1079f174cf3f6b390f4718a672119dfd43fa5d9dfac",
"inputs": {
"data/documents.json": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"data/chunks.json": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9"
},
"source_corpus": {
"sha256": "440da12a7a991824e923f70cada7c616c3fb3b0f12767ec7d75039a6a3231b10",
"document_count": 114,
"documents": {
"pages/interns/bogdan_paul_chis/README.md": "3b4b946eb3c10e9f9a4092e6cffa01c5080e7579aded2f6ed3120c41ed62b647",
"pages/interns/cesar_gutierrez/README.md": "b0be60576eed28d4b1920e9acf198684519373dfc13171567247c853b3ed699e",
"pages/interns/oliver_pejic/README.md": "0470f3c8465be8f02204b08e4c75ef47f68c7e4ad1038faa5641cdf1811f33d5",
"pages/interns/sevval_bulburu/README.md": "ff2a9d66253802f1af0fdfd74832e40fae41b6df23c5e61dc266f6cd0cc16c4d",
"pages/interns/yussef_ressaissi/README.md": "f997a65699da0db3222ef7a3c37652d05dfc3419ad4ec74683bafc2c897bb987",
"pages/students/2016/darius_lindvai/README.md": "fcaa47bb9708a6439eec5c4b7f7b4ed6c7359e423e1d940e1aed9f44957e6d62",
"pages/students/2016/darius_lindvai/dp2021/README.md": "be3d0fea2a6d592b225eecabd9b19445c2fce62b8289c07ab6af93835db12970",
"pages/students/2016/darius_lindvai/timovy_projekt/resers/README.md": "352162de1b1e192228fa5d26506c028ac47e567e90df53b1bbaa6ad736ec4766",
"pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md": "b4a81ff457624b089d618e4479afb1e3d748f6724f72104ef5518e535ae02fe5",
"pages/students/2016/dominik_nagy/README.md": "1c4037a158d84c45fa705a0c38e47142a9d46e226f0c138bc2d56d7c38f6d74d",
"pages/students/2016/dominik_nagy/timovy_projekt/README.md": "b81a4dab7316f7b846dcf36515f6aa9d44035c86f46f0e07b7a43308bcd31458",
"pages/students/2016/ediz_morochovic/README.md": "045e5eb9a3df2b84140808fbca0c3163266ad0efa144bc42d9d5c597881c6ac6",
"pages/students/2016/jakub_maruniak/README.md": "6734c41a95b9d0c1e3d226b393ad109ee4a7b82ebfffa73df2d7e0a743fefa14",
"pages/students/2016/jakub_maruniak/dp2021/README.md": "9cd884aec70705cdb2aaa0f514b513f16de263c37b749ced7f92916cd2f91c39",
"pages/students/2016/jakub_maruniak/dp2021/annotation/README.md": "debae0c50aea1accef0840912b9b1f37b2138f856002bb50e91d7ea2dbbe7963",
"pages/students/2016/jakub_maruniak/dp2021/train_experiments/README.md": "01dff38b37043414fa1db204394ddbe7aade00e14d02d495572b65c85d09b194",
"pages/students/2016/jakub_maruniak/timovy_projekt/README.md": "04114b1f3d84664aad5bdba5a0b614c5a6f23392be3c31fd9123d63db1be0a35",
"pages/students/2016/jan_holp/README.md": "80d6a8492766e5bfa61673a372e5f059d30c8e335de39842866c51485960e4d3",
"pages/students/2016/jan_holp/dp2021/README.md": "a62acb74009376766c7e2f9856ee01d21f84593eb9b3ce85cf2e6032aa412476",
"pages/students/2016/jan_holp/dp2021/zdrojove_subory/README.md": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"pages/students/2016/jan_holp/timovy_projekt/README.md": "79eebdfd3af7e8d5c279077e285a580731b1cf8cc9b4454f6596796e9915f70c",
"pages/students/2016/lukas_pokryvka/README.md": "f279456c5c277ff1ee0dd89a3464ed9f9f67b44bc185a5da9811ef5cfb675002",
"pages/students/2016/lukas_pokryvka/dp2021/README.md": "39b61376ce108e4a930f926ab2d6a997207457121bccfcf617129a3b210a15cf",
"pages/students/2016/lukas_pokryvka/timovy_projekt/README.md": "4de4b6c8cfed17b6c60090f82df1d55a66be70a79d3013850689b2baebdd801b",
"pages/students/2016/mark_feher/README.md": "bec37c0319fe0869a0377d771db23d090dbda93c27328f5697cdaeb3bbccfe9b",
"pages/students/2016/maros_harahus/README.md": "0da150e91918632c9e8cd1443f04da403595a372517e254928ce9f76c9d5033d",
"pages/students/2016/maros_harahus/timovy_projekt/README.md": "75af19f7c68f4b646fa1dbae75f3f3dfc52633a9f9da1400d21808bb20be0d83",
"pages/students/2016/patrik_pavlisin/README.md": "dc1c648d7cd9b0f0e021c57f9b98189e37c100871439e7e304390715913f2763",
"pages/students/2016/patrik_pavlisin/dp21/README.md": "ac1eb0fd0360ad34aa21b7c8da5bb010a6edf10be5721f28dedc3986cd1d8c54",
"pages/students/2016/patrik_pavlisin/dp22/README.md": "f2a8c619a812b99abab618ae388eb672407950e1f33fb8d7da7a528c4de81d50",
"pages/students/2016/patrik_pavlisin/tp20/README.md": "ae9c771f27e7e093bc29c6888c8d0b27ed61fb41519fd0d7e7f4f6cbb4626edc",
"pages/students/2016/tomas_kucharik/README.md": "6a1a215433eb1857660177b4cb2ecec03c6a86743d3a04cd6d5c9d9682c9e63b",
"pages/students/2017/dmytro_ushatenko/README.md": "65f4ac177d492162b08ca5da3d9eb048bd088a3c117a28a662113b7e4bc3a4ec",
"pages/students/2017/martin_jancura/README.md": "6a0a18d5095dcfb12abfdd72c9cfd8025d714fb5e29a7858cd2b285e4cb986b9",
"pages/students/2017/stanislav_matsunych/README.md": "90ffa11eff4a4360b0252dbc53a116c3a5777b3d5dd86ce5c89c02bdd2051c73",
"pages/students/2017/than_trung_thanh/README.md": "2150519b8580e7337520db4de64d2d1c82d2634380537d019f41c506dbf2979d",
"pages/students/2018/NaN1name_buday/README.md": "23c8c58090bdc954347e36f141bea91f43c8d5b1d777e346864a2ebe84c0bcce",
"pages/students/2018/adrian_remias/README.md": "e5bfe1d5d4e626beaa636a02d34ec58cf2cff9b0e57833905af35b80bf1465a2",
"pages/students/2018/david_ilas/README.md": "96c45ff9c5a821ddcb1a2fc86d487bd8f4729d1b76887914b0f5fbf99af45af0",
"pages/students/2018/david_omasta/README.md": "4164051870e1e1cf8949ed18a74196e9ea789dfbe4fee24bdeb736ca42848327",
"pages/students/2018/martin_wencel/README.md": "0e6a7a981689f22304f8baa5de97d4315c0695c94c9277de7851943eb012253b",
"pages/students/2018/oleh_bilykh/README.md": "4072c62dd8ee753a0c8fa43fdfbbdf566e394c2df13e6a79d08a302e332710b1",
"pages/students/2018/ondrej_megela/README.md": "969ff2e1a436b8f957e7168cf5eb678196b99aeda17701c502b016441f644065",
"pages/students/2018/samuel_sirotnik/README.md": "cc230ede67b6e8632511c92a5ed6d1e9deb42e70819280222bdfd369ee77fc70",
"pages/students/2019/alina_vitko/README.md": "9b071fe2aff470c275ae027be236fe836f741f774cabb9cbbb55304a8d5102cd",
"pages/students/2019/artem_yatsenko/README.md": "328b101e46c51c959e792cf7ca226942f81d2ab8fe9648ad6d0ecd07016429e2",
"pages/students/2019/david_stromp/README.md": "e46ca8691d5ec894128f6f6f2ed33e2c7bd141f80719bb0449b20f10f30cca09",
"pages/students/2019/dmytro_mural/README.md": "e881a47d847c3ef8d680844113fdbfbb87553b28982bf9eef4936f6d30923e93",
"pages/students/2019/filip_tomas/README.md": "75a1fe7b921e4bff0fdf9e2a957b4adad4b4a57aa7ebdffbb73dd82c02123f94",
"pages/students/2019/kristian_sopkovic/README.md": "3286960fb30f584df35455f9b6dfae81ca4827a1d711d2bcd6e48c3f150c9279",
"pages/students/2019/michal_stromko/README.md": "3ae1dd9b6638a6a249a94be3f3846286555666aefa225bb3a2c7288b35900ce7",
"pages/students/2019/michal_stromko/vp2021/README.md": "feb2e093fdd6461239e79b1e1f96e65a1c8cd96e8a2b70535ae556a9c198046a",
"pages/students/2019/michal_stromko/vp2021/tutorials/README.md": "40e3fa448c7329ef863e705435e23fe6b4788de440710f066fe0eed92d132bad",
"pages/students/2019/michal_stromko/vp2021/tutorials/img/README.md": "e8d3ac160b17dfdee2c3e996bf6a03af7a6f16f9140bf5cf7ee343a82dd1865f",
"pages/students/2019/patrik_pokrivcak/README.md": "3e3957ad0f7aca59d6eb64c38a3f5e21d8fd953200825d0d0e5cacabfbc35a7d",
"pages/students/2019/samuel_horani/README.md": "86e6959ccc612b069687b57996fa3f61660075d6a0d65bfec64f423d39a0a999",
"pages/students/2020/david_kostilnik/README.md": "2927524b75da59a64120fc364a0fc374986fa4d364d54e52c36800762c203264",
"pages/students/2020/jakub_kristian_lukas/README.md": "f139d9b1ebf5a3f45612f3df5703ecdd935b90701763c27c7c529044c0b0455a",
"pages/students/2020/kamil_tomcufcik/README.md": "9d865adf538bc6d7992ee4f4882713447da5c936fca17c43ca0f576072537048",
"pages/students/2020/matej_kobylan/README.md": "3d7b289e17b8f25678a07e9ee0e6741688485a0ccb1e50940a6a6acc6312a547",
"pages/students/2020/matus_suchanic/README.md": "770ca49538cca22df35f8e7ba736ee97483e4fe5788c221496553592b4e78dc6",
"pages/students/2020/pavol_hudak/README.md": "33d06fccc6451e5f19b27b0cad001a34e81635b4ea294a01dec7771c823abc40",
"pages/students/2020/vladyslav_krupko/README.md": "4c28e6d38651bb2cafcbf4a18e35210cd369aef2caff6c5b0aa52fac38f0fe7d",
"pages/students/2021/artur_hyrenko/README.md": "0568261f4ade4de5472edcfeb1ffed9abae18dc67e4050f4ce2ad088839a2bb3",
"pages/students/2021/eduard_matovka/README.md": "de2a7ae38ec87110e347956b37d0605a298ed2847f972961cc9b977da8221d43",
"pages/students/2021/manohar_gowdru_shridharu/README.md": "c00ceb620ed7087ead5d9448d476bbb98d273d1d9e7bf1bc4355c367b26c7a6b",
"pages/students/2021/martin_sarissky/README.md": "8e97b45a253745faa55f63c844e0b5dfe680aeaf6eb01fb8f9d06ac6e8ee8bcf",
"pages/students/2021/matej_novotny/README.md": "b083a7cadde532e3428dc9a57bc61456fcc5905bda91e011b96033d42bf5d9cb",
"pages/students/2021/matej_scislak/README.md": "b9a32cce8605e044a4f2828ff2d91a2995a62ef3a841e62f0a225c97221f0cc8",
"pages/students/2021/nikita_bodnar/README.md": "2016c0dd923c0f8e561d5cea42ac81da3610fdcc54a4527f56d078e34e53ce90",
"pages/students/2021/oleksandr_hryshchenko/README.md": "4b073992b715b6c75b79fd2c8f850a69f940966e3a61a82c625c32c945a93036",
"pages/students/2021/vladimir_ferko/README.md": "68e7d311896eda2171e5eec1ec3a8da90d326ed91acaace8058f68664a50d271",
"pages/students/2022/andrii_pervashov/README.md": "319dcaa1cc2e29dd7e3a710bab26cb3ac6944ab71b0b97e1de076b1199122334",
"pages/students/2022/daniil_huzenko/README.md": "390782fbff6220a9462060ce1a62959f2a2b88f42d06e2c51d1f406b75d73f2c",
"pages/students/2022/jakub_schwarc/README.md": "f541a8852c2b0f6f9d7182e01a34c4e3beee9bfa55427a06b331638b11eb8be6",
"pages/students/2022/jan_malinovsky/README.md": "8af021bfb131fa005a7aec3e2f2f12fc17fa5dc14e2675311adbbfc445a4bcab",
"pages/students/2022/jan_ptak/README.md": "131cf053764fd5423541ebe672c3b4f5da875cfb8e4b821f1335ddb967282282",
"pages/students/2022/oleh_poiasnik/README.md": "c2f7dc3f61f8b4a5b43ded1e3ae4c7d3e7f874439ee6b17dd9c8d45dabc25bc8",
"pages/students/2022/samuel_vasil/README.md": "67c807049147aacd04f3050f4ff0233a8b3f0a239ab173b430e13dfe113ad0a5",
"pages/students/2022/serhii_yemets/README.md": "ffbeac893947acafe855d576322a8826f9d4f87ac344cb31f8ceeefa244b9133",
"pages/students/2022/tetiana_mohorian/README.md": "c9efdc632bf1ca89a0bfde0dc00be7aaac8a85da3d92e2ce8506236c73e93cfb",
"pages/students/2022/valerii_kutsenko/README.md": "dc764c609ad3c1df59cfea997d4db82a591b1b1e0764453e319d841115507e4f",
"pages/students/2022/vladyslav_yanchenko/README.md": "4d1ce216e3b526ee437ddbc658e51d7bb83530087e43628c096a569a3854a47f",
"pages/students/2022/yevhenii_leonov/README.md": "b90e41db0cec2451027020a9102d32a24d777e8a38b604ccca87b36cd6b3a292",
"pages/students/2023/denis_le_thanh/README.md": "2cb83f942d86a02712b0c58778e46dc2211346d1a6eba606471dfa832245db61",
"pages/students/2023/maksym_zatirka/README.md": "7a7d2cbe276c352dacfbc6ba6de5f15303d30ecf34845ff8da8d085f3d67c3dd",
"pages/students/2023/oleksandr_dorybohov/README.md": "c24d6f941de0dafb4459cea8259cf147821cafc86ba65be5b528f419f734bc18",
"pages/students/2023/ronald_zalacko/README.md": "724b0f8e6e641236dba9bddbd77f4546e09514076c7dfb151807aaf4b882c96f",
"pages/students/2023/rostyslav_rodzhuk/README.md": "de356dd7427c8a0716c165a0f88ae47b691a86ee7a388f60e2d2e9306298415c",
"pages/students/2023/samuel_dzurina/README.md": "4818191d044b91219c796a35010c95da5cf2093e050d494f30b219b91e13fa0e",
"pages/students/2023/simona_bobrovcanova/README.md": "e473a1911dae10ab40e2cf6696a8be616a58c22aaf09af5953fbac475791a35e",
"pages/students/2023/yevhenii_medushivskyi/README.md": "5710ebf91c893a12fe6d19f5372311938d1e7c7bbdf203f2dc8ed2fe7c4f5a24",
"pages/topics/akopisat/README.md": "c7245e1fbfa4160618502c0baf8cc6bc47cd9a82b8a85bc1c645314829a7b76a",
"pages/topics/bert/README.md": "35e507f7ec2ad1d3526e2a15f94ea085ee25f680a02daf6df62fd429bd9c25c9",
"pages/topics/chatbot/README.md": "6f2f83ffb80109be47f9750cca9d3036ec3645abef9f2b590bd6825780c7c48a",
"pages/topics/faq/README.md": "15cfbd9c2a1649c80a7a58922e2b66dadab5c86a5ef33ef16c82bebd4d2dbf06",
"pages/topics/hatespeech/README.md": "364df94e70dd740c63544e2586ba7b957780f714fc0c941d1b04b02233e6cbc5",
"pages/topics/legal/README.md": "003bb9bfe1e0ca21652091c2d11496312eb04e0689fe459e14c2d5a738e8eeea",
"pages/topics/morphology/README.md": "e242cc5ff1a4cc98e227566c0ead999bbc95937b7c20be62cb2afb773d0d35c0",
"pages/topics/named-entity/README.md": "66b2e25f70211d432d596e377d552b3b8f0c525b14c31e3bc9c956c3aeadca84",
"pages/topics/named-entity/navod/README.md": "1ff87079a98411de0414754c60241e08d2214ddcd63ddf974e4340cce2ce5b02",
"pages/topics/nlp/README.md": "6f90abf9dc3f0ff33475283c13b4a08561abf631425ec2daacb076155a9e02c2",
"pages/topics/otvorene/README.md": "cd5544925553fe3840c0ea6f8ef60ab756e26ab6757bf810025c948d8d4f98fc",
"pages/topics/prodigy/README.md": "5e4f691f2754c2ec4afbaab2f41e6db77feb5b322dd9344465eb40c35f116499",
"pages/topics/python/README.md": "2ef073adba672142c39e9a59fa2e00a98ab52bda54233d46fbe6940ee23f9fcc",
"pages/topics/question/README.md": "408078a2a67cea2699e12359e85ab641136231e6104c2afdf874a705879525df",
"pages/topics/question/navod/README.md": "061e737c69184b2bd244316ca7ba841f7186fe01ee80ee993742a733dcad3ecd",
"pages/topics/question/nezodpovedatelne/README.md": "b6703d9690f4599b973efb27595ce157773ae0fc0d55cf2c0a09d9f50e8e4c92",
"pages/topics/question/validacie/README.md": "3216a7127abdfc6da50ed116f44ff2d6785e7da4af70d23aceda774e745ad562",
"pages/topics/resources/README.md": "09405397387f00cbcc5680f7d655f2d861e4d92e5df792a62d9dfd04f3534f48",
"pages/topics/seq2seq/README.md": "969d15c03a63a3f8d5edc9017fe5a0fb7311fdf4de4a3473a08266241f92c1e2",
"pages/topics/spacy/README.md": "f504e5db77cb8694f08a57fb8636e522d59a0e669216f4bc8273bc3c5f375f87",
"pages/topics/steel/README.md": "b4068cef2ed197711467ac47630d5b2064fdc8828f898f7af99664e94d833330",
"pages/topics/translation/README.md": "a7d4289e72199b307dc5ad8fc69cda39c1a33c27a6c7447af70aa9c3cc6b8635"
}
},
"retrieval_policy": {
"published_only": true,
"publication_policy": "explicit-false-hidden-v1",
"academic_preparation_is_work": false
}
},
"graph": {
"sha256": "566817ce7ddebb2638585192c148c41421071bdf719674fc6ddddb1c99a0e36f",
"nodes": 446,
"relationships": 877
},
"parameters": {
"limit": 40,
"max_entities": 8
},
"dev_summary": {
"questions": 1046,
"source_checked_count": 1046,
"all_facts_supported_count": 1046,
"strict_count": 1046,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.081848,
"latency_median_seconds": 0.071172,
"latency_p95_seconds": 0.113998
},
"source_audit_summary": {
"dev": {
"questions": 1046,
"source_consistent": 1046,
"issues": {}
},
"test": {
"questions": 80,
"source_consistent": 80,
"issues": {}
}
},
"policy": "Report all questions including negatives. No exclusions or holdout tuning.",
"dev_results_sha256": "d1a5d7b6d5aff3c0002f36a60cebc07627a79ec6d57de119dd587e8ab70143b0",
"audit_sha256": "7825d4f3e4e535eb5f49e975dfef363b9b813f1de5c97e2387e5c2f3e578e6a5"
}

143614
tools/graphrag/evaluation/graphrag_questions_v7.json Normal file
View File

File diff suppressed because it is too large Load Diff

1
tools/graphrag/evaluation/reproducibility.py
View File

@ -39,6 +39,7 @@ def content_fingerprints(questions: Path, root: Path = ROOT) -> dict[str, Any]:
'tools/index_state.py', 'tools/rag/common.py', 'tools/sync_indexes.py',
'tools/rag/rag_utils.py', 'tools/answer_evaluation/context_policy.py',
'tools/answer_evaluation/checked_generation.py', 'tools/answer_evaluation/answer_format.py',
'tools/answer_evaluation/render_answer.py',
'tools/rag/evaluation/rag_metrics.py', 'tools/rag/evaluation/rag_runner.py'])
return {
'implementation': {str(path.relative_to(root)): sha256_file(path) for path in files},

47
tools/graphrag/evaluation/results/answers_v7_dev_2026-10-26/answers_dev.csv Normal file
View File

@ -0,0 +1,47 @@
id,category,strict,error,latency_seconds,attempts,first_pass_strict
wide-001,shared_keyword_pair,True,,7.008382885003812,1,True
wide-002,shared_keyword_pair,True,,5.500854778001667,1,True
wide-041,same_academic_year_pair,True,,11.343971206995775,1,True
wide-042,same_academic_year_pair,True,,6.9671594820101745,1,True
wide-080,same_academic_year_pair,True,,9.73526496400882,1,True
wide-081,same_start_year_pair,True,,6.6912147020048,1,True
wide-082,same_start_year_pair,True,,9.779698412006837,1,True
wide-104,same_start_year_pair,True,,7.042951979994541,1,True
wide-113,person_title_year,True,,8.861812982009724,1,True
wide-114,person_title_year,True,,5.960790655997698,1,True
wide-116,person_title_year,True,,5.481083346996456,1,True
wide-119,person_title_year,True,,5.442881409995607,1,True
wide-129,person_title_year,True,,5.64509399299277,1,True
wide-137,person_keyword_title,True,,7.987423918006243,1,True
wide-138,person_keyword_title,True,,6.728769060995546,1,True
wide-139,person_keyword_title,True,,8.942678854000405,1,True
wide-141,person_keyword_title,True,,7.0407465109892655,1,True
wide-142,person_keyword_title,True,,7.21462029000395,1,True
wide-143,person_keyword_title,True,,8.128857211995637,1,True
wide-145,person_keyword_title,True,,3.167452590991161,1,True
wide-155,person_keyword_title,True,,6.265883696993114,1,True
wide-159,person_keyword_title,True,,9.182629139002529,1,True
wide-161,person_keyword_title,True,,7.464124964011717,1,True
wide-163,person_keyword_title,True,,10.472138197001186,1,True
wide-165,person_keyword_title,True,,3.6118825629964704,1,True
wide-169,document_author_category_multi_hop,True,,6.421270335005829,1,True
wide-170,document_author_category_multi_hop,True,,7.26794687200163,1,True
wide-178,document_author_category_multi_hop,True,,5.971922497003106,1,True
wide-181,document_author_category_multi_hop,True,,6.660555914000724,1,True
wide-193,person_multiple_categories,True,,5.167634055993403,1,True
wide-194,person_multiple_categories,True,,5.633185668993974,1,True
wide-209,person_multiple_categories,True,,4.679902340998524,1,True
wide-217,candidate_set_aggregation,True,,7.1928625760047,1,True
wide-218,candidate_set_aggregation,True,,7.220508996004355,1,True
wide-232,candidate_set_aggregation,True,,9.97075244000007,1,True
wide-236,candidate_set_aggregation,True,,8.210259957006201,1,True
wide-239,candidate_set_aggregation,True,,11.907442174997414,1,True
wide-249,conjunctive_disambiguation,True,,5.638401258998783,1,True
wide-250,conjunctive_disambiguation,True,,9.00651249299699,1,True
wide-283,conjunctive_disambiguation,True,,14.987600032996852,2,False
wide-289,negative_lookup,True,,5.1113616619986715,1,True
wide-290,negative_lookup,True,,2.8115369029983412,1,True
v7-dev-year-1999,negative_lookup,True,,2.0942367279931204,1,True
v7-dev-year-1899,negative_lookup,True,,2.1002495850116247,1,True
v7-dev-year-2100,negative_lookup,True,,2.0984817599965027,1,True
v7-dev-year-9999,negative_lookup,True,,2.9951046469941502,1,True
1 id category strict error latency_seconds attempts first_pass_strict
2 wide-001 shared_keyword_pair True 7.008382885003812 1 True
3 wide-002 shared_keyword_pair True 5.500854778001667 1 True
4 wide-041 same_academic_year_pair True 11.343971206995775 1 True
5 wide-042 same_academic_year_pair True 6.9671594820101745 1 True
6 wide-080 same_academic_year_pair True 9.73526496400882 1 True
7 wide-081 same_start_year_pair True 6.6912147020048 1 True
8 wide-082 same_start_year_pair True 9.779698412006837 1 True
9 wide-104 same_start_year_pair True 7.042951979994541 1 True
10 wide-113 person_title_year True 8.861812982009724 1 True
11 wide-114 person_title_year True 5.960790655997698 1 True
12 wide-116 person_title_year True 5.481083346996456 1 True
13 wide-119 person_title_year True 5.442881409995607 1 True
14 wide-129 person_title_year True 5.64509399299277 1 True
15 wide-137 person_keyword_title True 7.987423918006243 1 True
16 wide-138 person_keyword_title True 6.728769060995546 1 True
17 wide-139 person_keyword_title True 8.942678854000405 1 True
18 wide-141 person_keyword_title True 7.0407465109892655 1 True
19 wide-142 person_keyword_title True 7.21462029000395 1 True
20 wide-143 person_keyword_title True 8.128857211995637 1 True
21 wide-145 person_keyword_title True 3.167452590991161 1 True
22 wide-155 person_keyword_title True 6.265883696993114 1 True
23 wide-159 person_keyword_title True 9.182629139002529 1 True
24 wide-161 person_keyword_title True 7.464124964011717 1 True
25 wide-163 person_keyword_title True 10.472138197001186 1 True
26 wide-165 person_keyword_title True 3.6118825629964704 1 True
27 wide-169 document_author_category_multi_hop True 6.421270335005829 1 True
28 wide-170 document_author_category_multi_hop True 7.26794687200163 1 True
29 wide-178 document_author_category_multi_hop True 5.971922497003106 1 True
30 wide-181 document_author_category_multi_hop True 6.660555914000724 1 True
31 wide-193 person_multiple_categories True 5.167634055993403 1 True
32 wide-194 person_multiple_categories True 5.633185668993974 1 True
33 wide-209 person_multiple_categories True 4.679902340998524 1 True
34 wide-217 candidate_set_aggregation True 7.1928625760047 1 True
35 wide-218 candidate_set_aggregation True 7.220508996004355 1 True
36 wide-232 candidate_set_aggregation True 9.97075244000007 1 True
37 wide-236 candidate_set_aggregation True 8.210259957006201 1 True
38 wide-239 candidate_set_aggregation True 11.907442174997414 1 True
39 wide-249 conjunctive_disambiguation True 5.638401258998783 1 True
40 wide-250 conjunctive_disambiguation True 9.00651249299699 1 True
41 wide-283 conjunctive_disambiguation True 14.987600032996852 2 False
42 wide-289 negative_lookup True 5.1113616619986715 1 True
43 wide-290 negative_lookup True 2.8115369029983412 1 True
44 v7-dev-year-1999 negative_lookup True 2.0942367279931204 1 True
45 v7-dev-year-1899 negative_lookup True 2.1002495850116247 1 True
46 v7-dev-year-2100 negative_lookup True 2.0984817599965027 1 True
47 v7-dev-year-9999 negative_lookup True 2.9951046469941502 1 True

9719
tools/graphrag/evaluation/results/answers_v7_dev_2026-10-26/answers_dev.json Normal file
View File

File diff suppressed because it is too large Load Diff

225
tools/graphrag/evaluation/results/answers_v7_dev_2026-10-26/frozen_answers.json Normal file
View File

@ -0,0 +1,225 @@
{
"method": "Known-question development selection.",
"graph_freeze_sha256": "9e97794200520d0f2288e452c9e186c0291a54c63d2c1382f693a9753e4081cf",
"audit_sha256": "7825d4f3e4e535eb5f49e975dfef363b9b813f1de5c97e2387e5c2f3e578e6a5",
"generation_files": {
"tools/answer_evaluation/evaluate_graph_answers.py": "d81504e8fe51598c6bd1b0d482f61b9b6f51f87ba3f38b03ae1bc584675a3b76",
"tools/answer_evaluation/graph_answer_prompt.txt": "fc9f2f8d52e70a8d67b9f4814b3f15e6be64bf88afe53ce4db4f037bde00e064",
"tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47",
"tools/answer_evaluation/checked_generation.py": "cb821a65327424c19cbd996190b14e6aa30d9b909493f11716552f71249a0529",
"tools/answer_evaluation/render_answer.py": "609c534a37899b3febcea32d983da4b806c287de8f2ddf9bc0675e3280859002",
"tools/rag/evaluation/rag_runner.py": "031abf7d5b1a026eb199aa2a579bfb8baaa01175279d0018a364bfe50b76c3c1"
},
"model": "model120-fast",
"temperature": 0,
"workers": 3,
"dev_ids": [
"v7-dev-year-1899",
"v7-dev-year-1999",
"v7-dev-year-2100",
"v7-dev-year-9999",
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-116",
"wide-119",
"wide-129",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-165",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"test_ids": [
"fresh7-001",
"fresh7-002",
"fresh7-003",
"fresh7-004",
"fresh7-005",
"fresh7-006",
"fresh7-007",
"fresh7-008",
"fresh7-009",
"fresh7-010",
"fresh7-011",
"fresh7-012",
"fresh7-013",
"fresh7-014",
"fresh7-015",
"fresh7-016",
"fresh7-017",
"fresh7-018",
"fresh7-019",
"fresh7-020",
"fresh7-021",
"fresh7-022",
"fresh7-023",
"fresh7-024",
"fresh7-025",
"fresh7-026",
"fresh7-027",
"fresh7-028",
"fresh7-029",
"fresh7-030",
"fresh7-031",
"fresh7-032",
"fresh7-033",
"fresh7-034",
"fresh7-035",
"fresh7-036",
"fresh7-037",
"fresh7-038",
"fresh7-039",
"fresh7-040",
"fresh7-041",
"fresh7-042",
"fresh7-043",
"fresh7-044",
"fresh7-045",
"fresh7-046",
"fresh7-047",
"fresh7-048",
"fresh7-049",
"fresh7-050",
"fresh7-051",
"fresh7-052",
"fresh7-053",
"fresh7-054",
"fresh7-055",
"fresh7-056",
"fresh7-057",
"fresh7-058",
"fresh7-059",
"fresh7-060",
"fresh7-061",
"fresh7-062",
"fresh7-063",
"fresh7-064",
"fresh7-065",
"fresh7-066",
"fresh7-067",
"fresh7-068",
"fresh7-069",
"fresh7-070",
"fresh7-071",
"fresh7-072",
"fresh7-073",
"fresh7-074",
"fresh7-075",
"fresh7-076",
"fresh7-077",
"fresh7-078",
"fresh7-079",
"fresh7-080"
],
"dev_count": 46,
"test_count": 80,
"target_accuracy": 0.92,
"evaluation_kind": "development",
"rendering": "Verified structured claims rendered deterministically; raw model outputs preserved.",
"selected_split": "dev",
"selected_ids": [
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-116",
"wide-119",
"wide-129",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-165",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290",
"v7-dev-year-1999",
"v7-dev-year-1899",
"v7-dev-year-2100",
"v7-dev-year-9999"
],
"selected_count": 46,
"selected_dev_overlap": 46,
"corrective_request_limit": 1,
"format_repair_limit": 1,
"repair_policy": "At most one total corrective request for format, citations or retrieval-context violations. Same question/context, no gold; retain all attempts.",
"dev_results_sha256": null,
"policy": "Evaluate every declared fresh TEST case once; retain every failure; do not tune on this TEST.",
"selected_categories": {
"shared_keyword_pair": 2,
"same_academic_year_pair": 3,
"same_start_year_pair": 3,
"person_title_year": 5,
"person_keyword_title": 12,
"document_author_category_multi_hop": 4,
"person_multiple_categories": 3,
"candidate_set_aggregation": 5,
"conjunctive_disambiguation": 3,
"negative_lookup": 6
},
"limitations": [
"Known development questions; not a holdout.",
"Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.",
"Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose."
]
}

321
tools/graphrag/evaluation/results/answers_v7_regression_2026-10-26/answers_regression.csv Normal file
View File

@ -0,0 +1,321 @@
id,category,strict,error,latency_seconds,attempts,first_pass_strict
wide-001,shared_keyword_pair,True,,5.262378749001073,1,True
wide-002,shared_keyword_pair,True,,11.023844788011047,1,True
wide-003,shared_keyword_pair,True,,6.837112370994873,1,True
wide-004,shared_keyword_pair,True,,7.825073618005263,1,True
wide-005,shared_keyword_pair,True,,10.047133992004092,1,True
wide-006,shared_keyword_pair,True,,8.64593629699084,1,True
wide-007,shared_keyword_pair,True,,9.661823686998105,1,True
wide-008,shared_keyword_pair,True,,8.533432190000894,1,True
wide-009,shared_keyword_pair,True,,8.852785049995873,1,True
wide-010,shared_keyword_pair,True,,8.675380236003548,1,True
wide-011,shared_keyword_pair,True,,9.178905939988908,1,True
wide-012,shared_keyword_pair,True,,8.79627010099648,1,True
wide-013,shared_keyword_pair,True,,9.117540171995643,1,True
wide-014,shared_keyword_pair,True,,8.615218720005942,1,True
wide-015,shared_keyword_pair,True,,9.395387979006045,1,True
wide-016,shared_keyword_pair,True,,8.779454938994604,1,True
wide-017,shared_keyword_pair,True,,9.478577678994043,1,True
wide-018,shared_keyword_pair,True,,8.680933992000064,1,True
wide-019,shared_keyword_pair,True,,9.240019315999234,1,True
wide-020,shared_keyword_pair,True,,8.530438004003372,1,True
wide-021,shared_keyword_pair,True,,9.019007034992683,1,True
wide-022,shared_keyword_pair,True,,8.598487436000141,1,True
wide-023,shared_keyword_pair,True,,9.186067746006302,1,True
wide-024,shared_keyword_pair,True,,8.59094295601244,1,True
wide-025,shared_keyword_pair,True,,8.722591666009976,1,True
wide-026,shared_keyword_pair,True,,8.684057810998638,1,True
wide-027,shared_keyword_pair,True,,8.90965538599994,1,True
wide-028,shared_keyword_pair,True,,9.036431355998502,1,True
wide-029,shared_keyword_pair,True,,16.985941207996802,2,False
wide-030,shared_keyword_pair,True,,9.278765659008059,1,True
wide-031,shared_keyword_pair,True,,8.859797060998972,1,True
wide-032,shared_keyword_pair,True,,8.551712281012442,1,True
wide-033,shared_keyword_pair,True,,8.871659873999306,1,True
wide-034,shared_keyword_pair,True,,10.27713256199786,1,True
wide-035,shared_keyword_pair,True,,8.731808247001027,1,True
wide-036,shared_keyword_pair,True,,10.022244788997341,1,True
wide-037,shared_keyword_pair,True,,7.606950429995777,1,True
wide-038,shared_keyword_pair,True,,9.073549334003474,1,True
wide-039,shared_keyword_pair,True,,6.649941058989498,1,True
wide-040,shared_keyword_pair,True,,9.70009443500021,1,True
wide-041,same_academic_year_pair,True,,4.9646135219954886,1,True
wide-042,same_academic_year_pair,True,,8.739653348005959,1,True
wide-043,same_academic_year_pair,True,,8.06010635799612,1,True
wide-044,same_academic_year_pair,True,,10.40220917899569,1,True
wide-045,same_academic_year_pair,True,,8.44839003800007,1,True
wide-046,same_academic_year_pair,True,,9.635705161999795,1,True
wide-047,same_academic_year_pair,True,,11.822098678007023,1,True
wide-048,same_academic_year_pair,True,,8.148709733010037,1,True
wide-049,same_academic_year_pair,True,,18.129530342004728,2,False
wide-050,same_academic_year_pair,True,,8.248969623993617,1,True
wide-051,same_academic_year_pair,True,,10.277082996006357,1,True
wide-052,same_academic_year_pair,True,,10.281655037004384,1,True
wide-053,same_academic_year_pair,True,,5.898193639994133,1,True
wide-054,same_academic_year_pair,True,,7.909174476997578,1,True
wide-055,same_academic_year_pair,True,,6.191400594994775,1,True
wide-056,same_academic_year_pair,True,,10.229951275003259,1,True
wide-057,same_academic_year_pair,True,,5.858902864987613,1,True
wide-058,same_academic_year_pair,True,,9.390882232997683,1,True
wide-059,same_academic_year_pair,True,,7.287893086002441,1,True
wide-060,same_academic_year_pair,True,,10.045547063986305,1,True
wide-061,same_academic_year_pair,True,,7.13498594200064,1,True
wide-062,same_academic_year_pair,True,,8.994601780999801,1,True
wide-063,same_academic_year_pair,True,,6.288157966002473,1,True
wide-064,same_academic_year_pair,True,,9.025555226995493,1,True
wide-065,same_academic_year_pair,True,,6.9650647310045315,1,True
wide-066,same_academic_year_pair,True,,9.574780579001526,1,True
wide-067,same_academic_year_pair,True,,8.074011475997395,1,True
wide-068,same_academic_year_pair,True,,9.531611518992577,1,True
wide-069,same_academic_year_pair,True,,8.634510207004496,1,True
wide-070,same_academic_year_pair,True,,8.958400035000523,1,True
wide-071,same_academic_year_pair,True,,9.77839753999433,1,True
wide-072,same_academic_year_pair,True,,8.366447199005052,1,True
wide-073,same_academic_year_pair,True,,9.756975818992942,1,True
wide-074,same_academic_year_pair,True,,7.232408730007592,1,True
wide-075,same_academic_year_pair,True,,9.765164623997407,1,True
wide-076,same_academic_year_pair,True,,7.285313993008458,1,True
wide-077,same_academic_year_pair,True,,10.151919288997306,1,True
wide-078,same_academic_year_pair,True,,7.673378441002569,1,True
wide-079,same_academic_year_pair,True,,10.634893157999613,1,True
wide-080,same_academic_year_pair,True,,4.789500615996076,1,True
wide-081,same_start_year_pair,True,,8.585048984998139,1,True
wide-082,same_start_year_pair,True,,8.499404791000416,1,True
wide-083,same_start_year_pair,True,,5.630402820999734,1,True
wide-084,same_start_year_pair,True,,6.908654918996035,1,True
wide-085,same_start_year_pair,True,,5.944816776987864,1,True
wide-086,same_start_year_pair,True,,9.126075861990103,1,True
wide-087,same_start_year_pair,True,,5.440862012997968,1,True
wide-088,same_start_year_pair,True,,6.5059300099965185,1,True
wide-089,same_start_year_pair,True,,6.697157860995503,1,True
wide-090,same_start_year_pair,True,,3.26202636299422,1,True
wide-091,same_start_year_pair,True,,6.4398382079962175,1,True
wide-092,same_start_year_pair,True,,9.066495395993115,1,True
wide-093,same_start_year_pair,True,,7.360152271998231,1,True
wide-094,same_start_year_pair,True,,9.541269890003605,1,True
wide-095,same_start_year_pair,True,,7.569437859012396,1,True
wide-096,same_start_year_pair,True,,11.024270423004054,1,True
wide-097,same_start_year_pair,True,,5.715482076993794,1,True
wide-098,same_start_year_pair,True,,8.736371631996008,1,True
wide-099,same_start_year_pair,True,,6.249446924994118,1,True
wide-100,same_start_year_pair,True,,10.805812649006839,1,True
wide-101,same_start_year_pair,True,,5.220276794992969,1,True
wide-102,same_start_year_pair,True,,9.947560219006846,1,True
wide-103,same_start_year_pair,True,,7.779931154000224,1,True
wide-104,same_start_year_pair,True,,10.890818430008949,1,True
wide-105,same_start_year_pair,True,,3.6545174839993706,1,True
wide-106,same_start_year_pair,True,,7.7419946619920665,1,True
wide-107,same_start_year_pair,True,,4.410995238999021,1,True
wide-108,same_start_year_pair,True,,6.583789386000717,1,True
wide-109,same_start_year_pair,True,,6.958590795999044,1,True
wide-110,same_start_year_pair,True,,11.648579076994793,1,True
wide-111,same_start_year_pair,True,,3.022186320988112,1,True
wide-112,same_start_year_pair,True,,2.6744268070033286,1,True
wide-113,person_title_year,True,,3.9001984620117582,1,True
wide-114,person_title_year,True,,6.90090462200169,1,True
wide-115,person_title_year,True,,5.2774354059947655,1,True
wide-116,person_title_year,True,,5.405227045004722,1,True
wide-117,person_title_year,True,,4.662688102005632,1,True
wide-118,person_title_year,True,,4.6598335029993905,1,True
wide-119,person_title_year,True,,7.607376700005261,1,True
wide-120,person_title_year,True,,3.551628603003337,1,True
wide-121,person_title_year,True,,6.179311268002493,1,True
wide-122,person_title_year,True,,5.917100324993953,1,True
wide-123,person_title_year,True,,6.548046129988506,1,True
wide-124,person_title_year,True,,7.861738689010963,1,True
wide-125,person_title_year,True,,4.563962607993744,1,True
wide-126,person_title_year,True,,5.80828094399476,1,True
wide-127,person_title_year,True,,5.077105101998313,1,True
wide-128,person_title_year,True,,6.038464580007712,1,True
wide-129,person_title_year,True,,7.096872989000985,1,True
wide-130,person_title_year,True,,7.872525553000742,2,False
wide-131,person_title_year,True,,7.498921799997333,1,True
wide-132,person_title_year,True,,5.892531138000777,1,True
wide-133,person_title_year,True,,5.6638127920014085,1,True
wide-134,person_title_year,True,,5.172251234005671,1,True
wide-135,person_title_year,True,,5.744371400011005,1,True
wide-136,person_title_year,True,,4.798829792998731,1,True
wide-137,person_keyword_title,True,,6.910317715010024,1,True
wide-138,person_keyword_title,True,,6.913008176998119,1,True
wide-139,person_keyword_title,True,,8.655944869999075,1,True
wide-140,person_keyword_title,True,,7.773692659000517,1,True
wide-141,person_keyword_title,True,,9.624433186996612,1,True
wide-142,person_keyword_title,True,,3.844812393988832,1,True
wide-143,person_keyword_title,True,,7.311962670995854,1,True
wide-144,person_keyword_title,True,,8.996415396002703,1,True
wide-145,person_keyword_title,True,,4.76172783500806,1,True
wide-146,person_keyword_title,True,,6.710666915998445,1,True
wide-147,person_keyword_title,True,,7.200240036996547,1,True
wide-148,person_keyword_title,True,,8.873094891998335,1,True
wide-149,person_keyword_title,True,,7.56319696000719,1,True
wide-150,person_keyword_title,True,,8.975214990001405,1,True
wide-151,person_keyword_title,True,,7.537857122006244,1,True
wide-152,person_keyword_title,True,,8.62167695100652,1,True
wide-153,person_keyword_title,True,,6.811467941006413,1,True
wide-154,person_keyword_title,True,,9.136220678992686,1,True
wide-155,person_keyword_title,True,,7.490241913008504,1,True
wide-156,person_keyword_title,True,,9.619424235992483,1,True
wide-157,person_keyword_title,True,,7.075699581997469,1,True
wide-158,person_keyword_title,True,,9.402772761997767,1,True
wide-159,person_keyword_title,True,,7.3926155700028175,1,True
wide-160,person_keyword_title,True,,9.592219272002694,1,True
wide-161,person_keyword_title,True,,7.692930470002466,1,True
wide-162,person_keyword_title,True,,9.336959647000185,1,True
wide-163,person_keyword_title,True,,7.659145907993661,1,True
wide-164,person_keyword_title,True,,9.989820002010674,1,True
wide-165,person_keyword_title,True,,3.6757876240008045,1,True
wide-166,person_keyword_title,True,,6.71545403500204,1,True
wide-167,person_keyword_title,True,,8.453232906991616,1,True
wide-168,person_keyword_title,True,,7.789937220004504,1,True
wide-169,document_author_category_multi_hop,True,,8.230146304005757,1,True
wide-170,document_author_category_multi_hop,True,,6.4932332140015205,1,True
wide-171,document_author_category_multi_hop,True,,7.096607529994799,1,True
wide-172,document_author_category_multi_hop,True,,5.593765233003069,1,True
wide-173,document_author_category_multi_hop,True,,6.806600865995279,1,True
wide-174,document_author_category_multi_hop,True,,5.3536878140002955,1,True
wide-175,document_author_category_multi_hop,True,,7.072415729999193,1,True
wide-176,document_author_category_multi_hop,True,,6.418687598998076,1,True
wide-177,document_author_category_multi_hop,True,,7.768641275004484,1,True
wide-178,document_author_category_multi_hop,True,,6.695670635002898,1,True
wide-179,document_author_category_multi_hop,True,,6.646737192000728,1,True
wide-180,document_author_category_multi_hop,True,,5.989018792010029,1,True
wide-181,document_author_category_multi_hop,True,,6.119498893007403,1,True
wide-182,document_author_category_multi_hop,True,,5.9745991970121395,1,True
wide-183,document_author_category_multi_hop,True,,6.077126152988058,1,True
wide-184,document_author_category_multi_hop,True,,5.992435852997005,1,True
wide-185,document_author_category_multi_hop,True,,6.213222564998432,1,True
wide-186,document_author_category_multi_hop,True,,6.683954360996722,1,True
wide-187,document_author_category_multi_hop,True,,6.843328906004899,1,True
wide-188,document_author_category_multi_hop,True,,7.07258083600027,1,True
wide-189,document_author_category_multi_hop,True,,6.259952289008652,1,True
wide-190,document_author_category_multi_hop,True,,6.505296975999954,1,True
wide-191,document_author_category_multi_hop,True,,6.268639836998773,1,True
wide-192,document_author_category_multi_hop,True,,7.003398792003281,1,True
wide-193,person_multiple_categories,True,,6.194855802998063,1,True
wide-194,person_multiple_categories,True,,5.364437454001745,1,True
wide-195,person_multiple_categories,True,,4.545286751992535,1,True
wide-196,person_multiple_categories,True,,4.151337641000282,1,True
wide-197,person_multiple_categories,True,,8.146557949992712,2,False
wide-198,person_multiple_categories,True,,4.233387041007518,1,True
wide-199,person_multiple_categories,True,,6.291938512003981,1,True
wide-200,person_multiple_categories,True,,9.257844247011235,2,False
wide-201,person_multiple_categories,True,,3.8606335170043167,1,True
wide-202,person_multiple_categories,True,,4.706902887002798,1,True
wide-203,person_multiple_categories,True,,4.9716377110016765,1,True
wide-204,person_multiple_categories,True,,11.950999951994163,2,False
wide-205,person_multiple_categories,True,,5.60192243399797,1,True
wide-206,person_multiple_categories,True,,5.590988194002421,1,True
wide-207,person_multiple_categories,True,,7.846379655005876,2,False
wide-208,person_multiple_categories,True,,7.050330217010924,1,True
wide-209,person_multiple_categories,True,,5.981790188001469,1,True
wide-210,person_multiple_categories,True,,4.014929401004338,1,True
wide-211,person_multiple_categories,True,,5.048257627000567,1,True
wide-212,person_multiple_categories,True,,2.727644105994841,1,True
wide-213,person_multiple_categories,True,,4.587378602998797,1,True
wide-214,person_multiple_categories,True,,3.6640594680065988,1,True
wide-215,person_multiple_categories,True,,7.165378771998803,1,True
wide-216,person_multiple_categories,True,,6.913648023997666,2,False
wide-217,candidate_set_aggregation,True,,8.149436000996502,1,True
wide-218,candidate_set_aggregation,True,,7.928223324997816,1,True
wide-219,candidate_set_aggregation,True,,8.299362689998816,1,True
wide-220,candidate_set_aggregation,True,,9.450791957002366,1,True
wide-221,candidate_set_aggregation,True,,8.832846570992842,1,True
wide-222,candidate_set_aggregation,True,,9.567884083997342,1,True
wide-223,candidate_set_aggregation,True,,8.838387846000842,1,True
wide-224,candidate_set_aggregation,True,,9.386160062989802,1,True
wide-225,candidate_set_aggregation,True,,8.62494325399166,1,True
wide-226,candidate_set_aggregation,True,,9.428272984994692,1,True
wide-227,candidate_set_aggregation,True,,8.599248733007698,1,True
wide-228,candidate_set_aggregation,True,,9.487015202001203,1,True
wide-229,candidate_set_aggregation,True,,8.605383807007456,1,True
wide-230,candidate_set_aggregation,True,,9.61059859499801,1,True
wide-231,candidate_set_aggregation,True,,8.823051813000347,1,True
wide-232,candidate_set_aggregation,True,,9.740953706990695,1,True
wide-233,candidate_set_aggregation,True,,8.722386483990704,1,True
wide-234,candidate_set_aggregation,True,,9.489644886998576,1,True
wide-235,candidate_set_aggregation,True,,8.559914067009231,1,True
wide-236,candidate_set_aggregation,True,,9.524188774012146,1,True
wide-237,candidate_set_aggregation,True,,8.78702475198952,1,True
wide-238,candidate_set_aggregation,True,,9.779589921003208,1,True
wide-239,candidate_set_aggregation,True,,8.832212022010935,1,True
wide-240,candidate_set_aggregation,True,,9.979131497995695,1,True
wide-241,candidate_set_aggregation,True,,8.59488222800428,1,True
wide-242,candidate_set_aggregation,True,,9.823719104999327,1,True
wide-243,candidate_set_aggregation,True,,8.220046620001085,1,True
wide-244,candidate_set_aggregation,True,,10.194505384002696,1,True
wide-245,candidate_set_aggregation,True,,8.194634853003663,1,True
wide-246,candidate_set_aggregation,True,,10.07240725599695,1,True
wide-247,candidate_set_aggregation,True,,7.7612789029954,1,True
wide-248,candidate_set_aggregation,True,,11.560260020007263,1,True
wide-249,conjunctive_disambiguation,True,,5.476591879996704,1,True
wide-250,conjunctive_disambiguation,True,,8.843467402999522,1,True
wide-251,conjunctive_disambiguation,True,,7.115450201003114,1,True
wide-252,conjunctive_disambiguation,True,,8.416170042997692,1,True
wide-253,conjunctive_disambiguation,True,,6.920364450008492,1,True
wide-254,conjunctive_disambiguation,True,,8.249018574002548,1,True
wide-255,conjunctive_disambiguation,True,,6.858485064003617,1,True
wide-256,conjunctive_disambiguation,True,,8.336330271995394,1,True
wide-257,conjunctive_disambiguation,True,,6.96214788701036,1,True
wide-258,conjunctive_disambiguation,True,,8.381303394009592,1,True
wide-259,conjunctive_disambiguation,True,,7.045915052003693,1,True
wide-260,conjunctive_disambiguation,True,,8.721610349006369,1,True
wide-261,conjunctive_disambiguation,True,,6.935596920011449,1,True
wide-262,conjunctive_disambiguation,True,,8.776731780002592,1,True
wide-263,conjunctive_disambiguation,True,,6.759910934997606,1,True
wide-264,conjunctive_disambiguation,True,,9.799247552000452,1,True
wide-265,conjunctive_disambiguation,True,,5.189480711997021,1,True
wide-266,conjunctive_disambiguation,True,,8.85979151999345,1,True
wide-267,conjunctive_disambiguation,True,,6.7524094420077745,1,True
wide-268,conjunctive_disambiguation,True,,9.291388099998585,1,True
wide-269,conjunctive_disambiguation,True,,6.566982984004426,1,True
wide-270,conjunctive_disambiguation,True,,8.416923523996957,1,True
wide-271,conjunctive_disambiguation,True,,6.700699591994635,1,True
wide-272,conjunctive_disambiguation,True,,8.531122063999646,1,True
wide-273,conjunctive_disambiguation,True,,6.868541053991066,1,True
wide-274,conjunctive_disambiguation,True,,8.585297629004344,1,True
wide-275,conjunctive_disambiguation,True,,6.873942530000932,1,True
wide-276,conjunctive_disambiguation,True,,8.594346833997406,1,True
wide-277,conjunctive_disambiguation,True,,6.816038967008353,1,True
wide-278,conjunctive_disambiguation,True,,8.591339434002293,1,True
wide-279,conjunctive_disambiguation,True,,6.7824973950046115,1,True
wide-280,conjunctive_disambiguation,True,,8.697670733003179,1,True
wide-281,conjunctive_disambiguation,True,,6.762842822994571,1,True
wide-282,conjunctive_disambiguation,True,,8.647353213993483,1,True
wide-283,conjunctive_disambiguation,True,,18.08595217599941,2,False
wide-284,conjunctive_disambiguation,True,,7.201167734005139,1,True
wide-285,conjunctive_disambiguation,False,,28.01832078601001,2,False
wide-286,conjunctive_disambiguation,True,,5.158636806998402,1,True
wide-287,conjunctive_disambiguation,True,,11.181534039991675,1,True
wide-288,conjunctive_disambiguation,True,,5.375784352989285,1,True
wide-289,negative_lookup,True,,3.1519780079979682,1,True
wide-290,negative_lookup,True,,2.5652610169927357,1,True
wide-291,negative_lookup,True,,2.116932502001873,1,True
wide-292,negative_lookup,True,,2.1522221219929634,1,True
wide-293,negative_lookup,True,,2.2185270579939242,1,True
wide-294,negative_lookup,True,,2.3594186319969594,1,True
wide-295,negative_lookup,True,,3.4991669790033484,1,True
wide-296,negative_lookup,True,,3.309362318002968,1,True
wide-297,negative_lookup,True,,3.1218993680085987,1,True
wide-298,negative_lookup,True,,2.7141988500079606,1,True
wide-299,negative_lookup,True,,2.4447732209955575,1,True
wide-300,negative_lookup,True,,2.579182736008079,1,True
wide-301,negative_lookup,True,,2.425772479007719,1,True
wide-302,negative_lookup,True,,2.5622203609964345,1,True
wide-303,negative_lookup,True,,2.476981298008468,1,True
wide-304,negative_lookup,True,,2.6007943300064653,1,True
wide-305,negative_lookup,True,,2.457786357990699,1,True
wide-306,negative_lookup,True,,2.566486335999798,1,True
wide-307,negative_lookup,True,,2.6169559550035046,1,True
wide-308,negative_lookup,True,,2.7340419859974645,1,True
wide-309,negative_lookup,True,,2.619804314992507,1,True
wide-310,negative_lookup,True,,2.556121910994989,1,True
wide-311,negative_lookup,True,,2.4317323060095077,1,True
wide-312,negative_lookup,True,,2.506849172001239,1,True
wide-313,negative_lookup,True,,2.4141654220002238,1,True
wide-314,negative_lookup,True,,2.4877845580049325,1,True
wide-315,negative_lookup,True,,2.4770866070030024,1,True
wide-316,negative_lookup,True,,2.494239155013929,1,True
wide-317,negative_lookup,True,,2.461824482001248,1,True
wide-318,negative_lookup,True,,2.4353119149891427,1,True
wide-319,negative_lookup,True,,2.512933713005623,1,True
wide-320,negative_lookup,True,,1.8379592370038154,1,True
1 id category strict error latency_seconds attempts first_pass_strict
2 wide-001 shared_keyword_pair True 5.262378749001073 1 True
3 wide-002 shared_keyword_pair True 11.023844788011047 1 True
4 wide-003 shared_keyword_pair True 6.837112370994873 1 True
5 wide-004 shared_keyword_pair True 7.825073618005263 1 True
6 wide-005 shared_keyword_pair True 10.047133992004092 1 True
7 wide-006 shared_keyword_pair True 8.64593629699084 1 True
8 wide-007 shared_keyword_pair True 9.661823686998105 1 True
9 wide-008 shared_keyword_pair True 8.533432190000894 1 True
10 wide-009 shared_keyword_pair True 8.852785049995873 1 True
11 wide-010 shared_keyword_pair True 8.675380236003548 1 True
12 wide-011 shared_keyword_pair True 9.178905939988908 1 True
13 wide-012 shared_keyword_pair True 8.79627010099648 1 True
14 wide-013 shared_keyword_pair True 9.117540171995643 1 True
15 wide-014 shared_keyword_pair True 8.615218720005942 1 True
16 wide-015 shared_keyword_pair True 9.395387979006045 1 True
17 wide-016 shared_keyword_pair True 8.779454938994604 1 True
18 wide-017 shared_keyword_pair True 9.478577678994043 1 True
19 wide-018 shared_keyword_pair True 8.680933992000064 1 True
20 wide-019 shared_keyword_pair True 9.240019315999234 1 True
21 wide-020 shared_keyword_pair True 8.530438004003372 1 True
22 wide-021 shared_keyword_pair True 9.019007034992683 1 True
23 wide-022 shared_keyword_pair True 8.598487436000141 1 True
24 wide-023 shared_keyword_pair True 9.186067746006302 1 True
25 wide-024 shared_keyword_pair True 8.59094295601244 1 True
26 wide-025 shared_keyword_pair True 8.722591666009976 1 True
27 wide-026 shared_keyword_pair True 8.684057810998638 1 True
28 wide-027 shared_keyword_pair True 8.90965538599994 1 True
29 wide-028 shared_keyword_pair True 9.036431355998502 1 True
30 wide-029 shared_keyword_pair True 16.985941207996802 2 False
31 wide-030 shared_keyword_pair True 9.278765659008059 1 True
32 wide-031 shared_keyword_pair True 8.859797060998972 1 True
33 wide-032 shared_keyword_pair True 8.551712281012442 1 True
34 wide-033 shared_keyword_pair True 8.871659873999306 1 True
35 wide-034 shared_keyword_pair True 10.27713256199786 1 True
36 wide-035 shared_keyword_pair True 8.731808247001027 1 True
37 wide-036 shared_keyword_pair True 10.022244788997341 1 True
38 wide-037 shared_keyword_pair True 7.606950429995777 1 True
39 wide-038 shared_keyword_pair True 9.073549334003474 1 True
40 wide-039 shared_keyword_pair True 6.649941058989498 1 True
41 wide-040 shared_keyword_pair True 9.70009443500021 1 True
42 wide-041 same_academic_year_pair True 4.9646135219954886 1 True
43 wide-042 same_academic_year_pair True 8.739653348005959 1 True
44 wide-043 same_academic_year_pair True 8.06010635799612 1 True
45 wide-044 same_academic_year_pair True 10.40220917899569 1 True
46 wide-045 same_academic_year_pair True 8.44839003800007 1 True
47 wide-046 same_academic_year_pair True 9.635705161999795 1 True
48 wide-047 same_academic_year_pair True 11.822098678007023 1 True
49 wide-048 same_academic_year_pair True 8.148709733010037 1 True
50 wide-049 same_academic_year_pair True 18.129530342004728 2 False
51 wide-050 same_academic_year_pair True 8.248969623993617 1 True
52 wide-051 same_academic_year_pair True 10.277082996006357 1 True
53 wide-052 same_academic_year_pair True 10.281655037004384 1 True
54 wide-053 same_academic_year_pair True 5.898193639994133 1 True
55 wide-054 same_academic_year_pair True 7.909174476997578 1 True
56 wide-055 same_academic_year_pair True 6.191400594994775 1 True
57 wide-056 same_academic_year_pair True 10.229951275003259 1 True
58 wide-057 same_academic_year_pair True 5.858902864987613 1 True
59 wide-058 same_academic_year_pair True 9.390882232997683 1 True
60 wide-059 same_academic_year_pair True 7.287893086002441 1 True
61 wide-060 same_academic_year_pair True 10.045547063986305 1 True
62 wide-061 same_academic_year_pair True 7.13498594200064 1 True
63 wide-062 same_academic_year_pair True 8.994601780999801 1 True
64 wide-063 same_academic_year_pair True 6.288157966002473 1 True
65 wide-064 same_academic_year_pair True 9.025555226995493 1 True
66 wide-065 same_academic_year_pair True 6.9650647310045315 1 True
67 wide-066 same_academic_year_pair True 9.574780579001526 1 True
68 wide-067 same_academic_year_pair True 8.074011475997395 1 True
69 wide-068 same_academic_year_pair True 9.531611518992577 1 True
70 wide-069 same_academic_year_pair True 8.634510207004496 1 True
71 wide-070 same_academic_year_pair True 8.958400035000523 1 True
72 wide-071 same_academic_year_pair True 9.77839753999433 1 True
73 wide-072 same_academic_year_pair True 8.366447199005052 1 True
74 wide-073 same_academic_year_pair True 9.756975818992942 1 True
75 wide-074 same_academic_year_pair True 7.232408730007592 1 True
76 wide-075 same_academic_year_pair True 9.765164623997407 1 True
77 wide-076 same_academic_year_pair True 7.285313993008458 1 True
78 wide-077 same_academic_year_pair True 10.151919288997306 1 True
79 wide-078 same_academic_year_pair True 7.673378441002569 1 True
80 wide-079 same_academic_year_pair True 10.634893157999613 1 True
81 wide-080 same_academic_year_pair True 4.789500615996076 1 True
82 wide-081 same_start_year_pair True 8.585048984998139 1 True
83 wide-082 same_start_year_pair True 8.499404791000416 1 True
84 wide-083 same_start_year_pair True 5.630402820999734 1 True
85 wide-084 same_start_year_pair True 6.908654918996035 1 True
86 wide-085 same_start_year_pair True 5.944816776987864 1 True
87 wide-086 same_start_year_pair True 9.126075861990103 1 True
88 wide-087 same_start_year_pair True 5.440862012997968 1 True
89 wide-088 same_start_year_pair True 6.5059300099965185 1 True
90 wide-089 same_start_year_pair True 6.697157860995503 1 True
91 wide-090 same_start_year_pair True 3.26202636299422 1 True
92 wide-091 same_start_year_pair True 6.4398382079962175 1 True
93 wide-092 same_start_year_pair True 9.066495395993115 1 True
94 wide-093 same_start_year_pair True 7.360152271998231 1 True
95 wide-094 same_start_year_pair True 9.541269890003605 1 True
96 wide-095 same_start_year_pair True 7.569437859012396 1 True
97 wide-096 same_start_year_pair True 11.024270423004054 1 True
98 wide-097 same_start_year_pair True 5.715482076993794 1 True
99 wide-098 same_start_year_pair True 8.736371631996008 1 True
100 wide-099 same_start_year_pair True 6.249446924994118 1 True
101 wide-100 same_start_year_pair True 10.805812649006839 1 True
102 wide-101 same_start_year_pair True 5.220276794992969 1 True
103 wide-102 same_start_year_pair True 9.947560219006846 1 True
104 wide-103 same_start_year_pair True 7.779931154000224 1 True
105 wide-104 same_start_year_pair True 10.890818430008949 1 True
106 wide-105 same_start_year_pair True 3.6545174839993706 1 True
107 wide-106 same_start_year_pair True 7.7419946619920665 1 True
108 wide-107 same_start_year_pair True 4.410995238999021 1 True
109 wide-108 same_start_year_pair True 6.583789386000717 1 True
110 wide-109 same_start_year_pair True 6.958590795999044 1 True
111 wide-110 same_start_year_pair True 11.648579076994793 1 True
112 wide-111 same_start_year_pair True 3.022186320988112 1 True
113 wide-112 same_start_year_pair True 2.6744268070033286 1 True
114 wide-113 person_title_year True 3.9001984620117582 1 True
115 wide-114 person_title_year True 6.90090462200169 1 True
116 wide-115 person_title_year True 5.2774354059947655 1 True
117 wide-116 person_title_year True 5.405227045004722 1 True
118 wide-117 person_title_year True 4.662688102005632 1 True
119 wide-118 person_title_year True 4.6598335029993905 1 True
120 wide-119 person_title_year True 7.607376700005261 1 True
121 wide-120 person_title_year True 3.551628603003337 1 True
122 wide-121 person_title_year True 6.179311268002493 1 True
123 wide-122 person_title_year True 5.917100324993953 1 True
124 wide-123 person_title_year True 6.548046129988506 1 True
125 wide-124 person_title_year True 7.861738689010963 1 True
126 wide-125 person_title_year True 4.563962607993744 1 True
127 wide-126 person_title_year True 5.80828094399476 1 True
128 wide-127 person_title_year True 5.077105101998313 1 True
129 wide-128 person_title_year True 6.038464580007712 1 True
130 wide-129 person_title_year True 7.096872989000985 1 True
131 wide-130 person_title_year True 7.872525553000742 2 False
132 wide-131 person_title_year True 7.498921799997333 1 True
133 wide-132 person_title_year True 5.892531138000777 1 True
134 wide-133 person_title_year True 5.6638127920014085 1 True
135 wide-134 person_title_year True 5.172251234005671 1 True
136 wide-135 person_title_year True 5.744371400011005 1 True
137 wide-136 person_title_year True 4.798829792998731 1 True
138 wide-137 person_keyword_title True 6.910317715010024 1 True
139 wide-138 person_keyword_title True 6.913008176998119 1 True
140 wide-139 person_keyword_title True 8.655944869999075 1 True
141 wide-140 person_keyword_title True 7.773692659000517 1 True
142 wide-141 person_keyword_title True 9.624433186996612 1 True
143 wide-142 person_keyword_title True 3.844812393988832 1 True
144 wide-143 person_keyword_title True 7.311962670995854 1 True
145 wide-144 person_keyword_title True 8.996415396002703 1 True
146 wide-145 person_keyword_title True 4.76172783500806 1 True
147 wide-146 person_keyword_title True 6.710666915998445 1 True
148 wide-147 person_keyword_title True 7.200240036996547 1 True
149 wide-148 person_keyword_title True 8.873094891998335 1 True
150 wide-149 person_keyword_title True 7.56319696000719 1 True
151 wide-150 person_keyword_title True 8.975214990001405 1 True
152 wide-151 person_keyword_title True 7.537857122006244 1 True
153 wide-152 person_keyword_title True 8.62167695100652 1 True
154 wide-153 person_keyword_title True 6.811467941006413 1 True
155 wide-154 person_keyword_title True 9.136220678992686 1 True
156 wide-155 person_keyword_title True 7.490241913008504 1 True
157 wide-156 person_keyword_title True 9.619424235992483 1 True
158 wide-157 person_keyword_title True 7.075699581997469 1 True
159 wide-158 person_keyword_title True 9.402772761997767 1 True
160 wide-159 person_keyword_title True 7.3926155700028175 1 True
161 wide-160 person_keyword_title True 9.592219272002694 1 True
162 wide-161 person_keyword_title True 7.692930470002466 1 True
163 wide-162 person_keyword_title True 9.336959647000185 1 True
164 wide-163 person_keyword_title True 7.659145907993661 1 True
165 wide-164 person_keyword_title True 9.989820002010674 1 True
166 wide-165 person_keyword_title True 3.6757876240008045 1 True
167 wide-166 person_keyword_title True 6.71545403500204 1 True
168 wide-167 person_keyword_title True 8.453232906991616 1 True
169 wide-168 person_keyword_title True 7.789937220004504 1 True
170 wide-169 document_author_category_multi_hop True 8.230146304005757 1 True
171 wide-170 document_author_category_multi_hop True 6.4932332140015205 1 True
172 wide-171 document_author_category_multi_hop True 7.096607529994799 1 True
173 wide-172 document_author_category_multi_hop True 5.593765233003069 1 True
174 wide-173 document_author_category_multi_hop True 6.806600865995279 1 True
175 wide-174 document_author_category_multi_hop True 5.3536878140002955 1 True
176 wide-175 document_author_category_multi_hop True 7.072415729999193 1 True
177 wide-176 document_author_category_multi_hop True 6.418687598998076 1 True
178 wide-177 document_author_category_multi_hop True 7.768641275004484 1 True
179 wide-178 document_author_category_multi_hop True 6.695670635002898 1 True
180 wide-179 document_author_category_multi_hop True 6.646737192000728 1 True
181 wide-180 document_author_category_multi_hop True 5.989018792010029 1 True
182 wide-181 document_author_category_multi_hop True 6.119498893007403 1 True
183 wide-182 document_author_category_multi_hop True 5.9745991970121395 1 True
184 wide-183 document_author_category_multi_hop True 6.077126152988058 1 True
185 wide-184 document_author_category_multi_hop True 5.992435852997005 1 True
186 wide-185 document_author_category_multi_hop True 6.213222564998432 1 True
187 wide-186 document_author_category_multi_hop True 6.683954360996722 1 True
188 wide-187 document_author_category_multi_hop True 6.843328906004899 1 True
189 wide-188 document_author_category_multi_hop True 7.07258083600027 1 True
190 wide-189 document_author_category_multi_hop True 6.259952289008652 1 True
191 wide-190 document_author_category_multi_hop True 6.505296975999954 1 True
192 wide-191 document_author_category_multi_hop True 6.268639836998773 1 True
193 wide-192 document_author_category_multi_hop True 7.003398792003281 1 True
194 wide-193 person_multiple_categories True 6.194855802998063 1 True
195 wide-194 person_multiple_categories True 5.364437454001745 1 True
196 wide-195 person_multiple_categories True 4.545286751992535 1 True
197 wide-196 person_multiple_categories True 4.151337641000282 1 True
198 wide-197 person_multiple_categories True 8.146557949992712 2 False
199 wide-198 person_multiple_categories True 4.233387041007518 1 True
200 wide-199 person_multiple_categories True 6.291938512003981 1 True
201 wide-200 person_multiple_categories True 9.257844247011235 2 False
202 wide-201 person_multiple_categories True 3.8606335170043167 1 True
203 wide-202 person_multiple_categories True 4.706902887002798 1 True
204 wide-203 person_multiple_categories True 4.9716377110016765 1 True
205 wide-204 person_multiple_categories True 11.950999951994163 2 False
206 wide-205 person_multiple_categories True 5.60192243399797 1 True
207 wide-206 person_multiple_categories True 5.590988194002421 1 True
208 wide-207 person_multiple_categories True 7.846379655005876 2 False
209 wide-208 person_multiple_categories True 7.050330217010924 1 True
210 wide-209 person_multiple_categories True 5.981790188001469 1 True
211 wide-210 person_multiple_categories True 4.014929401004338 1 True
212 wide-211 person_multiple_categories True 5.048257627000567 1 True
213 wide-212 person_multiple_categories True 2.727644105994841 1 True
214 wide-213 person_multiple_categories True 4.587378602998797 1 True
215 wide-214 person_multiple_categories True 3.6640594680065988 1 True
216 wide-215 person_multiple_categories True 7.165378771998803 1 True
217 wide-216 person_multiple_categories True 6.913648023997666 2 False
218 wide-217 candidate_set_aggregation True 8.149436000996502 1 True
219 wide-218 candidate_set_aggregation True 7.928223324997816 1 True
220 wide-219 candidate_set_aggregation True 8.299362689998816 1 True
221 wide-220 candidate_set_aggregation True 9.450791957002366 1 True
222 wide-221 candidate_set_aggregation True 8.832846570992842 1 True
223 wide-222 candidate_set_aggregation True 9.567884083997342 1 True
224 wide-223 candidate_set_aggregation True 8.838387846000842 1 True
225 wide-224 candidate_set_aggregation True 9.386160062989802 1 True
226 wide-225 candidate_set_aggregation True 8.62494325399166 1 True
227 wide-226 candidate_set_aggregation True 9.428272984994692 1 True
228 wide-227 candidate_set_aggregation True 8.599248733007698 1 True
229 wide-228 candidate_set_aggregation True 9.487015202001203 1 True
230 wide-229 candidate_set_aggregation True 8.605383807007456 1 True
231 wide-230 candidate_set_aggregation True 9.61059859499801 1 True
232 wide-231 candidate_set_aggregation True 8.823051813000347 1 True
233 wide-232 candidate_set_aggregation True 9.740953706990695 1 True
234 wide-233 candidate_set_aggregation True 8.722386483990704 1 True
235 wide-234 candidate_set_aggregation True 9.489644886998576 1 True
236 wide-235 candidate_set_aggregation True 8.559914067009231 1 True
237 wide-236 candidate_set_aggregation True 9.524188774012146 1 True
238 wide-237 candidate_set_aggregation True 8.78702475198952 1 True
239 wide-238 candidate_set_aggregation True 9.779589921003208 1 True
240 wide-239 candidate_set_aggregation True 8.832212022010935 1 True
241 wide-240 candidate_set_aggregation True 9.979131497995695 1 True
242 wide-241 candidate_set_aggregation True 8.59488222800428 1 True
243 wide-242 candidate_set_aggregation True 9.823719104999327 1 True
244 wide-243 candidate_set_aggregation True 8.220046620001085 1 True
245 wide-244 candidate_set_aggregation True 10.194505384002696 1 True
246 wide-245 candidate_set_aggregation True 8.194634853003663 1 True
247 wide-246 candidate_set_aggregation True 10.07240725599695 1 True
248 wide-247 candidate_set_aggregation True 7.7612789029954 1 True
249 wide-248 candidate_set_aggregation True 11.560260020007263 1 True
250 wide-249 conjunctive_disambiguation True 5.476591879996704 1 True
251 wide-250 conjunctive_disambiguation True 8.843467402999522 1 True
252 wide-251 conjunctive_disambiguation True 7.115450201003114 1 True
253 wide-252 conjunctive_disambiguation True 8.416170042997692 1 True
254 wide-253 conjunctive_disambiguation True 6.920364450008492 1 True
255 wide-254 conjunctive_disambiguation True 8.249018574002548 1 True
256 wide-255 conjunctive_disambiguation True 6.858485064003617 1 True
257 wide-256 conjunctive_disambiguation True 8.336330271995394 1 True
258 wide-257 conjunctive_disambiguation True 6.96214788701036 1 True
259 wide-258 conjunctive_disambiguation True 8.381303394009592 1 True
260 wide-259 conjunctive_disambiguation True 7.045915052003693 1 True
261 wide-260 conjunctive_disambiguation True 8.721610349006369 1 True
262 wide-261 conjunctive_disambiguation True 6.935596920011449 1 True
263 wide-262 conjunctive_disambiguation True 8.776731780002592 1 True
264 wide-263 conjunctive_disambiguation True 6.759910934997606 1 True
265 wide-264 conjunctive_disambiguation True 9.799247552000452 1 True
266 wide-265 conjunctive_disambiguation True 5.189480711997021 1 True
267 wide-266 conjunctive_disambiguation True 8.85979151999345 1 True
268 wide-267 conjunctive_disambiguation True 6.7524094420077745 1 True
269 wide-268 conjunctive_disambiguation True 9.291388099998585 1 True
270 wide-269 conjunctive_disambiguation True 6.566982984004426 1 True
271 wide-270 conjunctive_disambiguation True 8.416923523996957 1 True
272 wide-271 conjunctive_disambiguation True 6.700699591994635 1 True
273 wide-272 conjunctive_disambiguation True 8.531122063999646 1 True
274 wide-273 conjunctive_disambiguation True 6.868541053991066 1 True
275 wide-274 conjunctive_disambiguation True 8.585297629004344 1 True
276 wide-275 conjunctive_disambiguation True 6.873942530000932 1 True
277 wide-276 conjunctive_disambiguation True 8.594346833997406 1 True
278 wide-277 conjunctive_disambiguation True 6.816038967008353 1 True
279 wide-278 conjunctive_disambiguation True 8.591339434002293 1 True
280 wide-279 conjunctive_disambiguation True 6.7824973950046115 1 True
281 wide-280 conjunctive_disambiguation True 8.697670733003179 1 True
282 wide-281 conjunctive_disambiguation True 6.762842822994571 1 True
283 wide-282 conjunctive_disambiguation True 8.647353213993483 1 True
284 wide-283 conjunctive_disambiguation True 18.08595217599941 2 False
285 wide-284 conjunctive_disambiguation True 7.201167734005139 1 True
286 wide-285 conjunctive_disambiguation False 28.01832078601001 2 False
287 wide-286 conjunctive_disambiguation True 5.158636806998402 1 True
288 wide-287 conjunctive_disambiguation True 11.181534039991675 1 True
289 wide-288 conjunctive_disambiguation True 5.375784352989285 1 True
290 wide-289 negative_lookup True 3.1519780079979682 1 True
291 wide-290 negative_lookup True 2.5652610169927357 1 True
292 wide-291 negative_lookup True 2.116932502001873 1 True
293 wide-292 negative_lookup True 2.1522221219929634 1 True
294 wide-293 negative_lookup True 2.2185270579939242 1 True
295 wide-294 negative_lookup True 2.3594186319969594 1 True
296 wide-295 negative_lookup True 3.4991669790033484 1 True
297 wide-296 negative_lookup True 3.309362318002968 1 True
298 wide-297 negative_lookup True 3.1218993680085987 1 True
299 wide-298 negative_lookup True 2.7141988500079606 1 True
300 wide-299 negative_lookup True 2.4447732209955575 1 True
301 wide-300 negative_lookup True 2.579182736008079 1 True
302 wide-301 negative_lookup True 2.425772479007719 1 True
303 wide-302 negative_lookup True 2.5622203609964345 1 True
304 wide-303 negative_lookup True 2.476981298008468 1 True
305 wide-304 negative_lookup True 2.6007943300064653 1 True
306 wide-305 negative_lookup True 2.457786357990699 1 True
307 wide-306 negative_lookup True 2.566486335999798 1 True
308 wide-307 negative_lookup True 2.6169559550035046 1 True
309 wide-308 negative_lookup True 2.7340419859974645 1 True
310 wide-309 negative_lookup True 2.619804314992507 1 True
311 wide-310 negative_lookup True 2.556121910994989 1 True
312 wide-311 negative_lookup True 2.4317323060095077 1 True
313 wide-312 negative_lookup True 2.506849172001239 1 True
314 wide-313 negative_lookup True 2.4141654220002238 1 True
315 wide-314 negative_lookup True 2.4877845580049325 1 True
316 wide-315 negative_lookup True 2.4770866070030024 1 True
317 wide-316 negative_lookup True 2.494239155013929 1 True
318 wide-317 negative_lookup True 2.461824482001248 1 True
319 wide-318 negative_lookup True 2.4353119149891427 1 True
320 wide-319 negative_lookup True 2.512933713005623 1 True
321 wide-320 negative_lookup True 1.8379592370038154 1 True

67440
tools/graphrag/evaluation/results/answers_v7_regression_2026-10-26/answers_regression.json Normal file
View File

File diff suppressed because it is too large Load Diff

260
tools/graphrag/evaluation/results/answers_v7_regression_2026-10-26/failures.json Normal file
View File

@ -0,0 +1,260 @@
{
"evaluation_kind": "historical_regression",
"questions": [
{
"id": "wide-285",
"category": "conjunctive_disambiguation",
"question": "Ktorá z osôb Dárius Lindvai a Dominik Nagy má kľúčové slovo pytorch a jej práca je z roku 2019?",
"graph_context": {
"status": "ok",
"reason": null,
"facts": [
{
"person": "Dárius Lindvai",
"year": 2019,
"document_path": "pages/students/2016/darius_lindvai/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
},
{
"person": "Dárius Lindvai",
"keyword": "pytorch",
"document_path": "pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
}
],
"sources": [
{
"source_id": "S1",
"document_path": "pages/students/2016/darius_lindvai/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
"title": "Dárius Lindvai"
},
{
"source_id": "S2",
"document_path": "pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial",
"title": "PYTORCH - LSTM TUTORIÁL"
},
{
"source_id": "S3",
"document_path": "pages/students/2016/dominik_nagy/README.md",
"source_url": "https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
"title": "Dominik Nagy"
}
],
"aggregation": null,
"complete": true,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"embedding_model": "intfloat/multilingual-e5-small",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"publication_policy": "explicit-false-hidden-v1",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"version": "33ea0794472c416e97c84371a2a6cdf8"
},
"answer_candidates": [
{
"person": "Dárius Lindvai",
"year": 2019,
"keyword": "pytorch",
"evidence_documents": [
"pages/students/2016/darius_lindvai/README.md",
"pages/students/2016/darius_lindvai/timovy_projekt/tutorial/README.md"
],
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
]
}
]
},
"strict": false,
"attempts": [
{
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n \"answer\": \"\",\n \"status\": \"ok\",\n \"claims\": [\n {\n \"person\": \"Dárius Lindvai\",\n \"year\": 2019,\n \"keyword\": \"pytorch\",\n \"title\": \"PYTORCH - LSTM TUTORIÁL\",\n \"source_urls\": [\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\",\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"\n ]\n }\n ],\n \"source_urls\": [\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\",\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"\n ],\n \"count\": null\n}"
}
}
],
"created": 1792974833,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 223,
"prompt_tokens": 1772,
"total_tokens": 1995,
"prompt_tokens_details": {
"cached_tokens": 815
}
},
"id": "chatcmpl-PTuYHRUKhpKd7EhYb0cDozPNHqVaQYxm",
"timings": {
"cache_n": 815,
"prompt_n": 957,
"prompt_ms": 2525.225,
"prompt_per_token_ms": 2.6386886102403344,
"prompt_per_second": 378.97613083982617,
"predicted_n": 223,
"predicted_ms": 7961.962,
"predicted_per_token_ms": 35.70386547085202,
"predicted_per_second": 28.00817185512817
}
},
"format_error": "ValidationError",
"strict": false
},
{
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n \"answer\": \"\",\n \"status\": \"ok\",\n \"claims\": [\n {\n \"person\": \"Dárius Lindvai\",\n \"title\": \"PYTORCH - LSTM TUTORIÁL\",\n \"year\": 2019,\n \"keyword\": \"pytorch\"\n }\n ],\n \"source_urls\": [\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\",\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"\n ],\n \"count\": null\n}"
}
}
],
"created": 1792974847,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 155,
"prompt_tokens": 2444,
"total_tokens": 2599,
"prompt_tokens_details": {
"cached_tokens": 1768
}
},
"id": "chatcmpl-EqJeUwzc7k5kwW0XpAbdx69edFgRW16H",
"timings": {
"cache_n": 1768,
"prompt_n": 676,
"prompt_ms": 946.839,
"prompt_per_token_ms": 1.4006494082840237,
"prompt_per_second": 713.9545371493991,
"predicted_n": 155,
"predicted_ms": 8280.544,
"predicted_per_token_ms": 53.42286451612903,
"predicted_per_second": 18.71857694373703
}
},
"answer": {
"answer": "",
"status": "ok",
"claims": [
{
"person": "Dárius Lindvai",
"title": "PYTORCH - LSTM TUTORIÁL",
"year": 2019,
"keyword": "pytorch"
}
],
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
]
},
"format_normalized": false,
"context_issues": [
"unsupported_academic_binding"
],
"strict": false
}
],
"accepted_for_delivery": false,
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n \"answer\": \"\",\n \"status\": \"ok\",\n \"claims\": [\n {\n \"person\": \"Dárius Lindvai\",\n \"title\": \"PYTORCH - LSTM TUTORIÁL\",\n \"year\": 2019,\n \"keyword\": \"pytorch\"\n }\n ],\n \"source_urls\": [\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\",\n \"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial\"\n ],\n \"count\": null\n}"
}
}
],
"created": 1792974847,
"model": "Qwen3.5-122B-A10B-Q4_K_M",
"system_fingerprint": "b8580-7c203670f",
"object": "chat.completion",
"usage": {
"completion_tokens": 155,
"prompt_tokens": 2444,
"total_tokens": 2599,
"prompt_tokens_details": {
"cached_tokens": 1768
}
},
"id": "chatcmpl-EqJeUwzc7k5kwW0XpAbdx69edFgRW16H",
"timings": {
"cache_n": 1768,
"prompt_n": 676,
"prompt_ms": 946.839,
"prompt_per_token_ms": 1.4006494082840237,
"prompt_per_second": 713.9545371493991,
"predicted_n": 155,
"predicted_ms": 8280.544,
"predicted_per_token_ms": 53.42286451612903,
"predicted_per_second": 18.71857694373703
}
},
"answer": {
"answer": "",
"status": "ok",
"claims": [
{
"person": "Dárius Lindvai",
"title": "PYTORCH - LSTM TUTORIÁL",
"year": 2019,
"keyword": "pytorch"
}
],
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai/timovy_projekt/tutorial"
]
},
"context_rejected": true,
"bindings_ok": true,
"unsupported_claims": [
{
"person": "Dárius Lindvai",
"title": "PYTORCH - LSTM TUTORIÁL",
"year": 2019,
"keyword": "pytorch"
}
],
"sources_ok": true,
"status_ok": true,
"count_ok": true,
"set_ok": true,
"citations_ok": true,
"unsupported_urls": [],
"context_issues": [
"empty_answer_text",
"unsupported_academic_binding"
],
"text_values_ok": false,
"missing_text_values": [
"Dárius Lindvai",
"pytorch",
"2019"
],
"contradictory_thesis_label": false,
"latency_seconds": 28.01832078601001
}
]
}

499
tools/graphrag/evaluation/results/answers_v7_regression_2026-10-26/frozen_answers.json Normal file
View File

@ -0,0 +1,499 @@
{
"method": "Previously exposed LLM questions; regression, not independent TEST.",
"graph_freeze_sha256": "9e97794200520d0f2288e452c9e186c0291a54c63d2c1382f693a9753e4081cf",
"audit_sha256": "7825d4f3e4e535eb5f49e975dfef363b9b813f1de5c97e2387e5c2f3e578e6a5",
"generation_files": {
"tools/answer_evaluation/evaluate_graph_answers.py": "d81504e8fe51598c6bd1b0d482f61b9b6f51f87ba3f38b03ae1bc584675a3b76",
"tools/answer_evaluation/graph_answer_prompt.txt": "fc9f2f8d52e70a8d67b9f4814b3f15e6be64bf88afe53ce4db4f037bde00e064",
"tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47",
"tools/answer_evaluation/checked_generation.py": "cb821a65327424c19cbd996190b14e6aa30d9b909493f11716552f71249a0529",
"tools/answer_evaluation/render_answer.py": "609c534a37899b3febcea32d983da4b806c287de8f2ddf9bc0675e3280859002",
"tools/rag/evaluation/rag_runner.py": "031abf7d5b1a026eb199aa2a579bfb8baaa01175279d0018a364bfe50b76c3c1"
},
"model": "model120-fast",
"temperature": 0,
"workers": 3,
"dev_ids": [
"v7-dev-year-1899",
"v7-dev-year-1999",
"v7-dev-year-2100",
"v7-dev-year-9999",
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-116",
"wide-119",
"wide-129",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-165",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"test_ids": [
"fresh7-001",
"fresh7-002",
"fresh7-003",
"fresh7-004",
"fresh7-005",
"fresh7-006",
"fresh7-007",
"fresh7-008",
"fresh7-009",
"fresh7-010",
"fresh7-011",
"fresh7-012",
"fresh7-013",
"fresh7-014",
"fresh7-015",
"fresh7-016",
"fresh7-017",
"fresh7-018",
"fresh7-019",
"fresh7-020",
"fresh7-021",
"fresh7-022",
"fresh7-023",
"fresh7-024",
"fresh7-025",
"fresh7-026",
"fresh7-027",
"fresh7-028",
"fresh7-029",
"fresh7-030",
"fresh7-031",
"fresh7-032",
"fresh7-033",
"fresh7-034",
"fresh7-035",
"fresh7-036",
"fresh7-037",
"fresh7-038",
"fresh7-039",
"fresh7-040",
"fresh7-041",
"fresh7-042",
"fresh7-043",
"fresh7-044",
"fresh7-045",
"fresh7-046",
"fresh7-047",
"fresh7-048",
"fresh7-049",
"fresh7-050",
"fresh7-051",
"fresh7-052",
"fresh7-053",
"fresh7-054",
"fresh7-055",
"fresh7-056",
"fresh7-057",
"fresh7-058",
"fresh7-059",
"fresh7-060",
"fresh7-061",
"fresh7-062",
"fresh7-063",
"fresh7-064",
"fresh7-065",
"fresh7-066",
"fresh7-067",
"fresh7-068",
"fresh7-069",
"fresh7-070",
"fresh7-071",
"fresh7-072",
"fresh7-073",
"fresh7-074",
"fresh7-075",
"fresh7-076",
"fresh7-077",
"fresh7-078",
"fresh7-079",
"fresh7-080"
],
"dev_count": 46,
"test_count": 80,
"target_accuracy": 0.92,
"evaluation_kind": "historical_regression",
"rendering": "Verified structured claims rendered deterministically; raw model outputs preserved.",
"selected_split": "regression",
"selected_ids": [
"wide-001",
"wide-002",
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-041",
"wide-042",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-081",
"wide-082",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-113",
"wide-114",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-137",
"wide-138",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-169",
"wide-170",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-193",
"wide-194",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-217",
"wide-218",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-249",
"wide-250",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-289",
"wide-290",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
],
"selected_count": 320,
"selected_dev_overlap": 42,
"corrective_request_limit": 1,
"format_repair_limit": 1,
"repair_policy": "At most one total corrective request for format, citations or retrieval-context violations. Same question/context, no gold; retain all attempts.",
"dev_results_sha256": "4af9b52c250c837707c0690d892ec833347b3d3ee71d338ec631f6450eef8045",
"policy": "Evaluate every declared regression case, including DEV overlap; retain all failures and stop tuning after this freeze.",
"selected_categories": {
"shared_keyword_pair": 40,
"same_academic_year_pair": 40,
"same_start_year_pair": 32,
"person_title_year": 24,
"person_keyword_title": 32,
"document_author_category_multi_hop": 24,
"person_multiple_categories": 24,
"candidate_set_aggregation": 32,
"conjunctive_disambiguation": 40,
"negative_lookup": 32
},
"limitations": [
"All questions and previous model outputs are known; this is historical regression.",
"Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.",
"Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose."
]
}

81
tools/graphrag/evaluation/results/answers_v7_test_2026-10-26/answers_test.csv Normal file
View File

@ -0,0 +1,81 @@
id,category,strict,error,latency_seconds,attempts,first_pass_strict
fresh7-001,shared_keyword_pair,True,,11.841625973989721,1,True
fresh7-002,shared_keyword_pair,True,,8.292787773010787,1,True
fresh7-003,shared_keyword_pair,True,,6.089656800992088,1,True
fresh7-004,shared_keyword_pair,True,,7.950799065991305,1,True
fresh7-005,shared_keyword_pair,True,,9.355126146008843,1,True
fresh7-006,shared_keyword_pair,True,,8.119235402002232,1,True
fresh7-007,shared_keyword_pair,True,,9.431924215008621,1,True
fresh7-008,shared_keyword_pair,True,,8.421721135993721,1,True
fresh7-009,shared_keyword_pair,True,,9.801845731999492,1,True
fresh7-010,shared_keyword_pair,True,,8.407548419010709,1,True
fresh7-011,same_academic_year_pair,True,,9.526154963998124,1,True
fresh7-012,same_academic_year_pair,True,,7.739766710001277,1,True
fresh7-013,same_academic_year_pair,True,,11.054396457999246,1,True
fresh7-014,same_academic_year_pair,True,,5.981948126995121,1,True
fresh7-015,same_academic_year_pair,True,,9.539100390000385,1,True
fresh7-016,same_academic_year_pair,True,,6.883040939006605,1,True
fresh7-017,same_academic_year_pair,True,,9.332280251008342,1,True
fresh7-018,same_academic_year_pair,True,,6.666328727005748,1,True
fresh7-019,same_academic_year_pair,True,,8.958197790008853,1,True
fresh7-020,same_academic_year_pair,True,,7.030446254997514,1,True
fresh7-021,same_start_year_pair,True,,9.740118704998167,1,True
fresh7-022,same_start_year_pair,True,,7.7155549000017345,1,True
fresh7-023,same_start_year_pair,True,,9.746938041003887,1,True
fresh7-024,same_start_year_pair,True,,7.16414850000001,1,True
fresh7-025,same_start_year_pair,True,,9.141140612002346,1,True
fresh7-026,same_start_year_pair,True,,6.95191725899349,1,True
fresh7-027,same_start_year_pair,True,,9.19188570100232,1,True
fresh7-028,same_start_year_pair,True,,7.348483943002066,1,True
fresh7-029,same_start_year_pair,True,,8.995747175998986,1,True
fresh7-030,same_start_year_pair,True,,6.975118151007337,1,True
fresh7-031,person_title_year,True,,7.311062619992299,1,True
fresh7-032,person_title_year,True,,5.878452272008872,1,True
fresh7-033,person_title_year,True,,5.801723632990615,1,True
fresh7-034,person_title_year,True,,5.779961151012685,1,True
fresh7-035,person_title_year,True,,5.503505647997372,1,True
fresh7-036,person_title_year,True,,5.739178684991202,1,True
fresh7-037,person_title_year,True,,5.415736306007602,1,True
fresh7-038,person_title_year,True,,7.437335399008589,1,True
fresh7-039,person_title_year,True,,4.408995279998635,1,True
fresh7-040,person_title_year,True,,6.882196734994068,1,True
fresh7-041,person_keyword_title,True,,5.83978758000012,1,True
fresh7-042,person_keyword_title,True,,8.581415363005362,1,True
fresh7-043,person_keyword_title,True,,7.124095003004186,1,True
fresh7-044,person_keyword_title,True,,9.642784591007512,1,True
fresh7-045,person_keyword_title,True,,7.515784107992658,1,True
fresh7-046,person_keyword_title,True,,10.188013713996042,1,True
fresh7-047,person_keyword_title,True,,8.431851733999792,1,True
fresh7-048,person_keyword_title,True,,9.907063878999907,1,True
fresh7-049,person_keyword_title,True,,8.099193093003123,1,True
fresh7-050,person_keyword_title,True,,8.852308599001844,1,True
fresh7-051,document_author_category_multi_hop,True,,7.668113538005855,1,True
fresh7-052,document_author_category_multi_hop,True,,8.831390542007284,1,True
fresh7-053,person_multiple_categories,True,,4.3355173820018535,1,True
fresh7-054,person_multiple_categories,True,,14.959355758997845,2,False
fresh7-055,person_multiple_categories,True,,4.788777031993959,1,True
fresh7-056,person_multiple_categories,True,,6.780442283998127,1,True
fresh7-057,person_multiple_categories,True,,5.894032968004467,1,True
fresh7-058,person_multiple_categories,True,,16.16342897800496,2,False
fresh7-059,person_multiple_categories,True,,8.475595613010228,2,False
fresh7-060,candidate_set_aggregation,True,,9.091813248000108,1,True
fresh7-061,candidate_set_aggregation,True,,8.758890958008124,1,True
fresh7-062,candidate_set_aggregation,True,,8.99064658000134,1,True
fresh7-063,candidate_set_aggregation,True,,9.600775625003735,1,True
fresh7-064,candidate_set_aggregation,True,,8.849686492001638,1,True
fresh7-065,candidate_set_aggregation,True,,9.296292756989715,1,True
fresh7-066,candidate_set_aggregation,True,,8.94247531799192,1,True
fresh7-067,conjunctive_disambiguation,True,,8.469535019001341,1,True
fresh7-068,conjunctive_disambiguation,True,,8.207272173007368,1,True
fresh7-069,conjunctive_disambiguation,True,,7.925748694004142,1,True
fresh7-070,conjunctive_disambiguation,True,,8.149811054012389,1,True
fresh7-071,conjunctive_disambiguation,True,,8.003814060997684,1,True
fresh7-072,conjunctive_disambiguation,True,,8.130394503008574,1,True
fresh7-073,negative_lookup,True,,4.625077007003711,1,True
fresh7-074,negative_lookup,True,,4.179091968006105,1,True
fresh7-075,negative_lookup,True,,3.3433215739933075,1,True
fresh7-076,negative_lookup,True,,3.1084746859996812,1,True
fresh7-077,negative_lookup,True,,2.6855998700048076,1,True
fresh7-078,negative_lookup,True,,2.4824905980058247,1,True
fresh7-079,negative_lookup,True,,2.805210263992194,1,True
fresh7-080,negative_lookup,True,,1.868269345999579,1,True
1 id category strict error latency_seconds attempts first_pass_strict
2 fresh7-001 shared_keyword_pair True 11.841625973989721 1 True
3 fresh7-002 shared_keyword_pair True 8.292787773010787 1 True
4 fresh7-003 shared_keyword_pair True 6.089656800992088 1 True
5 fresh7-004 shared_keyword_pair True 7.950799065991305 1 True
6 fresh7-005 shared_keyword_pair True 9.355126146008843 1 True
7 fresh7-006 shared_keyword_pair True 8.119235402002232 1 True
8 fresh7-007 shared_keyword_pair True 9.431924215008621 1 True
9 fresh7-008 shared_keyword_pair True 8.421721135993721 1 True
10 fresh7-009 shared_keyword_pair True 9.801845731999492 1 True
11 fresh7-010 shared_keyword_pair True 8.407548419010709 1 True
12 fresh7-011 same_academic_year_pair True 9.526154963998124 1 True
13 fresh7-012 same_academic_year_pair True 7.739766710001277 1 True
14 fresh7-013 same_academic_year_pair True 11.054396457999246 1 True
15 fresh7-014 same_academic_year_pair True 5.981948126995121 1 True
16 fresh7-015 same_academic_year_pair True 9.539100390000385 1 True
17 fresh7-016 same_academic_year_pair True 6.883040939006605 1 True
18 fresh7-017 same_academic_year_pair True 9.332280251008342 1 True
19 fresh7-018 same_academic_year_pair True 6.666328727005748 1 True
20 fresh7-019 same_academic_year_pair True 8.958197790008853 1 True
21 fresh7-020 same_academic_year_pair True 7.030446254997514 1 True
22 fresh7-021 same_start_year_pair True 9.740118704998167 1 True
23 fresh7-022 same_start_year_pair True 7.7155549000017345 1 True
24 fresh7-023 same_start_year_pair True 9.746938041003887 1 True
25 fresh7-024 same_start_year_pair True 7.16414850000001 1 True
26 fresh7-025 same_start_year_pair True 9.141140612002346 1 True
27 fresh7-026 same_start_year_pair True 6.95191725899349 1 True
28 fresh7-027 same_start_year_pair True 9.19188570100232 1 True
29 fresh7-028 same_start_year_pair True 7.348483943002066 1 True
30 fresh7-029 same_start_year_pair True 8.995747175998986 1 True
31 fresh7-030 same_start_year_pair True 6.975118151007337 1 True
32 fresh7-031 person_title_year True 7.311062619992299 1 True
33 fresh7-032 person_title_year True 5.878452272008872 1 True
34 fresh7-033 person_title_year True 5.801723632990615 1 True
35 fresh7-034 person_title_year True 5.779961151012685 1 True
36 fresh7-035 person_title_year True 5.503505647997372 1 True
37 fresh7-036 person_title_year True 5.739178684991202 1 True
38 fresh7-037 person_title_year True 5.415736306007602 1 True
39 fresh7-038 person_title_year True 7.437335399008589 1 True
40 fresh7-039 person_title_year True 4.408995279998635 1 True
41 fresh7-040 person_title_year True 6.882196734994068 1 True
42 fresh7-041 person_keyword_title True 5.83978758000012 1 True
43 fresh7-042 person_keyword_title True 8.581415363005362 1 True
44 fresh7-043 person_keyword_title True 7.124095003004186 1 True
45 fresh7-044 person_keyword_title True 9.642784591007512 1 True
46 fresh7-045 person_keyword_title True 7.515784107992658 1 True
47 fresh7-046 person_keyword_title True 10.188013713996042 1 True
48 fresh7-047 person_keyword_title True 8.431851733999792 1 True
49 fresh7-048 person_keyword_title True 9.907063878999907 1 True
50 fresh7-049 person_keyword_title True 8.099193093003123 1 True
51 fresh7-050 person_keyword_title True 8.852308599001844 1 True
52 fresh7-051 document_author_category_multi_hop True 7.668113538005855 1 True
53 fresh7-052 document_author_category_multi_hop True 8.831390542007284 1 True
54 fresh7-053 person_multiple_categories True 4.3355173820018535 1 True
55 fresh7-054 person_multiple_categories True 14.959355758997845 2 False
56 fresh7-055 person_multiple_categories True 4.788777031993959 1 True
57 fresh7-056 person_multiple_categories True 6.780442283998127 1 True
58 fresh7-057 person_multiple_categories True 5.894032968004467 1 True
59 fresh7-058 person_multiple_categories True 16.16342897800496 2 False
60 fresh7-059 person_multiple_categories True 8.475595613010228 2 False
61 fresh7-060 candidate_set_aggregation True 9.091813248000108 1 True
62 fresh7-061 candidate_set_aggregation True 8.758890958008124 1 True
63 fresh7-062 candidate_set_aggregation True 8.99064658000134 1 True
64 fresh7-063 candidate_set_aggregation True 9.600775625003735 1 True
65 fresh7-064 candidate_set_aggregation True 8.849686492001638 1 True
66 fresh7-065 candidate_set_aggregation True 9.296292756989715 1 True
67 fresh7-066 candidate_set_aggregation True 8.94247531799192 1 True
68 fresh7-067 conjunctive_disambiguation True 8.469535019001341 1 True
69 fresh7-068 conjunctive_disambiguation True 8.207272173007368 1 True
70 fresh7-069 conjunctive_disambiguation True 7.925748694004142 1 True
71 fresh7-070 conjunctive_disambiguation True 8.149811054012389 1 True
72 fresh7-071 conjunctive_disambiguation True 8.003814060997684 1 True
73 fresh7-072 conjunctive_disambiguation True 8.130394503008574 1 True
74 fresh7-073 negative_lookup True 4.625077007003711 1 True
75 fresh7-074 negative_lookup True 4.179091968006105 1 True
76 fresh7-075 negative_lookup True 3.3433215739933075 1 True
77 fresh7-076 negative_lookup True 3.1084746859996812 1 True
78 fresh7-077 negative_lookup True 2.6855998700048076 1 True
79 fresh7-078 negative_lookup True 2.4824905980058247 1 True
80 fresh7-079 negative_lookup True 2.805210263992194 1 True
81 fresh7-080 negative_lookup True 1.868269345999579 1 True

17309
tools/graphrag/evaluation/results/answers_v7_test_2026-10-26/answers_test.json Normal file
View File

File diff suppressed because it is too large Load Diff

259
tools/graphrag/evaluation/results/answers_v7_test_2026-10-26/frozen_answers.json Normal file
View File

@ -0,0 +1,259 @@
{
"method": "Fresh query-group holdout over known sources and templates; zero DEV query-group overlap.",
"graph_freeze_sha256": "9e97794200520d0f2288e452c9e186c0291a54c63d2c1382f693a9753e4081cf",
"audit_sha256": "7825d4f3e4e535eb5f49e975dfef363b9b813f1de5c97e2387e5c2f3e578e6a5",
"generation_files": {
"tools/answer_evaluation/evaluate_graph_answers.py": "d81504e8fe51598c6bd1b0d482f61b9b6f51f87ba3f38b03ae1bc584675a3b76",
"tools/answer_evaluation/graph_answer_prompt.txt": "fc9f2f8d52e70a8d67b9f4814b3f15e6be64bf88afe53ce4db4f037bde00e064",
"tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47",
"tools/answer_evaluation/checked_generation.py": "cb821a65327424c19cbd996190b14e6aa30d9b909493f11716552f71249a0529",
"tools/answer_evaluation/render_answer.py": "609c534a37899b3febcea32d983da4b806c287de8f2ddf9bc0675e3280859002",
"tools/rag/evaluation/rag_runner.py": "031abf7d5b1a026eb199aa2a579bfb8baaa01175279d0018a364bfe50b76c3c1"
},
"model": "model120-fast",
"temperature": 0,
"workers": 3,
"dev_ids": [
"v7-dev-year-1899",
"v7-dev-year-1999",
"v7-dev-year-2100",
"v7-dev-year-9999",
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-116",
"wide-119",
"wide-129",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-165",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
],
"test_ids": [
"fresh7-001",
"fresh7-002",
"fresh7-003",
"fresh7-004",
"fresh7-005",
"fresh7-006",
"fresh7-007",
"fresh7-008",
"fresh7-009",
"fresh7-010",
"fresh7-011",
"fresh7-012",
"fresh7-013",
"fresh7-014",
"fresh7-015",
"fresh7-016",
"fresh7-017",
"fresh7-018",
"fresh7-019",
"fresh7-020",
"fresh7-021",
"fresh7-022",
"fresh7-023",
"fresh7-024",
"fresh7-025",
"fresh7-026",
"fresh7-027",
"fresh7-028",
"fresh7-029",
"fresh7-030",
"fresh7-031",
"fresh7-032",
"fresh7-033",
"fresh7-034",
"fresh7-035",
"fresh7-036",
"fresh7-037",
"fresh7-038",
"fresh7-039",
"fresh7-040",
"fresh7-041",
"fresh7-042",
"fresh7-043",
"fresh7-044",
"fresh7-045",
"fresh7-046",
"fresh7-047",
"fresh7-048",
"fresh7-049",
"fresh7-050",
"fresh7-051",
"fresh7-052",
"fresh7-053",
"fresh7-054",
"fresh7-055",
"fresh7-056",
"fresh7-057",
"fresh7-058",
"fresh7-059",
"fresh7-060",
"fresh7-061",
"fresh7-062",
"fresh7-063",
"fresh7-064",
"fresh7-065",
"fresh7-066",
"fresh7-067",
"fresh7-068",
"fresh7-069",
"fresh7-070",
"fresh7-071",
"fresh7-072",
"fresh7-073",
"fresh7-074",
"fresh7-075",
"fresh7-076",
"fresh7-077",
"fresh7-078",
"fresh7-079",
"fresh7-080"
],
"dev_count": 46,
"test_count": 80,
"target_accuracy": 0.92,
"evaluation_kind": "independent_query_group_holdout",
"rendering": "Verified structured claims rendered deterministically; raw model outputs preserved.",
"selected_split": "test",
"selected_ids": [
"fresh7-001",
"fresh7-002",
"fresh7-003",
"fresh7-004",
"fresh7-005",
"fresh7-006",
"fresh7-007",
"fresh7-008",
"fresh7-009",
"fresh7-010",
"fresh7-011",
"fresh7-012",
"fresh7-013",
"fresh7-014",
"fresh7-015",
"fresh7-016",
"fresh7-017",
"fresh7-018",
"fresh7-019",
"fresh7-020",
"fresh7-021",
"fresh7-022",
"fresh7-023",
"fresh7-024",
"fresh7-025",
"fresh7-026",
"fresh7-027",
"fresh7-028",
"fresh7-029",
"fresh7-030",
"fresh7-031",
"fresh7-032",
"fresh7-033",
"fresh7-034",
"fresh7-035",
"fresh7-036",
"fresh7-037",
"fresh7-038",
"fresh7-039",
"fresh7-040",
"fresh7-041",
"fresh7-042",
"fresh7-043",
"fresh7-044",
"fresh7-045",
"fresh7-046",
"fresh7-047",
"fresh7-048",
"fresh7-049",
"fresh7-050",
"fresh7-051",
"fresh7-052",
"fresh7-053",
"fresh7-054",
"fresh7-055",
"fresh7-056",
"fresh7-057",
"fresh7-058",
"fresh7-059",
"fresh7-060",
"fresh7-061",
"fresh7-062",
"fresh7-063",
"fresh7-064",
"fresh7-065",
"fresh7-066",
"fresh7-067",
"fresh7-068",
"fresh7-069",
"fresh7-070",
"fresh7-071",
"fresh7-072",
"fresh7-073",
"fresh7-074",
"fresh7-075",
"fresh7-076",
"fresh7-077",
"fresh7-078",
"fresh7-079",
"fresh7-080"
],
"selected_count": 80,
"selected_dev_overlap": 0,
"corrective_request_limit": 1,
"format_repair_limit": 1,
"repair_policy": "At most one total corrective request for format, citations or retrieval-context violations. Same question/context, no gold; retain all attempts.",
"dev_results_sha256": "4af9b52c250c837707c0690d892ec833347b3d3ee71d338ec631f6450eef8045",
"policy": "Evaluate every declared fresh TEST case once; retain every failure; do not tune on this TEST.",
"selected_categories": {
"shared_keyword_pair": 10,
"same_academic_year_pair": 10,
"same_start_year_pair": 10,
"person_title_year": 10,
"person_keyword_title": 10,
"document_author_category_multi_hop": 2,
"person_multiple_categories": 7,
"candidate_set_aggregation": 7,
"conjunctive_disambiguation": 6,
"negative_lookup": 8
},
"limitations": [
"Known corpus/profiles and templates; these query groups were not previously evaluated.",
"Harness supplies graph context; native OpenWebUI tool choice and browser UI are outside this experiment.",
"Checks structured factual claims, their source support/citations and required prose values; not a general semantic judge of arbitrary prose."
]
}

147
tools/graphrag/evaluation/results/final_v7_2026-10-26/REPORT.md Normal file
View File

@ -0,0 +1,147 @@
# GraphRAG v7 — uzavretie bodu 3, 26. 10. 2026
Nový query-group TEST dosiahol **80/80** pri grafovom vyhľadávaní aj pri
finálnych odpovediach. Na prvý pokus modelu prešlo **77/80 — 96,25 %**;
tri otázky prešli po jednom povolenom opravnom požiadavku. Cieľ ≥92 % je splnený
už pri prvom pokuse aj pri finálnej pipeline. Celá sada má **591 úspešných
testov, bez preskočenia**.
**Bod 3 je technicky uzavretý pre podporované typy dotazov a tento korpus.**
Hodnotí sa modelový výber štruktúrovaných tvrdení, kontrola zdrojov a
renderer; nie ľubovoľná voľná próza LLM ani natívne OpenWebUI smerovanie.
100 % na 80 otázkach nie je zárukou všeobecnej pravdivosti.
## Výsledky
| Meranie | Správne | Úspešnosť |
|---|---:|---:|
| Graf DEV vrátane známych regresií | 1 046/1 046 | 100,00 % |
| Graf nový query-group TEST | 80/80 | 100,00 % |
| Graf historická podmnožina v DEV | 320/320 | 100,00 % |
| Model + renderer DEV, finálne | 46/46 | 100,00 % |
| Nový TEST, prvý pokus modelu + renderer | 77/80 | 96,25 % |
| Nový TEST, po jednej povolenej oprave | 80/80 | 100,00 % |
| Historická regresia, prvý pokus | 310/320 | 96.88 % |
| Historická regresia, finálne | 319/320 | 99.69 % |
| Živé syntetické funkčné scenáre mimo benchmarku | 9/9 | 100,00 % |
Modelový DEV mal prvý pokus 45/46 a jednu úspešnú opravu formátu.
V novom TESTe potreboval `fresh7-054` opravu schémy a `fresh7-058`,
`fresh7-059` odstránenie nepodloženého počtu pri neagregačnej otázke.
Opravu vyvolal iba formát alebo skutočný kontext; gold ani následné
skóre sa pri rozhodovaní o opakovaní nepoužívajú.
Finálna historická regresia má **0 chýb spracovania**,
**10 opravných požiadavkov**
(z toho formát 3, kontext 7).
Všetky odpovede, pôvodné pokusy a prípadné zlyhania ostávajú v menovateli.
Zostávajúci prípad `wide-285`: po chybe schémy model vložil názov dcérskeho
dokumentu PYTORCH - LSTM TUTORIÁL ako akademický názov k roku 2019.
Taká väzba nie je v akademických faktoch doložená; guard ju odmietol
(`unsupported_academic_binding`, `accepted_for_delivery=false`).
Odpoveď sa nedoručila. Povolený opravný pokus už bol vyčerpaný.
[Úplný záznam](../answers_v7_regression_2026-10-26/failures.json).
| Kategória nového TESTu | Finálne správne | Úspešnosť |
|---|---:|---:|
| `candidate_set_aggregation` | 7/7 | 100.00 % |
| `conjunctive_disambiguation` | 6/6 | 100.00 % |
| `document_author_category_multi_hop` | 2/2 | 100.00 % |
| `negative_lookup` | 8/8 | 100.00 % |
| `person_keyword_title` | 10/10 | 100.00 % |
| `person_multiple_categories` | 7/7 | 100.00 % |
| `person_title_year` | 10/10 | 100.00 % |
| `same_academic_year_pair` | 10/10 | 100.00 % |
| `same_start_year_pair` | 10/10 | 100.00 % |
| `shared_keyword_pair` | 10/10 | 100.00 % |
## Uzavreté podbody
| Podbod | Konkrétny výsledok |
|---|---|
| 3.1 Benchmark a zdroje | 1 126 zdrojovo skontrolovaných otázok; 10 opravených očakávaní po odstránení odhadu štúdia z adresára. Nová vzorka 20 otázok priamo skontrolovaná z Markdownov. |
| 3.2 Evaluator a doručenie | Rozpory v úplnosti, nepodložené počty/roky, chýbajúce claims/citácie a HTML/referenčné odkazy majú regresné testy. Finálny text vzniká iba z overených tvrdení. |
| 3.3 DEV | Kompletný grafový DEV 1 046/1 046, modelový DEV 46/46; staré prípady `wide-104`, `wide-119`, `wide-143`, `wide-176` prešli na prvý pokus aj v novej regresii. |
| 3.4 Zmrazenie | `frozen_v7.json`, deklarácie generovania a `frozen_release_v7.json`; explicitné CLI vstupy, odtlačky rendereru aj spoločných modulov. |
| 3.5 Finálne overenie | Nový TEST 80 skupín bez DEV prekryvu, jednorazovo po zmrazení; samostatná historická regresia 320 otázok a kontrola Dockeru. |
| 3.6 Metriky a správa | Oddelené prvotné a finálne odpovede, vyhľadávanie a modelová pipeline, nový TEST a regresia. V1–v6 ostávajú zachované. |
## Opravené chyby
- Rok 1999 sa už nevynechá: dotaz na Jána Holpa nevracia práce z 2019/2021.
Aj ostatné štvormiestne roky sa aplikujú ako obmedzenie. Nepodporovaný
číselný zápis po „roku“ sa explicitne odmieta.
- Veta „Výpis je neúplný“ neospravedlňuje iné kladné tvrdenie o úplnom zozname.
Pravdivé popretie úplnosti je povolené.
- Prirodzený text sa skladá deterministicky z validovaných claims/count a URL.
Pri regrade neočakávanej voľnej prózy dopísaný rok 2099 neprejde.
Zostávajúci číselný filter nie je univerzálny sémantický analyzátor;
používateľovi sa preto nedoručuje nekontrolovaný voľný text modelu.
- Počet bez úplnej agregácie sa odmietne. Pozitívne faktové odpovede vyžadujú
tvrdenia a citácie, vrátane zdrojov jednotlivých atribútov. Count=0 v úplnej
prázdnej agregácii je povolený.
- Kontrolujú sa aj referenčné Markdown a HTML odkazy, relatívne URL, entity
a korektné URL so zátvorkami. Nepovolené odkazy sa neobchádzajú upozornením.
- Začiatok štúdia je iba explicitne uvedený údaj. Graf eviduje pôvod a istotu,
chýbajúca hodnota ostáva neznáma; `wide-112` už nevyžaduje odhadovaný rok
Tomáša Kuchárika. Cohort nie je administratívny fakt o nástupe na štúdium.
- Skloňované meno sa v texte rozpoznáva obmedzenými príponami pri zachovaní
presnej štruktúrovanej identity. Canonical renderer používa pôvodné meno.
- Dataset, audit, manifest a DEV ID sa pri evaluácii vyberajú explicitne;
spúšťanie už potichu nevyberá v4.
## Nezávislosť a pôvod očakávaní
Všetkých 1 042 v6 otázok je odhalených a v7 ich označuje ako DEV. Štyri
nové auditové rokové regresie dopĺňajú počet na 1 046. Historických 320
sa vyhodnocuje z tejto množiny samostatne. Je to **regresia**, nie nový TEST.
Nový TEST má 80 kombinácií, 72 kladných a 8 záporných. Seed 20261026,
kvóty a ID vznikli z raw Markdownov pred čítaním predikcií. Normalizované
query skupiny ani formulácie sa s DEV neprekrývajú. Po zmrazení sa podľa
TESTu nemenil kód, prompt, gold ani výber otázok. Korpus, osoby a rodiny
šablón sú známe; nejde o unseen-profile experiment.
Priama AI kontrola 20 otázok pokrýva všetkých desať kategórií.
[Zdrojový hárok](../v7_preparation/source_review.csv) a
[výňatky](../v7_preparation/source_review_packet.json) sú uložené.
**Ľudská anotácia neprebehla.** Neistá adresárová konvencia bola odstránená,
nie potvrdená bez dôkazu. Autor–kategória má iba dve nové skupiny a výpis
kategórií sedem; nemožno tvrdiť rovnakú štatistickú silu každej kategórie.
## Regresia a zachované referencie
V6 mala strict 316/320. V7 mení gold pri 10 známych otázkach (sedem z
historických 320) a používa štruktúrovaný modelový výstup s rendererom.
Rozdiel percent nie je izolovaný odhad zlepšenia samotného LLM.
Všetky pôvodné baseline, otázky, finálne JSON/CSV a v1–v6 výsledky sa zachovávajú.
Prepočet citácií 56 archivovaných súborov má 6 044 odpovedí a päť výskytov
URL mimo podkladov, rovnako ako v6. Ide o prekrývajúce sa experimenty.
Klasický RAG sa novým modelovým TESTom nemeral. Prehratie piatich uložených
RAG odpovedí cez kontrolu upozornenia opäť prešlo 5/5; to nie je nová
presnosť modelu. Je to regresia prezentačnej vrstvy.
## Nasadenie a reprodukcia
Súbory, SQLite a Neo4j sú po koordinovanom `/sync` na spoločnej generácii
`33ea0794472c416e97c84371a2a6cdf8`; wiki Git je
`3e892a1d49e9bce118cd03478ce15e01c2bebb4e`. Index má 114 dokumentov a 887
chunkov, graf 446 uzlov a 877 vzťahov. Explicitné published=false sa skryje,
16 chýbajúcich hodnôt sa zahrnie; viditeľných je 113 dokumentov.
[Hostiteľské overenie](deployment_check_host.json) a
[overenie Dockeru](deployment_check_docker.json) potvrdzujú dostupnosť,
spoločnú generáciu, normalizované publikovanie a runtime odtlačky.
Modelové výstupy obsahujú celý skutočný grafový kontext a verziu dát.
[Príkazy a zmeny](../../../../../docs/graphrag_v7_2026-10-26_worklog.md).
Reprodukcia používa nový výstupný adresár a príslušný release checkout.
Git HEAD v metadátach merania je východiskový commit pred finálnym commitom;
konkrétny meraný kód určujú obsahové SHA. Model `model120-fast` je vzdialená
služba; parametre a odpovede sú uložené, váhy prevádzkuje poskytovateľ.
Ďalším blokom sú natívne nástroje a hlavný agent v OpenWebUI. Musia rešpektovať
odmietnutie doručenia a odovzdať overený text; voľné prepísanie rendererového
textu bez kontrol by uvedené vlastnosti obišlo. Toto smerovanie sa v bode 3
nemeralo a je súčasťou bodu 4.

130
tools/graphrag/evaluation/results/final_v7_2026-10-26/deployment_check_docker.json Normal file
View File

@ -0,0 +1,130 @@
{
"checked_at_utc": "2026-10-26T00:38:33.284887+00:00",
"health": {
"status": "ok",
"ready": true,
"database_exists": true,
"search_engine": "hybrid_fts5_embeddings",
"rag_enabled": true,
"zpwiki_exists": true,
"security_configured": true,
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
},
"tools": {
"rag": {
"ready": true,
"status": "ready",
"reason": null,
"embedding_model": "intfloat/multilingual-e5-small",
"dimensions": 384,
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
},
"neo4j": {
"ready": true,
"status": "ready",
"reason": null
},
"graphrag": {
"ready": true,
"status": "ready",
"reason": null,
"index_version": {
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"embedding_model": "intfloat/multilingual-e5-small",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"publication_policy": "explicit-false-hidden-v1",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"version": "33ea0794472c416e97c84371a2a6cdf8"
},
"nodes": 446
}
}
},
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
},
"frozen_implementation_inputs_and_graph_match": true,
"graph": {
"sha256": "566817ce7ddebb2638585192c148c41421071bdf719674fc6ddddb1c99a0e36f",
"nodes": 446,
"relationships": 877
},
"graph_cases": {
"known_title": {
"status": "ok",
"reason": null,
"complete": true
},
"generic_label_not_title": {
"status": "no_answer",
"reason": "no_matching_facts",
"complete": true
},
"unknown_person": {
"status": "no_answer",
"reason": "unknown_person",
"complete": true
},
"outside_century": {
"status": "no_answer",
"reason": "no_matching_facts",
"complete": true
},
"unsupported_year": {
"status": "no_answer",
"reason": "unsupported_year_constraint",
"complete": true
},
"unknown_start": {
"status": "no_answer",
"reason": "missing_documented_start_year",
"complete": true
}
},
"rag_live_query": "ok",
"release_code_match": true,
"rag_context_metadata_matches_contract": true,
"publication": {
"documents": 114,
"explicit_false": 1,
"missing_included": 16,
"visible_by_policy": 113,
"hidden_document_absent_from_rag": true
},
"runtime": {
"python": "3.11.17",
"fastapi": "0.136.3",
"pydantic": "2.13.4",
"neo4j": "6.3.1",
"sentence-transformers": "5.7.0",
"torch": "2.11.0+cpu"
},
"scope": "RAG HTTP API plus GraphRAG library/CLI; native OpenWebUI graph-tool routing is point 4.",
"docker_image_id": "sha256:10fd5e24b91cf2907a2b712ac8c2af796a26f5023daaf3834f3ca505c62f36f6"
}

129
tools/graphrag/evaluation/results/final_v7_2026-10-26/deployment_check_host.json Normal file
View File

@ -0,0 +1,129 @@
{
"checked_at_utc": "2026-10-26T00:38:25.612948+00:00",
"health": {
"status": "ok",
"ready": true,
"database_exists": true,
"search_engine": "hybrid_fts5_embeddings",
"rag_enabled": true,
"zpwiki_exists": true,
"security_configured": true,
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
},
"tools": {
"rag": {
"ready": true,
"status": "ready",
"reason": null,
"embedding_model": "intfloat/multilingual-e5-small",
"dimensions": 384,
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
},
"neo4j": {
"ready": true,
"status": "ready",
"reason": null
},
"graphrag": {
"ready": true,
"status": "ready",
"reason": null,
"index_version": {
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"embedding_model": "intfloat/multilingual-e5-small",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"publication_policy": "explicit-false-hidden-v1",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"version": "33ea0794472c416e97c84371a2a6cdf8"
},
"nodes": 446
}
}
},
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
},
"frozen_implementation_inputs_and_graph_match": true,
"graph": {
"sha256": "566817ce7ddebb2638585192c148c41421071bdf719674fc6ddddb1c99a0e36f",
"nodes": 446,
"relationships": 877
},
"graph_cases": {
"known_title": {
"status": "ok",
"reason": null,
"complete": true
},
"generic_label_not_title": {
"status": "no_answer",
"reason": "no_matching_facts",
"complete": true
},
"unknown_person": {
"status": "no_answer",
"reason": "unknown_person",
"complete": true
},
"outside_century": {
"status": "no_answer",
"reason": "no_matching_facts",
"complete": true
},
"unsupported_year": {
"status": "no_answer",
"reason": "unsupported_year_constraint",
"complete": true
},
"unknown_start": {
"status": "no_answer",
"reason": "missing_documented_start_year",
"complete": true
}
},
"rag_live_query": "ok",
"release_code_match": true,
"rag_context_metadata_matches_contract": true,
"publication": {
"documents": 114,
"explicit_false": 1,
"missing_included": 16,
"visible_by_policy": 113,
"hidden_document_absent_from_rag": true
},
"runtime": {
"python": "3.10.12",
"fastapi": "0.136.3",
"pydantic": "2.13.4",
"neo4j": "6.3.1",
"sentence-transformers": "5.7.0",
"torch": "2.11.0+cpu"
},
"scope": "RAG HTTP API plus GraphRAG library/CLI; native OpenWebUI graph-tool routing is point 4."
}

1047
tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_dev.csv Normal file
View File

File diff suppressed because one or more lines are too long

197906
tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_dev.json Normal file
View File

File diff suppressed because it is too large Load Diff

81
tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_test.csv Normal file
View File

File diff suppressed because one or more lines are too long

16132
tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_test.json Normal file
View File

File diff suppressed because it is too large Load Diff

348
tools/graphrag/evaluation/results/final_v7_2026-10-26/historical_graph_summary.json Normal file
View File

@ -0,0 +1,348 @@
{
"method": "Historical 320-case subset of the complete 1046-case DEV; no duplicated retrieval run.",
"source": "tools/graphrag/evaluation/results/final_v7_2026-10-26/graphrag_dev.json",
"source_sha256": "d1a5d7b6d5aff3c0002f36a60cebc07627a79ec6d57de119dd587e8ab70143b0",
"ids": [
"wide-001",
"wide-002",
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-041",
"wide-042",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-081",
"wide-082",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-113",
"wide-114",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-137",
"wide-138",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-169",
"wide-170",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-193",
"wide-194",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-217",
"wide-218",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-249",
"wide-250",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-289",
"wide-290",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
],
"summary": {
"questions": 320,
"source_checked_count": 320,
"all_facts_supported_count": 320,
"strict_count": 320,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.081877,
"latency_median_seconds": 0.078039,
"latency_p95_seconds": 0.119541
}
}

270
tools/graphrag/evaluation/results/final_v7_2026-10-26/summary.json Normal file
View File

@ -0,0 +1,270 @@
{
"release": "graphrag-v7-frozen",
"graph_dev": {
"questions": 1046,
"source_checked_count": 1046,
"all_facts_supported_count": 1046,
"strict_count": 1046,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.081848,
"latency_median_seconds": 0.071172,
"latency_p95_seconds": 0.113998
},
"graph_new_test": {
"questions": 80,
"source_checked_count": 80,
"all_facts_supported_count": 80,
"strict_count": 80,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.093012,
"latency_median_seconds": 0.091794,
"latency_p95_seconds": 0.127876
},
"graph_historical_regression": {
"questions": 320,
"source_checked_count": 320,
"all_facts_supported_count": 320,
"strict_count": 320,
"strict_accuracy": 1.0,
"error_count": 0,
"value_score": 1.0,
"entity_score": 1.0,
"relation_score": 1.0,
"property_score": 1.0,
"relation_choice_score": 1.0,
"document_recall": 1.0,
"source_recall": 1.0,
"binding_score": 1.0,
"path_score": 1.0,
"answerability_accuracy": 1.0,
"latency_mean_seconds": 0.081877,
"latency_median_seconds": 0.078039,
"latency_p95_seconds": 0.119541
},
"model_dev": {
"questions": 46,
"correct": 46,
"accuracy": 1.0,
"errors": 0,
"first_pass_correct": 45,
"format_repair_attempted": 1,
"corrective_request_count": 1,
"context_repair_attempted": 0,
"normalized_first_response": 0
},
"model_new_test": {
"questions": 80,
"correct": 80,
"accuracy": 1.0,
"errors": 0,
"first_pass_correct": 77,
"format_repair_attempted": 1,
"corrective_request_count": 3,
"context_repair_attempted": 2,
"normalized_first_response": 0
},
"model_historical_regression": {
"questions": 320,
"correct": 319,
"accuracy": 0.996875,
"errors": 0,
"first_pass_correct": 310,
"format_repair_attempted": 3,
"corrective_request_count": 10,
"context_repair_attempted": 7,
"normalized_first_response": 0
},
"model_new_test_by_category": {
"candidate_set_aggregation": {
"questions": 7,
"correct": 7,
"accuracy": 1.0
},
"conjunctive_disambiguation": {
"questions": 6,
"correct": 6,
"accuracy": 1.0
},
"document_author_category_multi_hop": {
"questions": 2,
"correct": 2,
"accuracy": 1.0
},
"negative_lookup": {
"questions": 8,
"correct": 8,
"accuracy": 1.0
},
"person_keyword_title": {
"questions": 10,
"correct": 10,
"accuracy": 1.0
},
"person_multiple_categories": {
"questions": 7,
"correct": 7,
"accuracy": 1.0
},
"person_title_year": {
"questions": 10,
"correct": 10,
"accuracy": 1.0
},
"same_academic_year_pair": {
"questions": 10,
"correct": 10,
"accuracy": 1.0
},
"same_start_year_pair": {
"questions": 10,
"correct": 10,
"accuracy": 1.0
},
"shared_keyword_pair": {
"questions": 10,
"correct": 10,
"accuracy": 1.0
}
},
"model_regression_by_category": {
"candidate_set_aggregation": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"conjunctive_disambiguation": {
"questions": 40,
"correct": 39,
"accuracy": 0.975
},
"document_author_category_multi_hop": {
"questions": 24,
"correct": 24,
"accuracy": 1.0
},
"negative_lookup": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"person_keyword_title": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"person_multiple_categories": {
"questions": 24,
"correct": 24,
"accuracy": 1.0
},
"person_title_year": {
"questions": 24,
"correct": 24,
"accuracy": 1.0
},
"same_academic_year_pair": {
"questions": 40,
"correct": 40,
"accuracy": 1.0
},
"same_start_year_pair": {
"questions": 32,
"correct": 32,
"accuracy": 1.0
},
"shared_keyword_pair": {
"questions": 40,
"correct": 40,
"accuracy": 1.0
}
},
"protocol": {
"seed": 20261026,
"dev_count": 1046,
"fresh_test_count": 80,
"test_categories": {
"shared_keyword_pair": 10,
"same_academic_year_pair": 10,
"same_start_year_pair": 10,
"person_title_year": 10,
"person_keyword_title": 10,
"document_author_category_multi_hop": 2,
"person_multiple_categories": 7,
"candidate_set_aggregation": 7,
"conjunctive_disambiguation": 6,
"negative_lookup": 8
},
"test_positives": 72,
"test_negatives": 8,
"available_groups": {
"shared_keyword_pair": 1883,
"same_academic_year_pair": 535,
"same_start_year_pair": 153,
"person_title_year": 30,
"person_keyword_title": 146,
"document_author_category_multi_hop": 2,
"person_multiple_categories": 7,
"candidate_set_aggregation": 54,
"conjunctive_disambiguation": 628,
"negative_lookup": 106
},
"old_gold_corrections": 10,
"regression_count": 320,
"llm_dev_count": 46,
"target": 0.92,
"policy": "Freeze after complete DEV; run fresh TEST once, retain every failure. Regression remains separate.",
"annotation": "Independent raw-source oracle plus AI source review; not human review."
},
"tests": {
"passed": 591,
"skipped": 0
},
"source_review": {
"sample": 20,
"supported": 20,
"reviewer": "AI direct raw Markdown review, not human"
},
"functional_scenarios": {
"passed": 9,
"total": 9
},
"historical_rag_replay": {
"passed": 5,
"total": 5,
"new_generation": false
},
"failed_test_ids": [],
"failed_regression_ids": [
"wide-285"
],
"limitations": [
"Known corpus, persons and template families; new query groups, not unseen profiles.",
"Source oracle plus AI review; no human annotation.",
"Only two fresh document-author/category groups and seven fresh category-list groups remain.",
"End-to-end native OpenWebUI tool routing is point 4.",
"Final prose is deterministic rendering of verified model-selected claims; this is not free-prose generation accuracy.",
"Gold corrections and changed delivery protocol prevent a controlled accuracy comparison with v6."
]
}

42
tools/graphrag/evaluation/results/final_v7_2026-10-26/verification.json Normal file
View File

@ -0,0 +1,42 @@
{
"test_command": "RUN_NEO4J_TESTS=1 RUN_LIVE_TESTS=1 RUN_LIVE_RAG_E2E=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest -q",
"logs": [
{
"name": "zp-v7-release-tests.log",
"sha256": "17f87f311f897da7a4170b102894668f13f7078eec8213a33f501ac8ca012081",
"output": "........................................................................ [ 12%]\n........................................................................ [ 24%]\n........................................................................ [ 36%]\n........................................................................ [ 48%]\n........................................................................ [ 60%]\n........................................................................ [ 73%]\n........................................................................ [ 85%]\n........................................................................ [ 97%]\n............... [100%]\n=============================== warnings summary ===============================\n.venv/lib/python3.10/site-packages/fastapi/testclient.py:1\n /home/janko/DP/zp-agent/.venv/lib/python3.10/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.\n from starlette.testclient import TestClient as TestClient # noqa\n\n-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html\n591 passed, 1 warning in 76.37s (0:01:16)\n"
},
{
"name": "zp-v7-targeted-final.log",
"sha256": "c53fc4867b28f28a687dcd19fbb04fb11dc7cc7c0a4a86abd2205b8233ad2b1c",
"output": "...................................................... [100%]\n54 passed in 2.06s\n"
}
],
"sync_response": {
"status": "ok",
"pull_git": false,
"duration_seconds": 163.9,
"counts": {
"documents": 114,
"chunks": 887,
"fts_chunks": 887,
"embedding_chunks": 887,
"tags": 2189,
"categories": 1193
},
"index_version": {
"version": "33ea0794472c416e97c84371a2a6cdf8",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-26T00:10:27.444513+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
},
"graph_counts": {
"nodes": 446,
"relationships": 877
}
}
}

43050
tools/graphrag/evaluation/results/v7_preparation/citation_regrade.json Normal file
View File

File diff suppressed because it is too large Load Diff

48
tools/graphrag/evaluation/results/v7_preparation/generation_dev_ids.json Normal file
View File

@ -0,0 +1,48 @@
[
"v7-dev-year-1899",
"v7-dev-year-1999",
"v7-dev-year-2100",
"v7-dev-year-9999",
"wide-001",
"wide-002",
"wide-041",
"wide-042",
"wide-080",
"wide-081",
"wide-082",
"wide-104",
"wide-113",
"wide-114",
"wide-116",
"wide-119",
"wide-129",
"wide-137",
"wide-138",
"wide-139",
"wide-141",
"wide-142",
"wide-143",
"wide-145",
"wide-155",
"wide-159",
"wide-161",
"wide-163",
"wide-165",
"wide-169",
"wide-170",
"wide-178",
"wide-181",
"wide-193",
"wide-194",
"wide-209",
"wide-217",
"wide-218",
"wide-232",
"wide-236",
"wide-239",
"wide-249",
"wide-250",
"wide-283",
"wide-289",
"wide-290"
]

417
tools/graphrag/evaluation/results/v7_preparation/live_scenarios.declaration.json Normal file
View File

@ -0,0 +1,417 @@
{
"method": "Live synthetic functional checks; separate from corpus accuracy metrics.",
"model": "model120-fast",
"hashes": {
"/home/janko/DP/zp-agent/tools/answer_evaluation/check_context_scenarios.py": "99dbae2747183622e0823cd8f6536f4ff38ef56d0e41dd7855096d47af3b1c37",
"/home/janko/DP/zp-agent/tools/answer_evaluation/graph_answer_prompt.txt": "fc9f2f8d52e70a8d67b9f4814b3f15e6be64bf88afe53ce4db4f037bde00e064",
"/home/janko/DP/zp-agent/tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47",
"/home/janko/DP/zp-agent/tools/answer_evaluation/checked_generation.py": "cb821a65327424c19cbd996190b14e6aa30d9b909493f11716552f71249a0529",
"/home/janko/DP/zp-agent/tools/answer_evaluation/answer_format.py": "86a27b4aebc6a5077e13794b81957a1cf063a1f3e070c583120ba53c1788d2f8",
"/home/janko/DP/zp-agent/tools/answer_evaluation/render_answer.py": "609c534a37899b3febcea32d983da4b806c287de8f2ddf9bc0675e3280859002"
},
"scenarios": [
{
"id": "complete",
"question": "Uveď názov práce osoby Eva Vzorová a rok práce.",
"context": {
"status": "ok",
"reason": null,
"complete": true,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"aggregation": null,
"answer_candidates": []
},
"expected": {
"status": "ok",
"claims": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026
}
]
}
},
{
"id": "incomplete_without_truncated",
"question": "Uveď názov práce osoby Eva Vzorová a rok práce.",
"context": {
"status": "ok",
"reason": null,
"complete": false,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"aggregation": null,
"answer_candidates": []
},
"expected": {
"status": "ok",
"claims": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026
}
]
}
},
{
"id": "truncated",
"question": "Uveď názov práce osoby Eva Vzorová a rok práce.",
"context": {
"status": "ok",
"reason": null,
"complete": false,
"truncated": true,
"truncation_reasons": [
"limit"
],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"aggregation": null,
"answer_candidates": []
},
"expected": {
"status": "ok",
"claims": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026
}
]
}
},
{
"id": "unknown_completeness",
"question": "Uveď názov práce osoby Eva Vzorová a rok práce.",
"context": {
"status": "ok",
"reason": null,
"complete": null,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"aggregation": null,
"answer_candidates": []
},
"expected": {
"status": "ok",
"claims": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026
}
]
}
},
{
"id": "exact_count_partial_evidence",
"question": "Aký je počet vyhovujúcich osôb?",
"context": {
"status": "ok",
"reason": null,
"complete": false,
"truncated": true,
"truncation_reasons": [
"evidence_limit"
],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"aggregation": {
"count": 2,
"complete": true
},
"answer_candidates": []
},
"expected": {
"status": "ok",
"count": 2
}
},
{
"id": "same_person_child_tag",
"question": "Uveď názov práce osoby Eva Vzorová v roku 2026 s kľúčovým slovom python.",
"context": {
"status": "ok",
"reason": null,
"complete": true,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
},
{
"source_id": "S2",
"document_path": "child.md",
"source_url": "https://example.test/child"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
},
{
"person": "Eva Vzorová",
"keyword": "python",
"document_path": "child.md",
"source_url": "https://example.test/child"
}
],
"aggregation": null,
"answer_candidates": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"keyword": "python",
"evidence_documents": [
"a.md",
"child.md"
],
"source_urls": [
"https://example.test/a",
"https://example.test/child"
]
}
]
},
"expected": {
"status": "ok",
"claims": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"keyword": "python"
}
]
}
},
{
"id": "same_person_year_child_tag",
"question": "Ktorá osoba spĺňa kľúčové slovo python a rok práce 2026?",
"context": {
"status": "ok",
"reason": null,
"complete": true,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
},
{
"source_id": "S2",
"document_path": "child.md",
"source_url": "https://example.test/child"
}
],
"facts": [
{
"person": "Eva Vzorová",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
},
{
"person": "Eva Vzorová",
"keyword": "python",
"document_path": "child.md",
"source_url": "https://example.test/child"
}
],
"aggregation": null,
"answer_candidates": [
{
"person": "Eva Vzorová",
"year": 2026,
"keyword": "python",
"evidence_documents": [
"a.md",
"child.md"
],
"source_urls": [
"https://example.test/a",
"https://example.test/child"
]
}
]
},
"expected": {
"status": "ok",
"claims": [
{
"person": "Eva Vzorová",
"year": 2026,
"keyword": "python"
}
]
}
},
{
"id": "type_is_not_keyword",
"question": "Uveď názov diplomového projektu osoby Eva Vzorová a rok.",
"context": {
"status": "ok",
"reason": null,
"complete": true,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [
{
"source_id": "S1",
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"facts": [
{
"person": "Eva Vzorová",
"title": "Vyhľadávanie informácií",
"year": 2026,
"document_path": "a.md",
"source_url": "https://example.test/a"
}
],
"aggregation": null,
"answer_candidates": []
},
"expected": {
"status": "ok",
"no_keyword": true
}
},
{
"id": "no_answer",
"question": "Akú prácu má Neexistujúca Osoba?",
"context": {
"status": "no_answer",
"reason": "unknown_person",
"complete": true,
"truncated": false,
"truncation_reasons": [],
"index_version": {
"version": "synthetic-functional-fixture"
},
"sources": [],
"facts": [],
"aggregation": null,
"answer_candidates": []
},
"expected": {
"status": "no_answer"
}
}
]
}

1293
tools/graphrag/evaluation/results/v7_preparation/live_scenarios.json Normal file
View File

File diff suppressed because it is too large Load Diff

37
tools/graphrag/evaluation/results/v7_preparation/protocol.json Normal file
View File

@ -0,0 +1,37 @@
{
"seed": 20261026,
"dev_count": 1046,
"fresh_test_count": 80,
"test_categories": {
"shared_keyword_pair": 10,
"same_academic_year_pair": 10,
"same_start_year_pair": 10,
"person_title_year": 10,
"person_keyword_title": 10,
"document_author_category_multi_hop": 2,
"person_multiple_categories": 7,
"candidate_set_aggregation": 7,
"conjunctive_disambiguation": 6,
"negative_lookup": 8
},
"test_positives": 72,
"test_negatives": 8,
"available_groups": {
"shared_keyword_pair": 1883,
"same_academic_year_pair": 535,
"same_start_year_pair": 153,
"person_title_year": 30,
"person_keyword_title": 146,
"document_author_category_multi_hop": 2,
"person_multiple_categories": 7,
"candidate_set_aggregation": 54,
"conjunctive_disambiguation": 628,
"negative_lookup": 106
},
"old_gold_corrections": 10,
"regression_count": 320,
"llm_dev_count": 46,
"target": 0.92,
"policy": "Freeze after complete DEV; run fresh TEST once, retain every failure. Regression remains separate.",
"annotation": "Independent raw-source oracle plus AI source review; not human review."
}

478
tools/graphrag/evaluation/results/v7_preparation/rag_delivery_replay.json Normal file
View File

@ -0,0 +1,478 @@
{
"configuration": {
"method": "Deterministic delivery replay on the same saved model responses, not new generation.",
"input": "tools/graphrag/evaluation/results/v6_preparation/rag_live_smoke.json",
"input_sha256": "674272fd911a5ae04690150ac899531feae90b4d44a1735a530a2838a0d32ff3",
"delivery_files": {
"/home/janko/DP/zp-agent/tools/answer_evaluation/rag_delivery.py": "b83f93d446f44fdd7d41c12eeeaae69c0b02908076624f4eb3f4f72b9b88dbc8",
"tools/rag/evaluation/evaluate_rag_answers.py": "945c2da655537fb37fe82a08e10dc3ed66a2d90980921f61f61efb171a4a26a2",
"tools/rag/evaluation/rag_metrics.py": "3e95922df90dd1b757a550040a2d9bd4f857e17d179f60170d7cc881a08f94b6",
"tools/rag/evaluation/rag_runner.py": "031abf7d5b1a026eb199aa2a579bfb8baaa01175279d0018a364bfe50b76c3c1",
"tools/answer_evaluation/context_policy.py": "0dc2a056ca0d886d7cc8248ed4632c910c5e35bebd03ef94573df41802297f47"
}
},
"results": [
{
"id": "q0053",
"question": "Aký je názov diplomová práca osoby Lukáš Pokrývka?",
"citations_ok": true,
"unsupported_urls": [],
"retrieval_coverage": [
{
"complete": false,
"truncated": true,
"truncation_reasons": [
"candidate_limit",
"per_document_limit",
"result_limit"
],
"index_version": {
"version": "10b78a8e70744e2792ffe4cf11d45571",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-25T21:10:39.160542+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
}
],
"context_issues": [],
"accepted_for_delivery": true,
"answer": "Výpis je neúplný.\n\nNázov diplomovej práce Lukáša Pokrývku je „Paralelné trénovanie neurónových sietí\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"tool_called": true,
"tool_call_count": 1,
"tool_calls": [
{
"name": "retrieve_zpwiki_context",
"arguments": {
"query": "Aký je názov diplomová práca osoby Lukáš Pokrývka?",
"limit": 5,
"published_only": true,
"max_per_document": 1
},
"latency_seconds": 8.6679,
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
"https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak",
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"
]
}
],
"rag_source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
"https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak",
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"
],
"first_model_latency_seconds": 5.65362,
"tool_latency_seconds": 8.6679,
"final_model_latency_seconds": 23.053733,
"total_latency_seconds": 37.375855,
"usage": {
"prompt_tokens": 19045,
"completion_tokens": 137,
"total_tokens": 19182
},
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
"citations_checked": true,
"answer_matches": [
true
],
"answer_contains_score": 1.0,
"source_matches": [
true
],
"source_match_count": 1,
"source_required_count": 1,
"source_url_score": 1.0,
"should_answer_ok": true,
"should_answer_score": 1.0,
"returned_no_answer": false,
"tool_score": 1.0,
"overall_score": 1.0,
"strict_pass": true,
"model_answer": "Názov diplomovej práce Lukáša Pokrývku je „Paralelné trénovanie neurónových sietí\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"first_pass_accepted_for_delivery": false,
"first_pass_context_issues": [
"missing_incompleteness_notice"
],
"display_notice_added": true
},
{
"id": "q0060",
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Lukáš Pokrývka.",
"citations_ok": true,
"unsupported_urls": [],
"retrieval_coverage": [
{
"complete": false,
"truncated": true,
"truncation_reasons": [
"candidate_limit",
"per_document_limit",
"result_limit"
],
"index_version": {
"version": "10b78a8e70744e2792ffe4cf11d45571",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-25T21:10:39.160542+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
}
],
"context_issues": [],
"accepted_for_delivery": true,
"answer": "Výpis je neúplný.\n\nRok začiatku štúdia osoby Lukáš Pokrývka je 2016, zatiaľ čo jeho diplomová práca bola realizovaná v roku 2021.\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"tool_called": true,
"tool_call_count": 1,
"tool_calls": [
{
"name": "retrieve_zpwiki_context",
"arguments": {
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Lukáš Pokrývka.",
"limit": 10,
"published_only": true,
"max_per_document": 5
},
"latency_seconds": 8.727633,
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
"https://zp.kemt.fei.tuke.sk/students/2020/jakub_kristian_lukas",
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
]
}
],
"rag_source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
"https://zp.kemt.fei.tuke.sk/students/2020/jakub_kristian_lukas",
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
],
"first_model_latency_seconds": 5.710692,
"tool_latency_seconds": 8.727633,
"final_model_latency_seconds": 40.777827,
"total_latency_seconds": 55.216681,
"usage": {
"prompt_tokens": 20160,
"completion_tokens": 155,
"total_tokens": 20315
},
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
"citations_checked": true,
"answer_matches": [
true,
true
],
"answer_contains_score": 1.0,
"source_matches": [
true
],
"source_match_count": 1,
"source_required_count": 1,
"source_url_score": 1.0,
"should_answer_ok": true,
"should_answer_score": 1.0,
"returned_no_answer": false,
"tool_score": 1.0,
"overall_score": 1.0,
"strict_pass": true,
"model_answer": "Rok začiatku štúdia osoby Lukáš Pokrývka je 2016, zatiaľ čo jeho diplomová práca bola realizovaná v roku 2021.\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
"first_pass_accepted_for_delivery": false,
"first_pass_context_issues": [
"missing_incompleteness_notice"
],
"display_notice_added": true
},
{
"id": "q0608",
"question": "Aký source_url patrí k stránke „Podpora slovenčiny v knižnici Spacy“?",
"citations_ok": true,
"unsupported_urls": [],
"retrieval_coverage": [
{
"complete": false,
"truncated": true,
"truncation_reasons": [
"candidate_limit",
"per_document_limit",
"result_limit"
],
"index_version": {
"version": "10b78a8e70744e2792ffe4cf11d45571",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-25T21:10:39.160542+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
}
],
"context_issues": [],
"accepted_for_delivery": true,
"answer": "Výpis je neúplný.\n\nStránka s názvom „Podpora slovenčiny v knižnici Spacy\" má pridelený zdrojový URL adresu https://zp.kemt.fei.tuke.sk/topics/spacy.\n\nZdroj: https://zp.kemt.fei.tuke.sk/topics/spacy",
"tool_called": true,
"tool_call_count": 1,
"tool_calls": [
{
"name": "retrieve_zpwiki_context",
"arguments": {
"query": "Aký source_url patrí k stránke „Podpora slovenčiny v knižnici Spacy“?",
"limit": 5,
"max_per_document": 1,
"published_only": true
},
"latency_seconds": 0.620349,
"source_urls": [
"https://zp.kemt.fei.tuke.sk/topics/spacy",
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
"https://zp.kemt.fei.tuke.sk/topics/resources"
]
}
],
"rag_source_urls": [
"https://zp.kemt.fei.tuke.sk/topics/spacy",
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
"https://zp.kemt.fei.tuke.sk/topics/resources"
],
"first_model_latency_seconds": 28.658617,
"tool_latency_seconds": 0.620349,
"final_model_latency_seconds": 35.815477,
"total_latency_seconds": 65.094973,
"usage": {
"prompt_tokens": 28510,
"completion_tokens": 141,
"total_tokens": 28651
},
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
"citations_checked": true,
"answer_matches": [
true
],
"answer_contains_score": 1.0,
"source_matches": [
true
],
"source_match_count": 1,
"source_required_count": 1,
"source_url_score": 1.0,
"should_answer_ok": true,
"should_answer_score": 1.0,
"returned_no_answer": false,
"tool_score": 1.0,
"overall_score": 1.0,
"strict_pass": true,
"model_answer": "Stránka s názvom „Podpora slovenčiny v knižnici Spacy\" má pridelený zdrojový URL adresu https://zp.kemt.fei.tuke.sk/topics/spacy.\n\nZdroj: https://zp.kemt.fei.tuke.sk/topics/spacy",
"first_pass_accepted_for_delivery": false,
"first_pass_context_issues": [
"missing_incompleteness_notice"
],
"display_notice_added": true
},
{
"id": "q1147",
"question": "Porovnaj záznamy osôb Ján Holp a Ondrej Megela. Pri osobe Ján Holp použi záznam typu „diplomová práca“ a pri osobe Ondrej Megela záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
"citations_ok": true,
"unsupported_urls": [],
"retrieval_coverage": [
{
"complete": false,
"truncated": true,
"truncation_reasons": [
"candidate_limit",
"per_document_limit",
"result_limit"
],
"index_version": {
"version": "10b78a8e70744e2792ffe4cf11d45571",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-25T21:10:39.160542+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
}
],
"context_issues": [],
"accepted_for_delivery": true,
"answer": "Výpis je neúplný.\n\nJán Holp vypracoval diplomovú prácu s názvom „Systém získavania informácií v slovenskom jazyku\" v roku 2021 a Ondrej Megela vypracoval bakalársku prácu s názvom „Neurónové jazykové modelovanie typu BERT\" v roku 2020.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/jan_holp\nhttps://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
"tool_called": true,
"tool_call_count": 1,
"tool_calls": [
{
"name": "retrieve_zpwiki_context",
"arguments": {
"query": "Porovnaj záznamy osôb Ján Holp a Ondrej Megela. Pri osobe Ján Holp použi záznam typu „diplomová práca“ a pri osobe Ondrej Megela záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
"limit": 5,
"published_only": true,
"max_per_document": 1
},
"latency_seconds": 1.001685,
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
"https://zp.kemt.fei.tuke.sk/topics/faq",
"https://zp.kemt.fei.tuke.sk/topics/bert"
]
}
],
"rag_source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
"https://zp.kemt.fei.tuke.sk/topics/faq",
"https://zp.kemt.fei.tuke.sk/topics/bert"
],
"first_model_latency_seconds": 14.535465,
"tool_latency_seconds": 1.001685,
"final_model_latency_seconds": 56.647723,
"total_latency_seconds": 72.185893,
"usage": {
"prompt_tokens": 25893,
"completion_tokens": 208,
"total_tokens": 26101
},
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
"citations_checked": true,
"answer_matches": [
true,
true,
true,
true
],
"answer_contains_score": 1.0,
"source_matches": [
true,
true
],
"source_match_count": 2,
"source_required_count": 2,
"source_url_score": 1.0,
"should_answer_ok": true,
"should_answer_score": 1.0,
"returned_no_answer": false,
"tool_score": 1.0,
"overall_score": 1.0,
"strict_pass": true,
"model_answer": "Ján Holp vypracoval diplomovú prácu s názvom „Systém získavania informácií v slovenskom jazyku\" v roku 2021 a Ondrej Megela vypracoval bakalársku prácu s názvom „Neurónové jazykové modelovanie typu BERT\" v roku 2020.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/jan_holp\nhttps://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
"first_pass_accepted_for_delivery": false,
"first_pass_context_issues": [
"missing_incompleteness_notice"
],
"display_notice_added": true
},
{
"id": "q0888",
"question": "Aký konkrétny finančný rozpočet bol uvedený pre projekt osoby Tomáš Kuchárik?",
"citations_ok": true,
"unsupported_urls": [],
"retrieval_coverage": [
{
"complete": false,
"truncated": true,
"truncation_reasons": [
"candidate_limit",
"per_document_limit",
"result_limit"
],
"index_version": {
"version": "10b78a8e70744e2792ffe4cf11d45571",
"git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
"created_at_utc": "2026-10-25T21:10:39.160542+00:00",
"source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
"documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
"chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
"embedding_model": "intfloat/multilingual-e5-small",
"publication_policy": "explicit-false-hidden-v1"
}
}
],
"context_issues": [],
"accepted_for_delivery": true,
"answer": "Výpis je neúplný.\n\nV dostupných dokumentoch ZP Wiki sa informácia o konkrétnom finančnom rozpočte pre projekt osoby Tomáš Kuchárik nepodarilo spoľahlivo nájsť.\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
"tool_called": true,
"tool_call_count": 1,
"tool_calls": [
{
"name": "retrieve_zpwiki_context",
"arguments": {
"query": "Aký konkrétny finančný rozpočet bol uvedený pre projekt osoby Tomáš Kuchárik?",
"limit": 5,
"published_only": true
},
"latency_seconds": 0.454629,
"source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
"https://zp.kemt.fei.tuke.sk/students/2020/matus_suchanic",
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"
]
}
],
"rag_source_urls": [
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
"https://zp.kemt.fei.tuke.sk/students/2020/matus_suchanic",
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"
],
"first_model_latency_seconds": 129.609931,
"tool_latency_seconds": 0.454629,
"final_model_latency_seconds": 20.02263,
"total_latency_seconds": 150.087617,
"usage": {
"prompt_tokens": 14022,
"completion_tokens": 136,
"total_tokens": 14158
},
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
"citations_checked": true,
"answer_matches": [],
"answer_contains_score": 1.0,
"source_matches": [
true
],
"source_match_count": 1,
"source_required_count": 0,
"source_url_score": 1.0,
"should_answer_ok": true,
"should_answer_score": 1.0,
"returned_no_answer": true,
"tool_score": 1.0,
"overall_score": 1.0,
"strict_pass": true,
"model_answer": "V dostupných dokumentoch ZP Wiki sa informácia o konkrétnom finančnom rozpočte pre projekt osoby Tomáš Kuchárik nepodarilo spoľahlivo nájsť.\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
"first_pass_accepted_for_delivery": false,
"first_pass_context_issues": [
"missing_incompleteness_notice"
],
"display_notice_added": true
}
],
"summary": {
"questions": 5,
"strict_pass": 5,
"display_notice_added": 5
}
}

322
tools/graphrag/evaluation/results/v7_preparation/regression_ids.json Normal file
View File

@ -0,0 +1,322 @@
[
"wide-001",
"wide-002",
"wide-003",
"wide-004",
"wide-005",
"wide-006",
"wide-007",
"wide-008",
"wide-009",
"wide-010",
"wide-011",
"wide-012",
"wide-013",
"wide-014",
"wide-015",
"wide-016",
"wide-017",
"wide-018",
"wide-019",
"wide-020",
"wide-021",
"wide-022",
"wide-023",
"wide-024",
"wide-025",
"wide-026",
"wide-027",
"wide-028",
"wide-029",
"wide-030",
"wide-031",
"wide-032",
"wide-033",
"wide-034",
"wide-035",
"wide-036",
"wide-037",
"wide-038",
"wide-039",
"wide-040",
"wide-041",
"wide-042",
"wide-043",
"wide-044",
"wide-045",
"wide-046",
"wide-047",
"wide-048",
"wide-049",
"wide-050",
"wide-051",
"wide-052",
"wide-053",
"wide-054",
"wide-055",
"wide-056",
"wide-057",
"wide-058",
"wide-059",
"wide-060",
"wide-061",
"wide-062",
"wide-063",
"wide-064",
"wide-065",
"wide-066",
"wide-067",
"wide-068",
"wide-069",
"wide-070",
"wide-071",
"wide-072",
"wide-073",
"wide-074",
"wide-075",
"wide-076",
"wide-077",
"wide-078",
"wide-079",
"wide-080",
"wide-081",
"wide-082",
"wide-083",
"wide-084",
"wide-085",
"wide-086",
"wide-087",
"wide-088",
"wide-089",
"wide-090",
"wide-091",
"wide-092",
"wide-093",
"wide-094",
"wide-095",
"wide-096",
"wide-097",
"wide-098",
"wide-099",
"wide-100",
"wide-101",
"wide-102",
"wide-103",
"wide-104",
"wide-105",
"wide-106",
"wide-107",
"wide-108",
"wide-109",
"wide-110",
"wide-111",
"wide-112",
"wide-113",
"wide-114",
"wide-115",
"wide-116",
"wide-117",
"wide-118",
"wide-119",
"wide-120",
"wide-121",
"wide-122",
"wide-123",
"wide-124",
"wide-125",
"wide-126",
"wide-127",
"wide-128",
"wide-129",
"wide-130",
"wide-131",
"wide-132",
"wide-133",
"wide-134",
"wide-135",
"wide-136",
"wide-137",
"wide-138",
"wide-139",
"wide-140",
"wide-141",
"wide-142",
"wide-143",
"wide-144",
"wide-145",
"wide-146",
"wide-147",
"wide-148",
"wide-149",
"wide-150",
"wide-151",
"wide-152",
"wide-153",
"wide-154",
"wide-155",
"wide-156",
"wide-157",
"wide-158",
"wide-159",
"wide-160",
"wide-161",
"wide-162",
"wide-163",
"wide-164",
"wide-165",
"wide-166",
"wide-167",
"wide-168",
"wide-169",
"wide-170",
"wide-171",
"wide-172",
"wide-173",
"wide-174",
"wide-175",
"wide-176",
"wide-177",
"wide-178",
"wide-179",
"wide-180",
"wide-181",
"wide-182",
"wide-183",
"wide-184",
"wide-185",
"wide-186",
"wide-187",
"wide-188",
"wide-189",
"wide-190",
"wide-191",
"wide-192",
"wide-193",
"wide-194",
"wide-195",
"wide-196",
"wide-197",
"wide-198",
"wide-199",
"wide-200",
"wide-201",
"wide-202",
"wide-203",
"wide-204",
"wide-205",
"wide-206",
"wide-207",
"wide-208",
"wide-209",
"wide-210",
"wide-211",
"wide-212",
"wide-213",
"wide-214",
"wide-215",
"wide-216",
"wide-217",
"wide-218",
"wide-219",
"wide-220",
"wide-221",
"wide-222",
"wide-223",
"wide-224",
"wide-225",
"wide-226",
"wide-227",
"wide-228",
"wide-229",
"wide-230",
"wide-231",
"wide-232",
"wide-233",
"wide-234",
"wide-235",
"wide-236",
"wide-237",
"wide-238",
"wide-239",
"wide-240",
"wide-241",
"wide-242",
"wide-243",
"wide-244",
"wide-245",
"wide-246",
"wide-247",
"wide-248",
"wide-249",
"wide-250",
"wide-251",
"wide-252",
"wide-253",
"wide-254",
"wide-255",
"wide-256",
"wide-257",
"wide-258",
"wide-259",
"wide-260",
"wide-261",
"wide-262",
"wide-263",
"wide-264",
"wide-265",
"wide-266",
"wide-267",
"wide-268",
"wide-269",
"wide-270",
"wide-271",
"wide-272",
"wide-273",
"wide-274",
"wide-275",
"wide-276",
"wide-277",
"wide-278",
"wide-279",
"wide-280",
"wide-281",
"wide-282",
"wide-283",
"wide-284",
"wide-285",
"wide-286",
"wide-287",
"wide-288",
"wide-289",
"wide-290",
"wide-291",
"wide-292",
"wide-293",
"wide-294",
"wide-295",
"wide-296",
"wide-297",
"wide-298",
"wide-299",
"wide-300",
"wide-301",
"wide-302",
"wide-303",
"wide-304",
"wide-305",
"wide-306",
"wide-307",
"wide-308",
"wide-309",
"wide-310",
"wide-311",
"wide-312",
"wide-313",
"wide-314",
"wide-315",
"wide-316",
"wide-317",
"wide-318",
"wide-319",
"wide-320"
]

8947
tools/graphrag/evaluation/results/v7_preparation/source_audit.json Normal file
View File

File diff suppressed because it is too large Load Diff

151
tools/graphrag/evaluation/results/v7_preparation/source_corrections.json Normal file
View File

@ -0,0 +1,151 @@
{
"questions": [
{
"id": "regression-0597",
"old_bindings": [
{
"person": "Stanislav Matsunych",
"start_year": 2017
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-dev-0630",
"old_bindings": [
{
"person": "Jakub Maruniak",
"start_year": 2016
},
{
"person": "Márk Fehér",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-dev-0631",
"old_bindings": [
{
"person": "Dominik Nagy",
"start_year": 2016
},
{
"person": "Patrik Pavlišin",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-083",
"old_bindings": [
{
"person": "Lukáš Pokrývka",
"start_year": 2016
},
{
"person": "Maroš Harahus",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-088",
"old_bindings": [
{
"person": "Ján Holp",
"start_year": 2016
},
{
"person": "Maroš Harahus",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-090",
"old_bindings": [
{
"person": "Martin Jancura",
"start_year": 2017
},
{
"person": "Stanislav Matsunych",
"start_year": 2017
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-105",
"old_bindings": [
{
"person": "Dominik Nagy",
"start_year": 2016
},
{
"person": "Maroš Harahus",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-107",
"old_bindings": [
{
"person": "Dominik Nagy",
"start_year": 2016
},
{
"person": "Márk Fehér",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-111",
"old_bindings": [
{
"person": "Dominik Nagy",
"start_year": 2016
},
{
"person": "Tomáš Kuchárik",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
},
{
"id": "wide-112",
"old_bindings": [
{
"person": "Jakub Maruniak",
"start_year": 2016
},
{
"person": "Tomáš Kuchárik",
"start_year": 2016
}
],
"new_status": "no_answer",
"reason": "Directory cohort is not a documented study-start year."
}
],
"policy": "Only explicit study-start text is a fact. Cohort remains separate metadata; missing means unknown."
}

21
tools/graphrag/evaluation/results/v7_preparation/source_review.csv Normal file
View File

@ -0,0 +1,21 @@
id,category,decision,reviewer,notes,sources,human_review
fresh7-060,candidate_set_aggregation,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2016/darius_lindvai/README.md; pages/students/2019/alina_vitko/README.md,not_performed
fresh7-066,candidate_set_aggregation,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2018/adrian_remias/README.md; pages/students/2019/patrik_pokrivcak/README.md,not_performed
fresh7-067,conjunctive_disambiguation,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2022/oleh_poiasnik/README.md,not_performed
fresh7-072,conjunctive_disambiguation,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2019/alina_vitko/README.md,not_performed
fresh7-051,document_author_category_multi_hop,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2016/darius_lindvai/README.md,not_performed
fresh7-052,document_author_category_multi_hop,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2020/vladyslav_krupko/README.md,not_performed
fresh7-073,negative_lookup,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2017/stanislav_matsunych/README.md,not_performed
fresh7-080,negative_lookup,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2021/oleksandr_hryshchenko/README.md,not_performed
fresh7-041,person_keyword_title,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2021/artur_hyrenko/README.md,not_performed
fresh7-050,person_keyword_title,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2019/kristian_sopkovic/README.md,not_performed
fresh7-053,person_multiple_categories,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2020/pavol_hudak/README.md,not_performed
fresh7-059,person_multiple_categories,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2018/adrian_remias/README.md,not_performed
fresh7-031,person_title_year,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2016/patrik_pavlisin/README.md,not_performed
fresh7-040,person_title_year,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2022/jan_ptak/README.md,not_performed
fresh7-011,same_academic_year_pair,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2018/adrian_remias/README.md; pages/students/2018/david_omasta/README.md,not_performed
fresh7-020,same_academic_year_pair,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2021/oleksandr_hryshchenko/README.md; pages/students/2022/jan_ptak/README.md,not_performed
fresh7-021,same_start_year_pair,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2020/david_kostilnik/README.md; pages/students/2020/kamil_tomcufcik/README.md,not_performed
fresh7-030,same_start_year_pair,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2021/artur_hyrenko/README.md; pages/students/2021/matej_novotny/README.md,not_performed
fresh7-001,shared_keyword_pair,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/interns/cesar_gutierrez/README.md; pages/students/2022/jakub_schwarc/README.md,not_performed
fresh7-010,shared_keyword_pair,supported_by_raw_markdown,AI direct raw-source review; not human,Literal taxonomy and explicit academic/study-start evidence checked; no directory inference.,pages/students/2021/eduard_matovka/README.md; pages/students/2021/vladimir_ferko/README.md,not_performed
1 id category decision reviewer notes sources human_review
2 fresh7-060 candidate_set_aggregation supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2016/darius_lindvai/README.md; pages/students/2019/alina_vitko/README.md not_performed
3 fresh7-066 candidate_set_aggregation supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2018/adrian_remias/README.md; pages/students/2019/patrik_pokrivcak/README.md not_performed
4 fresh7-067 conjunctive_disambiguation supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2022/oleh_poiasnik/README.md not_performed
5 fresh7-072 conjunctive_disambiguation supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2019/alina_vitko/README.md not_performed
6 fresh7-051 document_author_category_multi_hop supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2016/darius_lindvai/README.md not_performed
7 fresh7-052 document_author_category_multi_hop supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2020/vladyslav_krupko/README.md not_performed
8 fresh7-073 negative_lookup supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2017/stanislav_matsunych/README.md not_performed
9 fresh7-080 negative_lookup supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2021/oleksandr_hryshchenko/README.md not_performed
10 fresh7-041 person_keyword_title supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2021/artur_hyrenko/README.md not_performed
11 fresh7-050 person_keyword_title supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2019/kristian_sopkovic/README.md not_performed
12 fresh7-053 person_multiple_categories supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2020/pavol_hudak/README.md not_performed
13 fresh7-059 person_multiple_categories supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2018/adrian_remias/README.md not_performed
14 fresh7-031 person_title_year supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2016/patrik_pavlisin/README.md not_performed
15 fresh7-040 person_title_year supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2022/jan_ptak/README.md not_performed
16 fresh7-011 same_academic_year_pair supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2018/adrian_remias/README.md; pages/students/2018/david_omasta/README.md not_performed
17 fresh7-020 same_academic_year_pair supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2021/oleksandr_hryshchenko/README.md; pages/students/2022/jan_ptak/README.md not_performed
18 fresh7-021 same_start_year_pair supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2020/david_kostilnik/README.md; pages/students/2020/kamil_tomcufcik/README.md not_performed
19 fresh7-030 same_start_year_pair supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2021/artur_hyrenko/README.md; pages/students/2021/matej_novotny/README.md not_performed
20 fresh7-001 shared_keyword_pair supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/interns/cesar_gutierrez/README.md; pages/students/2022/jakub_schwarc/README.md not_performed
21 fresh7-010 shared_keyword_pair supported_by_raw_markdown AI direct raw-source review; not human Literal taxonomy and explicit academic/study-start evidence checked; no directory inference. pages/students/2021/eduard_matovka/README.md; pages/students/2021/vladimir_ferko/README.md not_performed

751
tools/graphrag/evaluation/results/v7_preparation/source_review_packet.json Normal file
View File

@ -0,0 +1,751 @@
[
{
"id": "fresh7-060",
"category": "candidate_set_aggregation",
"question": "Z kandidátov Alina Vitko, Dárius Lindvai, Adrián Remiáš vyber osoby s prácou v roku 2021. Koľko ich je?",
"expected_bindings": [
{
"person": "Alina Vitko",
"year": 2021
},
{
"person": "Dárius Lindvai",
"year": 2021
}
],
"raw_sources": [
{
"path": "pages/students/2019/alina_vitko/README.md",
"proof": {
"document_path": "pages/students/2019/alina_vitko/README.md",
"sha256": "9b071fe2aff470c275ae027be236fe836f741f774cabb9cbbb55304a8d5102cd",
"field": "academic_record",
"academic_kind": "Work",
"academic_type": "bachelor_project",
"title": "Vytovrenie prehľadu existujúcich systémov QA.",
"year": 2021,
"heading_line": 70,
"title_line": 72,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Alina Vitko\n3: published: true\n4: taxonomy:\n5: category: [bp2022]\n6: tag: [ir,qa,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n69: \n70: ## Bakalársky projekt 2021\n71: \n72: Vytovrenie prehľadu existujúcich systémov QA.\n73: "
},
{
"path": "pages/students/2016/darius_lindvai/README.md",
"proof": {
"document_path": "pages/students/2016/darius_lindvai/README.md",
"sha256": "fcaa47bb9708a6439eec5c4b7f7b4ed6c7359e423e1d940e1aed9f44957e6d62",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"title": "Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
"year": 2021,
"heading_line": null,
"title_line": 14,
"year_basis": "frontmatter_category"
},
"raw_excerpt": "1: ---\n2: title: Dárius Lindvai\n3: published: true\n4: taxonomy:\n5: category: [dp2021,bp2019]\n6: tag: [nn,interpunction,nlp]\n7: author: Daniel Hladek\n8: ---\n9: # Dárius Lindvai\n13: \n14: Názov: Obnovenie interpunkcie pomocou hlbokých neurónových sietí\n15: "
}
]
},
{
"id": "fresh7-066",
"category": "candidate_set_aggregation",
"question": "Z kandidátov Adrián Remiáš, Patrik Pokrivčák, Alina Vitko vyber osoby s prácou v roku 2024. Koľko ich je?",
"expected_bindings": [
{
"person": "Adrián Remiáš",
"year": 2024
},
{
"person": "Patrik Pokrivčák",
"year": 2024
}
],
"raw_sources": [
{
"path": "pages/students/2018/adrian_remias/README.md",
"proof": {
"document_path": "pages/students/2018/adrian_remias/README.md",
"sha256": "e5bfe1d5d4e626beaa636a02d34ec58cf2cff9b0e57833905af35b80bf1465a2",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"title": null,
"year": 2024,
"heading_line": 15,
"title_line": null,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Adrián Remiáš\n3: published: true\n4: date: 01-02-2014\n5: taxonomy:\n6: category: [dp2024]\n7: tag: [nlp, transformers, interpuction]\n8: author: Daniel Hladek\n9: ---\n14: \n15: # Diplomová práca 2024\n16: "
},
{
"path": "pages/students/2019/patrik_pokrivcak/README.md",
"proof": {
"document_path": "pages/students/2019/patrik_pokrivcak/README.md",
"sha256": "3e3957ad0f7aca59d6eb64c38a3f5e21d8fd953200825d0d0e5cacabfbc35a7d",
"field": "academic_record",
"academic_kind": "Work",
"academic_type": "diploma_project",
"title": null,
"year": 2024,
"heading_line": 64,
"title_line": null,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Patrik Pokrivčák\n3: published: true\n4: taxonomy:\n5: category: [dp2025]\n6: tag: [nlp, hate]\n7: author: Daniel Hladek\n8: ---\n9: \n63: \n64: ## Diplomový projekt 2024\n65: "
}
]
},
{
"id": "fresh7-067",
"category": "conjunctive_disambiguation",
"question": "Ktorá z osôb Oleh Poiasnik a Adrián Remiáš má kľúčové slovo nlp a jej práca je z roku 2025?",
"expected_bindings": [
{
"person": "Oleh Poiasnik",
"keyword": "nlp",
"year": 2025
}
],
"raw_sources": [
{
"path": "pages/students/2022/oleh_poiasnik/README.md",
"proof": {
"document_path": "pages/students/2022/oleh_poiasnik/README.md",
"sha256": "c2f7dc3f61f8b4a5b43ded1e3ae4c7d3e7f874439ee6b17dd9c8d45dabc25bc8",
"field": "taxonomy.tag",
"value": "nlp"
},
"raw_excerpt": "1: ---\n2: title: Oleh Poiasnik\n3: published: true\n4: taxonomy:\n5: category: [bp2025,dp2027]\n6: tag: [rag,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2022/oleh_poiasnik/README.md",
"proof": {
"document_path": "pages/students/2022/oleh_poiasnik/README.md",
"sha256": "c2f7dc3f61f8b4a5b43ded1e3ae4c7d3e7f874439ee6b17dd9c8d45dabc25bc8",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "bachelor_thesis",
"title": "Generovanie jazyka s podporou vyhľadávania v doménovo orientovaných dátach",
"year": 2025,
"heading_line": 80,
"title_line": 86,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Oleh Poiasnik\n3: published: true\n4: taxonomy:\n5: category: [bp2025,dp2027]\n6: tag: [rag,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n79: \n80: # Bakalárska práca 2025\n81: \n85: \n86: Generovanie jazyka s podporou vyhľadávania v doménovo orientovaných dátach\n87: "
}
]
},
{
"id": "fresh7-072",
"category": "conjunctive_disambiguation",
"question": "Ktorá z osôb Alina Vitko a Andrii Pervashov má kľúčové slovo nlp a jej práca je z roku 2022?",
"expected_bindings": [
{
"person": "Alina Vitko",
"keyword": "nlp",
"year": 2022
}
],
"raw_sources": [
{
"path": "pages/students/2019/alina_vitko/README.md",
"proof": {
"document_path": "pages/students/2019/alina_vitko/README.md",
"sha256": "9b071fe2aff470c275ae027be236fe836f741f774cabb9cbbb55304a8d5102cd",
"field": "taxonomy.tag",
"value": "nlp"
},
"raw_excerpt": "1: ---\n2: title: Alina Vitko\n3: published: true\n4: taxonomy:\n5: category: [bp2022]\n6: tag: [ir,qa,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2019/alina_vitko/README.md",
"proof": {
"document_path": "pages/students/2019/alina_vitko/README.md",
"sha256": "9b071fe2aff470c275ae027be236fe836f741f774cabb9cbbb55304a8d5102cd",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "bachelor_thesis",
"title": "Systémy pre generovanie odpovede na otázku v prirodzenom jazyku",
"year": 2022,
"heading_line": 12,
"title_line": 14,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Alina Vitko\n3: published: true\n4: taxonomy:\n5: category: [bp2022]\n6: tag: [ir,qa,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n11: \n12: # Bakalárska práca 2022\n13: \n14: Systémy pre generovanie odpovede na otázku v prirodzenom jazyku\n15: "
}
]
},
{
"id": "fresh7-051",
"category": "document_author_category_multi_hop",
"question": "Kto je autor dokumentu osoby Dárius Lindvai, ktorý patrí do kategórie dp2021?",
"expected_bindings": [
{
"person": "Dárius Lindvai",
"category": "dp2021",
"document_author": "Daniel Hladek",
"document_path": "pages/students/2016/darius_lindvai/README.md",
"same_document": true
}
],
"raw_sources": [
{
"path": "pages/students/2016/darius_lindvai/README.md",
"proof": {
"document_path": "pages/students/2016/darius_lindvai/README.md",
"sha256": "fcaa47bb9708a6439eec5c4b7f7b4ed6c7359e423e1d940e1aed9f44957e6d62",
"field": "taxonomy.category",
"value": "dp2021"
},
"raw_excerpt": "1: ---\n2: title: Dárius Lindvai\n3: published: true\n4: taxonomy:\n5: category: [dp2021,bp2019]\n6: tag: [nn,interpunction,nlp]\n7: author: Daniel Hladek\n8: ---\n9: # Dárius Lindvai"
},
{
"path": "pages/students/2016/darius_lindvai/README.md",
"proof": {
"document_path": "pages/students/2016/darius_lindvai/README.md",
"sha256": "fcaa47bb9708a6439eec5c4b7f7b4ed6c7359e423e1d940e1aed9f44957e6d62",
"field": "taxonomy.author",
"value": "Daniel Hladek"
},
"raw_excerpt": "1: ---\n2: title: Dárius Lindvai\n3: published: true\n4: taxonomy:\n5: category: [dp2021,bp2019]\n6: tag: [nn,interpunction,nlp]\n7: author: Daniel Hladek\n8: ---\n9: # Dárius Lindvai"
}
]
},
{
"id": "fresh7-052",
"category": "document_author_category_multi_hop",
"question": "Kto je autor dokumentu osoby Vladyslav Krupko, ktorý patrí do kategórie bp2024?",
"expected_bindings": [
{
"person": "Vladyslav Krupko",
"category": "bp2024",
"document_author": "Daniel Hladek",
"document_path": "pages/students/2020/vladyslav_krupko/README.md",
"same_document": true
}
],
"raw_sources": [
{
"path": "pages/students/2020/vladyslav_krupko/README.md",
"proof": {
"document_path": "pages/students/2020/vladyslav_krupko/README.md",
"sha256": "4c28e6d38651bb2cafcbf4a18e35210cd369aef2caff6c5b0aa52fac38f0fe7d",
"field": "taxonomy.category",
"value": "bp2024"
},
"raw_excerpt": "1: ---\n2: title: Vladyslav Krupko\n3: published: true\n4: taxonomy:\n5: category: [bp2024]\n6: tag: [spelling]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2020/vladyslav_krupko/README.md",
"proof": {
"document_path": "pages/students/2020/vladyslav_krupko/README.md",
"sha256": "4c28e6d38651bb2cafcbf4a18e35210cd369aef2caff6c5b0aa52fac38f0fe7d",
"field": "taxonomy.author",
"value": "Daniel Hladek"
},
"raw_excerpt": "1: ---\n2: title: Vladyslav Krupko\n3: published: true\n4: taxonomy:\n5: category: [bp2024]\n6: tag: [spelling]\n7: author: Daniel Hladek\n8: ---\n9: "
}
]
},
{
"id": "fresh7-073",
"category": "negative_lookup",
"question": "Aký názov práce má osoba Stanislav Matsunych v roku 2018?",
"expected_bindings": [],
"raw_sources": [
{
"path": "pages/students/2017/stanislav_matsunych/README.md",
"proof": {
"document_path": "pages/students/2017/stanislav_matsunych/README.md",
"sha256": "90ffa11eff4a4360b0252dbc53a116c3a5777b3d5dd86ce5c89c02bdd2051c73",
"field": "closed_world_metadata",
"tags": [
"nn",
"lm",
"nlp"
],
"categories": [
"bp2020"
]
},
"raw_excerpt": "1: ---\n2: title: Stanislav Matsunych\n3: published: true\n4: taxonomy:\n5: category: [bp2020]\n6: tag: [nn,lm,nlp]\n7: author: Daniel Hladek\n8: ---\n9: # Stanislav Matsunych"
}
]
},
{
"id": "fresh7-080",
"category": "negative_lookup",
"question": "Aký názov práce má osoba Oleksandr Hryshchenko v roku 2018?",
"expected_bindings": [],
"raw_sources": [
{
"path": "pages/students/2021/oleksandr_hryshchenko/README.md",
"proof": {
"document_path": "pages/students/2021/oleksandr_hryshchenko/README.md",
"sha256": "4b073992b715b6c75b79fd2c8f850a69f940966e3a61a82c625c32c945a93036",
"field": "closed_world_metadata",
"tags": [
"ir",
"nlp"
],
"categories": [
"dp2026"
]
},
"raw_excerpt": "1: ---\n2: title: Oleksandr Hryshchenko\n3: published: true\n4: taxonomy:\n5: category: [dp2026]\n6: tag: [ir,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
}
]
},
{
"id": "fresh7-041",
"category": "person_keyword_title",
"question": "Aký názov práce osoby Artur Hyrenko v roku 2026 súvisí s kľúčovým slovom dialog?",
"expected_bindings": [
{
"person": "Artur Hyrenko",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"year": 2026,
"title": "Zvýšenie bezpečnosti veľkých jazykových modelov",
"keyword": "dialog"
}
],
"raw_sources": [
{
"path": "pages/students/2021/artur_hyrenko/README.md",
"proof": {
"document_path": "pages/students/2021/artur_hyrenko/README.md",
"sha256": "0568261f4ade4de5472edcfeb1ffed9abae18dc67e4050f4ce2ad088839a2bb3",
"field": "taxonomy.tag",
"value": "dialog"
},
"raw_excerpt": "1: ---\n2: title: Artur Hyrenko\n3: published: true\n4: taxonomy:\n5: category: [dp2026]\n6: tag: [chatbot,rasa,dialog,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2021/artur_hyrenko/README.md",
"proof": {
"document_path": "pages/students/2021/artur_hyrenko/README.md",
"sha256": "0568261f4ade4de5472edcfeb1ffed9abae18dc67e4050f4ce2ad088839a2bb3",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"title": "Zvýšenie bezpečnosti veľkých jazykových modelov",
"year": 2026,
"heading_line": 13,
"title_line": 15,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Artur Hyrenko\n3: published: true\n4: taxonomy:\n5: category: [dp2026]\n6: tag: [chatbot,rasa,dialog,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n12: \n13: # Diplomová práca 2026\n14: \n15: Zvýšenie bezpečnosti veľkých jazykových modelov\n16: "
}
]
},
{
"id": "fresh7-050",
"category": "person_keyword_title",
"question": "Aký názov práce osoby Kristián Sopkovič v roku 2022 súvisí s kľúčovým slovom nlp?",
"expected_bindings": [
{
"person": "Kristián Sopkovič",
"academic_kind": "Thesis",
"academic_type": "bachelor_thesis",
"year": 2022,
"title": "Model Spacy pre spracovanie prirodzeného jazyka v slovenčine",
"keyword": "nlp"
}
],
"raw_sources": [
{
"path": "pages/students/2019/kristian_sopkovic/README.md",
"proof": {
"document_path": "pages/students/2019/kristian_sopkovic/README.md",
"sha256": "3286960fb30f584df35455f9b6dfae81ca4827a1d711d2bcd6e48c3f150c9279",
"field": "taxonomy.tag",
"value": "nlp"
},
"raw_excerpt": "1: ---\n2: title: Kristián Sopkovič\n3: published: true\n4: taxonomy:\n5: category: [bp2022,dp2024]\n6: tag: [spacy,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2019/kristian_sopkovic/README.md",
"proof": {
"document_path": "pages/students/2019/kristian_sopkovic/README.md",
"sha256": "3286960fb30f584df35455f9b6dfae81ca4827a1d711d2bcd6e48c3f150c9279",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "bachelor_thesis",
"title": "Model Spacy pre spracovanie prirodzeného jazyka v slovenčine",
"year": 2022,
"heading_line": 308,
"title_line": 312,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Kristián Sopkovič\n3: published: true\n4: taxonomy:\n5: category: [bp2022,dp2024]\n6: tag: [spacy,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n307: \n308: # Bakalárska práca 2022\n309: \n311: \n312: Model Spacy pre spracovanie prirodzeného jazyka v slovenčine\n313: "
}
]
},
{
"id": "fresh7-053",
"category": "person_multiple_categories",
"question": "Vypíš všetky kategórie dokumentov osoby Pavol Hudák.",
"expected_bindings": [
{
"person": "Pavol Hudák",
"category": "dp2025"
}
],
"raw_sources": [
{
"path": "pages/students/2020/pavol_hudak/README.md",
"proof": {
"document_path": "pages/students/2020/pavol_hudak/README.md",
"sha256": "33d06fccc6451e5f19b27b0cad001a34e81635b4ea294a01dec7771c823abc40",
"field": "taxonomy.category",
"value": "dp2025"
},
"raw_excerpt": "1: ---\n2: title: Pavol Hudák\n3: published: true\n4: taxonomy:\n5: category: [dp2025]\n6: tag: [nlp,qa]\n7: author: Daniel Hladek\n8: ---\n9: "
}
]
},
{
"id": "fresh7-059",
"category": "person_multiple_categories",
"question": "Vypíš všetky kategórie dokumentov osoby Adrián Remiáš.",
"expected_bindings": [
{
"person": "Adrián Remiáš",
"category": "dp2024"
}
],
"raw_sources": [
{
"path": "pages/students/2018/adrian_remias/README.md",
"proof": {
"document_path": "pages/students/2018/adrian_remias/README.md",
"sha256": "e5bfe1d5d4e626beaa636a02d34ec58cf2cff9b0e57833905af35b80bf1465a2",
"field": "taxonomy.category",
"value": "dp2024"
},
"raw_excerpt": "1: ---\n2: title: Adrián Remiáš\n3: published: true\n4: date: 01-02-2014\n5: taxonomy:\n6: category: [dp2024]\n7: tag: [nlp, transformers, interpuction]\n8: author: Daniel Hladek\n9: ---"
}
]
},
{
"id": "fresh7-031",
"category": "person_title_year",
"question": "Aký názov práce a rok práce sú pri osobe Patrik Pavlišin uvedené pre rok 2020?",
"expected_bindings": [
{
"person": "Patrik Pavlišin",
"academic_kind": "Thesis",
"academic_type": "bachelor_thesis",
"year": 2020,
"title": "Metódy automatického prekladu"
}
],
"raw_sources": [
{
"path": "pages/students/2016/patrik_pavlisin/README.md",
"proof": {
"document_path": "pages/students/2016/patrik_pavlisin/README.md",
"sha256": "dc1c648d7cd9b0f0e021c57f9b98189e37c100871439e7e304390715913f2763",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "bachelor_thesis",
"title": "Metódy automatického prekladu",
"year": 2020,
"heading_line": 242,
"title_line": 246,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Patrik Pavlišin\n3: published: true\n4: taxonomy:\n5: category: [dp2022,bp2020,tp2021]\n6: tag: [nmt,translation,nlp]\n7: author: Daniel Hladek\n8: ---\n9: # Patrik Pavlišin\n241: \n242: ## Bakalárska práca 2020\n243: \n245: \n246: Názov bakalárskej práce: Metódy automatického prekladu\n247: "
}
]
},
{
"id": "fresh7-040",
"category": "person_title_year",
"question": "Aký názov diplomového projektu a rok práce sú pri osobe Ján Pták uvedené pre rok 2026?",
"expected_bindings": [
{
"person": "Ján Pták",
"academic_kind": "Work",
"academic_type": "diploma_project",
"year": 2026,
"title": "Agent pre manažment záverečných prác"
}
],
"raw_sources": [
{
"path": "pages/students/2022/jan_ptak/README.md",
"proof": {
"document_path": "pages/students/2022/jan_ptak/README.md",
"sha256": "131cf053764fd5423541ebe672c3b4f5da875cfb8e4b821f1335ddb967282282",
"field": "academic_record",
"academic_kind": "Work",
"academic_type": "diploma_project",
"title": "Agent pre manažment záverečných prác",
"year": 2026,
"heading_line": 13,
"title_line": 18,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Ján Pták\n3: published: true\n4: taxonomy:\n5: category: [dp2027]\n6: tag: [rag,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n12: \n13: # Diplomový projekt 2026\n14: \n17: \n18: Agent pre manažment záverečných prác\n19: "
}
]
},
{
"id": "fresh7-011",
"category": "same_academic_year_pair",
"question": "Aké spoločné roky práce spájajú osoby Adrián Remiáš a Dávid Omasta?",
"expected_bindings": [
{
"person": "Adrián Remiáš",
"year": 2024
},
{
"person": "Dávid Omasta",
"year": 2024
}
],
"raw_sources": [
{
"path": "pages/students/2018/adrian_remias/README.md",
"proof": {
"document_path": "pages/students/2018/adrian_remias/README.md",
"sha256": "e5bfe1d5d4e626beaa636a02d34ec58cf2cff9b0e57833905af35b80bf1465a2",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"title": null,
"year": 2024,
"heading_line": 15,
"title_line": null,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Adrián Remiáš\n3: published: true\n4: date: 01-02-2014\n5: taxonomy:\n6: category: [dp2024]\n7: tag: [nlp, transformers, interpuction]\n8: author: Daniel Hladek\n9: ---\n14: \n15: # Diplomová práca 2024\n16: "
},
{
"path": "pages/students/2018/david_omasta/README.md",
"proof": {
"document_path": "pages/students/2018/david_omasta/README.md",
"sha256": "4164051870e1e1cf8949ed18a74196e9ea789dfbe4fee24bdeb736ca42848327",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"title": "Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku",
"year": 2024,
"heading_line": 19,
"title_line": 25,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Dávid Omasta\n3: published: true\n4: taxonomy:\n5: category: [dp2023,dp2024]\n6: tag: [lm]\n7: author: Daniel Hladek\n8: ---\n9: \n18: \n19: # Diplomová práca 2024\n20: \n24: \n25: Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku\n26: "
}
]
},
{
"id": "fresh7-020",
"category": "same_academic_year_pair",
"question": "Aké spoločné roky práce spájajú osoby Ján Pták a Oleksandr Hryshchenko?",
"expected_bindings": [
{
"person": "Ján Pták",
"year": 2026
},
{
"person": "Oleksandr Hryshchenko",
"year": 2026
}
],
"raw_sources": [
{
"path": "pages/students/2022/jan_ptak/README.md",
"proof": {
"document_path": "pages/students/2022/jan_ptak/README.md",
"sha256": "131cf053764fd5423541ebe672c3b4f5da875cfb8e4b821f1335ddb967282282",
"field": "academic_record",
"academic_kind": "Work",
"academic_type": "diploma_project",
"title": "Agent pre manažment záverečných prác",
"year": 2026,
"heading_line": 13,
"title_line": 18,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Ján Pták\n3: published: true\n4: taxonomy:\n5: category: [dp2027]\n6: tag: [rag,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n12: \n13: # Diplomový projekt 2026\n14: \n17: \n18: Agent pre manažment záverečných prác\n19: "
},
{
"path": "pages/students/2021/oleksandr_hryshchenko/README.md",
"proof": {
"document_path": "pages/students/2021/oleksandr_hryshchenko/README.md",
"sha256": "4b073992b715b6c75b79fd2c8f850a69f940966e3a61a82c625c32c945a93036",
"field": "academic_record",
"academic_kind": "Thesis",
"academic_type": "diploma_thesis",
"title": "Grafové neurónové siete pre získavanie informácií",
"year": 2026,
"heading_line": 13,
"title_line": 15,
"year_basis": "heading"
},
"raw_excerpt": "1: ---\n2: title: Oleksandr Hryshchenko\n3: published: true\n4: taxonomy:\n5: category: [dp2026]\n6: tag: [ir,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n12: \n13: # Diplomová práca 2026\n14: \n15: Grafové neurónové siete pre získavanie informácií\n16: "
}
]
},
{
"id": "fresh7-021",
"category": "same_start_year_pair",
"question": "Porovnaj rovnaký rok začiatku štúdia osôb Dávid Kostilník a Kamil Tomčufčík.",
"expected_bindings": [
{
"person": "Dávid Kostilník",
"start_year": 2020
},
{
"person": "Kamil Tomčufčík",
"start_year": 2020
}
],
"raw_sources": [
{
"path": "pages/students/2020/david_kostilnik/README.md",
"proof": {
"document_path": "pages/students/2020/david_kostilnik/README.md",
"sha256": "2927524b75da59a64120fc364a0fc374986fa4d364d54e52c36800762c203264",
"field": "start_year",
"value": 2020,
"basis": "explicit_text",
"confidence": "documented"
},
"raw_excerpt": "1: ---\n2: title: Dávid Kostilník\n3: published: true\n4: taxonomy:\n5: category: [dp2025]\n6: tag: [ir]\n7: author: Daniel Hladek\n8: ---\n9: \n10: rok začiatku štúdia: 2020"
},
{
"path": "pages/students/2020/kamil_tomcufcik/README.md",
"proof": {
"document_path": "pages/students/2020/kamil_tomcufcik/README.md",
"sha256": "9d865adf538bc6d7992ee4f4882713447da5c936fca17c43ca0f576072537048",
"field": "start_year",
"value": 2020,
"basis": "explicit_text",
"confidence": "documented"
},
"raw_excerpt": "1: ---\n2: title: Kamil Tomčufčík\n3: published: true\n4: taxonomy:\n5: category: [dp2025]\n6: tag: [lm]\n7: author: Daniel Hladek\n8: ---\n9: \n10: rok začiatku štúdia: 2020"
}
]
},
{
"id": "fresh7-030",
"category": "same_start_year_pair",
"question": "Porovnaj rovnaký rok začiatku štúdia osôb Artur Hyrenko a Matej Novotný.",
"expected_bindings": [
{
"person": "Artur Hyrenko",
"start_year": 2021
},
{
"person": "Matej Novotný",
"start_year": 2021
}
],
"raw_sources": [
{
"path": "pages/students/2021/artur_hyrenko/README.md",
"proof": {
"document_path": "pages/students/2021/artur_hyrenko/README.md",
"sha256": "0568261f4ade4de5472edcfeb1ffed9abae18dc67e4050f4ce2ad088839a2bb3",
"field": "start_year",
"value": 2021,
"basis": "explicit_text",
"confidence": "documented"
},
"raw_excerpt": "1: ---\n2: title: Artur Hyrenko\n3: published: true\n4: taxonomy:\n5: category: [dp2026]\n6: tag: [chatbot,rasa,dialog,nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n11: rok začiatku štúdia: 2021"
},
{
"path": "pages/students/2021/matej_novotny/README.md",
"proof": {
"document_path": "pages/students/2021/matej_novotny/README.md",
"sha256": "b083a7cadde532e3428dc9a57bc61456fcc5905bda91e011b96033d42bf5d9cb",
"field": "start_year",
"value": 2021,
"basis": "explicit_text",
"confidence": "documented"
},
"raw_excerpt": "1: ---\n2: title: Matej Novotný\n3: published: true\n4: taxonomy:\n5: category: [dp2026]\n6: tag: [nlp]\n7: author: Daniel Hladek\n8: ---\n9: \n11: rok začiatku štúdia: 2021"
}
]
},
{
"id": "fresh7-001",
"category": "shared_keyword_pair",
"question": "Over spoločné kľúčové slovo nlp pri osobách Cesar Abascal Gutierrez a Jakub Schwarc.",
"expected_bindings": [
{
"person": "Cesar Abascal Gutierrez",
"keyword": "nlp"
},
{
"person": "Jakub Schwarc",
"keyword": "nlp"
}
],
"raw_sources": [
{
"path": "pages/interns/cesar_gutierrez/README.md",
"proof": {
"document_path": "pages/interns/cesar_gutierrez/README.md",
"sha256": "b0be60576eed28d4b1920e9acf198684519373dfc13171567247c853b3ed699e",
"field": "taxonomy.tag",
"value": "nlp"
},
"raw_excerpt": "1: ---\n2: title: Cesar Abascal Gutierrez\n3: published: true\n4: taxonomy:\n5: category: [iaeste]\n6: tag: [ner,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2022/jakub_schwarc/README.md",
"proof": {
"document_path": "pages/students/2022/jakub_schwarc/README.md",
"sha256": "f541a8852c2b0f6f9d7182e01a34c4e3beee9bfa55427a06b331638b11eb8be6",
"field": "taxonomy.tag",
"value": "nlp"
},
"raw_excerpt": "1: ---\n2: title: Jakub Schwarc\n3: published: true\n4: taxonomy:\n5: category: [dp2027]\n6: tag: [nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
}
]
},
{
"id": "fresh7-010",
"category": "shared_keyword_pair",
"question": "Over spoločné kľúčové slovo dialog pri osobách Eduard Matovka a Vladimír Ferko.",
"expected_bindings": [
{
"person": "Eduard Matovka",
"keyword": "dialog"
},
{
"person": "Vladimír Ferko",
"keyword": "dialog"
}
],
"raw_sources": [
{
"path": "pages/students/2021/eduard_matovka/README.md",
"proof": {
"document_path": "pages/students/2021/eduard_matovka/README.md",
"sha256": "de2a7ae38ec87110e347956b37d0605a298ed2847f972961cc9b977da8221d43",
"field": "taxonomy.tag",
"value": "dialog"
},
"raw_excerpt": "1: ---\n2: title: Eduard Matovka\n3: published: true\n4: taxonomy:\n5: category: [bp2024]\n6: tag: [dialog,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
},
{
"path": "pages/students/2021/vladimir_ferko/README.md",
"proof": {
"document_path": "pages/students/2021/vladimir_ferko/README.md",
"sha256": "68e7d311896eda2171e5eec1ec3a8da90d326ed91acaace8058f68664a50d271",
"field": "taxonomy.tag",
"value": "dialog"
},
"raw_excerpt": "1: ---\n2: title: Vladimír Ferko\n3: published: true\n4: taxonomy:\n5: category: [bp2024]\n6: tag: [dialog,nlp]\n7: author: Daniel Hladek\n8: ---\n9: "
}
]
}
]

6
tools/graphrag/evaluation/source_oracle.py
View File

@ -231,13 +231,15 @@ def load_source_oracle(wiki_root: Path = DEFAULT_WIKI) -> dict:
if profile_path(path) != path or not document['title']:
continue
family = [doc for child, doc in documents.items() if profile_path(child) == path]
start = re.search(r'(?:rok\s+)?zaciat\w*\s+studi\w*\s*:?\s*(20\d{2})', norm('\n'.join(document['body_lines'])))
start = re.search(r'(?:rok\s+)?zaciat\w*\s+studi\w*\s*:?\s*([12]\d{3})', norm('\n'.join(document['body_lines'])))
if not start:
start = re.search(r'beginning of the study\s*(20\d{2})', norm('\n'.join(document['body_lines'])))
people[document['title']] = {'person': document['title'], 'profile': path, 'documents': [d['path'] for d in family],
'tags': sorted({tag for d in family for tag in d['tags']}, key=norm),
'categories': sorted({cat for d in family for cat in d['categories']}, key=norm),
'start_year': int(start[1]) if start else int(path.split('/')[2]) if path.startswith('pages/students/') else None,
'start_year': int(start[1]) if start else None,
'start_year_basis': 'explicit_text' if start else 'missing',
'cohort_year': int(path.split('/')[2]) if path.startswith('pages/students/') else None,
'academics': extract_academics(document),
'year_conflicts': [r for r in extract_academics(document)
if r['year_basis'] == 'heading' and any(category_type(cat)[0] == r['academic_type']

11
tools/graphrag/graph_search.py
View File

@ -756,7 +756,7 @@ def extract_query_constraints(
{
int(value)
for value in re.findall(
r"\b(20\d{2})\b",
r"\b(\d{4})\b",
normalized,
)
}
@ -2450,6 +2450,15 @@ def search_graph_with_session(
common = {"scope": GRAPH_SCOPE, "person_ids": person_ids, "limit": fetch_limit,
"published_only": published_only}
blocked_reasons = []
# A written year outside the current corpus is still a real constraint.
# Malformed/range notation is explicitly unsupported, never a global query.
explicit_year_tokens = re.findall(r'\b(?:roku|rokoch)\s+([^\s,?!]+)', clean_query, flags=re.I)
if any(re.match(r'\d', token) and not re.fullmatch(r'\d{4}[.]?', token) for token in explicit_year_tokens):
blocked_reasons.append('unsupported_year_constraint')
if constraints['mentions_start_year'] and any(
row.get('kind') == 'Person' and row.get('start_year') is None for row in resolved_entities
):
blocked_reasons.append('missing_documented_start_year')
if constraints["unresolved_keyword"]:
blocked_reasons.append("unknown_keyword")
if unknown_names:

18
tools/graphrag/graphrag.py
View File

@ -1003,9 +1003,8 @@ def extract_start_year(
chunks: list[dict[str, Any]],
) -> int | None:
pattern = re.compile(
r"rok\s+zaciatku\s+studia"
r"\s*\*?\s*:\s*\*?\s*"
r"(20\d{2})"
r"(?:(?:rok\s+)?zaciat\w*\s+studi\w*\s*\*?\s*:?\s*\*?\s*"
r"|beginning\s+of\s+the\s+study\s*:?\s*)([12]\d{3})"
)
for chunk in chunks:
@ -1022,16 +1021,6 @@ def extract_start_year(
match.group(1)
)
match = re.search(
r"pages/students/(20\d{2})/",
path.replace("\\", "/"),
)
if match:
return int(
match.group(1)
)
return None
@ -1685,6 +1674,9 @@ def build_graph_payload(
),
}
people[person_id]['start_year_basis'] = ('explicit_text' if people[person_id]['start_year'] is not None else 'missing')
people[person_id]['start_year_confidence'] = ('documented' if people[person_id]['start_year'] is not None else 'unknown')
person_documents.add(
(
person_id,