Kontroly a reporty

README.md

@@ -1377,7 +1377,72 @@ regresiu vrátane lokálneho Neo4j. Prepínač `--git-network` navyše iba
 načíta hlavičku nastavenej wiki vetvy cez existujúce SSH pripojenie.
 Počas pytestu sú externé spojenia zablokované a kľúč generujúceho modelu
 odstránený. HTTP transport testy povoľujú iba svoj dočasný loopback port.
-Výsledky a log ostávajú lokálne v `data/`.
+Výsledky ostávajú lokálne v `data/reports/verification/` a pytest logy
+v `data/logs/verification/`. Každé spustenie má vlastný časový identifikátor;
+existujúci report sa neprepisuje. Skript počká najviac 90 sekúnd na
+pripravenosť API po štarte Dockera (`--wait-ready-seconds`).
+
+### Reporty, logy a uchovávanie
+
+Prevádzkové indexy `documents.json`, `chunks.json`, `index_version.json`,
+SQLite databázy a zámky ostávajú priamo v `data/`. Reporty sa zapisujú
+do `data/reports/<oblasť>/`, logy do `data/logs/`. Výstupy inštalovateľných
+nástrojov zostávajú v `data/tool_packages/`.
+
+Päť najnovších dokončených kontrol projektu zostáva v aktívnom priečinku;
+staršie kontroly aj ich logy sa presunú do `archive/verification/`.
+Neúspešná dokončená kontrola sa tiež uchová. Rozpracované reporty sa
+automaticky nearchivujú. Staršie pôvodné výstupy sú v `archive/legacy/`.
+Súbory sa nemažú a ich obsah sa nemení; `data/reports/migrations/`
+obsahuje záznam pôvodnej a novej cesty spolu s SHA-256. Replay klienti
+vedia cez tento záznam načítať aj presunutý vstup.
+
+```bash
+.venv/bin/python scripts/manage_reports.py          # náhľad presunov
+.venv/bin/python scripts/manage_reports.py --apply  # presun, bez mazania
+```
+
+Formálne evaluačné výsledky v `tools/*/evaluation/results/`, baseline,
+GraphRAG verzie a podklady pre diplomovku sa týmto uchovávaním nemenia.
+RAG merania, audity a modelové odpovede v `reports/` sa automaticky
+nemažú ani nerotujú; uchovávajú sa kvôli reprodukcii. Celé `data/` zostáva
+ignorované Gitom. Parametre `--output` kontrolných skriptov sú voliteľné;
+staré ploché cesty `data/nazov.json` sa presmerujú do príslušnej oblasti.
+
+### Širšie overenie kontrolovaného RAG
+
+`tools/rag/evaluation/json_files/verified_rag_regression.json` obsahuje
+vopred vybraných 80 známych otázok z deviatich kategórií, vrátane 10
+neodpovedateľných otázok a 52 otázok s typovanými očakávaniami.
+Je to regresná vzorka; jej úspešnosť nie je nový nezávislý odhad presnosti.
+Očakávania overuje oddelený oracle priamo z pôvodných Markdownov.
+Zdrojové cesty, odtlačky a pôvod akademického roku sú uložené pri otázkach.
+Ľudská anotácia sa tým nenahrádza. Dve otázky na Dária Lindvaia používajú
+rok z kategórie `dp2021`; kontrolovaná cesta taký rok nepovažuje za
+explicitný dôkaz roku práce. Táto neistota je uvedená v datasete.
+
+```bash
+# Len lokálne vyhľadávanie a kontrola podkladov, bez generujúceho modelu:
+.venv/bin/python -m tools.answer_evaluation.evaluate_verified_rag --mode retrieve
+
+# Úmyselné chybné odpovede na uložených skutočných kontextoch, bez LLM:
+.venv/bin/python -m tools.answer_evaluation.check_verified_rag_guards --input <report.json>
+
+# Skutočné generovanie cez /answer: vyžaduje nakonfigurovaný model a povolené
+# odoslanie otázok aj wiki podkladov tomuto poskytovateľovi:
+.venv/bin/python -m tools.answer_evaluation.evaluate_verified_rag --mode deliver
+
+# Opätovné vyhodnotenie uložených odpovedí bez nového generovania:
+.venv/bin/python -m tools.answer_evaluation.evaluate_verified_rag --mode replay --input <report.json>
+```
+
+Runner posiela na backend iba otázku a parametre; očakávané odpovede model
+nedostane. Vyhodnocuje finálny stav, prijatie odpovede, citácie, úplné
+väzby osoba–práca–typ–rok a pokrytie očakávaných údajov vo výňatkoch.
+Nesprávne „nenájdené“, blokované odpovede a technické zlyhania sú vedené
+samostatne a zostávajú v menovateli. Doslovná citácia sama osebe nedokazuje
+relevantnosť. Automatická kontrola pokrytia známych údajov nenahrádza
+ľudské posúdenie významu všeobecných odpovedí.
 
 Kontrola syntaxe najdôležitejších upravovaných modulov:
 

app/routes.py

@@ -184,7 +184,8 @@ class AnswerRequest(StrictRequestModel):
 @router.post('/answer', dependencies=[Depends(require_search_api_key)], include_in_schema=False)
 def verified_answer(request: AnswerRequest) -> dict:
     try:
-        result = deliver_answer(request.query, engine=request.engine, limit=request.limit)
+        result = deliver_answer(request.query, engine=request.engine, limit=request.limit,
+                                include_diagnostics=request.include_diagnostics)
     except Exception:
         logger.exception('Príprava overenej odpovede zlyhala.')
         return {'status': 'error', 'reason': 'answer_unavailable', 'accepted_for_delivery': False,
@@ -196,6 +197,7 @@ def verified_answer(request: AnswerRequest) -> dict:
                   'claims': [], 'source_urls': []}
     if not request.include_diagnostics:
         result.pop('attempts', None)
+        result.pop('retrieval_context', None)
     return result
 
 

scripts/manage_reports.py

@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+"""Organize local reports and logs. Dry run by default; never delete files."""
+import argparse
+import json
+from pathlib import Path
+import sys
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from tools.runtime.report_files import migration_plan, retention_plan, apply_moves
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--apply', action='store_true')
+    parser.add_argument('--keep-verifications', type=int, default=5)
+    args = parser.parse_args()
+    moves = migration_plan() + retention_plan(keep=args.keep_verifications)
+    if args.apply and moves:
+        record = apply_moves(moves)
+        print(json.dumps({'moved': len(moves), 'deleted': 0, 'journal': str(record)}))
+    else:
+        print(json.dumps({'dry_run': True, 'moves': moves, 'deleted': 0}, ensure_ascii=False, indent=2))
+
+
+if __name__ == '__main__':
+    main()

scripts/verify_project.py

@@ -103,13 +103,17 @@ def main() -> int:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument('--docker-context', default='default')
     parser.add_argument('--git-network', action='store_true', help='Read-only probe of wiki origin over its configured SSH connection')
-    parser.add_argument('--output', type=Path, default=ROOT / 'data/project_verification.json')
+    parser.add_argument('--wait-ready-seconds', type=int, default=90,
+                        help='Bounded wait for API/model warm-up after Docker startup')
+    parser.add_argument('--output', type=Path)
     parser.add_argument('--test-worker', action='store_true', help=argparse.SUPPRESS)
     args = parser.parse_args()
     os.chdir(ROOT)
     sys.path.insert(0, str(ROOT))
     if args.test_worker:
         return local_test_worker()
+    from tools.runtime.report_files import report_output, report_log, retention_plan, apply_moves
+    args.output = report_output(args.output, 'verification')
     report = {'checks': {}, 'external_llm_calls': False}
     args.output.parent.mkdir(parents=True, exist_ok=True)
 
@@ -142,9 +146,17 @@ def main() -> int:
     check('compose_configuration', lambda: command(*compose, 'config', '--quiet'))
 
     def health():
-        with urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=15) as response:
+        deadline = time.monotonic() + max(0, min(args.wait_ready_seconds, 300))
+        while True:
+            try:
+                with urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3) as response:
                     value = json.load(response)
                 assert value['ready'] and all(tool['ready'] for tool in value['tools'].values())
+                break
+            except (OSError, AssertionError):
+                if time.monotonic() >= deadline:
+                    raise
+                time.sleep(1)
         assert value['tools']['rag']['index_version'] == value['tools']['graphrag']['index_version']
         return {'ready': True, 'index_version': value['index_version'],
                 'tools': {name: item['ready'] for name, item in value['tools'].items()}}
@@ -208,7 +220,7 @@ def main() -> int:
     check('openwebui_tool_packages', tool_packages)
 
     def tests():
-        log = args.output.with_suffix('.pytest.log')
+        log = report_log(args.output)
         environment = {**os.environ}
         environment.pop('OPENWEBUI_API_KEY', None)
         with log.open('w') as output:
@@ -223,6 +235,9 @@ def main() -> int:
     check('full_local_regression', tests)
     report['passed'] = all(item['passed'] for item in report['checks'].values())
     args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n')
+    moves = retention_plan()
+    if moves:
+        apply_moves(moves)
     print('Report: ' + str(args.output))
     return 0 if report['passed'] else 1
 

test/test_report_files.py

@@ -0,0 +1,75 @@
+import hashlib
+import json
+import os
+from pathlib import Path
+
+import pytest
+
+from tools.runtime.report_files import (apply_moves, migration_plan, report_log,
+    report_output, resolve_report_input, retention_plan)
+
+
+def test_migration_preserves_operational_data_and_report_bytes(tmp_path):
+    for name in ['documents.json', 'chunks.json', 'index_version.json', '.reindex.lock', 'zp_index.sqlite']:
+        (tmp_path/name).write_text('operational')
+    (tmp_path/'old.json').write_bytes(b'{"old":true}\n')
+    (tmp_path/'old.pytest.log').write_text('log')
+    plan = migration_plan(tmp_path)
+    assert {Path(m['source']).name for m in plan} == {'old.json', 'old.pytest.log'}
+    journal = apply_moves(plan, data_dir=tmp_path)
+    assert len(json.loads(journal.read_text())['moves']) == 2
+    for move in plan:
+        assert hashlib.sha256(Path(move['destination']).read_bytes()).hexdigest() == move['sha256']
+        assert resolve_report_input(Path(move['source']), data_dir=tmp_path) == Path(move['destination'])
+    assert (tmp_path/'documents.json').read_text() == 'operational'
+    assert not migration_plan(tmp_path)
+
+
+def test_no_overwrites_and_changed_plan_is_rejected_before_any_move(tmp_path):
+    for name in ['a.json','b.json']:
+        (tmp_path/name).write_text('{}')
+    plan = migration_plan(tmp_path)
+    (tmp_path/'b.json').write_text('{"changed":1}')
+    with pytest.raises(RuntimeError):
+        apply_moves(plan, data_dir=tmp_path)
+    assert (tmp_path/'a.json').exists()
+    output = report_output(None, 'verification', data_dir=tmp_path)
+    output.write_text('{}')
+    with pytest.raises(FileExistsError):
+        report_output(output, 'verification', data_dir=tmp_path)
+    for name in ['documents.json', 'chunks.json', 'index_version.json', 'zp_index.sqlite']:
+        with pytest.raises(ValueError):
+            report_output(tmp_path/name, 'verification', data_dir=tmp_path)
+
+
+def test_flat_legacy_output_is_redirected_and_unique_default_has_separate_log(tmp_path):
+    output = report_output(tmp_path/'old.json', 'verification', data_dir=tmp_path)
+    assert output == tmp_path/'reports/verification/old.json'
+    assert report_log(output, data_dir=tmp_path) == tmp_path/'logs/verification/old.pytest.log'
+    assert report_output(None, 'verification', data_dir=tmp_path) != report_output(None, 'verification', data_dir=tmp_path)
+    assert report_log(tmp_path/'custom/result.json', data_dir=tmp_path).parent == tmp_path/'logs/custom'
+
+
+def test_retention_archives_only_completed_verifications_and_follows_move_chain(tmp_path):
+    outputs = []
+    for i in range(8):
+        output = report_output(tmp_path/f'run{i}.json', 'verification', data_dir=tmp_path)
+        output.write_text(json.dumps({'passed': bool(i%2)}))
+        os.utime(output, (100+i,100+i))
+        report_log(output, data_dir=tmp_path).write_text('log')
+        outputs.append(output)
+    incomplete = tmp_path/'reports/verification/incomplete.json'
+    incomplete.write_text('{}')
+    experiment = report_output(None, 'rag_verified', data_dir=tmp_path)
+    experiment.write_text('{"passed":true}')
+    plan = retention_plan(tmp_path, keep=5)
+    assert len(plan) == 6
+    apply_moves(plan, data_dir=tmp_path)
+    assert incomplete.exists() and experiment.exists()
+    assert len(list((tmp_path/'reports/verification').glob('*.json'))) == 6
+    archived = resolve_report_input(outputs[0], data_dir=tmp_path)
+    assert archived.exists() and archived.parent == tmp_path/'reports/archive/verification'
+    next_target = tmp_path/'reports/archive/older'/archived.name
+    apply_moves([{'source':str(archived),'destination':str(next_target),
+                  'sha256':hashlib.sha256(archived.read_bytes()).hexdigest()}], data_dir=tmp_path)
+    assert resolve_report_input(outputs[0], data_dir=tmp_path) == next_target

test/test_verified_rag_extended.py

@@ -0,0 +1,132 @@
+import json
+
+import pytest
+
+from tools.answer_evaluation.evaluate_verified_rag import binding_matches, evaluate_case
+from tools.answer_evaluation.rag_evidence import rag_evidence_context
+from tools.answer_evaluation.verified_rag import rag_answer_issues, generate_verified_rag
+from tools.rag.rag_utils import build_context_text
+
+
+def payload(text, *, categories=None):
+    sources = [{'source_id':'S1', 'title':'Alena Nová',
+        'document_path':'pages/students/2020/alena_nova/README.md',
+        'source_url':'https://example.test/alena', 'categories':categories or [],
+        'text':text}]
+    return {'status':'ok', 'complete':None, 'sources':sources, 'context':build_context_text(sources)}
+
+
+def refusal():
+    return {'status':'no_answer','claims':[],'evidence_quotes':[],'source_urls':[]}
+
+
+@pytest.mark.parametrize('question', ['V ktorom roku začal študovať Alena Nová?',
+    'Aký je rok začiatku štúdia osoby Alena Nová?'])
+def test_explicit_study_start_cannot_be_falsely_refused(question):
+    p = payload('Dokument: Alena Nová\n\n*Rok začiatku štúdia:* 2020')
+    context = rag_evidence_context(question, p)
+    assert context['requirements']['start_year']
+    assert 'rejected_supported_academic_answer' in rag_answer_issues(refusal(), context)
+    result = generate_verified_rag(question, p, lambda messages: {'choices':[{'message':{'content':json.dumps(refusal())}}]})
+    assert not result['accepted_for_delivery']
+
+
+def test_study_start_never_comes_from_cohort_or_academic_year():
+    c = rag_evidence_context('Kedy začal študovať Alena Nová?',
+        payload('Sekcia: Bakalárska práca 2023\nNázov: Overené modelovanie textu', categories=['bp2023']))
+    assert not any('start_year' in row for row in c['facts'])
+    assert rag_answer_issues(refusal(), c) == []
+
+
+@pytest.mark.parametrize('year,unsupported', [('3000',False),('12345',True)])
+def test_explicit_year_is_honored_or_rejected_even_in_pre_rok_form(year,unsupported):
+    c = rag_evidence_context('Aký je názov diplomovej práce osoby Alena Nová pre rok '+year+'?',
+        payload('Sekcia: Diplomová práca 2025\nNázov: Overené modelovanie textu'))
+    assert c['requirements']['unsupported_year'] == unsupported
+    answer = {'status':'ok','claims':[{'person':'Alena Nová','work_type':'diploma_thesis','year':2025,
+        'title':'Overené modelovanie textu'}], 'source_urls':['https://example.test/alena']}
+    assert ('unsupported_requested_year' if unsupported else 'wrong_requested_academic_year') in rag_answer_issues(answer,c)
+
+
+def test_multiline_title_and_unlabelled_title_preceding_assignment():
+    p = payload('Sekcia: Diplomová práca 2025\nNázov práce:\n\nOverené modelovanie textu\n\nZadanie:\n1. Skúmajte dáta.')
+    c = rag_evidence_context('Aký je názov diplomovej práce osoby Alena Nová?', p)
+    assert any(row.get('title')=='Overené modelovanie textu' and row['year']==2025 for row in c['facts'])
+    p = payload('Sekcia: Diplomová práca 2025\nOverené modelovanie textu\nCieľ:\nZlepšiť model.')
+    assert any(row.get('title')=='Overené modelovanie textu' for row in rag_evidence_context('Práca?',p)['facts'])
+
+
+@pytest.mark.parametrize('text', [
+    'Sekcia: Diplomová práca 2025\nStretnutie po úspešných experimentoch\nStav:\nHotovo.',
+    'Sekcia: Diplomová práca 2025\nTéma:\nOverené modelovanie tex\n[OBSAH ZDROJA OREZANÝ]',
+    'Sekcia: Diplomová práca 2025\nNázov:\n\nĎALŠÍ DÔKAZ Z ROVNAKEJ ŠTUDENTSKEJ VETVY 2\nSekcia: Diplomová práca 2025\nStretnutie 12.3.'
+])
+def test_notes_clipped_titles_and_frame_labels_never_become_titles(text):
+    assert not any(row.get('title') for row in rag_evidence_context('Práca?',payload(text))['facts'])
+
+
+def test_sibling_scope_and_preparation_cannot_inherit_previous_work():
+    p = payload('# Diplomová práca 2025\nNázov: Overené modelovanie textu\n'
+                '# Poznámky\nNázov: Úplne iný dokument\n'
+                '# Príprava na diplomový projekt 2026\nNázov: Ďalší prípravný dokument')
+    c = rag_evidence_context('Aký je názov diplomovej práce osoby Alena Nová?',p)
+    titles = {row.get('title') for row in c['facts'] if row.get('work_type')}
+    assert titles == {'Overené modelovanie textu'}
+
+
+def test_typed_root_proposal_needs_one_matching_academic_scope():
+    p = payload('Dokument: Alena Nová\nNávrh na názov DP:\nOverené modelovanie textu\n'
+                'Sekcia: Diplomová práca 2025\nStretnutie 12.3.')
+    c = rag_evidence_context('Práca?',p)
+    assert any(r.get('title')=='Overené modelovanie textu' and r['year']==2025 for r in c['facts'])
+    p['sources'][0]['text'] += '\nSekcia: Diplomová práca 2024\nStretnutie 5.3.'
+    p['context'] = build_context_text(p['sources'])
+    assert not any(r.get('title') for r in rag_evidence_context('Práca?',p)['facts'])
+
+
+def test_quote_from_another_student_or_only_metadata_is_not_an_answer():
+    p = payload('Rok začiatku štúdia: 2020\nPoužíva Python.')
+    other = {**p['sources'][0], 'source_id':'S2','title':'Boris Nový',
+             'document_path':'pages/students/2020/boris_novy/README.md',
+             'source_url':'https://example.test/boris', 'text':'Používa Rust.'}
+    p['sources'].append(other); p['context']=build_context_text(p['sources'])
+    c = rag_evidence_context('Aký jazyk používa Alena Nová?',p)
+    a = {'status':'ok','claims':[],'source_urls':['https://example.test/boris'],
+         'evidence_quotes':[{'source_id':'S2','quote':'Používa Rust.'}]}
+    assert 'unrequested_quote_person' in rag_answer_issues(a,c)
+    a.update(source_urls=['https://example.test/alena'], evidence_quotes=[{'source_id':'S1','quote':'Názov dokumentu: Alena Nová'}])
+    assert 'metadata_quote_does_not_answer_fact' in rag_answer_issues(a,c)
+
+
+def test_evaluator_rejects_cross_person_join_wrong_tuple_and_false_refusal():
+    expected = {'person':'Alena Nová', 'title':'Overené modelovanie textu','year':2025,'work_type':'diploma_thesis'}
+    assert not binding_matches(expected,[{'person':'Alena Nová','title':expected['title']},
+        {'person':'Boris Nový','year':2025,'work_type':'diploma_thesis'}])
+    p = payload('Sekcia: Diplomová práca 2025\nNázov: Overené modelovanie textu')
+    case = {'id':'fixture','category':'fact_title','question':'Aký je názov diplomovej práce osoby Alena Nová?',
+        'should_answer':True,'expected_bindings':[expected], 'expected_source_urls':['https://example.test/alena'],
+        'expected_answer_contains':['Overené modelovanie textu']}
+    refused = {'status':'no_answer','accepted_for_delivery':True,'answer':'Nenájdené.','claims':[],'source_urls':[],'attempts':[]}
+    assert 'false_no_answer' in evaluate_case(case,p,refused)['issues']
+    bad = {**refused,'status':'ok','answer':'Overené modelovanie textu','source_urls':case['expected_source_urls'],
+           'claims':[{**expected,'year':2024}]}
+    assert 'unsupported_person_work_type_year_binding' in evaluate_case(case,p,bad)['issues']
+
+
+def test_evaluator_accepts_extra_supported_project_but_rejects_wrong_rendered_year():
+    from tools.answer_evaluation.verified_rag import render_rag_answer
+    p = payload('Sekcia: Diplomová práca 2025\nNázov: Overené modelovanie textu\n'
+                'Sekcia: Diplomový projekt 2024\nStretnutie 12.3.')
+    question = 'Aký je rok práce osoby Alena Nová?'
+    context = rag_evidence_context(question,p)
+    claims = [{'person':'Alena Nová','work_type':'diploma_thesis','year':2025},
+              {'person':'Alena Nová','work_type':'diploma_project','year':2024}]
+    parsed = {'status':'ok','claims':claims,'source_urls':['https://example.test/alena'],'evidence_quotes':[]}
+    delivered = {**parsed,'accepted_for_delivery':True,'answer':render_rag_answer(parsed,context),
+                 'attempts':[{'parsed_answer':parsed,'issues':[]}]}
+    case = {'id':'fixture','category':'fact_work_year','question':question,'should_answer':True,
+        'expected_source_urls':parsed['source_urls'], 'expected_answer_contains':['2025'],
+        'expected_bindings':[{'person':'Alena Nová','work_type':'diploma_thesis','year':2025}]}
+    assert evaluate_case(case,p,delivered)['passed']
+    delivered['answer'] += '\nPráca bola ukončená v roku 2099.'
+    assert 'delivered_text_differs_from_verified_renderer' in evaluate_case(case,p,delivered)['issues']

tools/answer_evaluation/check_context_scenarios.py

@@ -63,10 +63,10 @@ def scenarios():
 
 def main():
     parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--output', type=Path)
     args = parser.parse_args()
-    if args.output.exists():
-        parser.error('Refusing to overwrite results.')
+    from tools.runtime.report_files import report_output
+    args.output = report_output(args.output, 'context_scenarios')
     files = [Path(__file__), HERE/'graph_answer_prompt.txt', HERE/'context_policy.py', HERE/'checked_generation.py', HERE/'answer_format.py', HERE/'render_answer.py']
     hashes = {str(p): sha256_file(p) for p in files}
     declared = scenarios()

tools/answer_evaluation/check_deployment.py

@@ -18,13 +18,14 @@ from tools.rag.common import DB_FILE
 
 def main():
     parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--output', type=Path)
     parser.add_argument('--questions', type=Path, required=True)
     parser.add_argument('--audit', type=Path, required=True)
     parser.add_argument('--freeze-manifest', type=Path, required=True)
     parser.add_argument('--release-manifest', type=Path)
     args = parser.parse_args()
-    if args.output.exists(): parser.error('Refusing to overwrite deployment verification.')
+    from tools.runtime.report_files import report_output
+    args.output = report_output(args.output, 'deployment')
     base = Path('tools/graphrag/evaluation')
     release_path = args.release_manifest
     release_match = None

tools/answer_evaluation/check_rag_runtime.py

@@ -13,9 +13,10 @@ from .rag_delivery import finalize_rag_answer
 
 def main():
     parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--output', type=Path)
     args = parser.parse_args()
-    if args.output.exists(): parser.error('Refusing to overwrite smoke results.')
+    from tools.runtime.report_files import report_output
+    args.output = report_output(args.output, 'rag_runtime')
     source = Path('tools/rag/evaluation/results/test_classic_rag_frozen/rag_answers_test.json')
     historic = json.loads(source.read_text())['results']
     selected = ['q0053', 'q0060', 'q0608', 'q1147', 'q0888']

tools/answer_evaluation/check_verified_rag_guards.py

@@ -0,0 +1,96 @@
+"""Adversarial guard replay on real RAG contexts; no generative model calls.
+
+These are acceptance/rejection checks with deliberately constructed answers,
+not measured LLM accuracy. Known benchmark gold never enters production code.
+"""
+import argparse
+import copy
+import json
+from pathlib import Path
+
+from .evaluate_verified_rag import DATASET, FIELDS, binding_matches, digest
+from .rag_evidence import rag_evidence_context
+from .verified_rag import generate_verified_rag, rag_answer_issues
+from tools.runtime.report_files import report_output, resolve_report_input
+
+
+def check_cases(dataset: dict, saved: dict) -> list[dict]:
+    results = []
+    for case in dataset['cases']:
+        payload = saved[case['id']]['retrieval_context']
+        context = rag_evidence_context(case['question'], payload)
+        negative = {'status':'no_answer', 'answer':'', 'claims':[], 'evidence_quotes':[], 'source_urls':[]}
+
+        def check(name, answer, should_accept):
+            issues = rag_answer_issues(answer, context)
+            # Exercise the production generator's two-attempt rejection and
+            # deterministic renderer, with a controlled local completion stub.
+            calls = []
+            def complete(messages):
+                calls.append(messages)
+                return {'choices':[{'message':{'content':json.dumps(answer, ensure_ascii=False)}}]}
+            delivered = generate_verified_rag(case['question'], payload, complete)
+            actual = delivered['accepted_for_delivery']
+            passed = actual == should_accept and bool(not issues) == should_accept
+            if not should_accept:
+                passed = passed and len(calls) == 2 and delivered['status'] == 'error'
+            results.append({'id':case['id'], 'check':name, 'passed':passed,
+                            'expected_acceptance':should_accept, 'issues':issues})
+
+        if not case['should_answer']:
+            check('accept_supported_missing_fact_refusal', negative, True)
+            continue
+        bindings = case['expected_bindings']
+        if not bindings or not all(binding_matches(b, context['facts']) for b in bindings):
+            continue
+        claims = [{k:b[k] for k in FIELDS if k in b} for b in bindings]
+        for claim in claims:
+            matching = [row for row in context['facts'] if binding_matches(claim, [row])]
+            for field in ['title', 'year']:
+                values = {row[field] for row in matching if field in row}
+                if context['requirements'][field] and field not in claim and len(values) == 1:
+                    claim[field] = next(iter(values))
+        urls = sorted({row['source_url'] for row in context['facts'] if any(
+            row['person'] == b['person'] for b in bindings)})
+        answer = {'status':'ok', 'answer':'', 'claims':claims, 'evidence_quotes':[], 'source_urls':urls}
+        check('accept_supported_tuple', answer, True)
+        check('reject_false_refusal', negative, False)
+        for field, value in [('year',9876), ('title','Nepodložený názov z cudzieho dokumentu'),
+                             ('person','Neexistujúca Osoba'), ('work_type','doctoral_thesis')]:
+            bad = copy.deepcopy(answer)
+            if field == 'work_type' and bad['claims'][0].get(field) == value:
+                value = 'bachelor_thesis'
+            bad['claims'][0][field] = value
+            check('reject_wrong_'+field, bad, False)
+        bad = copy.deepcopy(answer)
+        bad['source_urls'] = ['https://example.invalid/not-provided']
+        check('reject_wrong_citation', bad, False)
+    return results
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--input', required=True, type=Path)
+    parser.add_argument('--dataset', type=Path, default=DATASET)
+    parser.add_argument('--output', type=Path)
+    args = parser.parse_args()
+    source = resolve_report_input(args.input)
+    saved_report = json.loads(source.read_text())
+    dataset = json.loads(args.dataset.read_text())
+    assert saved_report['configuration']['dataset_sha256'] == digest(args.dataset)
+    saved = {r['id']:r for r in saved_report['results']}
+    results = check_cases(dataset, saved)
+    output = report_output(args.output, 'rag_guards')
+    summary = {'checks':len(results), 'passed':sum(r['passed'] for r in results),
+               'contexts':len(saved), 'llm_calls':0,
+               'note':'Controlled fault injection, not an LLM accuracy measurement.'}
+    output.write_text(json.dumps({'method':__doc__, 'input':str(source), 'input_sha256':digest(source),
+        'summary':summary, 'results':results}, ensure_ascii=False, indent=2)+'\n')
+    print(json.dumps(summary))
+    print('Report: '+str(output))
+    if any(not r['passed'] for r in results):
+        raise SystemExit(1)
+
+
+if __name__ == '__main__':
+    main()

tools/answer_evaluation/evaluate_verified_rag.py

@@ -0,0 +1,218 @@
+"""Audit the real checked RAG path on a predeclared, known regression sample.
+
+Default mode only retrieves locally. `--mode deliver` explicitly invokes the
+configured backend model. Expected answers are used here, never sent to /answer.
+"""
+from __future__ import annotations
+
+import argparse
+from collections import Counter
+from concurrent.futures import ThreadPoolExecutor, as_completed
+import hashlib
+import json
+import os
+from pathlib import Path
+import time
+import urllib.request
+
+from dotenv import load_dotenv
+
+from tools.answer_evaluation.context_policy import normalized, citation_check, allowed_urls
+from tools.answer_evaluation.rag_evidence import rag_evidence_context
+from tools.answer_evaluation.verified_rag import rag_answer_issues, render_rag_answer
+from tools.graphrag.evaluation.source_oracle import DEFAULT_WIKI, load_source_oracle
+from tools.rag.evaluation.rag_metrics import expected_phrase_matches
+from tools.runtime.report_files import report_output, resolve_report_input
+
+ROOT = Path(__file__).resolve().parents[2]
+DATASET = ROOT / 'tools/rag/evaluation/json_files/verified_rag_regression.json'
+FIELDS = ['person', 'work_type', 'title', 'year', 'start_year']
+
+
+def digest(path):
+    return hashlib.sha256(Path(path).read_bytes()).hexdigest()
+
+
+def binding_matches(expected: dict, rows: list[dict]) -> bool:
+    # Study start can be a separate fact about the same person; the academic
+    # fields must all occur together in one actual academic row.
+    academic = {k: expected[k] for k in FIELDS if k in expected and k != 'start_year'}
+    academic_ok = any(all(normalized(row.get(k)) == normalized(v) for k, v in academic.items()) for row in rows)
+    start_ok = 'start_year' not in expected or any(
+        normalized(row.get('person')) == normalized(expected['person']) and
+        row.get('start_year') == expected['start_year'] for row in rows)
+    return academic_ok and start_ok
+
+
+def source_audit(case: dict, oracle: dict) -> list[str]:
+    issues = []
+    for item in case['source_checks']:
+        if digest(DEFAULT_WIKI / item['document_path']) != item['sha256']:
+            issues.append('changed_source_document')
+    for binding in case['expected_bindings']:
+        person = oracle['people'].get(binding['person'], {})
+        rows = [{'person': binding['person'], 'work_type': a['academic_type'], **a}
+                for a in person.get('academics', []) if a['document_path'] == binding['document_path']]
+        rows.append({'person': binding['person'], 'start_year': person.get('start_year')})
+        if not binding_matches(binding, rows):
+            issues.append('gold_binding_not_in_original_markdown')
+        if 'start_year' in binding and person.get('start_year_basis') != 'explicit_text':
+            issues.append('study_start_not_explicit')
+    return issues
+
+
+def evaluate_case(case: dict, payload: dict, delivered: dict | None = None) -> dict:
+    context = rag_evidence_context(case['question'], payload)
+    source_urls = allowed_urls(payload)
+    missing = [b for b in case['expected_bindings'] if not binding_matches(b, context['facts'])]
+    retrieval = {'expected_sources_found': bool(source_urls & set(case['expected_source_urls'])),
+                 'missing_bindings': missing,
+                 'expected_values_in_context': [v for v in case['expected_answer_contains']
+                     if expected_phrase_matches(v, payload.get('context', ''))]}
+    row = {'id': case['id'], 'category': case['category'], 'question': case['question'],
+           'should_answer': case['should_answer'], 'retrieval': retrieval,
+           'typed_expected': bool(case['expected_bindings']),
+           'index_version': payload.get('index_version')}
+    if delivered is None:
+        return row
+    issues = []
+    accepted = delivered.get('accepted_for_delivery') is True
+    status = delivered.get('status')
+    attempts = delivered.get('attempts', [])
+    parsed = attempts[-1].get('parsed_answer') if attempts else None
+    if accepted and parsed:
+        issues.extend(rag_answer_issues(parsed, context))
+        if delivered.get('answer') != render_rag_answer(parsed, context):
+            issues.append('delivered_text_differs_from_verified_renderer')
+        if delivered.get('claims', []) != parsed.get('claims', []):
+            issues.append('delivered_claims_differ_from_verified_selection')
+    if accepted and not citation_check(delivered, source_urls)['citations_ok']:
+        issues.append('delivered_unsupported_citation')
+    claims = delivered.get('claims', [])
+    if accepted:
+        # Validate every delivered claim against the actual retrieved tuple.
+        # Additional supported projects of the same person are not a wrong
+        # thesis year. Required benchmark tuples are checked separately below.
+        issues.extend(rag_answer_issues({'status':status, 'claims':claims,
+            'source_urls':delivered.get('source_urls', []),
+            'evidence_quotes':(parsed or {}).get('evidence_quotes', [])}, context))
+    if case['should_answer']:
+        if not accepted:
+            issues.append('delivery_blocked')
+        elif status != 'ok':
+            issues.append('false_no_answer' if status == 'no_answer' else 'nonpositive_answer')
+        if not all(binding_matches(b, claims) for b in case['expected_bindings']):
+            issues.append('missing_expected_binding')
+        # Exclude sources from factual scoring: a name/year in a URL cannot
+        # supply the requested fact. Quotes must cover the expected subject.
+        text = delivered.get('answer', '').split('\nZdroje:', 1)[0]
+        missing_values = [v for v in case['expected_answer_contains'] if not expected_phrase_matches(v, text)]
+        if missing_values:
+            issues.append('missing_expected_answer_value')
+        quotes = (parsed or {}).get('evidence_quotes', [])
+        if quotes:
+            content = '\n'.join(q['quote'] for q in quotes)
+            names = {s.get('title') for s in context['source_blocks']}
+            # Names in source headers are attribution, not answers to a factual
+            # question. Value coverage checks relevance beyond literal copying.
+            relevant = [v for v in case['expected_answer_contains'] if v not in names]
+            row['quote_expected_value_coverage'] = {v: expected_phrase_matches(v, content) for v in relevant}
+    elif not accepted or status != 'no_answer':
+        issues.append('incorrect_negative_answer')
+    row.update(passed=not issues, issues=sorted(set(issues)), delivered=delivered,
+               first_attempt_accepted=bool(attempts and not attempts[0].get('issues')))
+    return row
+
+
+def summarize(rows: list[dict]) -> dict:
+    counted = [r for r in rows if 'passed' in r]
+    total = len(counted)
+    passed = sum(r['passed'] for r in counted)
+    return {'total': len(rows), 'answers_evaluated': total, 'passed': passed,
+            'success_percent': round(100 * passed / total, 2) if total else None,
+            'false_no_answers': sum('false_no_answer' in r['issues'] for r in counted),
+            'blocked_deliveries': sum('delivery_blocked' in r['issues'] for r in counted),
+            'first_attempt_accepted': sum(r.get('first_attempt_accepted', False) for r in counted),
+            'failure_reasons': dict(Counter(i for r in counted for i in r['issues'])),
+            'source_coverage': sum(r['retrieval']['expected_sources_found'] for r in rows),
+            'answerable_source_coverage': {'passed': sum(r['retrieval']['expected_sources_found'] for r in rows if r['should_answer']),
+                                           'total': sum(r['should_answer'] for r in rows)},
+            'typed_binding_coverage': {'passed': sum(not r['retrieval']['missing_bindings'] for r in rows if r.get('typed_expected')),
+                                     'total': sum(r.get('typed_expected', False) for r in rows)},
+            'categories': {c: {'total': sum(r['category'] == c for r in rows),
+                'passed': sum(r.get('passed', False) for r in counted if r['category'] == c)} for c in sorted({r['category'] for r in rows})}}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--mode', choices=['retrieve', 'deliver', 'replay'], default='retrieve')
+    parser.add_argument('--input', type=Path, help='Saved report to regrade without model calls')
+    parser.add_argument('--dataset', type=Path, default=DATASET)
+    parser.add_argument('--backend', default='http://127.0.0.1:8000')
+    parser.add_argument('--output', type=Path)
+    args = parser.parse_args()
+    load_dotenv(ROOT / '.env', override=False)
+    dataset = json.loads(args.dataset.read_text())
+    assert digest(ROOT / dataset['source_file']) == dataset['source_sha256'], 'Historical baseline changed'
+    oracle = load_source_oracle()
+    audit = {c['id']: source_audit(c, oracle) for c in dataset['cases']}
+    assert not any(audit.values()), 'Expected answer/source audit failed'
+    output = report_output(args.output, 'rag_verified')
+    files = [Path(__file__), ROOT/'tools/answers/answer_delivery.py', ROOT/'app/routes.py',
+             *[(ROOT/'tools/answer_evaluation'/name) for name in ['verified_rag.py','rag_evidence.py','context_policy.py']],
+             ROOT/'tools/rag/rag_utils.py', ROOT/'tools/rag/llm_context.py']
+    metadata = {'method': dataset['method'], 'mode': args.mode, 'dataset_sha256': digest(args.dataset),
+                'limitations': dataset.get('limitations', []),
+                'source_audit': audit, 'source_corpus': oracle['source_corpus'],
+                'code_hashes': {str(p.relative_to(ROOT)): digest(p) for p in files},
+                'model_calls': args.mode == 'deliver', 'backend': args.backend,
+                'model': os.getenv('ANSWER_MODEL', 'model120-fast') if args.mode == 'deliver' else None}
+    output.with_suffix('.declaration.json').write_text(json.dumps(metadata, ensure_ascii=False, indent=2)+'\n')
+    key = os.getenv('SEARCH_API_KEY', '')
+    saved = None
+    if args.mode == 'replay':
+        if not args.input:
+            parser.error('--input is required for replay')
+        saved = {r['id']: r for r in json.loads(resolve_report_input(args.input).read_text())['results']}
+
+    def run(case):
+        started = time.monotonic()
+        if saved is not None:
+            old = saved[case['id']]
+            return evaluate_case(case, old['retrieval_context'], old.get('delivered')) | {'retrieval_context': old['retrieval_context']}
+        endpoint = '/answer' if args.mode == 'deliver' else '/rag'
+        body = {'query': case['question'], 'limit': 5}
+        if args.mode == 'deliver':
+            body.update(engine='rag', include_diagnostics=True)
+        req = urllib.request.Request(args.backend.rstrip('/')+endpoint, data=json.dumps(body).encode(),
+            headers={'Content-Type': 'application/json', 'X-API-Key': key})
+        try:
+            with urllib.request.urlopen(req, timeout=300) as response:
+                data = json.load(response)
+            payload = data.pop('retrieval_context') if args.mode == 'deliver' else data
+            result = evaluate_case(case, payload, data if args.mode == 'deliver' else None)
+            result.update(retrieval_context=payload, seconds=round(time.monotonic()-started, 2))
+            return result
+        except Exception as error:
+            # Count operational failures in the denominator; never retry only
+            # selected failed questions or expose raw HTTP/token-bearing errors.
+            return {'id': case['id'], 'category': case['category'], 'question': case['question'],
+                    'should_answer': case['should_answer'], 'passed': False, 'issues': ['request_or_evaluation_error'],
+                    'typed_expected': bool(case['expected_bindings']),
+                    'error_type': type(error).__name__, 'retrieval': {'expected_sources_found': False, 'missing_bindings': case['expected_bindings']}}
+
+    rows = []
+    with ThreadPoolExecutor(max_workers=2) as pool:
+        futures = {pool.submit(run, case): case for case in dataset['cases']}
+        for future in as_completed(futures):
+            rows.append(future.result())
+            result = {'configuration': metadata, 'summary': summarize(rows), 'results': sorted(rows, key=lambda r: r['id'])}
+            output.write_text(json.dumps(result, ensure_ascii=False, indent=2)+'\n')
+            print(f'{len(rows)}/{len(dataset["cases"])} {rows[-1]["id"]}: {rows[-1].get("issues", "retrieved")}', flush=True)
+    assert metadata['code_hashes'] == {name: digest(ROOT/name) for name in metadata['code_hashes']}, 'Code changed during measurement'
+    print(json.dumps(result['summary'], ensure_ascii=False))
+    print('Report: '+str(output))
+
+
+if __name__ == '__main__':
+    main()

tools/answer_evaluation/rag_delivery.py

@@ -34,9 +34,11 @@ def main():
 
     parser = argparse.ArgumentParser(description='Replay saved RAG outputs through the delivery layer; no model calls.')
     parser.add_argument('--input', type=Path, required=True)
-    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--output', type=Path)
     args = parser.parse_args()
-    if args.output.exists(): parser.error('Refusing to overwrite a replay.')
+    from tools.runtime.report_files import report_output, resolve_report_input
+    args.output = report_output(args.output, 'rag_replay')
+    args.input = resolve_report_input(args.input)
     original = json.loads(args.input.read_text())
     source = Path(original['configuration']['source_file'])
     assert sha256_file(source) == original['configuration']['source_sha256']

tools/answer_evaluation/rag_evidence.py

@@ -44,7 +44,7 @@ def work_type(text: str) -> str | None:
 def academic_scope(path: str) -> tuple[str, int | None] | None:
     for heading in reversed(path.split('>')):
         if normalized(heading).startswith(('priprava na ', 'priprava k ')):
-            continue
+            return None
         kind = work_type(heading)
         if kind:
             years = [int(year) for year in re.findall(r'(?<!\d)([12]\d{3})(?!\d)', heading)]
@@ -134,6 +134,7 @@ def rag_evidence_context(question: str, payload: dict) -> dict:
                 'document_path': source['document_path']}
         windows = []
         scope = None
+        heading_stack = []
         lines = []
         def flush():
             if lines:
@@ -141,23 +142,42 @@ def rag_evidence_context(question: str, payload: dict) -> dict:
                 lines.clear()
         for line in source['body'].splitlines():
             heading = re.match(r'^(?:Sekcia(?: dôkazu)?:|#{1,6}\s)(.*)', line)
-            if line.startswith('Dokument:'):
-                flush(); scope = None
+            if line.startswith(('RELEVANTNÁ SEKCIA ', 'DÔKAZ Z ', 'ĎALŠÍ DÔKAZ ',
+                                'ĎALŠÍ RELEVANTNÝ ', 'NAJRELEVANTNEJŠÍ ', 'ZAČIATOK RELEVANTNEJ ')):
+                flush(); scope = None; heading_stack.clear()
+            elif line.startswith('Dokument:'):
+                flush(); scope = None; heading_stack.clear()
             elif heading:
                 flush()
+                if line.startswith('Sekcia'):
+                    scope = academic_scope(heading[1])
+                    heading_stack = [(0, scope)]
+                else:
+                    level = len(line) - len(line.lstrip('#'))
+                    while heading_stack and heading_stack[-1][0] >= level:
+                        heading_stack.pop()
                     detected = academic_scope(heading[1])
-                if line.startswith('Sekcia') or detected:
-                    scope = detected
+                    scope = (None if normalized(heading[1]).startswith(('priprava na ', 'priprava k ')) else
+                             detected or (heading_stack[-1][1] if heading_stack else None))
+                    heading_stack.append((level, scope))
             else:
                 lines.append(line)
         flush()
         unscoped = []
         for section, text in windows:
             titles = []
-            for line in text.splitlines():
+            text_lines = text.splitlines()
+            for index, line in enumerate(text_lines):
                 if ':' in line:
                     label, value = line.split(':', 1)
                     if normalized(label) in TITLE_LABELS:
+                        value_index = index
+                        if not value.strip():
+                            value_index = next((j for j in range(index+1, len(text_lines)) if text_lines[j].strip()), index)
+                            value = text_lines[value_index] if value_index != index else ''
+                        following = next((v.strip() for v in text_lines[value_index+1:] if v.strip()), '')
+                        if following == '[OBSAH ZDROJA OREZANÝ]' or value.strip() == '[OBSAH ZDROJA OREZANÝ]':
+                            continue
                         title = clean_title(value)
                         if title:
                             titles.append(title)
@@ -166,6 +186,21 @@ def rag_evidence_context(question: str, payload: dict) -> dict:
                     facts.append({**base, 'start_year': int(start[1])})
             if section:
                 kind, year = section
+                if not titles:
+                    # Wiki also puts an unlabelled title immediately after its
+                    # academic heading (sometimes preceded by explicit study
+                    # start). Never promote meeting notes, tasks or list items.
+                    lead_lines = [v.strip() for v in text_lines if v.strip() and
+                                  not normalized(v).startswith('rok zaciatku studia')]
+                    lead = lead_lines[0] if lead_lines else ''
+                    next_line = lead_lines[1] if len(lead_lines) > 1 else ''
+                    if (lead and re.match(r'^(?:Cie[lľe]+\s*:|Zadanie\s*:|Myšlienky\s*:|1[.)]\s)', next_line, re.I) and
+                        not re.match(r'^(?:[-*+]\s|\d+[.)]\s|#{1,6}\s)', lead) and
+                        not normalized(lead).startswith(('stretnutie','virtualne stretnutie','stav','ulohy',
+                            'ciel','zadanie','zasobnik','rok ','zaciatok studia','projekt ','praca ','tema na '))):
+                        title = clean_title(lead)
+                        if title and len(normalized(title).split()) >= 3:
+                            titles.append(title)
                 record = {**base, 'work_type': kind}
                 if year is not None:
                     record['year'] = year
@@ -180,6 +215,18 @@ def rag_evidence_context(question: str, payload: dict) -> dict:
                 if section and normalized(title) in normalized(text):
                     facts.append({**base, 'title': title, 'work_type': section[0],
                                   **({'year': section[1]} if section[1] is not None else {})})
+            # A root label explicitly naming BP/DP licenses only that type;
+            # attach it only if the delivered evidence has exactly one such
+            # academic scope. Taxonomy categories still cannot supply a year.
+            for label, kind in [('navrh na nazov dp', 'diploma_thesis'),
+                                ('navrh na nazov bp', 'bachelor_thesis')]:
+                explicit = any(re.search(r'\b'+re.escape(label)+r'\s+'+re.escape(normalized(title))+r'\b',
+                                         normalized(text)) for section, text in windows if section is None)
+                scopes = {s for s, _ in windows if s and s[0] == kind}
+                if explicit and len(scopes) == 1:
+                    only = next(iter(scopes))
+                    facts.append({**base, 'title': title, 'work_type': kind,
+                                  **({'year': only[1]} if only[1] is not None else {})})
         for field, values in [('keyword', source.get('tags', [])), ('category', source.get('categories', [])),
                               ('document_author', [source['author']] if source.get('author') else [])]:
             facts.extend({**base, field: value} for value in values)
@@ -187,17 +234,19 @@ def rag_evidence_context(question: str, payload: dict) -> dict:
     requested = [person for person in people if text_contains_value(question, person, person=True)]
     expected_types = requested_types(question, requested)
     query = normalized(question)
-    academic = bool(TYPE_PATTERN.search(query) or re.search(r'\brok\w* prac\w*\b', query))
+    academic = bool(TYPE_PATTERN.search(query) or re.search(r'\b(?:rok\w* prac\w*|nazov\w* prac\w*)\b', query))
+    start_year = bool(re.search(r'\b(?:zaciat\w* stud\w*|zacal\w* stud\w*)\b', query))
     requirements = {'people': requested, 'work_types': expected_types,
                     'academic': academic, 'title': 'nazov' in query or 'tema' in query,
                     'year': academic and ('rok' in query or 'porovnaj' in query),
-                    'start_year': 'zaciat' in query and 'studi' in query}
-    requirements['years'] = ([int(value) for value in re.findall(r'\b(?:v roku|z roku|roku)\s+([12]\d{3})\b', query)]
-                              if academic and not requirements['start_year'] else [])
-    requirements['unsupported_year'] = academic and bool(re.search(r'\b(?:v roku|z roku|roku)\s+\d{5,}\b', query))
+                    'start_year': start_year}
+    stated_years = re.findall(r'\b(?:v roku|z roku|pre rok|na rok|roku|rok)\s+(\d+)\b', query) if academic else []
+    requirements['years'] = ([int(value) for value in stated_years if len(value) == 4]
+                              if not requirements['start_year'] else [])
+    requirements['unsupported_year'] = any(len(value) != 4 for value in stated_years)
     if requested:
         facts = [row for row in facts if row['person'] in requested]
-    return {**{key: payload.get(key) for key in ['status', 'reason', 'complete', 'truncated',
+    return {'question': question, **{key: payload.get(key) for key in ['status', 'reason', 'complete', 'truncated',
              'truncation_reasons', 'index_version']}, 'sources': payload.get('sources', []),
              'facts': facts, 'source_blocks': blocks, 'requirements': requirements,
              'aggregation': None, 'answer_candidates': []}

tools/answer_evaluation/regrade_citations.py

@@ -14,9 +14,11 @@ from .context_policy import citation_check, allowed_urls
 def main():
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument('--input', type=Path, action='append', required=True)
-    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--output', type=Path)
     args = parser.parse_args()
-    if args.output.exists(): parser.error('Refusing to overwrite regrading.')
+    from tools.runtime.report_files import report_output, resolve_report_input
+    args.output = report_output(args.output, 'citations')
+    args.input = [resolve_report_input(path) for path in args.input]
     reports = []
     for path in args.input:
         data = json.loads(path.read_text()); rows = []

tools/answer_evaluation/verified_rag.py

@@ -39,6 +39,8 @@ Ak sa otázka pýta na prácu osoby, vyplň claims: person, work_type a požadov
 Použi presný názov, typ a rok TOHO ISTÉHO záznamu danej osoby. Rok kategórie nie je rok práce.
 Pri porovnávaní odpovedz za každú požadovanú osobu; rešpektuj requirements.work_types.
 start_year je len explicitný začiatok štúdia, nikdy rok z adresára.
+Pri otázke na začiatok štúdia vyplň claims so start_year; pri rozlíšení rokov
+uveď start_year aj year a work_type. Citácia ani nesúvisiaci výňatok údaj nenahrádzajú.
 Pri všeobecnej otázke môžeš namiesto claims vybrať evidence_quotes: source_id a presný
 doslovný výňatok, ktorý odpovedá na otázku. Výňatok nesmieš parafrázovať ani meniť čísla.
 Cituj iba source_urls zo sources, ktoré dokladajú vybrané údaje. Názov profilu nie je názov práce.
@@ -92,7 +94,7 @@ def rag_answer_issues(answer: dict, context: dict) -> list[str]:
             issues.append('wrong_or_missing_requested_work_type')
         if requirements['years'] and claim.get('year') is not None and claim['year'] not in requirements['years']:
             issues.append('wrong_requested_academic_year')
-    if requirements['academic'] and answer['status'] == 'ok':
+    if (requirements['academic'] or requirements['start_year']) and answer['status'] == 'ok':
         if requirements['unsupported_year']:
             issues.append('unsupported_requested_year')
         if not requested:
@@ -102,7 +104,7 @@ def rag_answer_issues(answer: dict, context: dict) -> list[str]:
             for key in ['title', 'year', 'start_year']:
                 if requirements[key] and not any(claim.get(key) is not None for claim in matching):
                     issues.append('missing_requested_' + key)
-    if answer['status'] == 'no_answer' and requirements['academic'] and requested:
+    if answer['status'] == 'no_answer' and (requirements['academic'] or requirements['start_year']) and requested:
         # Reject an unsupported refusal only if every requested academic tuple
         # is explicitly present. Never invent a value to fill an absent field.
         supported = all(any(row['person'] == person and
@@ -110,7 +112,10 @@ def rag_answer_issues(answer: dict, context: dict) -> list[str]:
             (not requirements['years'] or row.get('year') in requirements['years']) and
             all(not requirements[key] or row.get(key) is not None for key in ['title', 'year'])
             for row in facts) for person in requested)
-        if supported and (requirements['title'] or requirements['year']) and not requirements['start_year'] and not requirements['unsupported_year']:
+        start_supported = all(any(row['person'] == person and row.get('start_year') is not None
+                                  for row in facts) for person in requested)
+        if (supported and (requirements['title'] or requirements['year'] or requirements['start_year'])
+                and (not requirements['start_year'] or start_supported) and not requirements['unsupported_year']):
             issues.append('rejected_supported_academic_answer')
     cited = set(answer.get('source_urls', []))
     blocks = {row['source_id']: row for row in context['source_blocks']}
@@ -120,6 +125,15 @@ def rag_answer_issues(answer: dict, context: dict) -> list[str]:
             issues.append('unsupported_evidence_quote')
         elif source.get('source_url') not in cited:
             issues.append('missing_quote_source')
+        if source:
+            if (requested and source.get('document_path', '').startswith(('pages/students/', 'pages/interns/')) and
+                    not any(source.get('document_path', '').startswith(row['document_path'].removesuffix('README.md'))
+                            for row in context['source_blocks'] if row.get('title') in requested)):
+                issues.append('unrequested_quote_person')
+            identity_question = bool(re.search(r'\b(?:url|odkaz\w*|dokument\w*)\b', normalized(context.get('question', ''))))
+            if (not identity_question and
+                    ' '.join(quote['quote'].split()) not in ' '.join(source['body'].split())):
+                issues.append('metadata_quote_does_not_answer_fact')
         # A quoted embedded URL is not automatically an allowed citation.
         if not citation_check({'answer': quote['quote'], 'source_urls': []}, allowed_urls(context))['citations_ok']:
             issues.append('unsupported_quote_citation')
@@ -139,7 +153,9 @@ def render_rag_answer(answer: dict, context: dict) -> str:
             values = [WORK_LABELS[claim['work_type']]] if claim.get('work_type') else []
             values += [f'{label}: {claim[key]}' for key, label in labels.items() if claim.get(key) is not None]
             lines.append(claim['person'] + ' — ' + '; '.join(values) + '.')
-        lines += ['Zo zdroja: „' + quote['quote'] + '“' for quote in answer.get('evidence_quotes', [])]
+        sources = {row['source_id']: row for row in context['source_blocks']}
+        lines += ['Zo zdroja ' + str(sources[quote['source_id']].get('title') or quote['source_id']) +
+                  ': „' + quote['quote'] + '“' for quote in answer.get('evidence_quotes', [])]
         if answer.get('count') is not None:
             lines.append(f'Počet: {answer["count"]}.')
     elif answer['status'] == 'no_answer':

tools/answers/answer_delivery.py

@@ -35,7 +35,7 @@ def complete_with_configured_model(messages: list[dict]) -> dict:
 
 
 def deliver_answer(question: str, *, engine: str = 'rag', limit: int = 5,
-                   complete=None, db_file: Path = DB_FILE) -> dict:
+                   complete=None, db_file: Path = DB_FILE, include_diagnostics: bool = False) -> dict:
     if not _capacity.acquire(blocking=False):
         return {'status': 'error', 'reason': 'answer_capacity_exceeded',
                 'accepted_for_delivery': False, 'answer': 'Agent práve spracúva iné odpovede. Skúste to o chvíľu.'}
@@ -44,6 +44,7 @@ def deliver_answer(question: str, *, engine: str = 'rag', limit: int = 5,
         if engine == 'rag':
             retrieved = compact_rag_payload(build_rag_context(db_file, question, limit=limit))
             result = generate_verified_rag(question, retrieved, model)
+            context = retrieved
         elif engine == 'graphrag':
             context = graph_context(search_graph(question))
             prompt = (Path(__file__).resolve().parents[1] / 'answer_evaluation/graph_answer_prompt.txt').read_text()
@@ -59,6 +60,7 @@ def deliver_answer(question: str, *, engine: str = 'rag', limit: int = 5,
                     'index_version': context.get('index_version'), 'attempts': generated['attempts']}
         else:
             raise ValueError('Unsupported answer engine')
-        return {**result, 'engine': engine}
+        return {**result, 'engine': engine,
+                **({'retrieval_context': context} if include_diagnostics else {})}
     finally:
         _capacity.release()

tools/rag/evaluation/results/verified_rag_2026-10-27/summary.json

@@ -0,0 +1,447 @@
+{
+  "date": "2026-10-27",
+  "scope": "Classic checked RAG regression and operational report organization. GraphRAG baseline/results unchanged.",
+  "status": {
+    "local_wider_verification": "complete",
+    "reports_and_logs": "complete",
+    "wider_80_model_answers": "awaiting_explicit_consent"
+  },
+  "dataset": {
+    "path": "tools/rag/evaluation/json_files/verified_rag_regression.json",
+    "sha256": "07d6dcaaf546c1df5a2a6c8074ad7502122175d4226a194cd782f5b5031a0b07"
+  },
+  "method": "80 predeclared known regression questions from nine strata, including 10 missing-fact negatives. No gold sent to production model. Not an independent TEST.",
+  "retrieval_and_typed_evidence": {
+    "total": 80,
+    "answers_evaluated": 0,
+    "passed": 0,
+    "success_percent": null,
+    "false_no_answers": 0,
+    "blocked_deliveries": 0,
+    "first_attempt_accepted": 0,
+    "failure_reasons": {},
+    "source_coverage": 80,
+    "answerable_source_coverage": {
+      "passed": 70,
+      "total": 70
+    },
+    "typed_binding_coverage": {
+      "passed": 50,
+      "total": 52
+    },
+    "categories": {
+      "disambiguation_year": {
+        "total": 8,
+        "passed": 0
+      },
+      "fact_start_year": {
+        "total": 8,
+        "passed": 0
+      },
+      "fact_title": {
+        "total": 10,
+        "passed": 0
+      },
+      "fact_work_year": {
+        "total": 10,
+        "passed": 0
+      },
+      "keyword_grounding": {
+        "total": 8,
+        "passed": 0
+      },
+      "multi_document": {
+        "total": 8,
+        "passed": 0
+      },
+      "single_document_synthesis": {
+        "total": 8,
+        "passed": 0
+      },
+      "specific_fact": {
+        "total": 10,
+        "passed": 0
+      },
+      "unanswerable_missing_fact": {
+        "total": 10,
+        "passed": 0
+      }
+    }
+  },
+  "unresolved_source_conventions": [
+    {
+      "id": "q0074",
+      "question": "V ktorom roku je pri osobe Dárius Lindvai uvedená diplomová práca?",
+      "missing_bindings": [
+        {
+          "person": "Dárius Lindvai",
+          "work_type": "diploma_thesis",
+          "document_path": "pages/students/2016/darius_lindvai/README.md",
+          "heading_line": null,
+          "title_line": 14,
+          "year": 2021,
+          "year_basis": "frontmatter_category"
+        }
+      ]
+    },
+    {
+      "id": "q1168",
+      "question": "Porovnaj záznamy osôb Dárius Lindvai a Tetiana Mohorian. Pri osobe Dárius Lindvai použi záznam typu „diplomová práca“ a pri osobe Tetiana Mohorian záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
+      "missing_bindings": [
+        {
+          "person": "Dárius Lindvai",
+          "work_type": "diploma_thesis",
+          "document_path": "pages/students/2016/darius_lindvai/README.md",
+          "heading_line": null,
+          "title_line": 14,
+          "title": "Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
+          "year": 2021,
+          "year_basis": "frontmatter_category"
+        }
+      ]
+    }
+  ],
+  "source_note": "Darius Lindvai diploma year 2021 comes from frontmatter category dp2021 without an explicit diploma heading. Runtime deliberately does not infer academic year from taxonomy.",
+  "guard_replay": {
+    "summary": {
+      "checks": 360,
+      "passed": 360,
+      "contexts": 80,
+      "llm_calls": 0,
+      "note": "Controlled fault injection, not an LLM accuracy measurement."
+    },
+    "by_check": {
+      "accept_supported_tuple": {
+        "passed": 50,
+        "total": 50
+      },
+      "reject_false_refusal": {
+        "passed": 50,
+        "total": 50
+      },
+      "reject_wrong_year": {
+        "passed": 50,
+        "total": 50
+      },
+      "reject_wrong_title": {
+        "passed": 50,
+        "total": 50
+      },
+      "reject_wrong_person": {
+        "passed": 50,
+        "total": 50
+      },
+      "reject_wrong_work_type": {
+        "passed": 50,
+        "total": 50
+      },
+      "reject_wrong_citation": {
+        "passed": 50,
+        "total": 50
+      },
+      "accept_supported_missing_fact_refusal": {
+        "passed": 10,
+        "total": 10
+      }
+    }
+  },
+  "authorized_five_model_smoke": {
+    "summary": {
+      "total": 5,
+      "answers_evaluated": 5,
+      "passed": 5,
+      "success_percent": 100.0,
+      "false_no_answers": 0,
+      "blocked_deliveries": 0,
+      "first_attempt_accepted": 5,
+      "failure_reasons": {},
+      "source_coverage": 5,
+      "answerable_source_coverage": {
+        "passed": 4,
+        "total": 4
+      },
+      "typed_binding_coverage": {
+        "passed": 3,
+        "total": 3
+      },
+      "categories": {
+        "disambiguation_year": {
+          "total": 1,
+          "passed": 1
+        },
+        "fact_title": {
+          "total": 1,
+          "passed": 1
+        },
+        "multi_document": {
+          "total": 1,
+          "passed": 1
+        },
+        "source_url": {
+          "total": 1,
+          "passed": 1
+        },
+        "unanswerable_missing_fact": {
+          "total": 1,
+          "passed": 1
+        }
+      }
+    },
+    "model": "model120-fast",
+    "provider": "https://ui.tukekemt.xyz",
+    "method": "Five actual /answer requests with at most one repair, using previously authorized questions. Same stored responses regraded after correcting evaluator false positives on additional supported academic works. No new generation during regrading.",
+    "results": [
+      {
+        "id": "q0053",
+        "question": "Aký je názov diplomová práca osoby Lukáš Pokrývka?",
+        "passed": true,
+        "issues": [],
+        "answer": "Lukáš Pokrývka — diplomová práca; názov práce: Paralelné trénovanie neurónových sietí; rok práce: 2021.\nVýpis je neúplný.\nZdroje: https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
+        "claims": [
+          {
+            "person": "Lukáš Pokrývka",
+            "title": "Paralelné trénovanie neurónových sietí",
+            "year": 2021,
+            "work_type": "diploma_thesis"
+          }
+        ],
+        "source_urls": [
+          "https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"
+        ],
+        "accepted_for_delivery": true
+      },
+      {
+        "id": "q0060",
+        "question": "Rozlíš rok začiatku štúdia a rok práce osoby Lukáš Pokrývka.",
+        "passed": true,
+        "issues": [],
+        "answer": "Lukáš Pokrývka — diplomová práca; rok práce: 2021; začiatok štúdia: 2016.\nLukáš Pokrývka — tímový projekt; rok práce: 2019; začiatok štúdia: 2016.\nLukáš Pokrývka — diplomový projekt; rok práce: 2020; začiatok štúdia: 2016.\nVýpis je neúplný.\nZdroje: https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
+        "claims": [
+          {
+            "person": "Lukáš Pokrývka",
+            "year": 2021,
+            "start_year": 2016,
+            "work_type": "diploma_thesis"
+          },
+          {
+            "person": "Lukáš Pokrývka",
+            "year": 2019,
+            "start_year": 2016,
+            "work_type": "team_project"
+          },
+          {
+            "person": "Lukáš Pokrývka",
+            "year": 2020,
+            "start_year": 2016,
+            "work_type": "diploma_project"
+          }
+        ],
+        "source_urls": [
+          "https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"
+        ],
+        "accepted_for_delivery": true
+      },
+      {
+        "id": "q0608",
+        "question": "Aký source_url patrí k stránke „Podpora slovenčiny v knižnici Spacy“?",
+        "passed": true,
+        "issues": [],
+        "answer": "Zo zdroja Podpora slovenčiny v knižnici Spacy: „Source URL: https://zp.kemt.fei.tuke.sk/topics/spacy“\nVýpis je neúplný.\nZdroje: https://zp.kemt.fei.tuke.sk/topics/spacy",
+        "claims": [],
+        "source_urls": [
+          "https://zp.kemt.fei.tuke.sk/topics/spacy"
+        ],
+        "accepted_for_delivery": true
+      },
+      {
+        "id": "q0888",
+        "question": "Aký konkrétny finančný rozpočet bol uvedený pre projekt osoby Tomáš Kuchárik?",
+        "passed": true,
+        "issues": [],
+        "answer": "V dostupných podkladoch sa požadovanú informáciu nepodarilo spoľahlivo nájsť.\nVýpis je neúplný.",
+        "claims": [],
+        "source_urls": [],
+        "accepted_for_delivery": true
+      },
+      {
+        "id": "q1147",
+        "question": "Porovnaj záznamy osôb Ján Holp a Ondrej Megela. Pri osobe Ján Holp použi záznam typu „diplomová práca“ a pri osobe Ondrej Megela záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
+        "passed": true,
+        "issues": [],
+        "answer": "Ján Holp — diplomová práca; názov práce: Systém získavania informácií v slovenskom jazyku; rok práce: 2021.\nOndrej Megela — bakalárska práca; názov práce: Neurónové jazykové modelovanie typu BERT; rok práce: 2020.\nVýpis je neúplný.\nZdroje: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp, https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
+        "claims": [
+          {
+            "person": "Ján Holp",
+            "title": "Systém získavania informácií v slovenskom jazyku",
+            "year": 2021,
+            "work_type": "diploma_thesis"
+          },
+          {
+            "person": "Ondrej Megela",
+            "title": "Neurónové jazykové modelovanie typu BERT",
+            "year": 2020,
+            "work_type": "bachelor_thesis"
+          }
+        ],
+        "source_urls": [
+          "https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
+          "https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
+        ],
+        "accepted_for_delivery": true
+      }
+    ]
+  },
+  "broad_model_accuracy": null,
+  "full_verification": {
+    "passed": true,
+    "checks": {
+      "python_and_shell": {
+        "passed": true,
+        "details": {
+          "compiled_python_files": 150,
+          "checked_shell_scripts": 4
+        },
+        "seconds": 0.5
+      },
+      "git_diff": {
+        "passed": true,
+        "details": "",
+        "seconds": 0.01
+      },
+      "git_integrity": {
+        "passed": true,
+        "details": "dangling tree 7a73ecae52355619e782cb822622ac3f69de7ae0\ndangling blob 5574a6fec5472a6e8f5e743a49da177b376eda10\ndangling tree fbb4fbc9fe2f927d118b4b3b97e886a316d1d8f1\ndangling blob a186d97f5ef3b6da4a0ef094f7d05c11b47d9350\ndangling tree a626128497ed8c0d3dc200285901748474ac7dd2\ndangling tree 3337aecd889bcdac7142dcb00e46f1c71aa0cf92\ndangling blob c0c87b808a2454b64badf9d191260458aa576653\ndangling tree ef38b2cbe6f6e24898cefe8a83450122dea5d4c6\ndangling blob f208469c41d2b62812c34bc4ace4f25fdbd627cf\ndangling blob 6489d7ce0bb130e424bbe8da6c73300a2e167b94\ndangling blob c949aad31632b94da867885110890f0a5c9ca2a5\ndangling tree 625a23eefd539c3844b515e71d0ac6929b7babe0\ndangling tree b4eaffde38b667198523282947dae5e4a4e4fb03\ndangling blob f54b96ee587ee0d697902604536f3803e6001ad2\ndangling blob c54c2eb64f9a1d522f8c7e335222fdac15b1d02b\ndangling blob c14d08a27284af6a3b91414aaf9f5a1773ed9240",
+        "seconds": 0.05
+      },
+      "compose_configuration": {
+        "passed": true,
+        "details": "",
+        "seconds": 0.11
+      },
+      "api_and_both_indexes": {
+        "passed": true,
+        "details": {
+          "ready": true,
+          "index_version": {
+            "version": "497fadb8f38c4af18d7d9e3b06cdb0a0",
+            "git_revision": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e",
+            "created_at_utc": "2026-10-26T18:15:17.983336+00:00",
+            "source_sha256": "007792c12f3467dd3203bb4951ccc197fdc0eef669580f1b1f893baf609ca363",
+            "documents_sha256": "0542af2a373771cf7c4a95f93d14f4f687f6a0187846892c959aa6efc4bade52",
+            "chunks_sha256": "437dde589be46e0c30842dd330de22d96a0173d2dbaff9c5f129568cb2bca4c9",
+            "embedding_model": "intfloat/multilingual-e5-small",
+            "publication_policy": "explicit-false-hidden-v1",
+            "embedding_fingerprint": "d066fb9a266a1a0c77c65da75dcf538cfab5c87d5c605ff26b1fd2da8a12aa92"
+          },
+          "tools": {
+            "rag": true,
+            "neo4j": true,
+            "graphrag": true
+          }
+        },
+        "seconds": 7.91
+      },
+      "sqlite_integrity": {
+        "passed": true,
+        "details": {
+          "documents": 114,
+          "chunks": 887,
+          "chunks_fts": 887,
+          "chunk_embeddings": 892
+        },
+        "seconds": 0.06
+      },
+      "container_readiness": {
+        "passed": true,
+        "details": "",
+        "seconds": 1.2
+      },
+      "wiki_git_ssh": {
+        "passed": true,
+        "details": "3e892a1d49e9bce118cd03478ce15e01c2bebb4e\trefs/heads/master",
+        "seconds": 0.83
+      },
+      "openwebui_tool_packages": {
+        "passed": true,
+        "details": {
+          "packages": [
+            {
+              "package_id": "zpwiki_rag",
+              "archive": "/home/janko/DP/zp-agent/data/tool_packages/zpwiki_rag.zip",
+              "sha256": "84c66cefbed33dbbabbe033f5fe5dc5e71cda380c9824b2ae7f15b30e304fdd6"
+            },
+            {
+              "package_id": "zpwiki_graphrag",
+              "archive": "/home/janko/DP/zp-agent/data/tool_packages/zpwiki_graphrag.zip",
+              "sha256": "6d6ab36bf8fe936a1ae9a1c1debb46cedb04116c860ee9d3216ca720e65cf2fe"
+            }
+          ],
+          "live_retrieval": true,
+          "same_data_version": true,
+          "negative_query": true,
+          "authorization": true,
+          "external_llm_calls": false
+        },
+        "seconds": 0.75
+      },
+      "full_local_regression": {
+        "passed": true,
+        "details": {
+          "passed": 734,
+          "log": "/home/janko/DP/zp-agent/data/logs/verification/20261027T005852Z-ef38a940.pytest.log",
+          "external_network_blocked": true
+        },
+        "seconds": 89.48
+      }
+    },
+    "deployed_python_modules_match_workspace": 125,
+    "offline_assets_build_probe": true,
+    "post_regression_both_tools_ready": true,
+    "index_version_unchanged": "497fadb8f38c4af18d7d9e3b06cdb0a0"
+  },
+  "report_organization": {
+    "moved_files": 26,
+    "deleted_files": 0,
+    "operational_json_hashes_unchanged": true,
+    "reports": "data/reports/<category>/",
+    "logs": "data/logs/",
+    "legacy_reports": "data/reports/archive/legacy/",
+    "legacy_logs": "data/logs/archive/legacy/",
+    "retention": "Keep five latest completed project verifications active, archive older reports and paired logs, preserve incomplete and experimental results. Never auto-delete.",
+    "journal": {
+      "path": "data/reports/migrations/20261027T004801Z-dfb5c510.json",
+      "sha256": "e6ff162d8d8164bdac07cb3c1781fc9d113990079d9d636203a8ed1ab48fce40"
+    }
+  },
+  "artifacts": {
+    "local_contexts": {
+      "path": "data/reports/rag_verified/20261027T005720Z-3186724c.json",
+      "sha256": "516276ce1f91f5b3e30c8d09d0a008021c777f067d74ef8fbd89faeba259af9d"
+    },
+    "guard_replay": {
+      "path": "data/reports/rag_guards/20261027T005846Z-cfbc62a7.json",
+      "sha256": "d2539d13a205af724a2aa95f7d11bc72d9a2e1f4176147eddcea78a6ab173e42"
+    },
+    "original_model_outputs": {
+      "path": "data/reports/rag_verified/20261027T005425Z-2e5ca17c.json",
+      "sha256": "e98bd32ccc9f5010fb09b674499cb7decf0417781f3b1d92d7da077050eca9e8"
+    },
+    "regraded_model_outputs": {
+      "path": "data/reports/rag_verified/20261027T005639Z-bb118881.json",
+      "sha256": "7383b72083e965c670cda2caadae268894eb8caead4185f48abde26c477e21c5"
+    },
+    "project_verification": {
+      "path": "data/reports/verification/20261027T005852Z-ef38a940.json",
+      "sha256": "eac6ef0caace933d42fd41c540a10c646699843aec6bc2bab28ef8be4727aa8f"
+    }
+  },
+  "limitations": [
+    "Known regression questions; not a new independent accuracy estimate.",
+    "Dárius Lindvai: the diploma year 2021 is inferred from taxonomy category dp2021; no explicit diploma heading. Runtime checked RAG deliberately does not use category years as work-year proof.",
+    "Source review is automated against original Markdown; independent human annotation has not been performed.",
+    "No 80-case model accuracy has been measured. Existing explicit permission covers five school-model questions; broader export approval is pending.",
+    "360 guard checks are controlled fault injection, not 360 generated model responses.",
+    "Finding the expected document or extracting a supported tuple does not by itself measure answer correctness.",
+    "Automatic quote relevance uses known expected-value coverage and literal support; independent human semantic review is still absent."
+  ]
+}

tools/runtime/report_files.py

@@ -0,0 +1,124 @@
+"""Locations and non-destructive retention for local operational reports."""
+from __future__ import annotations
+
+import hashlib
+import json
+import shutil
+from datetime import datetime, timezone
+from pathlib import Path
+from uuid import uuid4
+
+from tools.rag.common import DATA_DIR
+
+CORE_JSON = {'documents.json', 'chunks.json', 'index_version.json'}
+
+
+def run_id() -> str:
+    return datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + uuid4().hex[:8]
+
+
+def report_output(requested: Path | None, category: str, *, data_dir: Path = DATA_DIR) -> Path:
+    if not category.replace('_', '').isalnum():
+        raise ValueError('Invalid report category')
+    if requested is None:
+        target = data_dir / 'reports' / category / (run_id() + '.json')
+    else:
+        target = requested.resolve()
+        if target.parent == data_dir.resolve():
+            if target.name in CORE_JSON or target.name.startswith('.') or target.suffix != '.json':
+                raise ValueError('Report cannot overwrite operational data')
+            target = data_dir / 'reports' / category / target.name
+    if target.exists():
+        raise FileExistsError(f'Report already exists: {target}')
+    target.parent.mkdir(parents=True, exist_ok=True)
+    return target
+
+
+def report_log(report: Path, *, data_dir: Path = DATA_DIR) -> Path:
+    try:
+        relative = report.resolve().relative_to((data_dir / 'reports').resolve())
+    except ValueError:
+        identity = hashlib.sha256(str(report.absolute()).encode()).hexdigest()[:8]
+        relative = Path('custom') / (report.stem + '-' + identity + '.json')
+    target = data_dir / 'logs' / relative.with_suffix('.pytest.log')
+    if target.exists():
+        raise FileExistsError(f'Log already exists: {target}')
+    target.parent.mkdir(parents=True, exist_ok=True)
+    return target
+
+
+def resolve_report_input(path: Path, *, data_dir: Path = DATA_DIR) -> Path:
+    if path.exists():
+        return path
+    mappings = {}
+    for record in sorted((data_dir / 'reports/migrations').glob('*.json')):
+        for move in json.loads(record.read_text()).get('moves', []):
+            mappings[Path(move['source']).resolve()] = Path(move['destination'])
+    visited = set()
+    while not path.exists() and path.resolve() in mappings and path.resolve() not in visited:
+        visited.add(path.resolve())
+        path = mappings[path.resolve()]
+    return path
+
+
+def migration_plan(data_dir: Path = DATA_DIR) -> list[dict]:
+    moves = []
+    for path in sorted(data_dir.iterdir()):
+        if not path.is_file() or path.name.startswith('.') or path.name in CORE_JSON:
+            continue
+        if path.suffix == '.json':
+            target = data_dir / 'reports/archive/legacy' / path.name
+        elif path.suffix == '.log':
+            target = data_dir / 'logs/archive/legacy' / path.name
+        else:
+            continue
+        moves.append({'source': str(path.absolute()), 'destination': str(target.absolute()),
+                      'sha256': hashlib.sha256(path.read_bytes()).hexdigest()})
+    return moves
+
+
+def retention_plan(data_dir: Path = DATA_DIR, *, keep: int = 5) -> list[dict]:
+    if keep < 1:
+        raise ValueError('Keep at least one verification')
+    completed = []
+    for path in (data_dir / 'reports/verification').glob('*.json'):
+        try:
+            data = json.loads(path.read_text())
+            if isinstance(data, dict) and type(data.get('passed')) is bool:
+                completed.append(path)
+        except (ValueError, OSError):
+            continue
+    moves = []
+    for path in sorted(completed, key=lambda p: (p.stat().st_mtime_ns, p.name), reverse=True)[keep:]:
+        target = data_dir / 'reports/archive/verification' / path.name
+        moves.append({'source': str(path.absolute()), 'destination': str(target.absolute()),
+                      'sha256': hashlib.sha256(path.read_bytes()).hexdigest()})
+        log = data_dir / 'logs/verification' / path.with_suffix('.pytest.log').name
+        if log.exists():
+            target = data_dir / 'logs/archive/verification' / log.name
+            moves.append({'source': str(log.absolute()), 'destination': str(target.absolute()),
+                          'sha256': hashlib.sha256(log.read_bytes()).hexdigest()})
+    return moves
+
+
+def apply_moves(moves: list[dict], *, data_dir: Path = DATA_DIR) -> Path:
+    # Validate the entire plan before moving anything. Never replace existing files.
+    for move in moves:
+        source, destination = Path(move['source']), Path(move['destination'])
+        if destination.exists() or hashlib.sha256(source.read_bytes()).hexdigest() != move['sha256']:
+            raise RuntimeError('Report migration conflict; no files moved')
+    journal = data_dir / 'reports/migrations' / (run_id() + '.json')
+    journal.parent.mkdir(parents=True, exist_ok=True)
+    record = {'method': 'Move without rewriting content or deleting historical evidence', 'moves': []}
+    def write_journal():
+        temporary = journal.with_suffix('.tmp')
+        temporary.write_text(json.dumps(record, indent=2) + '\n')
+        temporary.replace(journal)
+    write_journal()
+    for move in moves:
+        destination = Path(move['destination'])
+        destination.parent.mkdir(parents=True, exist_ok=True)
+        shutil.move(move['source'], destination)
+        record['moves'].append(move)
+        write_journal()
+    return journal