Refactor RAG into standalone tool
This commit is contained in:
parent
043db73ee6
commit
921642a16a
@ -25,7 +25,6 @@ RUN python -m pip install --no-cache-dir \
|
||||
-r requirements.txt
|
||||
|
||||
COPY app ./app
|
||||
COPY scripts ./scripts
|
||||
COPY tools ./tools
|
||||
|
||||
RUN mkdir -p data
|
||||
|
||||
235
README.md
235
README.md
@ -13,8 +13,10 @@ OpenAPI Tool Server.
|
||||
Stabilná baseline retrieval vetva používa klasický hybridný RAG:
|
||||
FTS5/BM25 + embeddingy + RRF. Nad týmto baseline sa vo vetve GraphRAG
|
||||
dopĺňa knowledge graph uložený v Neo4j. Aktuálne je implementovaná
|
||||
grafová infraštruktúra, deterministický builder a vizualizácia grafu;
|
||||
graph retrieval a jeho spojenie s RAG vrstvou sú ďalší krok.
|
||||
grafová infraštruktúra, deterministický builder, vizualizácia grafu a
|
||||
samostatné grafové vyhľadávanie. Ďalším krokom je rozšírenie dátového
|
||||
modelu podľa požiadaviek projektu a spojenie grafového vyhľadávania s
|
||||
RAG vrstvou.
|
||||
|
||||
## Implementované
|
||||
|
||||
@ -69,6 +71,7 @@ graph retrieval a jeho spojenie s RAG vrstvou sú ďalší krok.
|
||||
- grafové vzťahy pre dokumenty, práce, témy, kategórie a autorstvo,
|
||||
- uniqueness constraints a opakovateľný rebuild GraphRAG grafu,
|
||||
- Neo4j Browser na vizualizáciu a kontrolu vytvoreného grafu,
|
||||
- samostatné grafové vyhľadávanie nad Neo4j s rozpoznaním entít, vzťahov, prác, tém a rokov,
|
||||
- retrieval evaluácia pre FTS, embeddings a hybridný retrieval,
|
||||
- answer-level RAG evaluácia cez OpenWebUI model a RAG tool,
|
||||
- oddelené answer-level overrides bez úpravy frozen retrieval
|
||||
@ -141,14 +144,16 @@ Neo4j
|
||||
↓
|
||||
grafové vzťahy + vizualizácia
|
||||
↓
|
||||
graph retrieval
|
||||
graph_search.py
|
||||
↓
|
||||
grafové entity + cesty + zdrojové dokumenty
|
||||
↓
|
||||
spojenie s klasickým RAG
|
||||
```
|
||||
|
||||
V aktuálnom stave je implementované vytvorenie a uloženie knowledge
|
||||
graphu. Samotný graph retrieval a jeho zapojenie do `/rag` sa ešte
|
||||
dopĺňajú.
|
||||
graphu aj samostatné grafové vyhľadávanie. Jeho zapojenie do `/rag` a
|
||||
OpenWebUI sa ešte dopĺňa.
|
||||
|
||||
Synchronizačná vetva:
|
||||
|
||||
@ -171,7 +176,7 @@ atomická výmena databázy
|
||||
Evaluačná vetva:
|
||||
|
||||
```text
|
||||
evaluation/json_files/*.json
|
||||
tools/rag/evaluation/json_files/*.json
|
||||
↓
|
||||
retrieval evaluator / RAG answer evaluator
|
||||
↓
|
||||
@ -179,11 +184,11 @@ runner
|
||||
↓
|
||||
metriky
|
||||
↓
|
||||
evaluation/results/
|
||||
tools/rag/evaluation/results/
|
||||
```
|
||||
|
||||
`evaluation/results/` nie je v nižšie uvedenom stromčeku, pretože
|
||||
obsahuje generované výstupy experimentov.
|
||||
`tools/rag/evaluation/results/` nie je v nižšie uvedenom stromčeku,
|
||||
pretože obsahuje generované výstupy experimentov.
|
||||
|
||||
## Štruktúra projektu
|
||||
|
||||
@ -198,38 +203,47 @@ zp-agent/
|
||||
│ ├── routes.py
|
||||
│ └── security.py
|
||||
│
|
||||
├── scripts/
|
||||
│ ├── common.py
|
||||
│ ├── scan_zpwiki.py
|
||||
│ ├── build_chunks.py
|
||||
│ ├── build_sqlite_index.py
|
||||
│ ├── embedding_utils.py
|
||||
│ ├── graph_db.py
|
||||
│ ├── graphrag.py
|
||||
│ ├── build_graphrag.py
|
||||
│ ├── neo4j_smoke.py
|
||||
│ ├── rag_utils.py
|
||||
│ ├── rebuild_index.py
|
||||
│ ├── search_db.py
|
||||
│ ├── search_core.py
|
||||
│ └── search_utils.py
|
||||
├── tools/
|
||||
│ ├── rag/
|
||||
│ │ ├── common.py
|
||||
│ │ ├── scan_zpwiki.py
|
||||
│ │ ├── build_chunks.py
|
||||
│ │ ├── build_sqlite_index.py
|
||||
│ │ ├── embedding_utils.py
|
||||
│ │ ├── rebuild_index.py
|
||||
│ │ ├── search_core.py
|
||||
│ │ ├── search_utils.py
|
||||
│ │ ├── search_db.py
|
||||
│ │ ├── rag_utils.py
|
||||
│ │ ├── rag_document_expansion.py
|
||||
│ │ ├── rag_query_evidence.py
|
||||
│ │ ├── rag_supplemental_expansion.py
|
||||
│ │ ├── run_e5_regression.sh
|
||||
│ │ ├── run_e5_1_regression.sh
|
||||
│ │ ├── evaluation/
|
||||
│ │ │ ├── evaluate_retrieval.py
|
||||
│ │ │ ├── evaluate_rag_answers.py
|
||||
│ │ │ ├── retrieval_runner.py
|
||||
│ │ │ ├── rag_runner.py
|
||||
│ │ │ ├── metrics.py
|
||||
│ │ │ ├── rag_metrics.py
|
||||
│ │ │ └── json_files/
|
||||
│ │ └── tests/
|
||||
│ │
|
||||
│ └── graphrag/
|
||||
│ ├── graph_db.py
|
||||
│ ├── graphrag.py
|
||||
│ ├── build_graphrag.py
|
||||
│ ├── graph_search.py
|
||||
│ ├── neo4j_smoke.py
|
||||
│ ├── evaluation/
|
||||
│ │ └── graphrag_questions.json
|
||||
│ └── tests/
|
||||
│
|
||||
├── evaluation/
|
||||
│ ├── evaluate_retrieval.py
|
||||
│ ├── retrieval_runner.py
|
||||
│ ├── metrics.py
|
||||
│ ├── evaluate_rag_answers.py
|
||||
│ ├── rag_runner.py
|
||||
│ ├── rag_metrics.py
|
||||
│ └── json_files/
|
||||
│ ├── questions.json
|
||||
│ ├── questions_before_ambiguity_cleanup.json
|
||||
│ ├── questions_before_validation.json
|
||||
│ ├── rag_answer_overrides.json
|
||||
│ ├── questions_extended_2000.json
|
||||
│ ├── graphrag_questions.json
|
||||
│ ├── robustness_questions.json
|
||||
│ └── performance_queries.json
|
||||
├── test/
|
||||
│ ├── test_api.py
|
||||
│ ├── test_configuration.py
|
||||
│ └── test_security.py
|
||||
│
|
||||
├── data/
|
||||
│ ├── documents.json
|
||||
@ -281,7 +295,7 @@ vypnuté. Pre integráciu OpenWebUI je dôležitý endpoint:
|
||||
- používa bezpečné porovnávanie autentifikačných hodnôt,
|
||||
- validuje očakávaný Gitea repozitár a súvisiacu konfiguráciu.
|
||||
|
||||
### `scripts/`
|
||||
### `tools/rag/`
|
||||
|
||||
`common.py`
|
||||
|
||||
@ -318,6 +332,8 @@ vyhľadávacou vrstvou.
|
||||
- normalizuje vektory,
|
||||
- poskytuje utility pre vektorové vyhľadávanie.
|
||||
|
||||
### `tools/graphrag/`
|
||||
|
||||
`graph_db.py`
|
||||
|
||||
- načítava konfiguráciu Neo4j,
|
||||
@ -340,6 +356,13 @@ vyhľadávacou vrstvou.
|
||||
- podporuje opakovateľný rebuild spravovanej časti grafu,
|
||||
- vypíše základné počty uzlov, vzťahov a kvalitu extrakcie názvov prác.
|
||||
|
||||
`graph_search.py`
|
||||
|
||||
- rozpoznáva relevantné osoby a obmedzenia v otázke,
|
||||
- vyhľadáva práce, témy, roky a spoločné vzťahy v Neo4j,
|
||||
- vracia grafové cesty, fakty a zdrojové dokumenty,
|
||||
- slúži ako základ budúceho GraphRAG nástroja pre OpenWebUI.
|
||||
|
||||
`neo4j_smoke.py`
|
||||
|
||||
Jednoduchý smoke test pripojenia k Neo4j používaný pri kontrole Docker
|
||||
@ -390,7 +413,7 @@ scan
|
||||
→ atomická výmena DB
|
||||
```
|
||||
|
||||
### `evaluation/`
|
||||
### `tools/rag/evaluation/`
|
||||
|
||||
`evaluate_retrieval.py`
|
||||
|
||||
@ -444,7 +467,7 @@ Počíta answer-level metriky, napríklad:
|
||||
- strict pass,
|
||||
- latency a tokenové údaje.
|
||||
|
||||
### `evaluation/json_files/`
|
||||
### `tools/rag/evaluation/json_files/`
|
||||
|
||||
`questions.json`
|
||||
|
||||
@ -486,36 +509,13 @@ Obsahuje kombináciu:
|
||||
Tento dataset je zatiaľ pripravený, ale nemá sa používať na priebežné
|
||||
ladenie frozen benchmarku.
|
||||
|
||||
`graphrag_questions.json`
|
||||
|
||||
Samostatný benchmark pripravený pre vývoj GraphRAG vetvy.
|
||||
|
||||
Obsahuje 500 otázok zameraných na:
|
||||
|
||||
- multi-hop vzťahy,
|
||||
- spoločné témy,
|
||||
- spoločné roky,
|
||||
- person → title → year,
|
||||
- person → topic → title,
|
||||
- person → category → author,
|
||||
- agregácie,
|
||||
- konjunktívnu disambiguáciu,
|
||||
- multi-document reasoning.
|
||||
|
||||
Dataset obsahuje aj GraphRAG metadata ako:
|
||||
GraphRAG benchmark je po oddelení nástrojov uložený samostatne v:
|
||||
|
||||
```text
|
||||
graph.task
|
||||
graph.hop_count
|
||||
graph.start_entities
|
||||
graph.expected_entities
|
||||
graph.expected_relations
|
||||
graph.expected_paths
|
||||
tools/graphrag/evaluation/graphrag_questions.json
|
||||
```
|
||||
|
||||
Dataset sa používa pre vývoj a neskoršie porovnanie GraphRAG vetvy.
|
||||
Knowledge graph a jeho Neo4j vrstva sú už implementované; samotný
|
||||
graph retrieval a answer-level GraphRAG tok sa ešte dopĺňajú.
|
||||
Obsahuje 500 otázok zameraných na grafové a viacvzťahové dopyty.
|
||||
|
||||
`robustness_questions.json`
|
||||
|
||||
@ -661,7 +661,7 @@ Kontrola spojenia z API kontajnera:
|
||||
|
||||
```bash
|
||||
docker compose exec zp-agent-api \
|
||||
python -m scripts.neo4j_smoke
|
||||
python -m tools.graphrag.neo4j_smoke
|
||||
```
|
||||
|
||||
## Reindexovanie
|
||||
@ -670,7 +670,7 @@ Celý proces načíta dokumenty, vytvorí chunky, obnoví FTS5 index a
|
||||
vytvorí embedding pre každý chunk:
|
||||
|
||||
```bash
|
||||
docker compose run --rm zp-agent-api python scripts/rebuild_index.py
|
||||
docker compose run --rm zp-agent-api python -m tools.rag.rebuild_index
|
||||
```
|
||||
|
||||
Vzniknú súbory:
|
||||
@ -718,7 +718,7 @@ Test z terminálu:
|
||||
|
||||
```bash
|
||||
docker compose run --rm zp-agent-api \
|
||||
python scripts/search_db.py "rag agent" --limit 5
|
||||
python -m tools.rag.search_db "rag agent" --limit 5
|
||||
```
|
||||
|
||||
Pred volaním API načítaj premenné z `.env`:
|
||||
@ -756,7 +756,7 @@ dokumentu.
|
||||
## RAG
|
||||
|
||||
Nad hybridným retrievalom je implementovaná RAG vrstva v
|
||||
`scripts/rag_utils.py`.
|
||||
`tools/rag/rag_utils.py`.
|
||||
|
||||
Endpoint `/rag` vyhľadá relevantné dokumenty a pripraví:
|
||||
|
||||
@ -827,17 +827,17 @@ Knowledge graph sa vytvorí príkazom:
|
||||
|
||||
```bash
|
||||
docker compose exec zp-agent-api \
|
||||
python -m scripts.build_graphrag
|
||||
python -m tools.graphrag.build_graphrag
|
||||
```
|
||||
|
||||
Builder vytvára constraints, zapisuje uzly a vzťahy a pri štandardnom
|
||||
spustení obnoví iba časť grafu spravovanú ZP Agentom. Opakované
|
||||
spustenie preto nevytvára duplicitné uzly.
|
||||
|
||||
Graf je možné vizuálne kontrolovať v Neo4j Browseri. GraphRAG zatiaľ
|
||||
nezasahuje do stabilného klasického `/rag` toku. Nasledujúca etapa
|
||||
doplní graph retrieval a následne spojenie grafového a klasického
|
||||
retrievalu.
|
||||
Graf je možné vizuálne kontrolovať v Neo4j Browseri. Samostatné
|
||||
grafové vyhľadávanie je implementované v `tools/graphrag/graph_search.py`.
|
||||
GraphRAG zatiaľ nezasahuje do stabilného klasického `/rag` toku; ďalšia
|
||||
etapa spojí grafové výsledky s klasickým RAG a OpenWebUI nástrojom.
|
||||
|
||||
## OpenWebUI
|
||||
|
||||
@ -874,7 +874,7 @@ Výsledná odpoveď má byť grounded v ZP Wiki a má používať relevantný
|
||||
Frozen retrieval benchmark sa spúšťa napríklad:
|
||||
|
||||
```bash
|
||||
python evaluation/evaluate_retrieval.py \
|
||||
python -m tools.rag.evaluation.evaluate_retrieval \
|
||||
--split dev \
|
||||
--strict-dataset
|
||||
```
|
||||
@ -912,7 +912,7 @@ TEST split sa nemá používať na priebežné ladenie retrieval konfigurácie.
|
||||
Answer-level evaluácia používa:
|
||||
|
||||
```bash
|
||||
python evaluation/evaluate_rag_answers.py --split dev
|
||||
python -m tools.rag.evaluation.evaluate_rag_answers --split dev
|
||||
```
|
||||
|
||||
Evaluuje kompletný model/tool/RAG tok a od retrieval benchmarku je
|
||||
@ -922,7 +922,7 @@ Ak je pôvodná otázka vhodná pre retrieval, ale nejednoznačná pre
|
||||
generatívnu odpoveď, jej answer-level formulácia sa upraví iba cez:
|
||||
|
||||
```text
|
||||
evaluation/json_files/rag_answer_overrides.json
|
||||
tools/rag/evaluation/json_files/rag_answer_overrides.json
|
||||
```
|
||||
|
||||
Pôvodný frozen `questions.json` tým ostáva nezmenený.
|
||||
@ -1014,10 +1014,10 @@ Kontrola syntaxe najdôležitejších upravovaných modulov:
|
||||
|
||||
```bash
|
||||
python -m py_compile \
|
||||
scripts/rag_utils.py \
|
||||
evaluation/evaluate_rag_answers.py \
|
||||
evaluation/rag_runner.py \
|
||||
evaluation/rag_metrics.py
|
||||
tools/rag/rag_utils.py \
|
||||
tools/rag/evaluation/evaluate_rag_answers.py \
|
||||
tools/rag/evaluation/rag_runner.py \
|
||||
tools/rag/evaluation/rag_metrics.py
|
||||
```
|
||||
|
||||
Pred checkpointom je vhodné použiť aj:
|
||||
@ -1048,47 +1048,48 @@ klasického baseline.
|
||||
|
||||
## Najbližší postup
|
||||
|
||||
Klasický RAG baseline je zmrazený a GraphRAG infraštruktúra s Neo4j je
|
||||
funkčná. Ďalší vývoj pokračuje nad samostatnou GraphRAG vetvou.
|
||||
Klasický RAG a GraphRAG sú oddelené do samostatných komponentov v
|
||||
`tools/rag/` a `tools/graphrag/`. Stabilný klasický RAG zostáva
|
||||
referenčným bodom.
|
||||
|
||||
### 1. Dokončenie kvality knowledge graphu
|
||||
### 1. Doplnenie GraphRAG dátového modelu
|
||||
|
||||
- doplniť bezpečné fallback pravidlá pre explicitné názvy prác,
|
||||
- ponechať `null` tam, kde zdroj názov skutočne neuvádza,
|
||||
- doplniť regresné testy pre graph builder.
|
||||
Podľa zadania doplniť alebo jasne oddeliť záverečnú prácu, autora práce,
|
||||
tému, kľúčové slová a študijný program tam, kde sú tieto údaje v zdroji
|
||||
reálne dostupné. Nevytvárať údaje, ktoré ZP Wiki neobsahuje.
|
||||
|
||||
### 2. Graph retrieval
|
||||
### 2. GraphRAG evaluácia
|
||||
|
||||
Vytvoriť vyhľadávaciu vrstvu nad Neo4j, ktorá z otázky identifikuje
|
||||
relevantné osoby, témy, kategórie, práce a roky a vráti:
|
||||
Spustiť samostatnú evaluáciu `graph_search.py` nad:
|
||||
|
||||
```text
|
||||
grafové entity
|
||||
→ grafové cesty
|
||||
→ relevantné dokumenty
|
||||
→ zdrojové chunky
|
||||
tools/graphrag/evaluation/graphrag_questions.json
|
||||
```
|
||||
|
||||
### 3. Spojenie s klasickým RAG
|
||||
a vyhodnotiť entity, vzťahy, cesty a nájdené dokumenty.
|
||||
|
||||
Grafový retrieval sa následne spojí s existujúcim FTS5 + embedding
|
||||
retrievalom. Klasický baseline zostane zachovaný ako samostatný
|
||||
referenčný bod.
|
||||
### 3. OpenWebUI nástroje
|
||||
|
||||
```text
|
||||
query
|
||||
↓
|
||||
├── classic retrieval
|
||||
└── graph retrieval
|
||||
↓
|
||||
context fusion
|
||||
↓
|
||||
RAG
|
||||
```
|
||||
Pripraviť samostatne inštalovateľný RAG nástroj a GraphRAG nástroj.
|
||||
Každý komponent bude mať vlastný adresár, rozhranie a testy.
|
||||
|
||||
### 4. GraphRAG evaluácia
|
||||
### 4. Hlavný agent
|
||||
|
||||
GraphRAG sa bude priebežne ladiť na DEV dátach a vyhodnocovať aj na
|
||||
samostatnom `graphrag_questions.json` datasete. Až po stabilizovaní
|
||||
celého systému budú nasledovať širšie embeddingové, robustness a
|
||||
performance experimenty.
|
||||
Pridať hlavný prompt/agenta, ktorý podľa otázky vyberie vhodný nástroj a
|
||||
bude vedieť kombinovať ich výsledky.
|
||||
|
||||
### 5. Git MCP server
|
||||
|
||||
Pridať samostatný MCP server pre prácu s Git repozitárom a spúšťať ho v
|
||||
Dockeri. Následne ho sprístupniť hlavnému agentovi ako ďalší nástroj.
|
||||
|
||||
### 6. Spojenie GraphRAG s klasickým RAG
|
||||
|
||||
Po stabilizovaní grafového vyhľadávania spojiť výsledky grafu s
|
||||
existujúcim FTS5 + embedding vyhľadávaním a porovnať klasický RAG s
|
||||
GraphRAG variantom.
|
||||
|
||||
### 7. Finálne experimenty
|
||||
|
||||
Až po stabilizovaní celého systému vykonať širšie porovnanie embedding
|
||||
modelov, robustnosti, výkonu a ostatných finálnych experimentov.
|
||||
|
||||
@ -28,7 +28,7 @@ from app.routes import router
|
||||
from app.security import (
|
||||
validate_security_configuration,
|
||||
)
|
||||
from scripts.embedding_utils import embed_query
|
||||
from tools.rag.embedding_utils import embed_query
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -36,16 +36,16 @@ from app.security import (
|
||||
verify_gitea_signature,
|
||||
webhook_should_pull_git,
|
||||
)
|
||||
from scripts.common import (
|
||||
from tools.rag.common import (
|
||||
DB_FILE,
|
||||
ZPWIKI_ROOT,
|
||||
)
|
||||
from scripts.rag_utils import build_rag_context
|
||||
from scripts.rebuild_index import (
|
||||
from tools.rag.rag_utils import build_rag_context
|
||||
from tools.rag.rebuild_index import (
|
||||
ReindexInProgressError,
|
||||
rebuild_index,
|
||||
)
|
||||
from scripts.search_utils import search_database
|
||||
from tools.rag.search_utils import search_database
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -9,7 +9,7 @@ from fastapi.testclient import TestClient
|
||||
|
||||
import app.main as main
|
||||
import app.routes as routes
|
||||
from scripts.rebuild_index import (
|
||||
from tools.rag.rebuild_index import (
|
||||
ReindexInProgressError,
|
||||
)
|
||||
|
||||
|
||||
@ -81,7 +81,11 @@ def test_runtime_requirements_contain_only_direct_dependencies() -> None:
|
||||
|
||||
|
||||
def test_all_application_python_files_compile() -> None:
|
||||
targets = [PROJECT_ROOT / "app", PROJECT_ROOT / "scripts"]
|
||||
targets = [
|
||||
PROJECT_ROOT / "app",
|
||||
PROJECT_ROOT / "tools" / "rag",
|
||||
PROJECT_ROOT / "tools" / "graphrag",
|
||||
]
|
||||
|
||||
for directory in targets:
|
||||
for path in directory.rglob("*.py"):
|
||||
|
||||
@ -10,13 +10,13 @@ from pathlib import Path
|
||||
from rich import print
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
|
||||
from scripts.common import (
|
||||
from tools.rag.common import (
|
||||
CHUNKS_FILE,
|
||||
PAGES_ROOT,
|
||||
ZPWIKI_ROOT,
|
||||
@ -10,19 +10,19 @@ from typing import Any
|
||||
from rich import print
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
|
||||
from scripts.common import (
|
||||
from tools.rag.common import (
|
||||
CHUNKS_FILE,
|
||||
DB_FILE,
|
||||
DOCUMENTS_FILE,
|
||||
read_json,
|
||||
)
|
||||
from scripts.embedding_utils import (
|
||||
from tools.rag.embedding_utils import (
|
||||
build_embedding_text,
|
||||
embed_passages,
|
||||
embedding_model_name,
|
||||
@ -10,7 +10,7 @@ from typing import Any
|
||||
import frontmatter
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
ZPWIKI_ROOT = Path(
|
||||
os.getenv(
|
||||
0
tools/rag/evaluation/__init__.py
Normal file
0
tools/rag/evaluation/__init__.py
Normal file
@ -9,7 +9,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(
|
||||
@ -18,7 +18,7 @@ if str(PROJECT_ROOT) not in sys.path:
|
||||
)
|
||||
|
||||
|
||||
from evaluation.rag_answer_data import (
|
||||
from tools.rag.evaluation.rag_answer_data import (
|
||||
DEFAULT_DELAY,
|
||||
apply_question_overrides,
|
||||
questions_fingerprint,
|
||||
@ -27,7 +27,7 @@ from evaluation.rag_answer_data import (
|
||||
validate_answer_questions,
|
||||
validate_string_list,
|
||||
)
|
||||
from evaluation.rag_answer_state import (
|
||||
from tools.rag.evaluation.rag_answer_state import (
|
||||
PARTIAL_SCHEMA_VERSION,
|
||||
build_base_result,
|
||||
build_error_result,
|
||||
@ -43,11 +43,11 @@ from evaluation.rag_answer_state import (
|
||||
save_json_results,
|
||||
validate_resume_compatibility,
|
||||
)
|
||||
from evaluation.rag_metrics import (
|
||||
from tools.rag.evaluation.rag_metrics import (
|
||||
evaluate_answer,
|
||||
summarize_results,
|
||||
)
|
||||
from evaluation.rag_runner import (
|
||||
from tools.rag.evaluation.rag_runner import (
|
||||
DEFAULT_BACKOFF_BASE,
|
||||
DEFAULT_BACKOFF_MAX,
|
||||
DEFAULT_MAX_ATTEMPTS,
|
||||
@ -59,14 +59,13 @@ from evaluation.rag_runner import (
|
||||
request_json,
|
||||
run_question,
|
||||
)
|
||||
from evaluation.retrieval_runner import (
|
||||
from tools.rag.evaluation.retrieval_runner import (
|
||||
load_questions,
|
||||
)
|
||||
|
||||
|
||||
EVALUATION_DIR = (
|
||||
PROJECT_ROOT
|
||||
/ "evaluation"
|
||||
Path(__file__).resolve().parent
|
||||
)
|
||||
|
||||
JSON_FILES_DIR = (
|
||||
@ -8,19 +8,19 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
|
||||
from evaluation.metrics import (
|
||||
from tools.rag.evaluation.metrics import (
|
||||
aggregate_by_category,
|
||||
aggregate_by_difficulty,
|
||||
aggregate_metrics,
|
||||
evaluate_question_mode,
|
||||
)
|
||||
from evaluation.retrieval_runner import (
|
||||
from tools.rag.evaluation.retrieval_runner import (
|
||||
EVALUATION_MODES,
|
||||
count_splits,
|
||||
filter_questions_by_split,
|
||||
@ -29,10 +29,10 @@ from evaluation.retrieval_runner import (
|
||||
retrieve_all_modes,
|
||||
validate_dataset,
|
||||
)
|
||||
from scripts.common import DB_FILE
|
||||
from tools.rag.common import DB_FILE
|
||||
|
||||
|
||||
EVALUATION_DIR = PROJECT_ROOT / "evaluation"
|
||||
EVALUATION_DIR = Path(__file__).resolve().parent
|
||||
JSON_FILES_DIR = EVALUATION_DIR / "json_files"
|
||||
|
||||
QUESTIONS_PATH = JSON_FILES_DIR / "questions.json"
|
||||
20833
tools/rag/evaluation/json_files/questions_before_override_merge.json
Normal file
20833
tools/rag/evaluation/json_files/questions_before_override_merge.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,138 @@
|
||||
{
|
||||
"q0005": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Ján Holp“?"
|
||||
},
|
||||
"q0010": {
|
||||
"question": "Je rok 2016 rokom diplomovej práce Jána Holpa? Ak nie, čo rok 2016 označuje a v ktorom roku je uvedená jeho diplomová práca?"
|
||||
},
|
||||
"q0020": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Maroš Harahus“?"
|
||||
},
|
||||
"q0031": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Tomáš Kuchárik“?"
|
||||
},
|
||||
"q0042": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Jakub Maruniak“?"
|
||||
},
|
||||
"q0056": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Lukáš Pokrývka“?"
|
||||
},
|
||||
"q0071": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Dárius Lindvai“?"
|
||||
},
|
||||
"q0086": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Dominik Nagy“?"
|
||||
},
|
||||
"q0097": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Márk Fehér“?"
|
||||
},
|
||||
"q0106": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Patrik Pavlišin“?"
|
||||
},
|
||||
"q0116": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Dmytro Ushatenko“?"
|
||||
},
|
||||
"q0126": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Ondrej Megela“?"
|
||||
},
|
||||
"q0136": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Martin Wencel“?"
|
||||
},
|
||||
"q0146": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Dávid Omasta“?"
|
||||
},
|
||||
"q0157": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Filip Tomáš“?"
|
||||
},
|
||||
"q0168": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Kristián Sopkovič“?"
|
||||
},
|
||||
"q0178": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Michal Stromko“?"
|
||||
},
|
||||
"q0189": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Alina Vitko“?"
|
||||
},
|
||||
"q0204": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Artem Yatsenko“?"
|
||||
},
|
||||
"q0218": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Martin Šarišský“?"
|
||||
},
|
||||
"q0233": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Matej Ščišľak“?"
|
||||
},
|
||||
"q0249": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Vladimír Ferko“?"
|
||||
},
|
||||
"q0264": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Eduard Matovka“?"
|
||||
},
|
||||
"q0279": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Matej Novotný“?"
|
||||
},
|
||||
"q0292": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Andrii Pervashov“?"
|
||||
},
|
||||
"q0307": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Valerii Kutsenko“?"
|
||||
},
|
||||
"q0321": {
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Oleh Poiasnik“?"
|
||||
},
|
||||
"q0337": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Tetiana Mohorian“?"
|
||||
},
|
||||
"q0352": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Vladyslav Yanchenko“?"
|
||||
},
|
||||
"q0367": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Daniil Huzenko“?"
|
||||
},
|
||||
"q0382": {
|
||||
"question": "Aká téma diplomového projektu je uvedená pri osobe „Ján Pták“?"
|
||||
},
|
||||
"q0398": {
|
||||
"question": "Aká téma diplomového projektu je uvedená pri osobe „Ján Malinovský“?"
|
||||
},
|
||||
"q0413": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Maksym Zatirka“?"
|
||||
},
|
||||
"q0427": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Rostyslav Rodzhuk“?"
|
||||
},
|
||||
"q0441": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Yevhenii Medushivskyi“?"
|
||||
},
|
||||
"q0456": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Ronald Zalacko“?"
|
||||
},
|
||||
"q0470": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Oleksandr Dorybohov“?"
|
||||
},
|
||||
"q0484": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Simona Bobrovčanová“?"
|
||||
},
|
||||
"q0499": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Samuel Džurina“?"
|
||||
},
|
||||
"q0514": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Samuel Vasiľ“?"
|
||||
},
|
||||
"q0529": {
|
||||
"question": "Aká téma bakalárskej práce je uvedená pri osobe „Denis Le Thanh“?"
|
||||
},
|
||||
"q0024": {
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval."
|
||||
}
|
||||
}
|
||||
@ -5,7 +5,7 @@ import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from evaluation.retrieval_runner import filter_questions_by_split
|
||||
from tools.rag.evaluation.retrieval_runner import filter_questions_by_split
|
||||
|
||||
|
||||
DEFAULT_DELAY = 0.5
|
||||
@ -8,13 +8,13 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from evaluation.rag_answer_data import (
|
||||
from tools.rag.evaluation.rag_answer_data import (
|
||||
questions_fingerprint,
|
||||
)
|
||||
from evaluation.rag_metrics import (
|
||||
from tools.rag.evaluation.rag_metrics import (
|
||||
summarize_results,
|
||||
)
|
||||
from evaluation.rag_runner import (
|
||||
from tools.rag.evaluation.rag_runner import (
|
||||
LOCAL_RAG_URL,
|
||||
OPENWEBUI_URL,
|
||||
)
|
||||
@ -12,7 +12,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
OPENWEBUI_URL = "https://ui.tukekemt.xyz/api/chat/completions"
|
||||
LOCAL_OPENAPI_URL = "http://localhost:8000/openapi.json"
|
||||
@ -0,0 +1,14 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Rok začiatku štúdia Dáriusa Lindvaia je 2016, zatiaľ čo rok vyjadrený kategóriou jeho diplomovej práce (dp2021) je 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",2.042902,1.160638,21.304558,24.508885,26576,149,26725,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021, zatiaľ čo Ján Pták pracoval na diplomovom projekte s názvom „Agent pre manažment záverečných prác"" v roku 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.904721,0.881586,9.992372,12.77906,10680,215,10895,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021 a Andrii Pervashov má uvedenú diplomovú prácu s názvom „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" v roku 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.748535,1.355357,35.011748,38.116826,41730,220,41950,
|
||||
|
@ -0,0 +1,348 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T09:14:38.732799+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "c8869f550ad5048fd873d9fa125cb4dddef7f77c7637ab8906394944d9ece16e",
|
||||
"selected_question_ids": [
|
||||
"q0982",
|
||||
"q1092",
|
||||
"q1125"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 3,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 25.134924,
|
||||
"prompt_tokens": 78986,
|
||||
"completion_tokens": 584,
|
||||
"total_tokens": 79570,
|
||||
"by_category": {
|
||||
"multi_document": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 25.447943,
|
||||
"prompt_tokens": 52410,
|
||||
"completion_tokens": 435,
|
||||
"total_tokens": 52845
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 24.508885,
|
||||
"prompt_tokens": 26576,
|
||||
"completion_tokens": 149,
|
||||
"total_tokens": 26725
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 25.134924,
|
||||
"prompt_tokens": 78986,
|
||||
"completion_tokens": 584,
|
||||
"total_tokens": 79570
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0982",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2016",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Rok začiatku štúdia Dáriusa Lindvaia je 2016, zatiaľ čo rok vyjadrený kategóriou jeho diplomovej práce (dp2021) je 2021.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"limit": 10,
|
||||
"max_per_document": 5
|
||||
},
|
||||
"latency_seconds": 1.160638,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 2.042902,
|
||||
"tool_latency_seconds": 1.160638,
|
||||
"final_model_latency_seconds": 21.304558,
|
||||
"total_latency_seconds": 24.508885,
|
||||
"usage": {
|
||||
"prompt_tokens": 26576,
|
||||
"completion_tokens": 149,
|
||||
"total_tokens": 26725
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1092",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Agent pre manažment záverečných prác",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" v roku 2021, zatiaľ čo Ján Pták pracoval na diplomovom projekte s názvom „Agent pre manažment záverečných prác\" v roku 2026.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.881586,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.904721,
|
||||
"tool_latency_seconds": 0.881586,
|
||||
"final_model_latency_seconds": 9.992372,
|
||||
"total_latency_seconds": 12.77906,
|
||||
"usage": {
|
||||
"prompt_tokens": 10680,
|
||||
"completion_tokens": 215,
|
||||
"total_tokens": 10895
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1125",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" v roku 2021 a Andrii Pervashov má uvedenú diplomovú prácu s názvom „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\" v roku 2027.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 10
|
||||
},
|
||||
"latency_seconds": 1.355357,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"
|
||||
],
|
||||
"first_model_latency_seconds": 1.748535,
|
||||
"tool_latency_seconds": 1.355357,
|
||||
"final_model_latency_seconds": 35.011748,
|
||||
"total_latency_seconds": 38.116826,
|
||||
"usage": {
|
||||
"prompt_tokens": 41730,
|
||||
"completion_tokens": 220,
|
||||
"total_tokens": 41950
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,15 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.6666666666666666,0.0,0.0,0.41666666666666663,False,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci existuje dokument o grafových neurónových sieťach s autorom Marošom Harahusom, v metadátach je uvedený ako autor dokumentu Daniel Hladek, zatiaľ čo Maroš Harahuš je študentom, ktorý na tomto dokumente pracoval a má v úlohách vypracovať prehľad o grafových neurónových sieťach. Naopak, dokument o grafových neurónových sieťach, kde je Maroš Harahuš uvedený ako autor dokumentu v metadátach, nebol nájdený.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",2.190052,0.482435,20.847032,23.520147,24189,260,24449,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Valerii Kutsenko začal štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.407052,0.583248,18.338193,20.329448,23011,119,23130,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.90643,0.532748,26.941411,29.381366,31564,237,31801,
|
||||
|
@ -0,0 +1,371 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T13:54:30.413355+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "98d0a80160067c196dd4d64ac213748519e34f9238df92e97c211f8731de8182",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0580"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 3,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.555556,
|
||||
"source_url_score": 0.333333,
|
||||
"should_answer_score": 0.333333,
|
||||
"overall_score": 0.555556,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.333333,
|
||||
"mean_latency_seconds": 24.41032,
|
||||
"prompt_tokens": 78764,
|
||||
"completion_tokens": 616,
|
||||
"total_tokens": 79380,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 20.329448,
|
||||
"prompt_tokens": 23011,
|
||||
"completion_tokens": 119,
|
||||
"total_tokens": 23130
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.666667,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.416667,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 23.520147,
|
||||
"prompt_tokens": 24189,
|
||||
"completion_tokens": 260,
|
||||
"total_tokens": 24449
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.381366,
|
||||
"prompt_tokens": 31564,
|
||||
"completion_tokens": 237,
|
||||
"total_tokens": 31801
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 20.329448,
|
||||
"prompt_tokens": 23011,
|
||||
"completion_tokens": 119,
|
||||
"total_tokens": 23130
|
||||
},
|
||||
"medium": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.333333,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.333333,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 26.450757,
|
||||
"prompt_tokens": 55753,
|
||||
"completion_tokens": 497,
|
||||
"total_tokens": 56250
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci existuje dokument o grafových neurónových sieťach s autorom Marošom Harahusom, v metadátach je uvedený ako autor dokumentu Daniel Hladek, zatiaľ čo Maroš Harahuš je študentom, ktorý na tomto dokumente pracoval a má v úlohách vypracovať prehľad o grafových neurónových sieťach. Naopak, dokument o grafových neurónových sieťach, kde je Maroš Harahuš uvedený ako autor dokumentu v metadátach, nebol nájdený.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.482435,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 2.190052,
|
||||
"tool_latency_seconds": 0.482435,
|
||||
"final_model_latency_seconds": 20.847032,
|
||||
"total_latency_seconds": 23.520147,
|
||||
"usage": {
|
||||
"prompt_tokens": 24189,
|
||||
"completion_tokens": 260,
|
||||
"total_tokens": 24449
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 0.6666666666666666,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.41666666666666663,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Valerii Kutsenko začal štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10
|
||||
},
|
||||
"latency_seconds": 0.583248,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.407052,
|
||||
"tool_latency_seconds": 0.583248,
|
||||
"final_model_latency_seconds": 18.338193,
|
||||
"total_latency_seconds": 20.329448,
|
||||
"usage": {
|
||||
"prompt_tokens": 23011,
|
||||
"completion_tokens": 119,
|
||||
"total_tokens": 23130
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nhttps://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka\nhttps://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.532748,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.90643,
|
||||
"tool_latency_seconds": 0.532748,
|
||||
"final_model_latency_seconds": 26.941411,
|
||||
"total_latency_seconds": 29.381366,
|
||||
"usage": {
|
||||
"prompt_tokens": 31564,
|
||||
"completion_tokens": 237,
|
||||
"total_tokens": 31801
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,12 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, keďže Maroš Harahus je v metadátach uvedený ako autor dokumentu pre študenta Dmytra Murala, ktorého téma je „Grafové neurónové siete pre vyhľadávanie na internete"", nie ako autor vlastnej študentskej práce o grafových neurónových sieťach.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.931615,0.662689,11.145896,13.740618,11938,208,12146,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Valerii Kutsenko začal štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.432716,0.583808,1.687614,3.705032,23011,119,23130,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci Andrii Pervashov pracoval na téme extrakcie štruktúrovaných informácií z medicínskej domény a Yevhenii Medushivskyi vytváral korpus medicínskych dát, v žiadnom z nájdených záznamov sa neobjavuje konkrétna práca s názvom alebo témou „multilinguálna extrakcia trojíc z medicínskych dát"".
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",1.715903,0.529436,27.760575,30.006754,31551,257,31808,
|
||||
|
@ -0,0 +1,364 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T13:55:18.011092+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "98d0a80160067c196dd4d64ac213748519e34f9238df92e97c211f8731de8182",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0580"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 3,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.666667,
|
||||
"source_url_score": 0.666667,
|
||||
"should_answer_score": 0.666667,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 15.817468,
|
||||
"prompt_tokens": 66500,
|
||||
"completion_tokens": 584,
|
||||
"total_tokens": 67084,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 3.705032,
|
||||
"prompt_tokens": 23011,
|
||||
"completion_tokens": 119,
|
||||
"total_tokens": 23130
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 13.740618,
|
||||
"prompt_tokens": 11938,
|
||||
"completion_tokens": 208,
|
||||
"total_tokens": 12146
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 30.006754,
|
||||
"prompt_tokens": 31551,
|
||||
"completion_tokens": 257,
|
||||
"total_tokens": 31808
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 3.705032,
|
||||
"prompt_tokens": 23011,
|
||||
"completion_tokens": 119,
|
||||
"total_tokens": 23130
|
||||
},
|
||||
"medium": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 0.5,
|
||||
"should_answer_score": 0.5,
|
||||
"overall_score": 0.625,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 21.873686,
|
||||
"prompt_tokens": 43489,
|
||||
"completion_tokens": 465,
|
||||
"total_tokens": 43954
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, keďže Maroš Harahus je v metadátach uvedený ako autor dokumentu pre študenta Dmytra Murala, ktorého téma je „Grafové neurónové siete pre vyhľadávanie na internete\", nie ako autor vlastnej študentskej práce o grafových neurónových sieťach.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.662689,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"first_model_latency_seconds": 1.931615,
|
||||
"tool_latency_seconds": 0.662689,
|
||||
"final_model_latency_seconds": 11.145896,
|
||||
"total_latency_seconds": 13.740618,
|
||||
"usage": {
|
||||
"prompt_tokens": 11938,
|
||||
"completion_tokens": 208,
|
||||
"total_tokens": 12146
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Valerii Kutsenko začal štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10
|
||||
},
|
||||
"latency_seconds": 0.583808,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.432716,
|
||||
"tool_latency_seconds": 0.583808,
|
||||
"final_model_latency_seconds": 1.687614,
|
||||
"total_latency_seconds": 3.705032,
|
||||
"usage": {
|
||||
"prompt_tokens": 23011,
|
||||
"completion_tokens": 119,
|
||||
"total_tokens": 23130
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci Andrii Pervashov pracoval na téme extrakcie štruktúrovaných informácií z medicínskej domény a Yevhenii Medushivskyi vytváral korpus medicínskych dát, v žiadnom z nájdených záznamov sa neobjavuje konkrétna práca s názvom alebo témou „multilinguálna extrakcia trojíc z medicínskych dát\".\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.529436,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.715903,
|
||||
"tool_latency_seconds": 0.529436,
|
||||
"final_model_latency_seconds": 27.760575,
|
||||
"total_latency_seconds": 30.006754,
|
||||
"usage": {
|
||||
"prompt_tokens": 31551,
|
||||
"completion_tokens": 257,
|
||||
"total_tokens": 31808
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,15 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,0.0,0.75,False,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci v dokumente o študentovi Dmytrovi Muralovi je Maroš Harahus uvedený ako autor stránky a téma práce sa týka grafových neurónových sietí, v metadátach dokumentu je Maroš Harahus uvedený ako autor stránky, nie ako autor študentského dokumentu o grafových neurónových sieťach, a v dokumente o Marošovi Harahusovi sa grafové neurónové siete spomínajú len ako úloha na vypracovanie prehľadu, nie ako názov jeho vlastnej záverečnej práce.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.475591,0.479066,22.127946,24.083269,24163,264,24427,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","Valerii Kutsenko začal štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.846186,0.357021,4.919796,7.123257,4664,145,4809,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Zdroje obsahujú spomienky na „multilinguálnu extrakciu"" v súvislosti s inými študentmi (napr. Michal Stromko pri vyhľadávaní alebo Maroš Harahus pri generovaní otázok) a spracovanie medicínskych dát (napr. Yevhenii Medushivskyi pri tvorbe korpusov alebo Andrii Pervashov pri extrakcii štruktúrovaných informácií), ale žiadny z nich explicitne neuvádza prácu s presnou témou „multilinguálna extrakcia trojíc z medicínskych dát"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.884738,0.55145,5.140703,7.577916,31564,369,31933,
|
||||
|
@ -0,0 +1,353 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T13:55:56.987787+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "98d0a80160067c196dd4d64ac213748519e34f9238df92e97c211f8731de8182",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0580"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 3,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.666667,
|
||||
"source_url_score": 0.666667,
|
||||
"should_answer_score": 0.333333,
|
||||
"overall_score": 0.666667,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.333333,
|
||||
"mean_latency_seconds": 12.928147,
|
||||
"prompt_tokens": 60391,
|
||||
"completion_tokens": 778,
|
||||
"total_tokens": 61169,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 7.123257,
|
||||
"prompt_tokens": 4664,
|
||||
"completion_tokens": 145,
|
||||
"total_tokens": 4809
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 24.083269,
|
||||
"prompt_tokens": 24163,
|
||||
"completion_tokens": 264,
|
||||
"total_tokens": 24427
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 7.577916,
|
||||
"prompt_tokens": 31564,
|
||||
"completion_tokens": 369,
|
||||
"total_tokens": 31933
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 7.123257,
|
||||
"prompt_tokens": 4664,
|
||||
"completion_tokens": 145,
|
||||
"total_tokens": 4809
|
||||
},
|
||||
"medium": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 0.5,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.5,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 15.830593,
|
||||
"prompt_tokens": 55727,
|
||||
"completion_tokens": 633,
|
||||
"total_tokens": 56360
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci v dokumente o študentovi Dmytrovi Muralovi je Maroš Harahus uvedený ako autor stránky a téma práce sa týka grafových neurónových sietí, v metadátach dokumentu je Maroš Harahus uvedený ako autor stránky, nie ako autor študentského dokumentu o grafových neurónových sieťach, a v dokumente o Marošovi Harahusovi sa grafové neurónové siete spomínajú len ako úloha na vypracovanie prehľadu, nie ako názov jeho vlastnej záverečnej práce.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural\nhttps://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.479066,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.475591,
|
||||
"tool_latency_seconds": 0.479066,
|
||||
"final_model_latency_seconds": 22.127946,
|
||||
"total_latency_seconds": 24.083269,
|
||||
"usage": {
|
||||
"prompt_tokens": 24163,
|
||||
"completion_tokens": 264,
|
||||
"total_tokens": 24427
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Valerii Kutsenko začal štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.357021,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"first_model_latency_seconds": 1.846186,
|
||||
"tool_latency_seconds": 0.357021,
|
||||
"final_model_latency_seconds": 4.919796,
|
||||
"total_latency_seconds": 7.123257,
|
||||
"usage": {
|
||||
"prompt_tokens": 4664,
|
||||
"completion_tokens": 145,
|
||||
"total_tokens": 4809
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Zdroje obsahujú spomienky na „multilinguálnu extrakciu\" v súvislosti s inými študentmi (napr. Michal Stromko pri vyhľadávaní alebo Maroš Harahus pri generovaní otázok) a spracovanie medicínskych dát (napr. Yevhenii Medushivskyi pri tvorbe korpusov alebo Andrii Pervashov pri extrakcii štruktúrovaných informácií), ale žiadny z nich explicitne neuvádza prácu s presnou témou „multilinguálna extrakcia trojíc z medicínskych dát\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.55145,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.884738,
|
||||
"tool_latency_seconds": 0.55145,
|
||||
"final_model_latency_seconds": 5.140703,
|
||||
"total_latency_seconds": 7.577916,
|
||||
"usage": {
|
||||
"prompt_tokens": 31564,
|
||||
"completion_tokens": 369,
|
||||
"total_tokens": 31933
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,66 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0004,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý rieši tému „Systém získavania informácií v slovenskom jazyku“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,1.0,0.5,False,"[""Ján Holp"", ""Systém získavania informácií v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","[""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2019/david_stromp""]","Študent Michal Stromko riešil v rámci svojho diplomového projektu 2 vytvorenie novej databázy pre vyhodnotenie získavania informácií v slovenskom jazyku a v bakalárskej práci 2022 sa venoval indexovaniu slovenského textu s cieľom vypracovať prehľad metód pre získavanie informácií.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",2.264907,0.939195,24.041229,27.246063,29443,203,29646,
|
||||
q0026,dev,specific_fact,medium,Akú konkrétnu skupinu neurónových sietí mal Maroš Harahus naštudovať v rámci úloh po úvodnom stretnutí s US Steel?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/topics/steel"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/dp21"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2019/samuel_horani"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","Maroš Harahus mal po úvodnom stretnutí s US Steel naštudovať a vypracovať prehľad aktuálnych metód grafových neurónových sietí (GNN).
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.820668,1.166803,37.166568,40.155286,44981,150,45131,
|
||||
q0040,dev,reverse_lookup,medium,Kto sa venuje práci s názvom „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,1.0,0.5,False,"[""Jakub Maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/topics/otvorene"", ""https://zp.kemt.fei.tuke.sk/students/2018/martin_wencel"", ""https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets"", ""https://zp.kemt.fei.tuke.sk/topics/spacy"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod""]","V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť konkrétnu študentskú prácu s presným názvom „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/otvorene",1.536128,0.410449,21.557901,23.505169,26017,136,26153,
|
||||
q0041,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý rieši tému „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,0.0,1.0,0.625,False,"[""Jakub Maruniak"", ""Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2018/martin_wencel"", ""https://zp.kemt.fei.tuke.sk/topics/spacy"", ""https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod""]","Študent Than Trung Thanh vypracoval bakalársky projekt v roku 2020 s názvom „Rozpoznávanie pomenovaných entít v slovenskom jazyku pomocou nástrojov Spacy a Prodigy"", pri ktorom sa venoval tvorbe korpusu, anotáciám a vývoju modelu pre túto úlohu. Ďalším študentom, ktorý rieši tému rozpoznávania pomenovaných entít v slovenčine, je Serhii Yemets, ktorý v rámci bakalárskej práce v roku 2025 pracuje na zlepšení slovenského modelu pre túto úlohu s využitím modelov typu BERT.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets",1.775508,0.357276,25.125982,27.259509,27920,286,28206,
|
||||
q0253,dev,disambiguation_year,hard,Rozlíš pri osobe Vladimír Ferko rok začiatku štúdia a rok uvedený v názve sekcie jeho bakalárskej práce.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2021"", ""2024""]","[""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko""]","[""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"", ""https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat""]","Vladimír Ferko začal štúdium v roku 2021, zatiaľ čo rok uvedený v názve sekcie jeho bakalárskej práce je 2024.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko",1.850572,0.503375,29.198301,31.553134,35152,149,35301,
|
||||
q0336,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý má bakalársku prácu s názvom „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Tetiana Mohorian"", ""Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","[""https://zp.kemt.fei.tuke.sk/students/2022/samuel_vasil"", ""https://zp.kemt.fei.tuke.sk/students/2023/oleksandr_dorybohov"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/samuel_vasil
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/oleksandr_dorybohov
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.919063,0.482241,24.748093,27.15017,27538,237,27775,
|
||||
q0570,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""anotácia pomenovaných entít""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","Cesar Abascal Gutierrez sa počas svojej stáže venoval téme automatickej annotácie entít v prekladoch reči (speech transcripts), kde jeho cieľom bolo vytvoriť manuálne annotovanú trénovaciu množinu dát a navrhnúť schému annotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.786802,0.267954,12.171385,14.226604,14010,166,14176,
|
||||
q0633,dev,topic_page_lookup,medium,"Ktorá tematická stránka ZP Wiki, nie študentská práca, sa priamo venuje téme „Strojový preklad“?",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Strojový preklad""]","[""https://zp.kemt.fei.tuke.sk/topics/translation""]","[""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/topics/otvorene""]","Tematícka stránka ZP Wiki, ktorá sa priamo venuje téme „Strojový preklad"" a nie je študentskou prácou, sa nazýva „Strojový preklad"" a nachádza sa pod URL adresou https://zp.kemt.fei.tuke.sk/topics/translation.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/translation",1.859107,0.970622,32.057544,34.888168,38357,167,38524,
|
||||
q0968,dev,single_document_synthesis,hard,Pri osobe Ján Pták uveď názov záznamu typu „diplomový projekt“ a rok uvedený v názve príslušnej sekcie. Neuvádzaj rok začiatku štúdia.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","Pre Ján Pták je názvom záznamu typu „diplomový projekt“ sekcia „Diplomový projekt 2026“ a rok uvedený v názve príslušnej sekcie je 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.39529,0.456783,22.790452,24.643161,27026,129,27155,
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca je vedená pod kategóriou roku 2020 (s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí""), pričom rok 2020 v sekcii „Diplomový projekt 2 2020"" označuje rok realizácie projektu a nie rok začiatku štúdia.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.627795,1.417725,38.438714,41.485325,44152,197,44349,
|
||||
q0990,dev,single_document_synthesis,hard,"V dokumente osoby Tomáš Kuchárik, ktorý súvisí s témou „strojový preklad“, uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"", ""2022""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura""]","Tomáš Kuchárik vypracovával diplomovú prácu s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"", ktorá je uvedená v roku 2022.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",1.920394,0.796818,21.817399,24.535242,25948,170,26118,
|
||||
q1022,dev,single_document_synthesis,hard,"V dokumente osoby Dárius Lindvai, ktorý súvisí s témou „interpunkcia“, uveď názov diplomovej práce a rok vyjadrený kategóriou tejto diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","Diplomová práca Dáriusa Lindvaiho s témou súvisiacou s interpunkciou sa volá „Obnovenie interpunkcie pomocou hlbokých neurónových sietí“ a jej kategóriou je rok 2021, ktorý je uvedený v názve repozitára a kontexte diplomového projektu (DP2021).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.895492,0.694162,8.408121,10.998024,8309,195,8504,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.75,1.0,1.0,0.9375,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky""]","Dárius Lindvai má v ZP Wiki evidovanú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v rámci diplomového projektu pre rok 2020, pričom v dokumentoch nie je explicitne uvedená sekcia označená ako „diplomová práca"" s iným rokom. Ján Pták má v zázname uvedenú prácu s názvom „Agent pre manažment záverečných prác"" v rámci diplomového projektu pre rok 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.963353,1.341374,14.677213,17.982364,16271,253,16524,
|
||||
q1109,dev,multi_document,hard,"Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agentové systémy pomocou veľkého jazykového modelu"", ""2026"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","V dostupných dokumentoch ZP Wiki sa pre Martina Šarišského v roku 2026 uvádza diplomová práca s názvom „Agentové systémy pomocou veľkého jazykového modelu“ a pre Andriiho Pervashova v roku 2027 je uvedená diplomová práca s návrhom názvu (témy) „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu“.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.876024,0.881262,24.716133,27.47413,28433,235,28668,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.75,1.0,1.0,0.9375,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaia nenašla žiadna záverečná práca typu „diplomová práca"", nakoľko jeho záznam uvádza len „Diplomový projekt 2020"" s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", a pre Andriiho Pervashova sa nenašla explicitne uvedená „diplomová práca"" s rokom v názve sekcie, keďže sekcia „Diplomová práca 2027"" obsahuje iba návrh na tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" bez potvrdeného roku ukončenia v názve práce.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",2.153057,0.951525,24.234206,27.339587,26186,325,26511,
|
||||
q1136,dev,multi_document,hard,"Porovnaj záznamy osôb Ján Malinovský a Yevhenii Medushivskyi. Pri Jánovi Malinovskom použi diplomový projekt a pri Yevhenii Medushivskom bakalársku prácu. Pri oboch uveď názov práce a rok uvedený v názve príslušnej sekcie, nie rok začiatku štúdia.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Grafové vyhľadávanie pre podporu vzdelávania"", ""2026"", ""Získavanie textu z webových stránok"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky""]","Ján Malinovský vypracováva diplomový projekt s názvom „Grafové vyhľadávanie pre podporu vzdelávania"" v roku 2026 a Yevhenii Medushivskyi vypracováva bakalársku prácu s názvom „Získavanie textu z webových stránok"" v roku 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",1.888024,0.29403,13.043088,15.225568,14164,217,14381,
|
||||
|
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,24 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,0.0,0.75,False,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Zdrojové dokumenty síce spomínajú Maroša Harahusa ako autora dokumentu pre študenta Dmytra Murala (tema: „Grafové neurónové siete pre vyhľadávanie na internete""), ale v metadátach dokumentu o Marošovi Harahusovi je uvedeným autorom Daniel Hladek a v texte jeho stránky sa o grafových neurónových sieťach hovorí len ako o úlohe na vypracovanie prehľadu, nie ako o názve jeho vlastnej záverečnej práce.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.185565,0.535819,20.569647,23.291825,24257,264,24521,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","Valerii Kutsenko začala štúdium v roku 2022 a jej diplomová práca bola realizovaná v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.832421,0.303168,4.572551,6.708397,4729,144,4873,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorej cieľom je vytvorenie vzdelávacích materiálov o Kubernetes a funkčného prototypu privátneho klastra v kombinácii s verejnými službami.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.553068,0.340111,13.728013,15.62183,16356,178,16534,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z viacjazyčných lekárskych dát s cieľom zostrojiť znalostný graf z balíčkových letákov liekov a využiť ho v inteligentnom agentovi na odporúčanie liečiv.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.773057,0.276874,10.683296,12.733637,12489,172,12661,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka
|
||||
https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",1.869678,0.482586,27.027717,29.380772,31731,238,31969,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiș mal zostaviť znalostný graf z lekárskych príbalových letákov (medical package inserts) v viacerých jazykoch v rámci svojej stáže (ERASMUS Intern).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.618991,0.285566,13.811282,15.71649,16511,141,16652,
|
||||
|
@ -0,0 +1,614 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T14:56:07.499665+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.833333,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.666667,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 17.242158,
|
||||
"prompt_tokens": 106073,
|
||||
"completion_tokens": 1137,
|
||||
"total_tokens": 107210,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.708397,
|
||||
"prompt_tokens": 4729,
|
||||
"completion_tokens": 144,
|
||||
"total_tokens": 4873
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 12.733637,
|
||||
"prompt_tokens": 12489,
|
||||
"completion_tokens": 172,
|
||||
"total_tokens": 12661
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 0.5,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 19.456827,
|
||||
"prompt_tokens": 40613,
|
||||
"completion_tokens": 442,
|
||||
"total_tokens": 41055
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.380772,
|
||||
"prompt_tokens": 31731,
|
||||
"completion_tokens": 238,
|
||||
"total_tokens": 31969
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.71649,
|
||||
"prompt_tokens": 16511,
|
||||
"completion_tokens": 141,
|
||||
"total_tokens": 16652
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.708397,
|
||||
"prompt_tokens": 4729,
|
||||
"completion_tokens": 144,
|
||||
"total_tokens": 4873
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.8,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.6,
|
||||
"overall_score": 0.8,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.6,
|
||||
"mean_latency_seconds": 19.348911,
|
||||
"prompt_tokens": 101344,
|
||||
"completion_tokens": 993,
|
||||
"total_tokens": 102337
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Zdrojové dokumenty síce spomínajú Maroša Harahusa ako autora dokumentu pre študenta Dmytra Murala (tema: „Grafové neurónové siete pre vyhľadávanie na internete\"), ale v metadátach dokumentu o Marošovi Harahusovi je uvedeným autorom Daniel Hladek a v texte jeho stránky sa o grafových neurónových sieťach hovorí len ako o úlohe na vypracovanie prehľadu, nie ako o názve jeho vlastnej záverečnej práce.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.535819,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 2.185565,
|
||||
"tool_latency_seconds": 0.535819,
|
||||
"final_model_latency_seconds": 20.569647,
|
||||
"total_latency_seconds": 23.291825,
|
||||
"usage": {
|
||||
"prompt_tokens": 24257,
|
||||
"completion_tokens": 264,
|
||||
"total_tokens": 24521
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Valerii Kutsenko začala štúdium v roku 2022 a jej diplomová práca bola realizovaná v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.303168,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"first_model_latency_seconds": 1.832421,
|
||||
"tool_latency_seconds": 0.303168,
|
||||
"final_model_latency_seconds": 4.572551,
|
||||
"total_latency_seconds": 6.708397,
|
||||
"usage": {
|
||||
"prompt_tokens": 4729,
|
||||
"completion_tokens": 144,
|
||||
"total_tokens": 4873
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorej cieľom je vytvorenie vzdelávacích materiálov o Kubernetes a funkčného prototypu privátneho klastra v kombinácii s verejnými službami.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.340111,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.553068,
|
||||
"tool_latency_seconds": 0.340111,
|
||||
"final_model_latency_seconds": 13.728013,
|
||||
"total_latency_seconds": 15.62183,
|
||||
"usage": {
|
||||
"prompt_tokens": 16356,
|
||||
"completion_tokens": 178,
|
||||
"total_tokens": 16534
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z viacjazyčných lekárskych dát s cieľom zostrojiť znalostný graf z balíčkových letákov liekov a využiť ho v inteligentnom agentovi na odporúčanie liečiv.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"published_only": true,
|
||||
"limit": 5,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.276874,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.773057,
|
||||
"tool_latency_seconds": 0.276874,
|
||||
"final_model_latency_seconds": 10.683296,
|
||||
"total_latency_seconds": 12.733637,
|
||||
"usage": {
|
||||
"prompt_tokens": 12489,
|
||||
"completion_tokens": 172,
|
||||
"total_tokens": 12661
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nhttps://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka\nhttps://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.482586,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.869678,
|
||||
"tool_latency_seconds": 0.482586,
|
||||
"final_model_latency_seconds": 27.027717,
|
||||
"total_latency_seconds": 29.380772,
|
||||
"usage": {
|
||||
"prompt_tokens": 31731,
|
||||
"completion_tokens": 238,
|
||||
"total_tokens": 31969
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiș mal zostaviť znalostný graf z lekárskych príbalových letákov (medical package inserts) v viacerých jazykoch v rámci svojej stáže (ERASMUS Intern).\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.285566,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.618991,
|
||||
"tool_latency_seconds": 0.285566,
|
||||
"final_model_latency_seconds": 13.811282,
|
||||
"total_latency_seconds": 15.71649,
|
||||
"usage": {
|
||||
"prompt_tokens": 16511,
|
||||
"completion_tokens": 141,
|
||||
"total_tokens": 16652
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,25 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, keďže dokumenty o grafových neurónových sieťach majú v metadátach ako autora dokumentu Daniel Hladek (dokumenty študentov Dmytro Mural a Oleksandr Hryshchenko) alebo súvisia s Marošom Harahusom ako vedúcim, nie ako autorom dokumentu, ktorý by bol o grafových neurónových sieťach.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",1.917333,0.435663,21.657875,24.011559,24252,279,24531,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu""]","Osoba Valerii Kutsenko začala štúdium v roku 2022 a vypracovávala diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.659944,0.329471,11.619301,13.609409,13872,134,14006,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Študent Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zaoberá hybridným cloudom.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.817412,0.360819,13.656469,15.835248,16369,163,16532,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala téme viacerojazyčnej extrakcie trojíc z lekárskych údajov, s cieľom zostrojiť znalostný graf z lekárskych príbalových letákov v rôznych jazykoch a využiť ho v inteligentnom agentovi pre odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.823478,0.198371,10.966602,12.988833,12489,184,12673,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Hoci sa v zdrojoch nachádzajú študenti pracujúci na témach súvisiacich s extrakciou informácií z medicínskych dát (Andrii Pervashov s tématikou extrakcie štruktúrovaných informácií) alebo s multilinguálnym spracovaním (Michal Stromko s multilinguálnym vyhľadávaním, Yevhenii Medushivskyi s vytváraním korpusov vrátane medicínskych), žiadny z nájdených záznamov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát"" ani meno študenta, ktorý by na nej pracoval.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",1.894309,0.520932,29.369732,31.785818,31629,364,31993,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiš mal zostaviť znalostný graf z viacjazyčných príbalových letákov na lieky v rámci svojej praxe Erasmus intern.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.420306,0.293604,13.747362,15.461826,16498,116,16614,
|
||||
|
@ -0,0 +1,621 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T14:58:01.449861+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.833333,
|
||||
"overall_score": 0.791667,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 18.948782,
|
||||
"prompt_tokens": 115109,
|
||||
"completion_tokens": 1240,
|
||||
"total_tokens": 116349,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 13.609409,
|
||||
"prompt_tokens": 13872,
|
||||
"completion_tokens": 134,
|
||||
"total_tokens": 14006
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 12.988833,
|
||||
"prompt_tokens": 12489,
|
||||
"completion_tokens": 184,
|
||||
"total_tokens": 12673
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 19.923403,
|
||||
"prompt_tokens": 40621,
|
||||
"completion_tokens": 442,
|
||||
"total_tokens": 41063
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 31.785818,
|
||||
"prompt_tokens": 31629,
|
||||
"completion_tokens": 364,
|
||||
"total_tokens": 31993
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 15.461826,
|
||||
"prompt_tokens": 16498,
|
||||
"completion_tokens": 116,
|
||||
"total_tokens": 16614
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 13.609409,
|
||||
"prompt_tokens": 13872,
|
||||
"completion_tokens": 134,
|
||||
"total_tokens": 14006
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.4,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.8,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.4,
|
||||
"mean_latency_seconds": 20.016657,
|
||||
"prompt_tokens": 101237,
|
||||
"completion_tokens": 1106,
|
||||
"total_tokens": 102343
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, keďže dokumenty o grafových neurónových sieťach majú v metadátach ako autora dokumentu Daniel Hladek (dokumenty študentov Dmytro Mural a Oleksandr Hryshchenko) alebo súvisia s Marošom Harahusom ako vedúcim, nie ako autorom dokumentu, ktorý by bol o grafových neurónových sieťach.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural\nZdroj: https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.435663,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.917333,
|
||||
"tool_latency_seconds": 0.435663,
|
||||
"final_model_latency_seconds": 21.657875,
|
||||
"total_latency_seconds": 24.011559,
|
||||
"usage": {
|
||||
"prompt_tokens": 24252,
|
||||
"completion_tokens": 279,
|
||||
"total_tokens": 24531
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala štúdium v roku 2022 a vypracovávala diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.329471,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"
|
||||
],
|
||||
"first_model_latency_seconds": 1.659944,
|
||||
"tool_latency_seconds": 0.329471,
|
||||
"final_model_latency_seconds": 11.619301,
|
||||
"total_latency_seconds": 13.609409,
|
||||
"usage": {
|
||||
"prompt_tokens": 13872,
|
||||
"completion_tokens": 134,
|
||||
"total_tokens": 14006
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Študent Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zaoberá hybridným cloudom.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.360819,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.817412,
|
||||
"tool_latency_seconds": 0.360819,
|
||||
"final_model_latency_seconds": 13.656469,
|
||||
"total_latency_seconds": 15.835248,
|
||||
"usage": {
|
||||
"prompt_tokens": 16369,
|
||||
"completion_tokens": 163,
|
||||
"total_tokens": 16532
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala téme viacerojazyčnej extrakcie trojíc z lekárskych údajov, s cieľom zostrojiť znalostný graf z lekárskych príbalových letákov v rôznych jazykoch a využiť ho v inteligentnom agentovi pre odporúčanie liekov.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.198371,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.823478,
|
||||
"tool_latency_seconds": 0.198371,
|
||||
"final_model_latency_seconds": 10.966602,
|
||||
"total_latency_seconds": 12.988833,
|
||||
"usage": {
|
||||
"prompt_tokens": 12489,
|
||||
"completion_tokens": 184,
|
||||
"total_tokens": 12673
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nHoci sa v zdrojoch nachádzajú študenti pracujúci na témach súvisiacich s extrakciou informácií z medicínskych dát (Andrii Pervashov s tématikou extrakcie štruktúrovaných informácií) alebo s multilinguálnym spracovaním (Michal Stromko s multilinguálnym vyhľadávaním, Yevhenii Medushivskyi s vytváraním korpusov vrátane medicínskych), žiadny z nájdených záznamov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát\" ani meno študenta, ktorý by na nej pracoval.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nZdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.520932,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.894309,
|
||||
"tool_latency_seconds": 0.520932,
|
||||
"final_model_latency_seconds": 29.369732,
|
||||
"total_latency_seconds": 31.785818,
|
||||
"usage": {
|
||||
"prompt_tokens": 31629,
|
||||
"completion_tokens": 364,
|
||||
"total_tokens": 31993
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiš mal zostaviť znalostný graf z viacjazyčných príbalových letákov na lieky v rámci svojej praxe Erasmus intern.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.293604,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.420306,
|
||||
"tool_latency_seconds": 0.293604,
|
||||
"final_model_latency_seconds": 13.747362,
|
||||
"total_latency_seconds": 15.461826,
|
||||
"usage": {
|
||||
"prompt_tokens": 16498,
|
||||
"completion_tokens": 116,
|
||||
"total_tokens": 16614
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,24 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,0.0,0.75,False,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci dokument o Dmytrovi Muralovi má v metadátach ako autora uvedeného Maroša Harahusa a pojednáva o grafových neurónových sieťach, otázka sa pýtala na dokument, kde je Maroš Harahus uvedený ako autor a ktorý by bol o grafových neurónových sieťach, no v jeho vlastnom dokumente (Maroš Harahus) sa táto téma objavuje len ako úloha na vypracovanie prehľadu, nie ako názov jeho záverečnej práce. Dokument Dmytra Murala spĺňa podmienku autora, ale nie je to dokument ""o"" Marošovi Harahusovi v zmysle toho, že by bol jeho prácou, ale prácou Dmytra Murala, ktorú autorizoval Maroš. Žiadny dokument v zozname nemá v metadátach autora Maroša Harahusa a zároveň je to práca, ktorú vypracoval Maroš Harahus o grafových neurónových sieťach.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.917805,0.46196,23.118097,25.498783,24254,368,24622,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Valerii Kutsenko začal štúdium v roku 2022 a jeho diplomová práca bola vypracovaná v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.731198,0.565053,18.886992,21.184424,23094,137,23231,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zameriava na kombináciu verejných a privátnych cloudových prostredí.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.778938,0.381112,14.23094,16.391602,16368,171,16539,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z lekárskych dát v viacerých jazykoch, konkrétne s cieľom zostrojiť znalostný graf z balíčkových letákov liekov a využiť ho v inteligentnom agentovi na odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.781939,0.262713,12.352614,14.397747,14474,177,14651,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.935774,0.543638,27.495506,29.975693,31630,253,31883,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiș mal zostaviť znalostný graf z lekárskeho obsahu, konkrétne z príbalových letákov liekov (medical package inserts) v niekoľkých jazykoch.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.230288,0.357349,13.867596,15.456182,16486,112,16598,
|
||||
|
@ -0,0 +1,632 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T15:00:04.555143+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.833333,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.666667,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 20.484072,
|
||||
"prompt_tokens": 126306,
|
||||
"completion_tokens": 1218,
|
||||
"total_tokens": 127524,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 21.184424,
|
||||
"prompt_tokens": 23094,
|
||||
"completion_tokens": 137,
|
||||
"total_tokens": 23231
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 14.397747,
|
||||
"prompt_tokens": 14474,
|
||||
"completion_tokens": 177,
|
||||
"total_tokens": 14651
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 0.5,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 20.945192,
|
||||
"prompt_tokens": 40622,
|
||||
"completion_tokens": 539,
|
||||
"total_tokens": 41161
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.975693,
|
||||
"prompt_tokens": 31630,
|
||||
"completion_tokens": 253,
|
||||
"total_tokens": 31883
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.456182,
|
||||
"prompt_tokens": 16486,
|
||||
"completion_tokens": 112,
|
||||
"total_tokens": 16598
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 21.184424,
|
||||
"prompt_tokens": 23094,
|
||||
"completion_tokens": 137,
|
||||
"total_tokens": 23231
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.8,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.6,
|
||||
"overall_score": 0.8,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.6,
|
||||
"mean_latency_seconds": 20.344001,
|
||||
"prompt_tokens": 103212,
|
||||
"completion_tokens": 1081,
|
||||
"total_tokens": 104293
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci dokument o Dmytrovi Muralovi má v metadátach ako autora uvedeného Maroša Harahusa a pojednáva o grafových neurónových sieťach, otázka sa pýtala na dokument, kde je Maroš Harahus uvedený ako autor a ktorý by bol o grafových neurónových sieťach, no v jeho vlastnom dokumente (Maroš Harahus) sa táto téma objavuje len ako úloha na vypracovanie prehľadu, nie ako názov jeho záverečnej práce. Dokument Dmytra Murala spĺňa podmienku autora, ale nie je to dokument \"o\" Marošovi Harahusovi v zmysle toho, že by bol jeho prácou, ale prácou Dmytra Murala, ktorú autorizoval Maroš. Žiadny dokument v zozname nemá v metadátach autora Maroša Harahusa a zároveň je to práca, ktorú vypracoval Maroš Harahus o grafových neurónových sieťach.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.46196,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.917805,
|
||||
"tool_latency_seconds": 0.46196,
|
||||
"final_model_latency_seconds": 23.118097,
|
||||
"total_latency_seconds": 25.498783,
|
||||
"usage": {
|
||||
"prompt_tokens": 24254,
|
||||
"completion_tokens": 368,
|
||||
"total_tokens": 24622
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Valerii Kutsenko začal štúdium v roku 2022 a jeho diplomová práca bola vypracovaná v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.565053,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.731198,
|
||||
"tool_latency_seconds": 0.565053,
|
||||
"final_model_latency_seconds": 18.886992,
|
||||
"total_latency_seconds": 21.184424,
|
||||
"usage": {
|
||||
"prompt_tokens": 23094,
|
||||
"completion_tokens": 137,
|
||||
"total_tokens": 23231
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zameriava na kombináciu verejných a privátnych cloudových prostredí.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.381112,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.778938,
|
||||
"tool_latency_seconds": 0.381112,
|
||||
"final_model_latency_seconds": 14.23094,
|
||||
"total_latency_seconds": 16.391602,
|
||||
"usage": {
|
||||
"prompt_tokens": 16368,
|
||||
"completion_tokens": 171,
|
||||
"total_tokens": 16539
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z lekárskych dát v viacerých jazykoch, konkrétne s cieľom zostrojiť znalostný graf z balíčkových letákov liekov a využiť ho v inteligentnom agentovi na odporúčanie liekov.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.262713,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.781939,
|
||||
"tool_latency_seconds": 0.262713,
|
||||
"final_model_latency_seconds": 12.352614,
|
||||
"total_latency_seconds": 14.397747,
|
||||
"usage": {
|
||||
"prompt_tokens": 14474,
|
||||
"completion_tokens": 177,
|
||||
"total_tokens": 14651
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nZdroj: https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.543638,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.935774,
|
||||
"tool_latency_seconds": 0.543638,
|
||||
"final_model_latency_seconds": 27.495506,
|
||||
"total_latency_seconds": 29.975693,
|
||||
"usage": {
|
||||
"prompt_tokens": 31630,
|
||||
"completion_tokens": 253,
|
||||
"total_tokens": 31883
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiș mal zostaviť znalostný graf z lekárskeho obsahu, konkrétne z príbalových letákov liekov (medical package inserts) v niekoľkých jazykoch.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?"
|
||||
},
|
||||
"latency_seconds": 0.357349,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.230288,
|
||||
"tool_latency_seconds": 0.357349,
|
||||
"final_model_latency_seconds": 13.867596,
|
||||
"total_latency_seconds": 15.456182,
|
||||
"usage": {
|
||||
"prompt_tokens": 16486,
|
||||
"completion_tokens": 112,
|
||||
"total_tokens": 16598
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,21 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,0.0,0.75,False,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/dp21"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2019/samuel_horani""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci dokument o študentovi Marošovi Harahusovi obsahuje úlohu vypracovať prehľad metód grafových neurónových sietí, žiadny študentský dokument s ním ako autorom nezoznamuje tému grafových neurónových sietí ako svoju vlastnú záverečnú prácu. Naopak, študent Dmytro Mural má v metadátach ako autora dokumentu uvedeného Maroša Harahusa a jeho práca sa týka grafových neurónových sietí pre vyhľadávanie na internete.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.184397,1.253753,39.554929,42.994515,46122,265,46387,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu""]","Osoba Valerii Kutsenko začala študovať v roku 2022 a vypracováva diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.225255,0.33161,11.757217,13.314717,13782,108,13890,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.77286,0.466661,13.173697,15.413718,16304,149,16453,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala problému viacjazyčného extrahovania trojíc (triplet extraction) z medicínskych údajov, pričom cieľom bolo vytvoriť znalostný graf z lekárskych príbalových letákov a využiť ho v inteligentnom agentovi na odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.804197,0.21156,10.771943,12.788098,12423,182,12605,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci niektoré zdroje spomínajú prácu s medicínskymi dátami (napr. Yevhenii Medushivskyi, ktorý plánoval vytvoriť korpus medicínskych dát, alebo Andrii Pervashov, ktorý sa zameriava na extrakciu informácií z medicínskej domény), žiadny z vrátených dokumentov explicitne nezaznamenáva prácu konkrétnej osoby na téme „multilinguálna extrakcia trojíc z medicínskych dát"".
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.90076,2.02751,27.928895,31.858084,31666,288,31954,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiš mal zostaviť znalostný graf z viacjazyčných príbalových letákov liekov (medical package inserts) v rámci svojej stáže ERASMUS.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.402649,0.314172,13.699309,15.416748,16433,121,16554,
|
||||
|
@ -0,0 +1,629 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T14:14:16.324361+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.583333,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.666667,
|
||||
"overall_score": 0.770833,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.333333,
|
||||
"mean_latency_seconds": 21.964313,
|
||||
"prompt_tokens": 136730,
|
||||
"completion_tokens": 1113,
|
||||
"total_tokens": 137843,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 13.314717,
|
||||
"prompt_tokens": 13782,
|
||||
"completion_tokens": 108,
|
||||
"total_tokens": 13890
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 12.788098,
|
||||
"prompt_tokens": 12423,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 12605
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.75,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 0.5,
|
||||
"overall_score": 0.8125,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.204116,
|
||||
"prompt_tokens": 62426,
|
||||
"completion_tokens": 414,
|
||||
"total_tokens": 62840
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 31.858084,
|
||||
"prompt_tokens": 31666,
|
||||
"completion_tokens": 288,
|
||||
"total_tokens": 31954
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.416748,
|
||||
"prompt_tokens": 16433,
|
||||
"completion_tokens": 121,
|
||||
"total_tokens": 16554
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 13.314717,
|
||||
"prompt_tokens": 13782,
|
||||
"completion_tokens": 108,
|
||||
"total_tokens": 13890
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.6,
|
||||
"overall_score": 0.725,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.2,
|
||||
"mean_latency_seconds": 23.694233,
|
||||
"prompt_tokens": 122948,
|
||||
"completion_tokens": 1005,
|
||||
"total_tokens": 123953
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci dokument o študentovi Marošovi Harahusovi obsahuje úlohu vypracovať prehľad metód grafových neurónových sietí, žiadny študentský dokument s ním ako autorom nezoznamuje tému grafových neurónových sietí ako svoju vlastnú záverečnú prácu. Naopak, študent Dmytro Mural má v metadátach ako autora dokumentu uvedeného Maroša Harahusa a jeho práca sa týka grafových neurónových sietí pre vyhľadávanie na internete.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 10,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 1.253753,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/dp21",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/samuel_horani"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/dp21",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/samuel_horani"
|
||||
],
|
||||
"first_model_latency_seconds": 2.184397,
|
||||
"tool_latency_seconds": 1.253753,
|
||||
"final_model_latency_seconds": 39.554929,
|
||||
"total_latency_seconds": 42.994515,
|
||||
"usage": {
|
||||
"prompt_tokens": 46122,
|
||||
"completion_tokens": 265,
|
||||
"total_tokens": 46387
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala študovať v roku 2022 a vypracováva diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko."
|
||||
},
|
||||
"latency_seconds": 0.33161,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"
|
||||
],
|
||||
"first_model_latency_seconds": 1.225255,
|
||||
"tool_latency_seconds": 0.33161,
|
||||
"final_model_latency_seconds": 11.757217,
|
||||
"total_latency_seconds": 13.314717,
|
||||
"usage": {
|
||||
"prompt_tokens": 13782,
|
||||
"completion_tokens": 108,
|
||||
"total_tokens": 13890
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.466661,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.77286,
|
||||
"tool_latency_seconds": 0.466661,
|
||||
"final_model_latency_seconds": 13.173697,
|
||||
"total_latency_seconds": 15.413718,
|
||||
"usage": {
|
||||
"prompt_tokens": 16304,
|
||||
"completion_tokens": 149,
|
||||
"total_tokens": 16453
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.5,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala problému viacjazyčného extrahovania trojíc (triplet extraction) z medicínskych údajov, pričom cieľom bolo vytvoriť znalostný graf z lekárskych príbalových letákov a využiť ho v inteligentnom agentovi na odporúčanie liekov.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"published_only": true,
|
||||
"limit": 5,
|
||||
"max_per_document": 5
|
||||
},
|
||||
"latency_seconds": 0.21156,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.804197,
|
||||
"tool_latency_seconds": 0.21156,
|
||||
"final_model_latency_seconds": 10.771943,
|
||||
"total_latency_seconds": 12.788098,
|
||||
"usage": {
|
||||
"prompt_tokens": 12423,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 12605
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci niektoré zdroje spomínajú prácu s medicínskymi dátami (napr. Yevhenii Medushivskyi, ktorý plánoval vytvoriť korpus medicínskych dát, alebo Andrii Pervashov, ktorý sa zameriava na extrakciu informácií z medicínskej domény), žiadny z vrátených dokumentov explicitne nezaznamenáva prácu konkrétnej osoby na téme „multilinguálna extrakcia trojíc z medicínskych dát\".\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 2.02751,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.90076,
|
||||
"tool_latency_seconds": 2.02751,
|
||||
"final_model_latency_seconds": 27.928895,
|
||||
"total_latency_seconds": 31.858084,
|
||||
"usage": {
|
||||
"prompt_tokens": 31666,
|
||||
"completion_tokens": 288,
|
||||
"total_tokens": 31954
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiš mal zostaviť znalostný graf z viacjazyčných príbalových letákov liekov (medical package inserts) v rámci svojej stáže ERASMUS.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.314172,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.402649,
|
||||
"tool_latency_seconds": 0.314172,
|
||||
"final_model_latency_seconds": 13.699309,
|
||||
"total_latency_seconds": 15.416748,
|
||||
"usage": {
|
||||
"prompt_tokens": 16433,
|
||||
"completion_tokens": 121,
|
||||
"total_tokens": 16554
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,23 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť študentský dokument, ktorý by mal o grafových neurónových sieťach a zároveň mal v metadátach ako autora dokumentu uvedeného Maroša Harahusa, keďže Maroš Harahus je autorom dokumentu o študentovi Dmytrovi Muralovi, ale téma grafových neurónových sietí je priradená k Dmytrovi Muralovi, nie k Marošovi Harahusovi ako autorovi práce.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.675409,0.726721,11.301008,13.703528,11938,254,12192,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","Osoba Valerii Kutsenko začala štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.802983,0.305261,4.362708,6.471183,4664,147,4811,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zaoberá hybridným cloudom a kombináciou verejných a privátnych riešení.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.539961,0.348596,13.023603,14.912815,16290,160,16450,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala téme viacjazyčnej extrakcie trojíc z medicínskych údajov s cieľom zostrojiť znalostný graf z lekárskych letákov a využiť ho v inteligentnom agente na odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.768136,0.188296,10.48419,12.441029,12424,171,12595,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci sa v záznamoch nachádzajú študenti pracujúci na témach súvisiacich s extrakciou informácií z medicínskych dát (napr. Andrii Pervashov s témou extrakcie štruktúrovaných informácií alebo Yevhenii Medushivskyi s vytváraním korpusu medicínskych dát), žiadny dokument explicitne neuvádza tému s presným názvom „multilinguálna extrakcia trojíc z medicínskych dát"".
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",1.87733,0.483072,27.634379,29.995575,31666,290,31956,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiș mal zostaviť znalostný graf z príbalových letákov liekov v niekoľkých jazykoch.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.572851,0.27095,13.431892,15.276197,16446,122,16568,
|
||||
|
@ -0,0 +1,608 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T14:42:47.683367+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.833333,
|
||||
"overall_score": 0.791667,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 15.466721,
|
||||
"prompt_tokens": 93428,
|
||||
"completion_tokens": 1144,
|
||||
"total_tokens": 94572,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.471183,
|
||||
"prompt_tokens": 4664,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 4811
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 12.441029,
|
||||
"prompt_tokens": 12424,
|
||||
"completion_tokens": 171,
|
||||
"total_tokens": 12595
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 14.308172,
|
||||
"prompt_tokens": 28228,
|
||||
"completion_tokens": 414,
|
||||
"total_tokens": 28642
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.995575,
|
||||
"prompt_tokens": 31666,
|
||||
"completion_tokens": 290,
|
||||
"total_tokens": 31956
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 15.276197,
|
||||
"prompt_tokens": 16446,
|
||||
"completion_tokens": 122,
|
||||
"total_tokens": 16568
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.471183,
|
||||
"prompt_tokens": 4664,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 4811
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.4,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.8,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.4,
|
||||
"mean_latency_seconds": 17.265829,
|
||||
"prompt_tokens": 88764,
|
||||
"completion_tokens": 997,
|
||||
"total_tokens": 89761
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť študentský dokument, ktorý by mal o grafových neurónových sieťach a zároveň mal v metadátach ako autora dokumentu uvedeného Maroša Harahusa, keďže Maroš Harahus je autorom dokumentu o študentovi Dmytrovi Muralovi, ale téma grafových neurónových sietí je priradená k Dmytrovi Muralovi, nie k Marošovi Harahusovi ako autorovi práce.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\nhttps://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.726721,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"first_model_latency_seconds": 1.675409,
|
||||
"tool_latency_seconds": 0.726721,
|
||||
"final_model_latency_seconds": 11.301008,
|
||||
"total_latency_seconds": 13.703528,
|
||||
"usage": {
|
||||
"prompt_tokens": 11938,
|
||||
"completion_tokens": 254,
|
||||
"total_tokens": 12192
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.305261,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"first_model_latency_seconds": 1.802983,
|
||||
"tool_latency_seconds": 0.305261,
|
||||
"final_model_latency_seconds": 4.362708,
|
||||
"total_latency_seconds": 6.471183,
|
||||
"usage": {
|
||||
"prompt_tokens": 4664,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 4811
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zaoberá hybridným cloudom a kombináciou verejných a privátnych riešení.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.348596,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.539961,
|
||||
"tool_latency_seconds": 0.348596,
|
||||
"final_model_latency_seconds": 13.023603,
|
||||
"total_latency_seconds": 14.912815,
|
||||
"usage": {
|
||||
"prompt_tokens": 16290,
|
||||
"completion_tokens": 160,
|
||||
"total_tokens": 16450
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala téme viacjazyčnej extrakcie trojíc z medicínskych údajov s cieľom zostrojiť znalostný graf z lekárskych letákov a využiť ho v inteligentnom agente na odporúčanie liekov.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.188296,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.768136,
|
||||
"tool_latency_seconds": 0.188296,
|
||||
"final_model_latency_seconds": 10.48419,
|
||||
"total_latency_seconds": 12.441029,
|
||||
"usage": {
|
||||
"prompt_tokens": 12424,
|
||||
"completion_tokens": 171,
|
||||
"total_tokens": 12595
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci sa v záznamoch nachádzajú študenti pracujúci na témach súvisiacich s extrakciou informácií z medicínskych dát (napr. Andrii Pervashov s témou extrakcie štruktúrovaných informácií alebo Yevhenii Medushivskyi s vytváraním korpusu medicínskych dát), žiadny dokument explicitne neuvádza tému s presným názvom „multilinguálna extrakcia trojíc z medicínskych dát\".\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.483072,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.87733,
|
||||
"tool_latency_seconds": 0.483072,
|
||||
"final_model_latency_seconds": 27.634379,
|
||||
"total_latency_seconds": 29.995575,
|
||||
"usage": {
|
||||
"prompt_tokens": 31666,
|
||||
"completion_tokens": 290,
|
||||
"total_tokens": 31956
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiș mal zostaviť znalostný graf z príbalových letákov liekov v niekoľkých jazykoch.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.27095,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.572851,
|
||||
"tool_latency_seconds": 0.27095,
|
||||
"final_model_latency_seconds": 13.431892,
|
||||
"total_latency_seconds": 15.276197,
|
||||
"usage": {
|
||||
"prompt_tokens": 16446,
|
||||
"completion_tokens": 122,
|
||||
"total_tokens": 16568
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,26 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,0.0,0.75,False,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci existuje dokument o grafových neurónových sieťach s názvom „Grafové neurónové siete pre vyhľadávanie na internete"", ktorého vedúcim je Maroš Harahus, v metadátach tohto dokumentu je ako autor uvedený Dmytro Mural, nie Maroš Harahus. Maroš Harahus je v metadátach uvedený ako autor dokumentu pre študenta Dmytra Murala, nie ako študent vypracúvajúci prácu o grafových sieťach.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.878275,0.46288,21.417009,23.758927,24189,257,24446,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Osoba Valerii Kutsenko začala štúdium v roku 2022 a pracuje na diplomovej práci v roku 2027.
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.835251,0.599982,15.874489,18.310612,19133,145,19278,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zameriava na testovanie hybridného cloudu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.370989,0.352234,13.612118,15.335857,16277,137,16414,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdana Paula Chiša sa venovala téme viacjazyčnej extrakcie trojíc z medicínskych údajov, s cieľom zostrojiť znalostný graf z balíčkových letákov liekov v rôznych jazykoch a využiť ho v inteligentnom agentovi pre odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.814325,0.178706,12.371844,14.36533,14409,181,14590,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci v zdrojoch figurujú študenti pracujúci na témach súvisiacich s extrakciou informácií z medicínskych dát (Andrii Pervashov) alebo s multilinguálnym vyhľadávaním (Michal Stromko), žiadny z vrátených dokumentov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát"" ani meno študenta, ktorý by ju vypracoval.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",2.032631,0.583528,28.940407,31.557285,31572,355,31927,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiš mal zostaviť znalostný graf z príbalových letákov liekov v niekoľkých jazykoch.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.411554,0.282168,13.567631,15.261889,16433,109,16542,
|
||||
|
@ -0,0 +1,628 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T14:44:46.425959+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.666667,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.333333,
|
||||
"mean_latency_seconds": 19.764983,
|
||||
"prompt_tokens": 122013,
|
||||
"completion_tokens": 1184,
|
||||
"total_tokens": 123197,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 18.310612,
|
||||
"prompt_tokens": 19133,
|
||||
"completion_tokens": 145,
|
||||
"total_tokens": 19278
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 14.36533,
|
||||
"prompt_tokens": 14409,
|
||||
"completion_tokens": 181,
|
||||
"total_tokens": 14590
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 0.5,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 19.547392,
|
||||
"prompt_tokens": 40466,
|
||||
"completion_tokens": 394,
|
||||
"total_tokens": 40860
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 31.557285,
|
||||
"prompt_tokens": 31572,
|
||||
"completion_tokens": 355,
|
||||
"total_tokens": 31927
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 15.261889,
|
||||
"prompt_tokens": 16433,
|
||||
"completion_tokens": 109,
|
||||
"total_tokens": 16542
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 18.310612,
|
||||
"prompt_tokens": 19133,
|
||||
"completion_tokens": 145,
|
||||
"total_tokens": 19278
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.4,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.6,
|
||||
"overall_score": 0.7,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.2,
|
||||
"mean_latency_seconds": 20.055858,
|
||||
"prompt_tokens": 102880,
|
||||
"completion_tokens": 1039,
|
||||
"total_tokens": 103919
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci existuje dokument o grafových neurónových sieťach s názvom „Grafové neurónové siete pre vyhľadávanie na internete\", ktorého vedúcim je Maroš Harahus, v metadátach tohto dokumentu je ako autor uvedený Dmytro Mural, nie Maroš Harahus. Maroš Harahus je v metadátach uvedený ako autor dokumentu pre študenta Dmytra Murala, nie ako študent vypracúvajúci prácu o grafových sieťach.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.46288,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.878275,
|
||||
"tool_latency_seconds": 0.46288,
|
||||
"final_model_latency_seconds": 21.417009,
|
||||
"total_latency_seconds": 23.758927,
|
||||
"usage": {
|
||||
"prompt_tokens": 24189,
|
||||
"completion_tokens": 257,
|
||||
"total_tokens": 24446
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala štúdium v roku 2022 a pracuje na diplomovej práci v roku 2027.\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 2
|
||||
},
|
||||
"latency_seconds": 0.599982,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 1.835251,
|
||||
"tool_latency_seconds": 0.599982,
|
||||
"final_model_latency_seconds": 15.874489,
|
||||
"total_latency_seconds": 18.310612,
|
||||
"usage": {
|
||||
"prompt_tokens": 19133,
|
||||
"completion_tokens": 145,
|
||||
"total_tokens": 19278
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zameriava na testovanie hybridného cloudu.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.352234,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.370989,
|
||||
"tool_latency_seconds": 0.352234,
|
||||
"final_model_latency_seconds": 13.612118,
|
||||
"total_latency_seconds": 15.335857,
|
||||
"usage": {
|
||||
"prompt_tokens": 16277,
|
||||
"completion_tokens": 137,
|
||||
"total_tokens": 16414
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdana Paula Chiša sa venovala téme viacjazyčnej extrakcie trojíc z medicínskych údajov, s cieľom zostrojiť znalostný graf z balíčkových letákov liekov v rôznych jazykoch a využiť ho v inteligentnom agentovi pre odporúčanie liekov.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.178706,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.814325,
|
||||
"tool_latency_seconds": 0.178706,
|
||||
"final_model_latency_seconds": 12.371844,
|
||||
"total_latency_seconds": 14.36533,
|
||||
"usage": {
|
||||
"prompt_tokens": 14409,
|
||||
"completion_tokens": 181,
|
||||
"total_tokens": 14590
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci v zdrojoch figurujú študenti pracujúci na témach súvisiacich s extrakciou informácií z medicínskych dát (Andrii Pervashov) alebo s multilinguálnym vyhľadávaním (Michal Stromko), žiadny z vrátených dokumentov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát\" ani meno študenta, ktorý by ju vypracoval.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nhttps://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka\nhttps://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.583528,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 2.032631,
|
||||
"tool_latency_seconds": 0.583528,
|
||||
"final_model_latency_seconds": 28.940407,
|
||||
"total_latency_seconds": 31.557285,
|
||||
"usage": {
|
||||
"prompt_tokens": 31572,
|
||||
"completion_tokens": 355,
|
||||
"total_tokens": 31927
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiš mal zostaviť znalostný graf z príbalových letákov liekov v niekoľkých jazykoch.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.282168,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.411554,
|
||||
"tool_latency_seconds": 0.282168,
|
||||
"final_model_latency_seconds": 13.567631,
|
||||
"total_latency_seconds": 15.261889,
|
||||
"usage": {
|
||||
"prompt_tokens": 16433,
|
||||
"completion_tokens": 109,
|
||||
"total_tokens": 16542
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,24 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože Maroš Harahus je uvedený ako autor dokumentu o študentovi Dmytro Mural, ktorého téma je „Grafové neurónové siete pre vyhľadávanie na internete"", no nie ako autor vlastnej záverečnej práce s touto témou.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.952053,0.463194,20.366225,22.782111,24192,202,24394,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Valerii Kutsenko začal štúdium v roku 2022 a pracoval na diplomovom projekte v roku 2024.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.660631,0.572996,18.837965,21.072531,23024,131,23155,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zameriava na vytvorenie vzdelávacích materiálov o Kubernetes a funkčného prototypu privátneho klastra.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.803924,0.439361,14.046647,16.29046,16304,182,16486,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiș sa venovala téme extrakcie trojíc z lekárskych dát vo viacerých jazykoch, pričom cieľom bolo vybudovať znalostný graf z lekárskych balíčkov a využiť ho v inteligentnom agentovi na odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.794308,0.193581,10.762881,12.751199,12424,175,12599,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Hoci sa v zdrojoch nachádzajú spomienky na medicínske dáta a extrakciu informácií, žiadny dokument explicitne nesúvisí s témou „multilinguálna extrakcia trojíc z medicínskych dát"" ako celým názvom práce ani ako hlavnou témou konkrétneho študenta.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.881908,0.516101,27.33766,29.736441,31564,244,31808,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Bogdan Paul Chiš mal zostaviť znalostný graf z medicínskych letákov liekov (medical package inserts) v niekoľkých jazykoch. Tento projekt sa realizoval v rámci jeho praxe Erasmus intern Spring 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.816935,0.21273,9.406038,11.436042,10427,160,10587,
|
||||
|
@ -0,0 +1,629 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T14:46:40.730090+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.75,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.833333,
|
||||
"overall_score": 0.854167,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 19.011464,
|
||||
"prompt_tokens": 117935,
|
||||
"completion_tokens": 1094,
|
||||
"total_tokens": 119029,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 21.072531,
|
||||
"prompt_tokens": 23024,
|
||||
"completion_tokens": 131,
|
||||
"total_tokens": 23155
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 12.751199,
|
||||
"prompt_tokens": 12424,
|
||||
"completion_tokens": 175,
|
||||
"total_tokens": 12599
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 19.536285,
|
||||
"prompt_tokens": 40496,
|
||||
"completion_tokens": 384,
|
||||
"total_tokens": 40880
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.736441,
|
||||
"prompt_tokens": 31564,
|
||||
"completion_tokens": 244,
|
||||
"total_tokens": 31808
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 11.436042,
|
||||
"prompt_tokens": 10427,
|
||||
"completion_tokens": 160,
|
||||
"total_tokens": 10587
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 21.072531,
|
||||
"prompt_tokens": 23024,
|
||||
"completion_tokens": 131,
|
||||
"total_tokens": 23155
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.8,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.8,
|
||||
"overall_score": 0.85,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.8,
|
||||
"mean_latency_seconds": 18.599251,
|
||||
"prompt_tokens": 94911,
|
||||
"completion_tokens": 963,
|
||||
"total_tokens": 95874
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože Maroš Harahus je uvedený ako autor dokumentu o študentovi Dmytro Mural, ktorého téma je „Grafové neurónové siete pre vyhľadávanie na internete\", no nie ako autor vlastnej záverečnej práce s touto témou.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.463194,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.952053,
|
||||
"tool_latency_seconds": 0.463194,
|
||||
"final_model_latency_seconds": 20.366225,
|
||||
"total_latency_seconds": 22.782111,
|
||||
"usage": {
|
||||
"prompt_tokens": 24192,
|
||||
"completion_tokens": 202,
|
||||
"total_tokens": 24394
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Valerii Kutsenko začal štúdium v roku 2022 a pracoval na diplomovom projekte v roku 2024.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.572996,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.660631,
|
||||
"tool_latency_seconds": 0.572996,
|
||||
"final_model_latency_seconds": 18.837965,
|
||||
"total_latency_seconds": 21.072531,
|
||||
"usage": {
|
||||
"prompt_tokens": 23024,
|
||||
"completion_tokens": 131,
|
||||
"total_tokens": 23155
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.5,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zameriava na vytvorenie vzdelávacích materiálov o Kubernetes a funkčného prototypu privátneho klastra.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.439361,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.803924,
|
||||
"tool_latency_seconds": 0.439361,
|
||||
"final_model_latency_seconds": 14.046647,
|
||||
"total_latency_seconds": 16.29046,
|
||||
"usage": {
|
||||
"prompt_tokens": 16304,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 16486
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiș sa venovala téme extrakcie trojíc z lekárskych dát vo viacerých jazykoch, pričom cieľom bolo vybudovať znalostný graf z lekárskych balíčkov a využiť ho v inteligentnom agentovi na odporúčanie liekov.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.193581,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.794308,
|
||||
"tool_latency_seconds": 0.193581,
|
||||
"final_model_latency_seconds": 10.762881,
|
||||
"total_latency_seconds": 12.751199,
|
||||
"usage": {
|
||||
"prompt_tokens": 12424,
|
||||
"completion_tokens": 175,
|
||||
"total_tokens": 12599
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nHoci sa v zdrojoch nachádzajú spomienky na medicínske dáta a extrakciu informácií, žiadny dokument explicitne nesúvisí s témou „multilinguálna extrakcia trojíc z medicínskych dát\" ako celým názvom práce ani ako hlavnou témou konkrétneho študenta.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.516101,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.881908,
|
||||
"tool_latency_seconds": 0.516101,
|
||||
"final_model_latency_seconds": 27.33766,
|
||||
"total_latency_seconds": 29.736441,
|
||||
"usage": {
|
||||
"prompt_tokens": 31564,
|
||||
"completion_tokens": 244,
|
||||
"total_tokens": 31808
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiš mal zostaviť znalostný graf z medicínskych letákov liekov (medical package inserts) v niekoľkých jazykoch. Tento projekt sa realizoval v rámci jeho praxe Erasmus intern Spring 2026.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 5
|
||||
},
|
||||
"latency_seconds": 0.21273,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.816935,
|
||||
"tool_latency_seconds": 0.21273,
|
||||
"final_model_latency_seconds": 9.406038,
|
||||
"total_latency_seconds": 11.436042,
|
||||
"usage": {
|
||||
"prompt_tokens": 10427,
|
||||
"completion_tokens": 160,
|
||||
"total_tokens": 10587
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
2082
tools/rag/evaluation/results/dev_final_freeze/rag_answers_dev.csv
Normal file
2082
tools/rag/evaluation/results/dev_final_freeze/rag_answers_dev.csv
Normal file
File diff suppressed because it is too large
Load Diff
48602
tools/rag/evaluation/results/dev_final_freeze/rag_answers_dev.json
Normal file
48602
tools/rag/evaluation/results/dev_final_freeze/rag_answers_dev.json
Normal file
File diff suppressed because it is too large
Load Diff
2095
tools/rag/evaluation/results/dev_final_full/rag_answers_dev.csv
Normal file
2095
tools/rag/evaluation/results/dev_final_full/rag_answers_dev.csv
Normal file
File diff suppressed because it is too large
Load Diff
48612
tools/rag/evaluation/results/dev_final_full/rag_answers_dev.json
Normal file
48612
tools/rag/evaluation/results/dev_final_full/rag_answers_dev.json
Normal file
File diff suppressed because it is too large
Load Diff
2113
tools/rag/evaluation/results/dev_final_full_v2/rag_answers_dev.csv
Normal file
2113
tools/rag/evaluation/results/dev_final_full_v2/rag_answers_dev.csv
Normal file
File diff suppressed because it is too large
Load Diff
48523
tools/rag/evaluation/results/dev_final_full_v2/rag_answers_dev.json
Normal file
48523
tools/rag/evaluation/results/dev_final_full_v2/rag_answers_dev.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,57 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0004,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý rieši tému „Systém získavania informácií v slovenskom jazyku“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Ján Holp"", ""Systém získavania informácií v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/topics/legal"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko""]","Študentom, ktorý rieši tému „Systém získavania informácií v slovenskom jazyku"", je Ján Holp, ktorý vypracoval diplomovú prácu s presne takýmto názvom v roku 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",2.100296,0.538287,22.383795,25.023141,27575,169,27744,
|
||||
q0026,dev,specific_fact,medium,Akú konkrétnu skupinu neurónových sietí mal Maroš Harahus naštudovať v rámci úloh po úvodnom stretnutí s US Steel?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","Maroš Harahus mal po úvodnom stretnutí s US Steel naštudovať konkrétne aktuálne metódy grafových neurónových sietí a nájsť a vyskúšať toolkit na GNN (Graph Neural Networks).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",2.03264,0.78379,8.814131,11.630922,9866,178,10044,
|
||||
q0040,dev,reverse_lookup,medium,Kto sa venuje práci s názvom „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Jakub Maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh""]","Práci s názvom „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"" vypracoval Jakub Maruniak.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",1.760059,0.439057,19.554143,21.754194,24650,141,24791,
|
||||
q0041,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý rieši tému „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Jakub Maruniak"", ""Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/topics/otvorene"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko""]","Študent Jakub Maruniak vypracoval diplomovú prácu s návrhom názvu „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",1.97136,0.388183,20.22358,22.583788,24738,161,24899,
|
||||
q0253,dev,disambiguation_year,hard,Rozlíš pri osobe Vladimír Ferko rok začiatku štúdia a rok uvedený v názve sekcie jeho bakalárskej práce.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2021"", ""2024""]","[""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko""]","[""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic""]","Vladimír Ferko začal študovať v roku 2021, zatiaľ čo rok uvedený v názve sekcie jeho bakalárskej práce je 2024.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko",2.014831,0.435108,19.676963,22.127614,23721,162,23883,
|
||||
q0336,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý má bakalársku prácu s názvom „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tetiana Mohorian"", ""Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian"", ""https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak"", ""https://zp.kemt.fei.tuke.sk/students/2020/jakub_kristian_lukas"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura""]","Študentka Tetiana Mohorian vypracováva bakalársku prácu s názvom „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian",2.133266,0.454499,24.128348,26.716858,29251,169,29420,
|
||||
q0570,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""anotácia entít""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","Cesar Abascal Gutierrez sa počas svojej stáže venoval problémom v oblasti anotácie pomenovaných entít (Named Entity Recognition), pričom jeho cieľom bolo vytvoriť manuálne anotačný trénovací súbor z prepisov reči a navrhnúť schému anotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.852512,0.296243,12.454928,14.604139,14276,170,14446,
|
||||
q0633,dev,topic_page_lookup,medium,"Ktorá tematická stránka ZP Wiki, nie študentská práca, sa priamo venuje téme „Strojový preklad“?",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Strojový preklad""]","[""https://zp.kemt.fei.tuke.sk/topics/translation""]","[""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/topics/otvorene""]","Tematická stránka ZP Wiki, ktorá sa priamo venuje téme „Strojový preklad"" a nie je študentskou prácou, sa nazýva „Strojový preklad"" a nachádza sa pod cestou `pages/topics/translation`.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/translation",1.991916,0.99058,32.381462,35.364991,38868,162,39030,
|
||||
q0968,dev,single_document_synthesis,hard,Pri osobe Ján Pták uveď názov témy alebo práce uvedenej v zázname typu „diplomový projekt“ a rok uvedený v názve príslušnej sekcie. Neuvádzaj rok začiatku štúdia.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Pri osobe Ján Pták je v zázname typu „diplomový projekt"" uvedená téma „Agent pre manažment záverečných prác"" a rok v názve príslušnej sekcie je 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.78334,0.895301,41.802253,44.482299,48248,152,48400,
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","Rok začiatku štúdia Dária Lindvaia je 2016 a rok vyjadrený kategóriou jeho diplomovej práce je 2021, čo sa líši od roku sekcie diplomového projektu 2020.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.550172,1.343518,38.468043,41.363045,44805,143,44948,
|
||||
q0990,dev,single_document_synthesis,hard,"V dokumente osoby Tomáš Kuchárik, ktorý súvisí s témou „strojový preklad“, uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"", ""2022""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/translation""]","Diplomová práca Tomáša Kuchárika súvisiaca s témou strojového prekladu sa volá „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“ a je uvedená v roku 2022.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",1.94433,0.594338,13.68606,16.225244,15424,180,15604,
|
||||
q1022,dev,single_document_synthesis,hard,"V dokumente osoby Dárius Lindvai, ktorý súvisí s témou „interpunkcia“, uveď názov diplomovej práce a rok vyjadrený kategóriou tejto diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/adrian_remias"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","Dárius Lindvai vypracoval diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" a kategória tejto práce uvádza rok 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.931365,1.211395,18.225585,21.368993,22207,162,22369,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v kategórii diplomovej práce pre rok 2021, zatiaľ čo Ján Pták má pri diplomovom projekte uvedený rok 2026 a tému „Agent pre manažment záverečných prác"".
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.589001,0.750022,20.144654,22.484322,23223,194,23417,
|
||||
q1109,dev,multi_document,hard,"Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agentové systémy pomocou veľkého jazykového modelu"", ""2026"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Diplomová práca Martina Šarišského má názov „Agentové systémy pomocou veľkého jazykového modelu"" a je na ZP Wiki uvedená pre rok 2026, zatiaľ čo diplomová práca Andrii Pervashova má názov „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" a je uvedená pre rok 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.876281,0.654705,25.051578,27.583365,28844,230,29074,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021, čo vyplýva z kategórie dp2021. Andrii Pervashov má ako diplomovú prácu sekciu s názvom „Diplomová práca 2027"" a návrh na tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", pričom rok je v názve tejto sekcie uvedený ako 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.543741,0.878946,21.647388,24.070721,23880,248,24128,
|
||||
q1136,dev,multi_document,hard,"Porovnaj záznamy osôb Ján Malinovský a Yevhenii Medushivskyi. Pri Jánovi Malinovskom použi diplomový projekt a pri Yevhenii Medushivskom bakalársku prácu. Pri oboch uveď názov práce a rok uvedený v názve príslušnej sekcie, nie rok začiatku štúdia.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Grafové vyhľadávanie pre podporu vzdelávania"", ""2026"", ""Získavanie textu z webových stránok"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Z dostupných dokumentov sa pre Ján Malinovský v časti Diplomový projekt 2026 uvádza téma „Grafové vyhľadávanie pre podporu vzdelávania"" a nie explicitný názov práce, zatiaľ čo pre Yevhenii Medushivskyi sa v časti Bakalárska práca 2026 uvádza téma „Získavanie textu z webových stránok"" bez explicitne uvedeného názvu práce.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",1.682368,0.478387,22.401935,24.563452,25612,226,25838,
|
||||
|
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,27 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože v metadátach žiadneho študentského dokumentu o grafových neurónových sieťach nie je Maroš Harahus uvedený ako autor dokumentu, ale skôr ako autor dokumentu pre študenta Dmytro Mural, ktorého práca sa zaoberá grafovými neurónovými sieťami.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.115666,0.974012,27.830161,30.921054,34196,211,34407,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Osoba Valerii Kutsenko začala štúdium v roku 2022 a pracuje na diplomovej práci v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.407377,0.565645,18.476198,20.449999,23076,118,23194,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa venuje testovaniu hybridného cloudového prostredia.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.540706,0.382902,13.596292,15.52047,16355,150,16505,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z lekárskych dát v niekoľkých jazykoch, s cieľom zostrojiť graf poznania z lekárskych balíčkových letákov a využiť ho v inteligentnom agentovi odporúčajúcom lieky.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.785095,0.252294,10.784058,12.821861,12489,175,12664,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Hoci sa v nájdených záznamoch spomínajú študenti pracujúci s medicínskymi dátami (Yevhenii Medushivskyi pri tvorbe korpusu a Andrii Pervashov pri extrakcii štruktúrovaných informácií z medicínskej domény) alebo s multilinguálnymi úlohami (Michal Stromko), žiadny z vrátených dokumentov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát"" ani meno konkrétnej osoby, ktorá by na nej pracovala.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka
|
||||
https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",1.873627,0.476028,29.470153,31.820596,31731,379,32110,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Bogdan Paul Chiš mal zostaviť znalostný graf z medicínskych príbalových letákov v rôznych jazykoch.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.828582,0.208718,9.092169,11.129802,10493,139,10632,
|
||||
|
@ -0,0 +1,631 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T15:15:32.604477+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.666667,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.833333,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 20.443964,
|
||||
"prompt_tokens": 128340,
|
||||
"completion_tokens": 1172,
|
||||
"total_tokens": 129512,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 20.449999,
|
||||
"prompt_tokens": 23076,
|
||||
"completion_tokens": 118,
|
||||
"total_tokens": 23194
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 12.821861,
|
||||
"prompt_tokens": 12489,
|
||||
"completion_tokens": 175,
|
||||
"total_tokens": 12664
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 23.220762,
|
||||
"prompt_tokens": 50551,
|
||||
"completion_tokens": 361,
|
||||
"total_tokens": 50912
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 31.820596,
|
||||
"prompt_tokens": 31731,
|
||||
"completion_tokens": 379,
|
||||
"total_tokens": 32110
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 11.129802,
|
||||
"prompt_tokens": 10493,
|
||||
"completion_tokens": 139,
|
||||
"total_tokens": 10632
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 20.449999,
|
||||
"prompt_tokens": 23076,
|
||||
"completion_tokens": 118,
|
||||
"total_tokens": 23194
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.6,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.8,
|
||||
"overall_score": 0.8,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.6,
|
||||
"mean_latency_seconds": 20.442757,
|
||||
"prompt_tokens": 105264,
|
||||
"completion_tokens": 1054,
|
||||
"total_tokens": 106318
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože v metadátach žiadneho študentského dokumentu o grafových neurónových sieťach nie je Maroš Harahus uvedený ako autor dokumentu, ale skôr ako autor dokumentu pre študenta Dmytro Mural, ktorého práca sa zaoberá grafovými neurónovými sieťami.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 3
|
||||
},
|
||||
"latency_seconds": 0.974012,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"
|
||||
],
|
||||
"first_model_latency_seconds": 2.115666,
|
||||
"tool_latency_seconds": 0.974012,
|
||||
"final_model_latency_seconds": 27.830161,
|
||||
"total_latency_seconds": 30.921054,
|
||||
"usage": {
|
||||
"prompt_tokens": 34196,
|
||||
"completion_tokens": 211,
|
||||
"total_tokens": 34407
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala štúdium v roku 2022 a pracuje na diplomovej práci v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10
|
||||
},
|
||||
"latency_seconds": 0.565645,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.407377,
|
||||
"tool_latency_seconds": 0.565645,
|
||||
"final_model_latency_seconds": 18.476198,
|
||||
"total_latency_seconds": 20.449999,
|
||||
"usage": {
|
||||
"prompt_tokens": 23076,
|
||||
"completion_tokens": 118,
|
||||
"total_tokens": 23194
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa venuje testovaniu hybridného cloudového prostredia.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.382902,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.540706,
|
||||
"tool_latency_seconds": 0.382902,
|
||||
"final_model_latency_seconds": 13.596292,
|
||||
"total_latency_seconds": 15.52047,
|
||||
"usage": {
|
||||
"prompt_tokens": 16355,
|
||||
"completion_tokens": 150,
|
||||
"total_tokens": 16505
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z lekárskych dát v niekoľkých jazykoch, s cieľom zostrojiť graf poznania z lekárskych balíčkových letákov a využiť ho v inteligentnom agentovi odporúčajúcom lieky.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"published_only": true,
|
||||
"limit": 5,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.252294,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.785095,
|
||||
"tool_latency_seconds": 0.252294,
|
||||
"final_model_latency_seconds": 10.784058,
|
||||
"total_latency_seconds": 12.821861,
|
||||
"usage": {
|
||||
"prompt_tokens": 12489,
|
||||
"completion_tokens": 175,
|
||||
"total_tokens": 12664
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nHoci sa v nájdených záznamoch spomínajú študenti pracujúci s medicínskymi dátami (Yevhenii Medushivskyi pri tvorbe korpusu a Andrii Pervashov pri extrakcii štruktúrovaných informácií z medicínskej domény) alebo s multilinguálnymi úlohami (Michal Stromko), žiadny z vrátených dokumentov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát\" ani meno konkrétnej osoby, ktorá by na nej pracovala.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nhttps://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka\nhttps://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.476028,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.873627,
|
||||
"tool_latency_seconds": 0.476028,
|
||||
"final_model_latency_seconds": 29.470153,
|
||||
"total_latency_seconds": 31.820596,
|
||||
"usage": {
|
||||
"prompt_tokens": 31731,
|
||||
"completion_tokens": 379,
|
||||
"total_tokens": 32110
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiš mal zostaviť znalostný graf z medicínskych príbalových letákov v rôznych jazykoch.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.208718,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.828582,
|
||||
"tool_latency_seconds": 0.208718,
|
||||
"final_model_latency_seconds": 9.092169,
|
||||
"total_latency_seconds": 11.129802,
|
||||
"usage": {
|
||||
"prompt_tokens": 10493,
|
||||
"completion_tokens": 139,
|
||||
"total_tokens": 10632
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,24 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože v metadátach dokumentu o grafových neurónových sieťach (bakalárska práca Dmytra Murula) je ako autor uvedený Maroš Harahus, ale nie ako autor dokumentu v zmysle metadát stránky, kde je autorom Daniel Hladek, a žiadny dokument s Marošom Harahusom ako autorom metadát neobsahuje tému grafových neurónových sietí.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.915382,0.460945,20.86888,23.24588,24254,236,24490,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","Osoba Valerii Kutsenko začala štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.824298,0.325987,4.751043,6.901548,4729,147,4876,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zameriava na tému hybridného cloudu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.35686,0.359045,13.371848,15.088362,16343,138,16481,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiș sa venovala téme (multilingválne) extrakcie trojíc z medicínskych dát, pričom cieľom bolo zostrojiť znalostný graf z medicínskych letákov v viacerých jazykoch a využiť ho v inteligentnom agentovi odporúčajúcom lieky.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.79586,0.187609,12.492747,14.4767,14474,182,14656,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",2.024733,0.523568,27.291054,29.840178,31636,244,31880,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiš mal zostaviť znalostný graf z príbalových letákov liekov v viacerých jazykoch počas svojej stáže.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.628926,0.301374,13.67717,15.608101,16511,128,16639,
|
||||
|
@ -0,0 +1,615 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T15:17:17.977158+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.833333,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.833333,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 5,
|
||||
"strict_pass_rate": 0.833333,
|
||||
"mean_latency_seconds": 17.526795,
|
||||
"prompt_tokens": 107947,
|
||||
"completion_tokens": 1075,
|
||||
"total_tokens": 109022,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.901548,
|
||||
"prompt_tokens": 4729,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 4876
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 14.4767,
|
||||
"prompt_tokens": 14474,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 14656
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 19.167121,
|
||||
"prompt_tokens": 40597,
|
||||
"completion_tokens": 374,
|
||||
"total_tokens": 40971
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 29.840178,
|
||||
"prompt_tokens": 31636,
|
||||
"completion_tokens": 244,
|
||||
"total_tokens": 31880
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.608101,
|
||||
"prompt_tokens": 16511,
|
||||
"completion_tokens": 128,
|
||||
"total_tokens": 16639
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.901548,
|
||||
"prompt_tokens": 4729,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 4876
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.8,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.8,
|
||||
"overall_score": 0.85,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.8,
|
||||
"mean_latency_seconds": 19.651844,
|
||||
"prompt_tokens": 103218,
|
||||
"completion_tokens": 928,
|
||||
"total_tokens": 104146
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože v metadátach dokumentu o grafových neurónových sieťach (bakalárska práca Dmytra Murula) je ako autor uvedený Maroš Harahus, ale nie ako autor dokumentu v zmysle metadát stránky, kde je autorom Daniel Hladek, a žiadny dokument s Marošom Harahusom ako autorom metadát neobsahuje tému grafových neurónových sietí.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.460945,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.915382,
|
||||
"tool_latency_seconds": 0.460945,
|
||||
"final_model_latency_seconds": 20.86888,
|
||||
"total_latency_seconds": 23.24588,
|
||||
"usage": {
|
||||
"prompt_tokens": 24254,
|
||||
"completion_tokens": 236,
|
||||
"total_tokens": 24490
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala štúdium v roku 2022 a vypracováva diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.325987,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"first_model_latency_seconds": 1.824298,
|
||||
"tool_latency_seconds": 0.325987,
|
||||
"final_model_latency_seconds": 4.751043,
|
||||
"total_latency_seconds": 6.901548,
|
||||
"usage": {
|
||||
"prompt_tokens": 4729,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 4876
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zameriava na tému hybridného cloudu.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.359045,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.35686,
|
||||
"tool_latency_seconds": 0.359045,
|
||||
"final_model_latency_seconds": 13.371848,
|
||||
"total_latency_seconds": 15.088362,
|
||||
"usage": {
|
||||
"prompt_tokens": 16343,
|
||||
"completion_tokens": 138,
|
||||
"total_tokens": 16481
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiș sa venovala téme (multilingválne) extrakcie trojíc z medicínskych dát, pričom cieľom bolo zostrojiť znalostný graf z medicínskych letákov v viacerých jazykoch a využiť ho v inteligentnom agentovi odporúčajúcom lieky.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.187609,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.79586,
|
||||
"tool_latency_seconds": 0.187609,
|
||||
"final_model_latency_seconds": 12.492747,
|
||||
"total_latency_seconds": 14.4767,
|
||||
"usage": {
|
||||
"prompt_tokens": 14474,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 14656
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov\nhttps://zp.kemt.fei.tuke.sk/students/2019/michal_stromko\nhttps://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka\nhttps://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.523568,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 2.024733,
|
||||
"tool_latency_seconds": 0.523568,
|
||||
"final_model_latency_seconds": 27.291054,
|
||||
"total_latency_seconds": 29.840178,
|
||||
"usage": {
|
||||
"prompt_tokens": 31636,
|
||||
"completion_tokens": 244,
|
||||
"total_tokens": 31880
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiš mal zostaviť znalostný graf z príbalových letákov liekov v viacerých jazykoch počas svojej stáže.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.301374,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.628926,
|
||||
"tool_latency_seconds": 0.301374,
|
||||
"final_model_latency_seconds": 13.67717,
|
||||
"total_latency_seconds": 15.608101,
|
||||
"usage": {
|
||||
"prompt_tokens": 16511,
|
||||
"completion_tokens": 128,
|
||||
"total_tokens": 16639
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,20 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť študentský dokument o grafových neurónových sieťach, ktorý by mal v metadátach ako autora dokumentu uvedeného Maroša Harahusa; v skutočnosti je Maroš Harahus autorom dokumentu o bakalárskej práci Dmytra Murala s témou „Grafové neurónové siete pre vyhľadávanie na internete"", zatiaľ čo dokument s názvom „Maroš Harahus"" má ako autora Daniela Hladka a obsahuje len úlohy týkajúce sa grafových neurónových sietí v rámci jeho vlastného štúdia.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",1.903565,0.459758,21.384463,23.748487,24254,256,24510,
|
||||
q0311,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko""]","Osoba Valerii Kutsenko začala študovať v roku 2022 a vypracovávala diplomovú prácu v roku 2027.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",1.840469,0.556885,13.408471,15.806565,16270,148,16418,
|
||||
q0375,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Daniil Huzenko"", ""hybridný cloud""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina""]","Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia"", ktorá sa zameriava na vytvorenie vzdelávacích materiálov o Kubernetes a funkčného prototypu privátneho klastra v kombinácii s verejnými službami.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.577905,0.405217,14.153553,16.137211,16355,178,16533,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Stáž osoby Bogdan Paul Chiš sa venovala problému multijazyčného extrahovania trojíc z lekárskych údajov s cieľom zostrojiť znalostný graf z lekárskych letáčkov liekov a využiť tento graf v inteligentnom agentovi, ktorý odporúča lieky.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.774716,0.201767,2.144249,4.121239,14474,176,14650,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,0.0,0.0,0.25,False,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci sa v záznamoch spomína práca s medicínskymi dátami (napr. vytváranie korpusu medicínskych dát alebo extrakcia štruktúrovaných informácií z medicínskej domény), žiadny z nájdených dokumentov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát"" ani nešpecifikuje osobu, ktorá by na takejto konkrétnej téme pracovala.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.905308,0.520546,27.989848,30.416476,31629,284,31913,
|
||||
q0586,dev,specific_fact,medium,Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""medical package inserts""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik""]","Bogdan Paul Chiș mal zostaviť znalostný graf z lekárskych príbalových letákov v rôznych jazykoch.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.250758,0.302373,13.805922,15.359807,16486,100,16586,
|
||||
|
@ -0,0 +1,624 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T15:19:03.730945+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "b80c4d9cc4cb16d12a971cba13bde7de98de7aac4d1ad308a405fe0f1ae29905",
|
||||
"selected_question_ids": [
|
||||
"q0024",
|
||||
"q0311",
|
||||
"q0375",
|
||||
"q0579",
|
||||
"q0580",
|
||||
"q0586"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.666667,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 0.833333,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 17.598298,
|
||||
"prompt_tokens": 119468,
|
||||
"completion_tokens": 1142,
|
||||
"total_tokens": 120610,
|
||||
"by_category": {
|
||||
"disambiguation_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.806565,
|
||||
"prompt_tokens": 16270,
|
||||
"completion_tokens": 148,
|
||||
"total_tokens": 16418
|
||||
},
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 4.121239,
|
||||
"prompt_tokens": 14474,
|
||||
"completion_tokens": 176,
|
||||
"total_tokens": 14650
|
||||
},
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 19.942849,
|
||||
"prompt_tokens": 40609,
|
||||
"completion_tokens": 434,
|
||||
"total_tokens": 41043
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.0,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_score": 0.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 30.416476,
|
||||
"prompt_tokens": 31629,
|
||||
"completion_tokens": 284,
|
||||
"total_tokens": 31913
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.359807,
|
||||
"prompt_tokens": 16486,
|
||||
"completion_tokens": 100,
|
||||
"total_tokens": 16586
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.806565,
|
||||
"prompt_tokens": 16270,
|
||||
"completion_tokens": 148,
|
||||
"total_tokens": 16418
|
||||
},
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.6,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 0.8,
|
||||
"overall_score": 0.8,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.6,
|
||||
"mean_latency_seconds": 17.956644,
|
||||
"prompt_tokens": 103198,
|
||||
"completion_tokens": 994,
|
||||
"total_tokens": 104192
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť študentský dokument o grafových neurónových sieťach, ktorý by mal v metadátach ako autora dokumentu uvedeného Maroša Harahusa; v skutočnosti je Maroš Harahus autorom dokumentu o bakalárskej práci Dmytra Murala s témou „Grafové neurónové siete pre vyhľadávanie na internete\", zatiaľ čo dokument s názvom „Maroš Harahus\" má ako autora Daniela Hladka a obsahuje len úlohy týkajúce sa grafových neurónových sietí v rámci jeho vlastného štúdia.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.459758,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.903565,
|
||||
"tool_latency_seconds": 0.459758,
|
||||
"final_model_latency_seconds": 21.384463,
|
||||
"total_latency_seconds": 23.748487,
|
||||
"usage": {
|
||||
"prompt_tokens": 24254,
|
||||
"completion_tokens": 256,
|
||||
"total_tokens": 24510
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0311",
|
||||
"split": "dev",
|
||||
"category": "disambiguation_year",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/valerii_kutsenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2022",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Kontrola proti zámene roku začiatku štúdia s rokom záverečnej práce alebo projektu.",
|
||||
"answer": "Osoba Valerii Kutsenko začala študovať v roku 2022 a vypracovávala diplomovú prácu v roku 2027.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš rok začiatku štúdia a rok práce osoby Valerii Kutsenko.",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 5
|
||||
},
|
||||
"latency_seconds": 0.556885,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"
|
||||
],
|
||||
"first_model_latency_seconds": 1.840469,
|
||||
"tool_latency_seconds": 0.556885,
|
||||
"final_model_latency_seconds": 13.408471,
|
||||
"total_latency_seconds": 15.806565,
|
||||
"usage": {
|
||||
"prompt_tokens": 16270,
|
||||
"completion_tokens": 148,
|
||||
"total_tokens": 16418
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0375",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/daniil_huzenko/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Daniil Huzenko",
|
||||
"hybridný cloud"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Daniil Huzenko vypracováva bakalársku prácu s predbežným názvom „Testovanie hybridného klaudu s využitím kombinácie verejného a privátneho riešenia\", ktorá sa zameriava na vytvorenie vzdelávacích materiálov o Kubernetes a funkčného prototypu privátneho klastra v kombinácii s verejnými službami.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktorý dokument alebo študent súvisí s témou hybridný cloud a osobou Daniil Huzenko?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.405217,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"
|
||||
],
|
||||
"first_model_latency_seconds": 1.577905,
|
||||
"tool_latency_seconds": 0.405217,
|
||||
"final_model_latency_seconds": 14.153553,
|
||||
"total_latency_seconds": 16.137211,
|
||||
"usage": {
|
||||
"prompt_tokens": 16355,
|
||||
"completion_tokens": 178,
|
||||
"total_tokens": 16533
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0579",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"multilinguálna extrakcia trojíc z medicínskych dát"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Stáž osoby Bogdan Paul Chiš sa venovala problému multijazyčného extrahovania trojíc z lekárskych údajov s cieľom zostrojiť znalostný graf z lekárskych letáčkov liekov a využiť tento graf v inteligentnom agentovi, ktorý odporúča lieky.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.201767,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.774716,
|
||||
"tool_latency_seconds": 0.201767,
|
||||
"final_model_latency_seconds": 2.144249,
|
||||
"total_latency_seconds": 4.121239,
|
||||
"usage": {
|
||||
"prompt_tokens": 14474,
|
||||
"completion_tokens": 176,
|
||||
"total_tokens": 14650
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0580",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Bogdan Paul Chiș"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Hoci sa v záznamoch spomína práca s medicínskymi dátami (napr. vytváranie korpusu medicínskych dát alebo extrakcia štruktúrovaných informácií z medicínskej domény), žiadny z nájdených dokumentov explicitne neuvádza tému „multilinguálna extrakcia trojíc z medicínskych dát\" ani nešpecifikuje osobu, ktorá by na takejto konkrétnej téme pracovala.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.520546,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.905308,
|
||||
"tool_latency_seconds": 0.520546,
|
||||
"final_model_latency_seconds": 27.989848,
|
||||
"total_latency_seconds": 30.416476,
|
||||
"usage": {
|
||||
"prompt_tokens": 31629,
|
||||
"completion_tokens": 284,
|
||||
"total_tokens": 31913
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_match_count": 0,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": false,
|
||||
"should_answer_score": 0.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.25,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0586",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/bogdan_paul_chis/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"medical package inserts"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Bogdan Paul Chiș mal zostaviť znalostný graf z lekárskych príbalových letákov v rôznych jazykoch.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Z akého typu dokumentov mal Bogdan Paul Chiș zostaviť znalostný graf?"
|
||||
},
|
||||
"latency_seconds": 0.302373,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.250758,
|
||||
"tool_latency_seconds": 0.302373,
|
||||
"final_model_latency_seconds": 13.805922,
|
||||
"total_latency_seconds": 15.359807,
|
||||
"usage": {
|
||||
"prompt_tokens": 16486,
|
||||
"completion_tokens": 100,
|
||||
"total_tokens": 16586
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,20 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0570,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""anotácia entít""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","Cesar Abascal Gutierrez sa počas svojej stáže v lete 2019 venoval problému rozpoznávania neznámych pomenovaných entít, pričom jeho cieľom bolo vytvoriť manuálne anotovanú trénovaciu sadu z prepisov reči a navrhnúť schému pre túto anotáciu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",2.0651,0.371596,11.762852,14.200271,14145,182,14327,
|
||||
q0968,dev,single_document_synthesis,hard,Pri osobe Ján Pták uveď názov témy alebo práce uvedenej v zázname typu „diplomový projekt“ a rok uvedený v názve príslušnej sekcie. Neuvádzaj rok začiatku štúdia.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Ján Pták vypracováva diplomový projekt s názvom „Agent pre manažment záverečných prác"" a rok uvedený v názve sekcie je 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.744308,1.116938,39.190319,42.052811,48117,141,48258,
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Rok začiatku štúdia Dáriusa Lindvaiho je 2016, zatiaľ čo jeho diplomová práca je vyjadrená kategóriou z roku 2021 (dp2021).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.704977,1.031873,36.216216,38.954341,43045,150,43195,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Dárius Lindvai vypracovával diplomovú prácu s názvom Obnovenie interpunkcie pomocou hlbokých neurónových sietí v roku 2021, čo je rok vyjadrený kategóriou diplomovej práce v metadátach, zatiaľ čo Ján Pták má v zázname uvedený diplomový projekt v roku 2026 s témou Agent pre manažment záverečných prác.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.81292,0.817265,23.109547,25.740566,26229,225,26454,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.75,1.0,1.0,0.9375,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021""]","V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaia nenašla diplomová práca, ale iba diplomový projekt s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" z roku 2020, zatiaľ čo Andrii Pervashov má v sekciách uvedený návrh na tému diplomovej práce „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" s označením roka 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.539209,0.747723,21.225615,23.513234,23749,229,23978,
|
||||
|
@ -0,0 +1,544 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T08:18:58.226348+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "d24250557bb62dd4ce9e418a8054d56aea5d4f3a7b0fbc2265b30e9d10853194",
|
||||
"selected_question_ids": [
|
||||
"q0570",
|
||||
"q0968",
|
||||
"q0982",
|
||||
"q1092",
|
||||
"q1125"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 5,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.95,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.9875,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.8,
|
||||
"mean_latency_seconds": 28.892245,
|
||||
"prompt_tokens": 155285,
|
||||
"completion_tokens": 927,
|
||||
"total_tokens": 156212,
|
||||
"by_category": {
|
||||
"internship_topic": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 14.200271,
|
||||
"prompt_tokens": 14145,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 14327
|
||||
},
|
||||
"multi_document": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.875,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.96875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 24.6269,
|
||||
"prompt_tokens": 49978,
|
||||
"completion_tokens": 454,
|
||||
"total_tokens": 50432
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 40.503576,
|
||||
"prompt_tokens": 91162,
|
||||
"completion_tokens": 291,
|
||||
"total_tokens": 91453
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.9375,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.984375,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.75,
|
||||
"mean_latency_seconds": 32.565238,
|
||||
"prompt_tokens": 141140,
|
||||
"completion_tokens": 745,
|
||||
"total_tokens": 141885
|
||||
},
|
||||
"medium": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 14.200271,
|
||||
"prompt_tokens": 14145,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 14327
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0570",
|
||||
"split": "dev",
|
||||
"category": "internship_topic",
|
||||
"difficulty": "medium",
|
||||
"question": "Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/cesar_gutierrez/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"anotácia entít"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Cesar Abascal Gutierrez sa počas svojej stáže v lete 2019 venoval problému rozpoznávania neznámych pomenovaných entít, pričom jeho cieľom bolo vytvoriť manuálne anotovanú trénovaciu sadu z prepisov reči a navrhnúť schému pre túto anotáciu.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?",
|
||||
"published_only": true,
|
||||
"limit": 5,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.371596,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 2.0651,
|
||||
"tool_latency_seconds": 0.371596,
|
||||
"final_model_latency_seconds": 11.762852,
|
||||
"total_latency_seconds": 14.200271,
|
||||
"usage": {
|
||||
"prompt_tokens": 14145,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 14327
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0968",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Pri osobe Ján Pták uveď názov témy alebo práce uvedenej v zázname typu „diplomový projekt“ a rok uvedený v názve príslušnej sekcie. Neuvádzaj rok začiatku štúdia.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Agent pre manažment záverečných prác",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Ján Pták vypracováva diplomový projekt s názvom „Agent pre manažment záverečných prác\" a rok uvedený v názve sekcie je 2026.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Pri osobe Ján Pták uveď názov témy alebo práce uvedenej v zázname typu „diplomový projekt“ a rok uvedený v názve príslušnej sekcie. Neuvádzaj rok začiatku štúdia.",
|
||||
"published_only": false,
|
||||
"limit": 10
|
||||
},
|
||||
"latency_seconds": 1.116938,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 1.744308,
|
||||
"tool_latency_seconds": 1.116938,
|
||||
"final_model_latency_seconds": 39.190319,
|
||||
"total_latency_seconds": 42.052811,
|
||||
"usage": {
|
||||
"prompt_tokens": 48117,
|
||||
"completion_tokens": 141,
|
||||
"total_tokens": 48258
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0982",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2016",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Rok začiatku štúdia Dáriusa Lindvaiho je 2016, zatiaľ čo jeho diplomová práca je vyjadrená kategóriou z roku 2021 (dp2021).\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"limit": 10,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 1.031873,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.704977,
|
||||
"tool_latency_seconds": 1.031873,
|
||||
"final_model_latency_seconds": 36.216216,
|
||||
"total_latency_seconds": 38.954341,
|
||||
"usage": {
|
||||
"prompt_tokens": 43045,
|
||||
"completion_tokens": 150,
|
||||
"total_tokens": 43195
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1092",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Agent pre manažment záverečných prác",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom Obnovenie interpunkcie pomocou hlbokých neurónových sietí v roku 2021, čo je rok vyjadrený kategóriou diplomovej práce v metadátach, zatiaľ čo Ján Pták má v zázname uvedený diplomový projekt v roku 2026 s témou Agent pre manažment záverečných prác.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.817265,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
],
|
||||
"first_model_latency_seconds": 1.81292,
|
||||
"tool_latency_seconds": 0.817265,
|
||||
"final_model_latency_seconds": 23.109547,
|
||||
"total_latency_seconds": 25.740566,
|
||||
"usage": {
|
||||
"prompt_tokens": 26229,
|
||||
"completion_tokens": 225,
|
||||
"total_tokens": 26454
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1125",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaia nenašla diplomová práca, ale iba diplomový projekt s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" z roku 2020, zatiaľ čo Andrii Pervashov má v sekciách uvedený návrh na tému diplomovej práce „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\" s označením roka 2027.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.747723,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak/dp2021"
|
||||
],
|
||||
"first_model_latency_seconds": 1.539209,
|
||||
"tool_latency_seconds": 0.747723,
|
||||
"final_model_latency_seconds": 21.225615,
|
||||
"total_latency_seconds": 23.513234,
|
||||
"usage": {
|
||||
"prompt_tokens": 23749,
|
||||
"completion_tokens": 229,
|
||||
"total_tokens": 23978
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 0.75,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.9375,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,10 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0020,dev,paraphrase,medium,Aká téma diplomovej práce je uvedená pri osobe „Maroš Harahus“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Neurónová morfologická anotácia slovenského jazyka""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2019/david_stromp"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus/timovy_projekt""]","Témou diplomovej práce Maroša Harahusa je Neurónová morfologická anotácia slovenského jazyka.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",2.342913,0.14733,7.150825,9.641468,8601,108,8709,
|
||||
q0021,dev,fact_work_year,easy,V ktorom roku je pri osobe Maroš Harahus uvedená diplomová práca?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/david_stromp""]","Pri osobe Maroš Harahus je uvedená diplomová práca v roku 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.366555,0.047068,4.463115,5.876927,4912,89,5001,
|
||||
q0028,dev,fact_title,easy,Aký je názov diplomová práca osoby Tomáš Kuchárik?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Osoba Tomáš Kuchárik má v ZP Wiki zaznamenané dve diplomové práce: z roku 2021 s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou crowdsourcingu"" a z roku 2022 s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",2.045989,0.041427,5.559473,7.647097,4850,203,5053,
|
||||
|
@ -0,0 +1,334 @@
|
||||
{
|
||||
"generated_at": "2026-08-16T18:04:36.438031+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "93c28885ab63887ff45d64509f2ad5e0e36b088066af8647a7e37bd47eb7ef7a",
|
||||
"selected_question_ids": [
|
||||
"q0020",
|
||||
"q0021",
|
||||
"q0028"
|
||||
],
|
||||
"applied_override_ids": [
|
||||
"q0020"
|
||||
],
|
||||
"split": "dev",
|
||||
"selected_question_count": 3,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 7.721831,
|
||||
"prompt_tokens": 18363,
|
||||
"completion_tokens": 400,
|
||||
"total_tokens": 18763,
|
||||
"by_category": {
|
||||
"fact_title": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 7.647097,
|
||||
"prompt_tokens": 4850,
|
||||
"completion_tokens": 203,
|
||||
"total_tokens": 5053
|
||||
},
|
||||
"fact_work_year": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 5.876927,
|
||||
"prompt_tokens": 4912,
|
||||
"completion_tokens": 89,
|
||||
"total_tokens": 5001
|
||||
},
|
||||
"paraphrase": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 9.641468,
|
||||
"prompt_tokens": 8601,
|
||||
"completion_tokens": 108,
|
||||
"total_tokens": 8709
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"easy": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 6.762012,
|
||||
"prompt_tokens": 9762,
|
||||
"completion_tokens": 292,
|
||||
"total_tokens": 10054
|
||||
},
|
||||
"medium": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 9.641468,
|
||||
"prompt_tokens": 8601,
|
||||
"completion_tokens": 108,
|
||||
"total_tokens": 8709
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0020",
|
||||
"split": "dev",
|
||||
"category": "paraphrase",
|
||||
"difficulty": "medium",
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Maroš Harahus“?",
|
||||
"override_applied": true,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Neurónová morfologická anotácia slovenského jazyka"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Témou diplomovej práce Maroša Harahusa je Neurónová morfologická anotácia slovenského jazyka.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Maroš Harahus diplomová práca téma",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.14733,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus/timovy_projekt"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus/timovy_projekt"
|
||||
],
|
||||
"first_model_latency_seconds": 2.342913,
|
||||
"tool_latency_seconds": 0.14733,
|
||||
"final_model_latency_seconds": 7.150825,
|
||||
"total_latency_seconds": 9.641468,
|
||||
"usage": {
|
||||
"prompt_tokens": 8601,
|
||||
"completion_tokens": 108,
|
||||
"total_tokens": 8709
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0021",
|
||||
"split": "dev",
|
||||
"category": "fact_work_year",
|
||||
"difficulty": "easy",
|
||||
"question": "V ktorom roku je pri osobe Maroš Harahus uvedená diplomová práca?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Pri osobe Maroš Harahus je uvedená diplomová práca v roku 2021.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Maroš Harahus diplomová práca rok"
|
||||
},
|
||||
"latency_seconds": 0.047068,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp"
|
||||
],
|
||||
"first_model_latency_seconds": 1.366555,
|
||||
"tool_latency_seconds": 0.047068,
|
||||
"final_model_latency_seconds": 4.463115,
|
||||
"total_latency_seconds": 5.876927,
|
||||
"usage": {
|
||||
"prompt_tokens": 4912,
|
||||
"completion_tokens": 89,
|
||||
"total_tokens": 5001
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0028",
|
||||
"split": "dev",
|
||||
"category": "fact_title",
|
||||
"difficulty": "easy",
|
||||
"question": "Aký je názov diplomová práca osoby Tomáš Kuchárik?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/tomas_kucharik/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Osoba Tomáš Kuchárik má v ZP Wiki zaznamenané dve diplomové práce: z roku 2021 s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou crowdsourcingu\" a z roku 2022 s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Tomáš Kuchárik diplomová práca názov",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.041427,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 2.045989,
|
||||
"tool_latency_seconds": 0.041427,
|
||||
"final_model_latency_seconds": 5.559473,
|
||||
"total_latency_seconds": 7.647097,
|
||||
"usage": {
|
||||
"prompt_tokens": 4850,
|
||||
"completion_tokens": 203,
|
||||
"total_tokens": 5053
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,4 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/topics/steel""]","Študentský dokument, ktorý sa venuje témam grafových neurónových sietí a má v metadátach uvedeného Maroša Harahusa ako autora dokumentu, je bakalárska práca s názvom „Grafové neurónové siete pre vyhľadávanie na internete"" študenta Dmytra Murala.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.445964,0.090987,9.778916,12.316244,10764,187,10951,
|
||||
|
@ -0,0 +1,159 @@
|
||||
{
|
||||
"generated_at": "2026-08-16T18:51:58.675909+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "52c161098d40fc722b049238a90bb77cee69f6872504dbf4fbbb3d101a36246c",
|
||||
"selected_question_ids": [
|
||||
"q0024"
|
||||
],
|
||||
"applied_override_ids": [
|
||||
"q0024"
|
||||
],
|
||||
"split": "dev",
|
||||
"selected_question_count": 1,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 12.316244,
|
||||
"prompt_tokens": 10764,
|
||||
"completion_tokens": 187,
|
||||
"total_tokens": 10951,
|
||||
"by_category": {
|
||||
"keyword_grounding": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 12.316244,
|
||||
"prompt_tokens": 10764,
|
||||
"completion_tokens": 187,
|
||||
"total_tokens": 10951
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"medium": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 12.316244,
|
||||
"prompt_tokens": 10764,
|
||||
"completion_tokens": 187,
|
||||
"total_tokens": 10951
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?",
|
||||
"override_applied": true,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Dmytro Mural",
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Answer-level disambiguation: študentský dokument patrí Dmytrovi Muralovi; Maroš Harahus je v jeho metadátach uvedený ako autor dokumentu. Nemá sa zamieňať autor dokumentu so študentom, ktorý prácu vypracoval.",
|
||||
"answer": "Študentský dokument, ktorý sa venuje témam grafových neurónových sietí a má v metadátach uvedeného Maroša Harahusa ako autora dokumentu, je bakalárska práca s názvom „Grafové neurónové siete pre vyhľadávanie na internete\" študenta Dmytra Murala.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Maroš Harahus autor študentský dokument grafové neuronové siete",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.090987,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/steel"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/steel"
|
||||
],
|
||||
"first_model_latency_seconds": 2.445964,
|
||||
"tool_latency_seconds": 0.090987,
|
||||
"final_model_latency_seconds": 9.778916,
|
||||
"total_latency_seconds": 12.316244,
|
||||
"usage": {
|
||||
"prompt_tokens": 10764,
|
||||
"completion_tokens": 187,
|
||||
"total_tokens": 10951
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,16 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0013,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou slovenský internet a osobou Ján Holp?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Ján Holp"", ""slovenský internet""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","S osobou Ján Holp súvisí študentský dokument (stránka) s názvom „Ján Holp"", v ktorom sú uvedené jeho záverečné práce týkajúce sa slovenského internetu. Ide o bakalársku prácu s názvom „Vyhľadávač na slovenskom internete"" z roku 2019 a diplomový projekt z roku 2020, ktorý sa zameriava na využitie veľkých slovenských dát z internetu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",1.889331,0.12737,5.917284,7.934235,4973,191,5164,
|
||||
q0024,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou grafové neurónové siete a osobou Maroš Harahus?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,0.0,1.0,0.75,False,"[""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","S témou grafové neurónové siete a osobou Maroš Harahus súvisí študentský dokument o študentovi Dmytro Mural, kde Maroš Harahus je uvedený ako autor dokumentu. Tento dokument obsahuje bakalársku prácu s témou „Grafové neurónové siete pre vyhľadávanie na internete“ a zároveň spomína štúdium grafových neurónových sietí v rámci bakalárskeho projektu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.084415,0.044731,6.389972,8.519334,5963,213,6176,
|
||||
q0026,dev,specific_fact,medium,Aké metódy mal Maroš Harahus študovať v súvislosti s US Steel?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V súvislosti s US Steel mal Maroš Harahus za úlohu naštudovať dáta z tejto firmy a vypracovať prehľad aktuálnych metód grafových neurónových sietí, pričom mal tiež nájsť a vyskúšať toolkit na GNN.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.40073,0.134669,5.826493,7.362124,5737,131,5868,
|
||||
q0029,dev,reverse_lookup,medium,Kto sa venuje práci s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tomáš Kuchárik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura""]","Pracou s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“ sa zaoberá Tomáš Kuchárik vo svojej diplomovej práci z roku 2022.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",2.280178,0.147059,10.601254,13.028955,12129,180,12309,
|
||||
q0030,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý rieši tému „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tomáš Kuchárik"", ""Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Tému „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"" rieši študent Tomáš Kuchárik vo svojej diplomovej práci z roku 2022.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",1.873471,0.053306,5.462488,7.389482,5158,148,5306,
|
||||
|
@ -0,0 +1,468 @@
|
||||
{
|
||||
"generated_at": "2026-08-16T18:49:10.965775+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "2e756dedb0431ae8ff368936ff0fbf6157eb90819b468b688d64276e07a91108",
|
||||
"selected_question_ids": [
|
||||
"q0013",
|
||||
"q0024",
|
||||
"q0026",
|
||||
"q0029",
|
||||
"q0030"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 5,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.95,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.8,
|
||||
"mean_latency_seconds": 8.846826,
|
||||
"prompt_tokens": 33960,
|
||||
"completion_tokens": 863,
|
||||
"total_tokens": 34823,
|
||||
"by_category": {
|
||||
"keyword_grounding": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 0.5,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 8.226785,
|
||||
"prompt_tokens": 10936,
|
||||
"completion_tokens": 404,
|
||||
"total_tokens": 11340
|
||||
},
|
||||
"reverse_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 13.028955,
|
||||
"prompt_tokens": 12129,
|
||||
"completion_tokens": 180,
|
||||
"total_tokens": 12309
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 7.362124,
|
||||
"prompt_tokens": 5737,
|
||||
"completion_tokens": 131,
|
||||
"total_tokens": 5868
|
||||
},
|
||||
"topic_lookup": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 7.389482,
|
||||
"prompt_tokens": 5158,
|
||||
"completion_tokens": 148,
|
||||
"total_tokens": 5306
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"medium": {
|
||||
"total": 5,
|
||||
"completed": 5,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 0.8,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.95,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.8,
|
||||
"mean_latency_seconds": 8.846826,
|
||||
"prompt_tokens": 33960,
|
||||
"completion_tokens": 863,
|
||||
"total_tokens": 34823
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0013",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou slovenský internet a osobou Ján Holp?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/jan_holp/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Ján Holp",
|
||||
"slovenský internet"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "S osobou Ján Holp súvisí študentský dokument (stránka) s názvom „Ján Holp\", v ktorom sú uvedené jeho záverečné práce týkajúce sa slovenského internetu. Ide o bakalársku prácu s názvom „Vyhľadávač na slovenskom internete\" z roku 2019 a diplomový projekt z roku 2020, ktorý sa zameriava na využitie veľkých slovenských dát z internetu.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "slovenský internet Ján Holp dokument študent",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.12737,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
],
|
||||
"first_model_latency_seconds": 1.889331,
|
||||
"tool_latency_seconds": 0.12737,
|
||||
"final_model_latency_seconds": 5.917284,
|
||||
"total_latency_seconds": 7.934235,
|
||||
"usage": {
|
||||
"prompt_tokens": 4973,
|
||||
"completion_tokens": 191,
|
||||
"total_tokens": 5164
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0024",
|
||||
"split": "dev",
|
||||
"category": "keyword_grounding",
|
||||
"difficulty": "medium",
|
||||
"question": "Ktorý dokument alebo študent súvisí s témou grafové neurónové siete a osobou Maroš Harahus?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Maroš Harahus",
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "S témou grafové neurónové siete a osobou Maroš Harahus súvisí študentský dokument o študentovi Dmytro Mural, kde Maroš Harahus je uvedený ako autor dokumentu. Tento dokument obsahuje bakalársku prácu s témou „Grafové neurónové siete pre vyhľadávanie na internete“ a zároveň spomína štúdium grafových neurónových sietí v rámci bakalárskeho projektu.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Maroš Harahus grafové neurónové siete dokument študent",
|
||||
"limit": 10,
|
||||
"published_only": false,
|
||||
"max_per_document": 5
|
||||
},
|
||||
"latency_seconds": 0.044731,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"
|
||||
],
|
||||
"first_model_latency_seconds": 2.084415,
|
||||
"tool_latency_seconds": 0.044731,
|
||||
"final_model_latency_seconds": 6.389972,
|
||||
"total_latency_seconds": 8.519334,
|
||||
"usage": {
|
||||
"prompt_tokens": 5963,
|
||||
"completion_tokens": 213,
|
||||
"total_tokens": 6176
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
false
|
||||
],
|
||||
"source_url_score": 0.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0026",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Aké metódy mal Maroš Harahus študovať v súvislosti s US Steel?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"grafové neurónové siete"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V súvislosti s US Steel mal Maroš Harahus za úlohu naštudovať dáta z tejto firmy a vypracovať prehľad aktuálnych metód grafových neurónových sietí, pričom mal tiež nájsť a vyskúšať toolkit na GNN.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Maroš Harahus US Steel metódy"
|
||||
},
|
||||
"latency_seconds": 0.134669,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"
|
||||
],
|
||||
"first_model_latency_seconds": 1.40073,
|
||||
"tool_latency_seconds": 0.134669,
|
||||
"final_model_latency_seconds": 5.826493,
|
||||
"total_latency_seconds": 7.362124,
|
||||
"usage": {
|
||||
"prompt_tokens": 5737,
|
||||
"completion_tokens": 131,
|
||||
"total_tokens": 5868
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0029",
|
||||
"split": "dev",
|
||||
"category": "reverse_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Kto sa venuje práci s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/tomas_kucharik/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Tomáš Kuchárik"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Pracou s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“ sa zaoberá Tomáš Kuchárik vo svojej diplomovej práci z roku 2022.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu autor",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.147059,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"
|
||||
],
|
||||
"first_model_latency_seconds": 2.280178,
|
||||
"tool_latency_seconds": 0.147059,
|
||||
"final_model_latency_seconds": 10.601254,
|
||||
"total_latency_seconds": 13.028955,
|
||||
"usage": {
|
||||
"prompt_tokens": 12129,
|
||||
"completion_tokens": 180,
|
||||
"total_tokens": 12309
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0030",
|
||||
"split": "dev",
|
||||
"category": "topic_lookup",
|
||||
"difficulty": "medium",
|
||||
"question": "Nájdi študenta alebo dokument, ktorý rieši tému „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu“.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/tomas_kucharik/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Tomáš Kuchárik",
|
||||
"Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Tému „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu\" rieši študent Tomáš Kuchárik vo svojej diplomovej práci z roku 2022.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.053306,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"first_model_latency_seconds": 1.873471,
|
||||
"tool_latency_seconds": 0.053306,
|
||||
"final_model_latency_seconds": 5.462488,
|
||||
"total_latency_seconds": 7.389482,
|
||||
"usage": {
|
||||
"prompt_tokens": 5158,
|
||||
"completion_tokens": 148,
|
||||
"total_tokens": 5306
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,29 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0039,dev,fact_title,easy,Aký je názov diplomová práca osoby Jakub Maruniak?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc""]","Názov diplomovej práce Jakuba Maruniaka je „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",2.316807,0.445829,19.392639,22.156105,22596,142,22738,
|
||||
q0042,dev,paraphrase,medium,Aká téma diplomovej práce je uvedená pri osobe „Jakub Maruniak“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity"", ""https://zp.kemt.fei.tuke.sk/topics/prodigy""]","Pri osobe Jakub Maruniak je uvedený návrh na názov diplomovej práce „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",1.942664,0.331068,13.590749,15.865577,15594,147,15741,
|
||||
q0393,dev,specific_fact,medium,Akú databázu a backend už Ján Pták podľa stavu používal?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""SQLite"", ""FastAPI""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o tom, akú databázu a backend Ján Pták už používal. Z dostupných zdrojov vyplýva, že Ján Pták pracuje na diplomovom projekte s témou „Agent pre manažment záverečných prác"", kde má za cieľ vytvoriť agenta využívajúceho technológie ako RAG a GraphRAG, pričom ako rozhranie sa plánuje použitie OpenWebUI, ale konkrétne uvedené databázy alebo backendové technológie, ktoré by už boli implementované, sa v záznamoch neuvádzajú.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.926943,0.249945,25.411096,27.588697,26953,249,27202,
|
||||
q0575,dev,specific_fact,medium,Aké tri hlavné ciele mal Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.3333333333333333,1.0,1.0,0.8333333333333334,False,"[""recognize unknown named entities"", ""manually annotated training set"", ""annotation schema""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Cesar Abascal Gutierrez mal počas svojej praxe v lete 2019 tri hlavné ciele: byť schopný rozpoznať neznáme menované entity, vytvoriť manuálne anotovanú tréningovú sadu z transkriptov reči a navrhnúť schému anotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.96392,0.123979,11.664886,13.753221,12078,175,12253,
|
||||
q0624,dev,specific_fact,medium,Koľko otázok má vzniknúť pre každý odsek pri anotácii otázok?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""5""]","[""https://zp.kemt.fei.tuke.sk/topics/question""]","[""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/topics/question/navod"", ""https://zp.kemt.fei.tuke.sk/topics/question/validacie"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","Podľa dokumentácie projektu Question Answering sa má pri anotácii vytvoriť 5 otázok pre každý odsek.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/question",1.858985,0.379721,24.122253,26.361773,28087,114,28201,
|
||||
q0689,dev,multi_document,hard,Ktoré dokumenty sa venujú rozpoznávaniu pomenovaných entít alebo ich anotácii?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""pomenované entity"", ""NER""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod""]","[""https://zp.kemt.fei.tuke.sk/topics/named-entity"", ""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/topics/spacy"", ""https://zp.kemt.fei.tuke.sk/topics/resources"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic""]","Na rozpoznávanie pomenovaných entít a ich anotáciu sa vo ZP Wiki venuje niekoľko dokumentov vrátane témy Pomenované entity, ktorá popisuje úlohy a nástroje ako Prodigy, ako aj študentských prác a projektov. Konkrétne ide o bakalársku prácu Artema Yatsenka s názvom Rozpoznávanie pomenovaných entít v slovenskom jazyku, diplomový projekt Jakuba Maruniaka s návrhom názvu Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku, bakalársky projekt Than Trung Thanh o rozpoznávaní pomenovaných entít pomocou nástrojov Spacy a Prodigy, bakalársku prácu Ediza Morochoviča o štatistickom rozpoznávaní pomenovaných entít a bakalársku prácu Serhiiho Yemets zameranú na zlepšenie slovenského modelu pre rozpoznávanie pomenovaných entít. Ďalej sa na tému pracuje v internom projekte Cesar Abascal Gutierrez a v témach Podpora slovenčiny v knižnici Spacy a Slovenské jazykové zdroje, ktoré spomínajú anotáciu pomenovaných entít a existujúce korpusy.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/topics/named-entity
|
||||
https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez
|
||||
https://zp.kemt.fei.tuke.sk/topics/spacy
|
||||
https://zp.kemt.fei.tuke.sk/topics/resources
|
||||
https://zp.kemt.fei.tuke.sk/topics/named-entity/navod
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak
|
||||
https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic",1.889659,0.435265,41.331604,43.657548,37934,567,38501,
|
||||
|
@ -0,0 +1,646 @@
|
||||
{
|
||||
"generated_at": "2026-08-16T21:42:32.047540+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "18d354c819cd8f062cf463e43908b7ae3af3195431e664f69848bc3fd744c6db",
|
||||
"selected_question_ids": [
|
||||
"q0039",
|
||||
"q0042",
|
||||
"q0393",
|
||||
"q0575",
|
||||
"q0624",
|
||||
"q0689"
|
||||
],
|
||||
"applied_override_ids": [
|
||||
"q0042"
|
||||
],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.722222,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.930556,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 24.897153,
|
||||
"prompt_tokens": 143242,
|
||||
"completion_tokens": 1394,
|
||||
"total_tokens": 144636,
|
||||
"by_category": {
|
||||
"fact_title": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 22.156105,
|
||||
"prompt_tokens": 22596,
|
||||
"completion_tokens": 142,
|
||||
"total_tokens": 22738
|
||||
},
|
||||
"multi_document": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 43.657548,
|
||||
"prompt_tokens": 37934,
|
||||
"completion_tokens": 567,
|
||||
"total_tokens": 38501
|
||||
},
|
||||
"paraphrase": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.865577,
|
||||
"prompt_tokens": 15594,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 15741
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.444444,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.861111,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.333333,
|
||||
"mean_latency_seconds": 22.567897,
|
||||
"prompt_tokens": 67118,
|
||||
"completion_tokens": 538,
|
||||
"total_tokens": 67656
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"easy": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 22.156105,
|
||||
"prompt_tokens": 22596,
|
||||
"completion_tokens": 142,
|
||||
"total_tokens": 22738
|
||||
},
|
||||
"hard": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 43.657548,
|
||||
"prompt_tokens": 37934,
|
||||
"completion_tokens": 567,
|
||||
"total_tokens": 38501
|
||||
},
|
||||
"medium": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.583333,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.895833,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 20.892317,
|
||||
"prompt_tokens": 82712,
|
||||
"completion_tokens": 685,
|
||||
"total_tokens": 83397
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0039",
|
||||
"split": "dev",
|
||||
"category": "fact_title",
|
||||
"difficulty": "easy",
|
||||
"question": "Aký je názov diplomová práca osoby Jakub Maruniak?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/jakub_maruniak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Názov diplomovej práce Jakuba Maruniaka je „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Aký je názov diplomová práca osoby Jakub Maruniak?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.445829,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc"
|
||||
],
|
||||
"first_model_latency_seconds": 2.316807,
|
||||
"tool_latency_seconds": 0.445829,
|
||||
"final_model_latency_seconds": 19.392639,
|
||||
"total_latency_seconds": 22.156105,
|
||||
"usage": {
|
||||
"prompt_tokens": 22596,
|
||||
"completion_tokens": 142,
|
||||
"total_tokens": 22738
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0042",
|
||||
"split": "dev",
|
||||
"category": "paraphrase",
|
||||
"difficulty": "medium",
|
||||
"question": "Aká téma diplomovej práce je uvedená pri osobe „Jakub Maruniak“?",
|
||||
"override_applied": true,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/jakub_maruniak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Pri osobe Jakub Maruniak je uvedený návrh na názov diplomovej práce „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku\".\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Aká téma diplomovej práce je uvedená pri osobe „Jakub Maruniak“?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.331068,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/prodigy"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/prodigy"
|
||||
],
|
||||
"first_model_latency_seconds": 1.942664,
|
||||
"tool_latency_seconds": 0.331068,
|
||||
"final_model_latency_seconds": 13.590749,
|
||||
"total_latency_seconds": 15.865577,
|
||||
"usage": {
|
||||
"prompt_tokens": 15594,
|
||||
"completion_tokens": 147,
|
||||
"total_tokens": 15741
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0393",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"SQLite",
|
||||
"FastAPI"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o tom, akú databázu a backend Ján Pták už používal. Z dostupných zdrojov vyplýva, že Ján Pták pracuje na diplomovom projekte s témou „Agent pre manažment záverečných prác\", kde má za cieľ vytvoriť agenta využívajúceho technológie ako RAG a GraphRAG, pričom ako rozhranie sa plánuje použitie OpenWebUI, ale konkrétne uvedené databázy alebo backendové technológie, ktoré by už boli implementované, sa v záznamoch neuvádzajú.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.249945,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.926943,
|
||||
"tool_latency_seconds": 0.249945,
|
||||
"final_model_latency_seconds": 25.411096,
|
||||
"total_latency_seconds": 27.588697,
|
||||
"usage": {
|
||||
"prompt_tokens": 26953,
|
||||
"completion_tokens": 249,
|
||||
"total_tokens": 27202
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false,
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0575",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/cesar_gutierrez/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"recognize unknown named entities",
|
||||
"manually annotated training set",
|
||||
"annotation schema"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Cesar Abascal Gutierrez mal počas svojej praxe v lete 2019 tri hlavné ciele: byť schopný rozpoznať neznáme menované entity, vytvoriť manuálne anotovanú tréningovú sadu z transkriptov reči a navrhnúť schému anotácie.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.123979,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"
|
||||
],
|
||||
"first_model_latency_seconds": 1.96392,
|
||||
"tool_latency_seconds": 0.123979,
|
||||
"final_model_latency_seconds": 11.664886,
|
||||
"total_latency_seconds": 13.753221,
|
||||
"usage": {
|
||||
"prompt_tokens": 12078,
|
||||
"completion_tokens": 175,
|
||||
"total_tokens": 12253
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false,
|
||||
false,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 0.3333333333333333,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.8333333333333334,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0624",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Koľko otázok má vzniknúť pre každý odsek pri anotácii otázok?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/topics/question/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"5"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Podľa dokumentácie projektu Question Answering sa má pri anotácii vytvoriť 5 otázok pre každý odsek.\n\nZdroj: https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Koľko otázok má vzniknúť pre každý odsek pri anotácii otázok?",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.379721,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/navod",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/validacie",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/navod",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question/validacie",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.858985,
|
||||
"tool_latency_seconds": 0.379721,
|
||||
"final_model_latency_seconds": 24.122253,
|
||||
"total_latency_seconds": 26.361773,
|
||||
"usage": {
|
||||
"prompt_tokens": 28087,
|
||||
"completion_tokens": 114,
|
||||
"total_tokens": 28201
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0689",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Ktoré dokumenty sa venujú rozpoznávaniu pomenovaných entít alebo ich anotácii?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/jakub_maruniak/README.md",
|
||||
"pages/students/2019/artem_yatsenko/README.md",
|
||||
"pages/interns/cesar_gutierrez/README.md",
|
||||
"pages/topics/named-entity/README.md",
|
||||
"pages/topics/named-entity/navod/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity/navod"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"pomenované entity",
|
||||
"NER"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Na rozpoznávanie pomenovaných entít a ich anotáciu sa vo ZP Wiki venuje niekoľko dokumentov vrátane témy Pomenované entity, ktorá popisuje úlohy a nástroje ako Prodigy, ako aj študentských prác a projektov. Konkrétne ide o bakalársku prácu Artema Yatsenka s názvom Rozpoznávanie pomenovaných entít v slovenskom jazyku, diplomový projekt Jakuba Maruniaka s návrhom názvu Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku, bakalársky projekt Than Trung Thanh o rozpoznávaní pomenovaných entít pomocou nástrojov Spacy a Prodigy, bakalársku prácu Ediza Morochoviča o štatistickom rozpoznávaní pomenovaných entít a bakalársku prácu Serhiiho Yemets zameranú na zlepšenie slovenského modelu pre rozpoznávanie pomenovaných entít. Ďalej sa na tému pracuje v internom projekte Cesar Abascal Gutierrez a v témach Podpora slovenčiny v knižnici Spacy a Slovenské jazykové zdroje, ktoré spomínajú anotáciu pomenovaných entít a existujúce korpusy.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/topics/named-entity\nhttps://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez\nhttps://zp.kemt.fei.tuke.sk/topics/spacy\nhttps://zp.kemt.fei.tuke.sk/topics/resources\nhttps://zp.kemt.fei.tuke.sk/topics/named-entity/navod\nhttps://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko\nhttps://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak\nhttps://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh\nhttps://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets\nhttps://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Ktoré dokumenty sa venujú rozpoznávaniu pomenovaných entít alebo ich anotácii?",
|
||||
"limit": 10,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.435265,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/spacy",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/resources",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity/navod",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity",
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/spacy",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/resources",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/named-entity/navod",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"
|
||||
],
|
||||
"first_model_latency_seconds": 1.889659,
|
||||
"tool_latency_seconds": 0.435265,
|
||||
"final_model_latency_seconds": 41.331604,
|
||||
"total_latency_seconds": 43.657548,
|
||||
"usage": {
|
||||
"prompt_tokens": 37934,
|
||||
"completion_tokens": 567,
|
||||
"total_tokens": 38501
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 5,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,7 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0393,dev,specific_fact,medium,Akú databázu a backend už Ján Pták podľa stavu používal?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""SQLite"", ""FastAPI""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informácie o tom, akú databázu a backend Ján Pták už podľa stavu používal, keďže dostupný záznam pre jeho diplomový projekt uvádza iba ciele, tématické oblasti a úlohy (ako je použitie OpenWebUI, Docker, LangChain a GraphRAG) bez explicitného spomenutia konkrétnej implementovanej databázy alebo backendového frameworku.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",2.987626,0.364613,21.934543,25.28763,26926,182,27108,
|
||||
q0575,dev,specific_fact,medium,Aké tri hlavné ciele mal Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.6666666666666666,1.0,1.0,0.9166666666666666,False,"[""recognize unknown named entities"", ""manually annotated training set"", ""annotation schema""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Cesar Abascal Gutierrez mal počas svojej stáže v lete 2019 stanovené tri hlavné ciele: byť schopný rozpoznať neznáme menované entity, vytvoriť manuálne anotovanú trénovaciu množinu z prepisov reči a navrhnúť anotáciu schému.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.907934,0.111388,10.773955,12.793719,12079,176,12255,
|
||||
|
@ -0,0 +1,232 @@
|
||||
{
|
||||
"generated_at": "2026-09-27T09:47:47.360199+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "8f4f7bf2d60a3d58fbc338932663d33046dde1b7951ca82402bc30f0ce627b7b",
|
||||
"selected_question_ids": [
|
||||
"q0393",
|
||||
"q0575"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 2,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.333333,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 19.040675,
|
||||
"prompt_tokens": 39005,
|
||||
"completion_tokens": 358,
|
||||
"total_tokens": 39363,
|
||||
"by_category": {
|
||||
"specific_fact": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.333333,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 19.040675,
|
||||
"prompt_tokens": 39005,
|
||||
"completion_tokens": 358,
|
||||
"total_tokens": 39363
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"medium": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.333333,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.833333,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 19.040675,
|
||||
"prompt_tokens": 39005,
|
||||
"completion_tokens": 358,
|
||||
"total_tokens": 39363
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0393",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"SQLite",
|
||||
"FastAPI"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informácie o tom, akú databázu a backend Ján Pták už podľa stavu používal, keďže dostupný záznam pre jeho diplomový projekt uvádza iba ciele, tématické oblasti a úlohy (ako je použitie OpenWebUI, Docker, LangChain a GraphRAG) bez explicitného spomenutia konkrétnej implementovanej databázy alebo backendového frameworku.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.364613,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 2.987626,
|
||||
"tool_latency_seconds": 0.364613,
|
||||
"final_model_latency_seconds": 21.934543,
|
||||
"total_latency_seconds": 25.28763,
|
||||
"usage": {
|
||||
"prompt_tokens": 26926,
|
||||
"completion_tokens": 182,
|
||||
"total_tokens": 27108
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false,
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0575",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/cesar_gutierrez/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"recognize unknown named entities",
|
||||
"manually annotated training set",
|
||||
"annotation schema"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Cesar Abascal Gutierrez mal počas svojej stáže v lete 2019 stanovené tri hlavné ciele: byť schopný rozpoznať neznáme menované entity, vytvoriť manuálne anotovanú trénovaciu množinu z prepisov reči a navrhnúť anotáciu schému.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.111388,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"
|
||||
],
|
||||
"first_model_latency_seconds": 1.907934,
|
||||
"tool_latency_seconds": 0.111388,
|
||||
"final_model_latency_seconds": 10.773955,
|
||||
"total_latency_seconds": 12.793719,
|
||||
"usage": {
|
||||
"prompt_tokens": 12079,
|
||||
"completion_tokens": 176,
|
||||
"total_tokens": 12255
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 0.6666666666666666,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.9166666666666666,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,162 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0039,dev,fact_title,easy,Aký je názov diplomová práca osoby Jakub Maruniak?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc"", ""https://zp.kemt.fei.tuke.sk/students/2016/mark_feher"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť názov diplomovej práce osoby Jakub Maruniak.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",2.24246,0.389789,13.368636,16.00156,16348,138,16486,
|
||||
q0042,dev,paraphrase,medium,Aká téma diplomovej práce je uvedená pri osobe „Jakub Maruniak“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity"", ""https://zp.kemt.fei.tuke.sk/topics/prodigy""]","V dostupných dokumentoch ZP Wiki sa explicitný názov témy diplomovej práce pri osobe Jakub Maruniak nenašiel; dokumenty uvádzajú iba sekciu „Diplomová práca 2021"" a obsahujú informácie o anotáciách dát, trénovaní modelov a dosahovanej presnosti, ale chýba tu konkrétny formulovaný názov témy.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",1.933275,0.206227,13.48626,15.626358,14790,198,14988,
|
||||
q0046,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Jakub Maruniak.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2020/jakub_kristian_lukas"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2022/jakub_schwarc"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2018/adrian_remias""]","Osoba Jakub Maruniak začala štúdium v roku 2016 a jeho diplomový projekt bol evidovaný v roku 2020 a 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak",1.587601,0.654635,25.131313,27.374407,29215,125,29340,
|
||||
q0105,dev,topic_lookup,medium,"Nájdi študenta alebo dokument, ktorý rieši tému „Metódy automatického prekladu“.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Patrik Pavlišin"", ""Metódy automatického prekladu""]","[""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Študent Patrik Pavlišin vypracoval bakalársku prácu s názvom „Metódy automatického prekladu"" v roku 2020.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",1.698637,0.405739,21.307265,23.412287,24567,129,24696,
|
||||
q0130,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou neurónové siete a osobou Ondrej Megela?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Ondrej Megela"", ""neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","[""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko""]","Ondrej Megela vypracoval bakalársku prácu s názvom „Neurónové jazykové modelovanie typu BERT"", ktorá sa zaoberá neurónovými sieťami.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",1.985063,0.335115,16.745465,19.066164,19007,148,19155,
|
||||
q0143,dev,fact_title,easy,Aký je názov diplomová práca osoby Dávid Omasta?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_ilas"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/mark_feher"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Dávid Omasta má v rámci diplomovej práce roku 2024 ako návrh na názov zadané „Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku"", pričom v predchádzajúcom roku 2023 bol uvedený návrh názvu „Generatívne modely slovenského jazyka"".
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2018/david_omasta",1.980822,0.189392,12.825413,14.996126,13515,190,13705,
|
||||
q0146,dev,paraphrase,medium,Aká téma diplomovej práce je uvedená pri osobe „Dávid Omasta“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_ilas"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/mark_feher""]","Pri osobe Dávid Omasta je v dokumente uvedená téma diplomovej práce pre rok 2023 ako „Dotrénovanie slovenského generatívneho jazykového modelu"" a pre rok 2024 je uvedený návrh na názov „Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2018/david_omasta",1.947583,0.345171,12.631562,14.924875,13483,187,13670,
|
||||
q0243,dev,specific_fact,medium,Koľko otázok bolo vytvorených na testovanie systému Mateja Ščišľaka?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""110""]","[""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","[""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","Na testovanie systému, ktorý vyvíja Matej Ščišľak v rámci svojej diplomovej práce, bolo vytvorených 110 otázok.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",1.601706,0.397502,20.154136,22.15397,23287,119,23406,
|
||||
q0334,dev,fact_title,easy,Aký je názov bakalárska práca osoby Tetiana Mohorian?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"", ""https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova""]","Názov bakalárskej práce Tetiany Mohorian je „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian",1.987856,0.14545,12.774859,14.908613,14358,142,14500,
|
||||
q0337,dev,paraphrase,medium,Aká téma bakalárskej práce je uvedená pri osobe „Tetiana Mohorian“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian"", ""https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak"", ""https://zp.kemt.fei.tuke.sk/students/2016/ediz_morochovic"", ""https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Téma bakalárskej práce pri osobe Tetiana Mohorian je „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov“.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian",1.995099,0.196022,11.963397,14.155102,13461,145,13606,
|
||||
q0341,dev,disambiguation_year,hard,Rozlíš rok začiatku štúdia a rok práce osoby Tetiana Mohorian.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2025""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian"", ""https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/topics/legal""]","Podľa dostupných informácií Tetiana Mohorian začala štúdium v roku 2022 a jej bakalárska práca je plánovaná na rok 2025. V dostupných dokumentoch sa nepodarilo nájsť explicitne uvedený rok, v ktorom prácu vykonáva, ale z kontextu sekcie „Bakalárska práca 2025"" vyplýva, že ide o prácu plánovanú na tento rok.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian",1.829271,0.301089,14.547317,16.67824,15290,201,15491,
|
||||
q0345,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou prompting a osobou Tetiana Mohorian?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Tetiana Mohorian"", ""prompting""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","[""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian""]","Tetiana Mohorian vypracováva bakalársku prácu s názvom „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov"", v ktorej sa zaoberá aj témou prompting, keďže jej úlohou je aplikovať existujúce modely na detekciu nenávistnej reči s využitím prompting a metódy LORA.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian",1.917566,0.138939,8.036479,10.093246,7594,191,7785,
|
||||
q0376,dev,specific_fact,medium,Aký hardvér mal Daniil Huzenko použiť na vlastný klaster?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Raspberry PI""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko""]","[""https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko"", ""https://zp.kemt.fei.tuke.sk/students/2018/samuel_sirotnik"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2020/matus_suchanic""]","V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť konkrétne informácie o type hardvéru, ktorý mal Daniil Huzenko použiť na svoj vlastný klaster, hoci z dokumentu vyplýva, že mal zostaviť klaster z dosiek Raspberry Pi a nainštalovať naň platformu Kubernetes.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/daniil_huzenko",1.774413,0.210807,18.742771,20.728521,20928,171,21099,
|
||||
q0387,dev,adversarial_year,hard,Je rok 2022 rokom záverečnej práce osoby Ján Pták?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa rok 2022 uvádza ako rok začiatku štúdia pre Jána Ptáka, ktorý vypracovával diplomový projekt s témou „Agent pre manažment záverečných prác"", pričom rok ukončenia tejto práce je uvedený ako 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",2.090508,0.208552,12.388959,14.688544,12953,181,13134,
|
||||
q0393,dev,specific_fact,medium,Akú databázu a backend už Ján Pták podľa stavu používal?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,0.0,0.5,False,"[""SQLite"", ""FastAPI""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť. Zdroj pre Ján Pták uvádza tému „Agent pre manažment záverečných prác"", ciele, nástroje ako OpenWebUI, Docker, LangChain a úlohy súvisiace s RAG a GraphRAG, ale explicitne nešpecifikuje použitú databázu ani backendovú technológiu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.947817,0.221708,17.529245,19.699329,18930,199,19129,
|
||||
q0435,dev,keyword_grounding,medium,Ktorý dokument alebo študent súvisí s témou LLM a osobou Rostyslav Rodzhuk?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Rostyslav Rodzhuk"", ""LLM""]","[""https://zp.kemt.fei.tuke.sk/students/2023/rostyslav_rodzhuk""]","[""https://zp.kemt.fei.tuke.sk/students/2023/rostyslav_rodzhuk""]","Študent Rostyslav Rodzhuk vypracováva bakalársku prácu s témou „Inteligentný agent pre podporu práce špeciálneho pedagóga"", ktorá priamo využíva veľké jazykové modely (LLM) na vytvorenie agenta, a zároveň sa venuje vedeckému projektu s názvom „Veľký jazykový model ako inteligentný agent"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/rostyslav_rodzhuk",2.022361,0.225358,7.918054,10.16611,7232,200,7432,
|
||||
q0438,dev,fact_title,easy,Aký je názov bakalárska práca osoby Yevhenii Medushivskyi?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Získavanie textu z webových stránok""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/yevhenii_leonov"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko""]","Bakalárska práca Yevhenii Medushivskyi má názov „Získavanie textu z webových stránok"".
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",2.051048,0.177561,17.339599,19.568962,19832,148,19980,
|
||||
q0439,dev,reverse_lookup,medium,Kto sa venuje práci s názvom „Získavanie textu z webových stránok“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Yevhenii Medushivskyi""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi""]","[""https://zp.kemt.fei.tuke.sk/students/2019/david_stromp"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko""]","Pomena práce s presným názvom „Získavanie textu z webových stránok"" sa venuje študent Yevhenii Medushivskyi v rámci svojej bakalárskej práce pre rok 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",2.008235,0.45602,21.990257,24.455158,24814,164,24978,
|
||||
q0450,dev,specific_fact,medium,Približne koľko textu mal Yevhenii Medushivskyi zozbieraného v marci 2026?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""450MB""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi""]","[""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi""]","Yevhenii Medushivskyi mal zozbieraného približne 450 MB textu k marcu 2026, konkrétne k dátumu 13. marca 2026, ako je uvedené v zázname o stretnutí.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi",2.206095,0.171027,8.437544,10.81491,8355,186,8541,
|
||||
q0551,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Oliver Pejic?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Hate Speech a MTEB evaluácia""]","[""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic""]","[""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko""]","Oliver Pejic sa počas svojej stáže v lete 2024 venoval problematike hodnotenia modelov vkladania viet (sentence transformer models) pomocou nástroja MTEB a príprave nových úloh pre hodnenie v oblasti detekcie nenávistných prejavov (Slovak HATE speech) a hľadania odpovedí na otázky (Slovak question answering).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",1.965366,0.139742,7.717434,9.822848,7278,191,7469,
|
||||
q0556,dev,specific_fact,medium,Na akých dvoch MTEB úlohách mal Oliver Pejic pracovať?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Slovak HATE speech"", ""Slovak question answering""]","[""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic""]","[""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic"", ""https://zp.kemt.fei.tuke.sk/students/2020/david_kostilnik"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic""]","Oliver Pejic mal počas svojej internship práce pracovať na príprave dvoch úloh pre MTEB (Massive Text Embedding Benchmark): úlohy pre slovenský korpus zložený z dát o nenávistných prejavoch (Slovak Hate Speech) a úlohy pre slovenské vyhľadávanie odpovedí na otázky (Slovak Question Answering/SK-QUAD).
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/interns/oliver_pejic",1.528918,0.272722,22.326568,24.128859,24882,168,25050,
|
||||
q0570,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""anotácia pomenovaných entít""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus""]","Stáž osoby Cesar Abascal Gutierrez sa venovala problémom v oblasti označovania named entity (identifikácia neznámych pojmov) a tvorbe manuálne anotovanej trénovacej množiny z prepisov reči.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.997514,0.267191,11.498318,13.763515,12349,156,12505,
|
||||
q0572,dev,keyword_grounding,medium,Ktorá stáž osoby Cesar Abascal Gutierrez súvisela s témou named entities?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Cesar Abascal Gutierrez"", ""named entities""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko""]","Stáž osoby Cesar Abascal Gutierrez súvisela s témou anotácie pomenovaných entít (Named entity annotations), kde mal za cieľ rozpoznávať neznáme pomenované entity a vytvárať manuálne anotačné trénovacie súbory z prepisov reči.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",2.017878,0.184736,15.52677,17.72995,17444,169,17613,
|
||||
q0575,dev,specific_fact,medium,Aké tri hlavné ciele mal Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.6666666666666666,1.0,1.0,0.9166666666666666,False,"[""recognize unknown named entities"", ""manually annotated training set"", ""annotation schema""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Cesar Abascal Gutierrez mal počas svojej praxe v lete 2019 stanovené tri hlavné ciele: byť schopný rozpoznať neznáme menované entity, vytvoriť manuálne anotovanú trénovaciu množinu z prepisov reči a navrhnúť schému anotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.972809,0.128692,8.400085,10.502176,8493,176,8669,
|
||||
q0579,dev,internship_topic,medium,Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""multilinguálna extrakcia trojíc z medicínskych dát""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh"", ""https://zp.kemt.fei.tuke.sk/topics/question/nezodpovedatelne"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka""]","Stáž osoby Bogdan Paul Chiš sa venovala problému extrakcie trojíc (triplet extraction) z lekárskych dát, pričom cieľom bolo zostrojiť graf znalostí z viacjazyčných lekárnických letáčkov a využiť ho v inteligentnom agentovi odporúčajúcom lieky.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.322095,0.212376,13.382888,14.91784,14769,137,14906,
|
||||
q0580,dev,reverse_lookup,medium,Kto pracoval na téme multilinguálna extrakcia trojíc z medicínskych dát?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Bogdan Paul Chiș""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"", ""https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko""]","Na téme multilinguálnej extrakcie trojíc z medicínskych dát pracoval Bogdan Paul Chiš počas svojho praxe v rámci programu ERASMUS na jar 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",2.099069,0.536416,20.346367,22.982487,23303,157,23460,
|
||||
q0582,dev,keyword_grounding,medium,Ktorá stáž osoby Bogdan Paul Chiș súvisela s témou knowledge graph?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Bogdan Paul Chiș"", ""knowledge graph""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh""]","Stáž osoby Bogdan Paul Chiš s názvom „(multilingual) Triplet extraction from medical data"", ktorá sa konala v rámci programu ERASMUS Intern Spring 2026, súvisela s témou knowledge graph, pričom cieľom bolo zostaviť znalostný graf z lekárskych balíčkových letákov v viacerých jazykoch a využiť ho v inteligentnom agentovi na odporúčanie liekov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis",1.717973,0.233427,13.696612,15.648477,14211,186,14397,
|
||||
q0597,dev,topic_page_lookup,easy,Nájdi projektovú alebo informačnú stránku „Dialógový systém“.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dialógový systém""]","[""https://zp.kemt.fei.tuke.sk/topics/chatbot""]","[""https://zp.kemt.fei.tuke.sk/topics/chatbot"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2019/samuel_horani"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko""]","Projektová a informačná stránka pre tému „Dialógový systém"" (tiež označená ako chatbot) sa nachádza na adrese https://zp.kemt.fei.tuke.sk/topics/chatbot.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/chatbot",1.976041,0.269402,19.289668,21.53587,22242,143,22385,
|
||||
q0598,dev,topic_page_lookup,medium,Ktorý dokument v ZP Wiki sa priamo venuje téme „Dialógový systém“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dialógový systém""]","[""https://zp.kemt.fei.tuke.sk/topics/chatbot""]","[""https://zp.kemt.fei.tuke.sk/topics/chatbot"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2019/samuel_horani"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko""]","Priamo téme „Dialógový systém"" sa v ZP Wiki venuje dokument s názvom „Dialógový systém"", ktorý je projektovou stránkou pre tvorbu chatbotov.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/chatbot",1.541873,0.298781,20.033891,21.875108,22903,111,23014,
|
||||
q0604,dev,specific_fact,medium,S akým ďalším typom systému sa má chatbot integrovať?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""QA systémom""]","[""https://zp.kemt.fei.tuke.sk/topics/chatbot""]","[""https://zp.kemt.fei.tuke.sk/topics/bert"", ""https://zp.kemt.fei.tuke.sk/topics/chatbot"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","Podľa dostupných informácií sa chatbot má integrovať s QA systémom (systémom na odpovedanie na otázky).
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/topics/chatbot
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",1.888498,0.352899,11.373793,13.615559,12322,146,12468,
|
||||
q0616,dev,topic_page_lookup,medium,Ktorý dokument v ZP Wiki sa priamo venuje téme „Question Answering“?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Question Answering""]","[""https://zp.kemt.fei.tuke.sk/topics/question""]","[""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic""]","Dokument v ZP Wiki, ktorý sa priamo venuje téme „Question Answering“, má názov „Question Answering“ a zameriava sa na vytvorenie slovenského klonu datasetu SQuaD 2.0, prípravu anotácií na dátach z Wikipédie a vývoj anotačnej infraštruktúry.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/question",1.468455,0.399011,22.955516,24.823639,25822,147,25969,
|
||||
q0621,dev,specific_fact,medium,Aký dataset chce projekt Question Answering vytvoriť v slovenskom jazyku?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""SQuaD 2.0""]","[""https://zp.kemt.fei.tuke.sk/topics/question""]","[""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2018/oleh_bilykh""]","Projekt Question Answering chce vytvoriť slovenskú kópiu datasetu SQuAD 2.0 na základe údajov zo slovenskej Wikipédie. Tento dataset bude mať finálny výstup v SQUAD formáte a zahŕňať bude aj neobslúhovateľné otázky.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/question",1.777923,0.280932,20.842367,22.90183,23248,154,23402,
|
||||
q0622,dev,specific_fact,medium,Aký anotačný nástroj používa projekt Question Answering?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Prodigy""]","[""https://zp.kemt.fei.tuke.sk/topics/question""]","[""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2019/alina_vitko""]","Projekt Question Answering používa na anotáciu nástroj Prodigy.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/question",1.949462,0.166443,11.343564,13.459861,13205,108,13313,
|
||||
q0624,dev,specific_fact,medium,Koľko otázok má vzniknúť pre každý odsek pri anotácii otázok?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,0.0,0.5,False,"[""5""]","[""https://zp.kemt.fei.tuke.sk/topics/question""]","[""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/topics/question/navod"", ""https://zp.kemt.fei.tuke.sk/topics/question/validacie"", ""https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/topics/question/navod
|
||||
https://zp.kemt.fei.tuke.sk/topics/question/validacie",2.076011,0.340891,18.827692,21.245177,21598,144,21742,
|
||||
q0632,dev,topic_page_lookup,easy,Nájdi projektovú alebo informačnú stránku „Strojový preklad“.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Strojový preklad""]","[""https://zp.kemt.fei.tuke.sk/topics/translation""]","[""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/otvorene"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy""]","Projektová alebo informačná stránka s názvom „Strojový preklad“ sa nachádza na adrese https://zp.kemt.fei.tuke.sk/topics/translation.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/translation",2.019691,0.228377,17.607882,19.856772,20168,135,20303,
|
||||
q0687,dev,multi_document,hard,"Nájdi viacero dokumentov, ktoré riešia RAG alebo GraphRAG.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""RAG"", ""GraphRAG""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2023/maksym_zatirka"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","V databáze ZP Wiki sa našlo viacero dokumentov zaoberajúcich sa témami RAG a GraphRAG, medzi ktoré patria diplomová práca Olega Poiasnika s názvom „Expertný agentový systém na podporu rozhodovania v lekárni"", ktorá využíva LightRAG a pracuje so znalostným grafom, či diplomová práca Mateja Ščišľaka s názvom „Znalostné grafy pre zlepšenie vyhľadávania právnych informácií"", kde bol vytvorený systém GraphRAG kombinujúci FAISS a Neo4J.
|
||||
|
||||
Ďalej sa tu nachádza diplomový projekt Jána Malinovského s názvom „Grafové vyhľadávanie pre podporu vzdelávania"", ktorý sa zameriava na naučenie a aplikáciu GraphRAG, diplomový projekt Jána Ptáka „Agent pre manažment záverečných prác"", ktorý si vyžaduje naučiť sa GraphRAG, ako aj bakalársku prácu Maksyma Zatiroku „Grafová databáza pre podporu generovania slovenského jazyka"", ktorá skúma využitie grafov pri RAG.
|
||||
|
||||
Medzi ostatné relevantné dokumenty patrí aj stážista Bogdan Paul Chiș, ktorý riešil tému extrakcie trojíc z medicínskych dát s využitím RAG a GraphRAG, a diplomový projekt Martina Šarišského, ktorý skúma, ako znalostný graf a Graph RAG dokážu vylepšiť vyhľadávanie v zákonoch.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/maksym_zatirka
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak
|
||||
https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",1.941236,0.388186,30.433983,32.764117,25987,597,26584,
|
||||
q0688,dev,multi_document,hard,"Nájdi dokumenty, ktoré spájajú znalostné grafy s RAG alebo inteligentnými agentmi.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""znalostný graf"", ""RAG"", ""agent""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis""]","[""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2023/maksym_zatirka"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2023/rostyslav_rodzhuk""]","V dostupných dokumentoch ZP Wiki sa nachádza viacero projektov, ktoré spájajú znalostné grafy s technológiami RAG (Retrieval Augmented Generation) alebo inteligentnými agentmi.
|
||||
|
||||
Diplomová práca Valeria Kutsenka (2027) sa zameriava na zlepšenie vyhľadávania pomocou znalostných grafov, kde agent buduje vnútornú reprezentáciu v podobe znalostného grafu a využíva systémy ako LightRAG a MiniRAG. Diplomová práca Mateja Ščišľaka (2026) s názvom „Znalostné grafy pre zlepšenie vyhľadávania právnych informácií"" implementuje systém GraphRAG využívajúci databázy Neo4J a FAISS na vyhľadávanie v právnych textoch a vytvorila aj agenta na báze Ollama. Diplomová práca Olega Poiasnika (2027) navrhuje expertný agentový systém pre lekáreň, ktorý využíva znalostný graf zostavený z databázy liekov (pomocou LightRAG a Neo4J) na podporu rozhodovania o interakciách a kontraindikáciách.
|
||||
|
||||
Ďalšie projekty zahŕňajú diplomový projekt Jána Malinovského (2026) „Grafové vyhľadávanie pre podporu vzdelávania"", ktorý vytvára agenta vyhľadávajúceho v znalostnom grafe vzdelávacích konceptov, a bakalársku prácu Maxyma Zatirku (2026) „Grafová databáza pre podporu generovania slovenského jazyka"", kde inteligentný agent využíva znalostný graf slovenského práva. Stážista Bogdan Paul Chiš pracoval na výskyte s cieľom využiť znalostný graf z viacjazyčných lekárskych príbalových letákov v inteligentnom agentovi na odporúčanie liekov. Projekt Jána Ptáka (2026) sa zaoberá agentom pre manažment záverečných prác s cieľom využiť znalostný graf pre vysvetliteľnosť modelu, a diplomový projekt Martina Šarišského skúma vplyv znalostných grafov na Graph RAG v oblasti právnych predpisov.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/maksym_zatirka
|
||||
https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",1.588763,0.531688,48.146049,50.267509,40940,779,41719,
|
||||
q0689,dev,multi_document,hard,Ktoré dokumenty sa venujú rozpoznávaniu pomenovaných entít alebo ich anotácii?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""pomenované entity"", ""NER""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod""]","[""https://zp.kemt.fei.tuke.sk/topics/named-entity"", ""https://zp.kemt.fei.tuke.sk/topics/prodigy"", ""https://zp.kemt.fei.tuke.sk/topics/spacy"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/topics/resources"", ""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/topics/named-entity/navod"", ""https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak""]","Dokumenty, ktoré sa venujú rozpoznávaniu pomenovaných entít alebo ich anotácii, zahŕňajú témy Pomenované entity a Anotácia textových dát, ako aj študentské práce, ako je diplomová práca Jakuba Maruniaka s názvom Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku, bakalárska práca Artema Yatsenka s názvom Rozpoznávanie pomenovaných entít v slovenskom jazyku, bakalársky projekt Than Trung Thana s názvom Rozpoznávanie pomenovaných entít v slovenskom jazyku pomocou nástrojov Spacy a Prodigy a bakalárska práca Vladimíra Ferku s názvom Anotácia a vyhodnotenie slovenskej databázy nenávistnej reči, ktorá zahŕňa anotáciu podobnú rozpoznávaniu pomenovaných entít.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/topics/named-entity
|
||||
https://zp.kemt.fei.tuke.sk/topics/prodigy
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko
|
||||
https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko",2.12482,0.4767,34.991346,37.593919,33996,427,34423,
|
||||
q0690,dev,multi_document,hard,Nájdi dokumenty súvisiace so strojovým alebo automatickým prekladom.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""strojový preklad"", ""automatický preklad""]","[""..."", ""..."", ""..."", ""..."", ""...""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko"", ""https://zp.kemt.fei.tuke.sk/students/2021/nikita_bodnar"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"", ""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic""]","Dokumenty súvisiace so strojovým alebo automatickým prekladom zahŕňajú bakalársku prácu Patrika Pavlišina s názvom „Metódy automatického prekladu"" a jeho diplomovú prácu s predbežným názvom „Neurónový strojový preklad"", ako aj bakalársku prácu Martina Jancuru „Webová aplikácia pre demonštráciu strojového prekladu"". Ďalej je tu diplomová práca Tomáša Kuchára s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"", bakalárska práca Vladyslava Krupka, ktorá sa zaoberá otázkou, ako funguje neurónový strojový preklad, a bakalárska práca Nikity Bodnar, ktorá tiež skúma neurónový strojový preklad v kontexte korekcie textu. Ďalej Dominik Nagy vypracoval diplomovú prácu „Neurónový strojový preklad pomocou knižnice Fairseq"" a Kristián Sopkovič pracoval na diplomovom projekte využívajúcom strojovo preložené databázy pre cross-lingual information retrieval. Maroš Harahus sa v rámci doktorandského štúdia a dizertačnej práce zaoberal aj experimentmi so strojovým prekladom a nástrojom fairseq.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin
|
||||
https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik
|
||||
https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/nikita_bodnar
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.876814,0.636482,52.696152,55.210624,49141,582,49723,
|
||||
|
File diff suppressed because it is too large
Load Diff
889
tools/rag/evaluation/results/full_dev_1148_dataset/failures.txt
Normal file
889
tools/rag/evaluation/results/full_dev_1148_dataset/failures.txt
Normal file
@ -0,0 +1,889 @@
|
||||
FAILS: 46
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0004
|
||||
CATEGORY: topic_lookup
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Nájdi študenta alebo dokument, ktorý rieši tému „Systém získavania informácií v slovenskom jazyku“.
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa táto presná téma „Systém získavania informácií v slovenskom jazyku" nenašla pod týmto názvom, no nájdené sú študentské práce s tematikou agentov a systémov pre získavanie informácií alebo vyhľadávanie v slovenskom jazyku. Napríklad Kristián Sopkovič má v rámci dizertácie (2028) návrh témy „Agentový prístup k získavaniu informácií", kde agent iniciatívne získava potrebné informácie, a v rámci diplomovej práce (2024) pracoval na „cross-lingual information retrieval" (kroslinguálne vyhľadávanie), vrátane modelov pre slovenský jazyk. Ďalším príkladom je Dávid Stromp, ktorý mal v diplomovke cieľ „Zlepšiť proces získavania informácií" (2022), a Oleh Poiasnik, ktorý riešil „Generovanie jazyka s podporou vyhľadávania v doménovo orientovaných dátach" (2025), vrátane vyhľadávania v slovenských zákonoch.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/david_stromp
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik
|
||||
EXPECTED: ['Ján Holp', 'Systém získavania informácií v slovenskom jazyku']
|
||||
ANSWER MATCHES: [False, True]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jan_holp']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0026
|
||||
CATEGORY: specific_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Aké metódy mal Maroš Harahus študovať v súvislosti s US Steel?
|
||||
ANSWER: Maroš Harahus mal pri spolupráci s US Steel za úlohu naštudovať dáta z vysokej pece a následne navrhnúť a vyskúšať metódy na predikciu, vrátane rekurentných neurónových sietí (RNN, GRU, LSTM), Transformer network, Generative Adversarial Network (GAN), variáneho autoenkódera (VAE) a klasických štatistických modelov, ako aj metódy konverzie dát na vektor príznakov a výpočtu chybovej funkcie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus
|
||||
EXPECTED: ['grafové neurónové siete']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0040
|
||||
CATEGORY: reverse_lookup
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Kto sa venuje práci s názvom „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku“?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť záverečnú prácu s presným názvom „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku" a jej autora. Hoci sa v záznamoch objavuje téma rozpoznávania pomenovaných entít v slovenčine pri študentoch Martin Wencel (bakalárska práca 2020) a Serhii Yemets (bakalárska práca 2025), žiadny z nich nemá uvedený tento konkrétny názov práce.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2018/martin_wencel
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/topics/otvorene
|
||||
EXPECTED: ['Jakub Maruniak']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0041
|
||||
CATEGORY: topic_lookup
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Nájdi študenta alebo dokument, ktorý rieši tému „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku“.
|
||||
ANSWER: Na tému „Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku" riešil študent Than Trung Thanh v rámci svojho bakalárskeho projektu v roku 2020, pri ktorom vytváral korpus a model pomocou nástrojov Spacy a Prodigy. Ďalšími študentmi, ktorí sa touto problematikou zaoberali, sú Martin Wencel (bakalárska práca 2020 o demonštrácii spracovania jazyka vrátane rozpoznávania entít) a Serhii Yemets (bakalárska práca 2025 o zlepšení slovenského modelu pre rozpoznávanie pomenovaných entít).
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh
|
||||
https://zp.kemt.fei.tuke.sk/students/2018/martin_wencel
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/serhii_yemets
|
||||
EXPECTED: ['Jakub Maruniak', 'Anotácia a rozpoznávanie pomenovaných entít v slovenskom jazyku']
|
||||
ANSWER MATCHES: [False, True]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0173
|
||||
CATEGORY: specific_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Aký bol hlavný cieľ bakalárskej práce Kristiána Sopkoviča?
|
||||
ANSWER: Hlavným cieľom bakalárskej práce Kristiána Sopkoviča bolo zlepšiť presnosť modelu Spacy pre spracovanie slovenčine, čo zahŕňalo natrénovanie modelu, vykonanie experimentov a identifikáciu slabých miest na jeho následné vylepšenie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic
|
||||
EXPECTED: ['Zlepšiť presnosť modelu Spacy pre slovenčinu']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0253
|
||||
CATEGORY: disambiguation_year
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Rozlíš rok začiatku štúdia a rok práce osoby Vladimír Ferko.
|
||||
ANSWER: Z dostupných dokumentov sa nepodarilo nájsť rok, v ktorom Vladimír Ferko začal pracovať, je však uvedené, že rok začiatku štúdia bol 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko
|
||||
EXPECTED: ['2021', '2024']
|
||||
ANSWER MATCHES: [True, False]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0336
|
||||
CATEGORY: topic_lookup
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Nájdi študenta alebo dokument, ktorý rieši tému „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov“.
|
||||
ANSWER: Podľa dostupných informácií v ZP Wiki sa téma „Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov" (alebo podobne sformulovaná ako „Rozpoznávanie nenávistnej reči pomocou transfomers") objavila ako jeden z nápadov na tému diplomovej práce študenta Martina Jancuru, pričom jeho finálna diplomová práca bola pomenovaná „Rozpoznávanie emócií v texte" a zameriavala sa na sentiment a emócie v texte.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura
|
||||
EXPECTED: ['Tetiana Mohorian', 'Rozpoznávanie nenávistnej reči pomocou veľkých jazykových modelov']
|
||||
ANSWER MATCHES: [False, True]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0393
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Akú databázu a backend už Ján Pták podľa stavu používal?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o konkrétnej databáze alebo backendovom technologickom riešení, ktoré Ján Pták už používal. Dokument o jeho diplomovom projekte z roku 2026 s témou „Agent pre manažment záverečných prác" uvádza cieľ nasadiť riešenie pomocou Dockeru a rozhrania OpenWebUI, no neobsahuje explicitné uvedenie konkrétnej databázy ani backendového jazyka či frameworku, ktorý by už bol v stave projektu implementovaný.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0494
|
||||
CATEGORY: specific_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Aký problém mali malé modely pri práci Simony Bobrovčanovej?
|
||||
ANSWER: Pri práci Simony Bobrovčanovej, konkrétne pri jej bakalárskej práci zamerané na automatickú sumarizáciu slovenských právnych textov, mali malé modely problém s príliš krátkou dĺžkou kontextu, vďaka čomu často vynechávali informácie nachádzajúce sa na konci textu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova
|
||||
EXPECTED: ['príliš maly kontext']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0556
|
||||
CATEGORY: specific_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Na akých dvoch MTEB úlohách mal Oliver Pejic pracovať?
|
||||
ANSWER: Oliver Pejic mal počas svojej stáže pracovať na vytvorení MTEB vyhodnocovacích úloh pre slovenskú detekciu nenávistných prejavov a pre slovenské vyhľadávanie odpovedí na otázky (retrieval).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/oliver_pejic
|
||||
EXPECTED: ['Slovak HATE speech', 'Slovak question answering']
|
||||
ANSWER MATCHES: [False, False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/interns/oliver_pejic']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0570
|
||||
CATEGORY: internship_topic
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Akému problému alebo téme sa venovala stáž osoby Cesar Abascal Gutierrez?
|
||||
ANSWER: Stáž osoby Cesar Abascal Gutierrez sa venovala problému rozpoznávania neznámych entít v prenesených transkriptoch reči, pričom cieľom bolo vytvoriť manuálne anotovanú trénovaciu sadu a navrhnúť schému pre anotáciu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez
|
||||
EXPECTED: ['anotácia pomenovaných entít']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0575
|
||||
CATEGORY: specific_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Aké tri hlavné ciele mal Cesar Abascal Gutierrez?
|
||||
ANSWER: Cesar Abascal Gutierrez mal počas svojej stáže v lete 2019 stanovené tieto tri hlavné ciele: byť schopný rozpoznať neznáme menované entity, vytvoriť manuálne anotovanú trénovaciu množinu z prepisov reči a navrhnúť schému anotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez
|
||||
EXPECTED: ['recognize unknown named entities', 'manually annotated training set', 'annotation schema']
|
||||
ANSWER MATCHES: [False, True, True]
|
||||
ANSWER SCORE: 0.6666666666666666
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0579
|
||||
CATEGORY: internship_topic
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Akému problému alebo téme sa venovala stáž osoby Bogdan Paul Chiș?
|
||||
ANSWER: Stáž osoby Bogdan Paul Chiš sa venovala téme extrakcie trojíc z lekárskych dát v mnohých jazykoch s cieľom zostaviť znalostný graf z balíčkových letákov liekov a využiť ho v inteligentnom agentovi odporúčajúcom lieky.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis
|
||||
EXPECTED: ['multilinguálna extrakcia trojíc z medicínskych dát']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0584
|
||||
CATEGORY: specific_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Aká bola téma stáže Bogdana Paula Chișa?
|
||||
ANSWER: Témai Bogdana Paula Chișa počas jeho stáže bolo (multilingválne) extrahovanie trojíc z lekárskeho obsahu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis
|
||||
EXPECTED: ['Triplet extraction from medical data']
|
||||
ANSWER MATCHES: [False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/interns/bogdan_paul_chis']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0633
|
||||
CATEGORY: topic_page_lookup
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Ktorý dokument v ZP Wiki sa priamo venuje téme „Strojový preklad“?
|
||||
ANSWER: Dokument s názvom „Strojový preklad", ktorý sa priamo a podrobne venuje tejto téme, je študentská práca Patrika Pavlišina, kde sú rozebrané prístupy ako neurónový strojový preklad, architektúra Encoder-Decoder a modely ako BERT.
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/tp20
|
||||
EXPECTED: ['Strojový preklad']
|
||||
ANSWER MATCHES: [True]
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/topics/translation']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0827
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko GB RAM mal počítač alebo server, ktorý Jakub Maruniak používal pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa informácia o počte GB RAM počítača alebo servera, ktorý Jakub Maruniak používal pri práci, nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0829
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Jakub Maruniak pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o konkrétnom modeli GPU, ktorý použil Jakub Maruniak pri práci. Zdrojové dokumenty spomínajú použitie knižnice Spacy a nástroja Prodigy, ako aj vykonávanie experimentov s trénovaním modelu, ale nezisťuje sa v nich špecifikácia hardvéru, konkrétne modelu grafickej karty.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jakub_maruniak']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0832
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Dominik Nagy pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o konkrétnom modeli GPU, ktorý Dominik Nagy pri práci použil; v záznamoch je uvedené len, že prebiehalo trénovanie na GPU bez použitia balíka Anaconda, ale bez špecifikácie typu hardvéru.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0834
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko konkrétnych používateľov testovalo výsledný systém osoby Patrik Pavlišin?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o počte konkrétnych používateľov, ktorí testovali výsledný systém patriaci osobe Patrik Pavlišin.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0838
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Dávid Omasta pri práci?
|
||||
ANSWER: Z dostupných dokumentov ZP Wiki nie je uvedený konkrétny model GPU, ktorý Dávid Omasta použil, iba sa spomína, že experimenty s modelmi LLAMA a mt5-base sa nedarili realizovať kvôli obmedzeniam GPU.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2018/david_omasta
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2018/david_omasta']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0841
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Kristián Sopkovič pri práci?
|
||||
ANSWER: Z dostupných dokumentov ZP Wiki sa nepodarilo nájsť informáciu o konkrétnom modeli GPU, ktorý Kristián Sopkovič pri práci používal. Zdroje síce spomínajú, že pri trénovaní modelov pracoval s hardvérom s 24 GB pamäte (v kontexte 1x24 GB Titan) a nasadzoval modely na Titan, ale nezverejňujú presný model GPU.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0843
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko konkrétnych používateľov testovalo výsledný systém osoby Michal Stromko?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o počte konkrétnych používateľov, ktorí testovali výsledný systém Michala Stromka.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0844
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Michal Stromko pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o konkrétnom modeli GPU, ktorý Michal Stromko pri svojej práci použil. Zdroje uvádzajú používanie softvérových modelov a nástrojov, ako sú mnlr, MPnet, LABSE, SlovakBERT, FAISS a JinaDocArray, ale neobsahujú špecifikáciu hardvéru ani konkrétny typ grafickej karty.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2019/michal_stromko']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0845
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko GB RAM mal počítač alebo server, ktorý Artem Yatsenko používal pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: True
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0847
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Artem Yatsenko pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa informáciu o konkrétnom modeli GPU, ktorý Artem Yatsenko pri práci využíval, nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2019/artem_yatsenko']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0848
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko GB RAM mal počítač alebo server, ktorý Matej Ščišľak používal pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak']
|
||||
SOURCE MATCHES: [False]
|
||||
SOURCE SCORE: 0.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: True
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0850
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Matej Ščišľak pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o konkrétnom modeli GPU, ktorý Matej Ščišľak pri práci použil.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0852
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko konkrétnych používateľov testovalo výsledný systém osoby Oleh Poiasnik?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o presnom počte konkrétnych používateľov, ktorí testovali výsledný systém od osoby Oleh Poiasnik. Dokumenty síce obsahujú úlohy osloviť kolegov na testovanie a slovne vyhodnotiť systém, ale neuvádzajú konkrétny počet účastníkov testovania.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0853
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Oleh Poiasnik pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o konkrétnom modeli GPU, ktorý Oleh Poiasnik použil pri práci. Zdroje uvádzajú len použitie softvérových nástrojov (ako vllm, ollama) a modelov (Mistral, Qwen2.5, MiniLM), ale nie hardvéru GPU.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0854
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko GB RAM mal počítač alebo server, ktorý Tetiana Mohorian používal pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o tom, koľko GB RAM mal počítač alebo server, ktorý Tetiana Mohorian používala pri práci. Zdroj uvádza len, že ak by jej GPU počítača nestačilo, vedúci jej pridelí prístup na školský server alebo môže vyskúšať Google Colab, konkrétne hodnoty pamäte však nie sú uvedené.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0856
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Tetiana Mohorian pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o konkrétnom modeli GPU, ktorý Tetiana Mohorian použila pri práci. Dokument uvádza len všeobecnú možnosť, že ak jej GPU vlastného počítača nebude stačiť, môže získať prístup na školský server alebo využiť Google Colab.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/tetiana_mohorian']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0864
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko konkrétnych používateľov testovalo výsledný systém osoby Samuel Džurina?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o presnom počte konkrétnych používateľov, ktorí testovali výsledný systém Samuela Džurinu. Dokumenty uvádzajú, že agenta je potrebné otestovať a vyhodnotiť formou dotazníka a že existujúci chatbot je použiteľný, ale nešpecifikujú počet testujúcich.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0865
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Samuel Džurina pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o konkrétnom modeli GPU, ktorý Samuel Džurina používal pri práci; v zázname je síce spomenutá otázka o tom, či firma disponuje hardvérom s GPU, no odpoveď uvádza, že je možné riešenie externe a nie je špecifikovaný žiadny konkrétny model.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0870
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Valerii Kutsenko pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože v uvedených záznamoch o Valerii Kutsenkovi sa síce spomínajú modely (napr. Slovak T5, mT5, Slovak Mistral, Slovak Falcon) a nástroje (Ollama), ale nie je explicitne uvedený konkrétny model GPU hardvéru, ktorý pri práci používal.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0873
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny model GPU používal Andrii Pervashov pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože hoci je v zázname o Andrii Pervashovi uvedené, že pri trénovaní modelu sa stretol s problémami s pamäťou GPU a pracoval s obmedzením 128 znakov kvôli dostupnej pamäti, konkrétny model GPU nie je v zdrojoch explicitne uvedený.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0874
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko GB RAM mal počítač alebo server, ktorý Ján Pták používal pri práci?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo nájsť informáciu o tom, koľko GB RAM mal počítač alebo server, ktorý Ján Pták používal pri práci.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0879
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Koľko strán mala finálna záverečná práca osoby Ján Holp?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o počte strán finálnej záverečnej práce osoby Ján Holp.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jan_holp']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0882
|
||||
CATEGORY: unanswerable_missing_fact
|
||||
DIFFICULTY: medium
|
||||
SHOULD ANSWER: False
|
||||
QUESTION: Aký konkrétny finančný rozpočet bol uvedený pre projekt osoby Ján Holp?
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa informáciu o konkrétnom finančnom rozpočte pre projekt Ján Holpa nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp
|
||||
EXPECTED: []
|
||||
ANSWER MATCHES: []
|
||||
ANSWER SCORE: 1.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/jan_holp']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: False
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0968
|
||||
CATEGORY: single_document_synthesis
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Pri osobe Ján Pták uveď názov záznamu typu „diplomový projekt“ a rok, pri ktorom je táto práca na ZP Wiki uvedená.
|
||||
ANSWER: Ján Pták vypracováva diplomový projekt s názvom „Agent pre manažment záverečných prác“ a rok začatia štúdia pre túto prácu je 2022.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak
|
||||
EXPECTED: ['Agent pre manažment záverečných prác', '2026']
|
||||
ANSWER MATCHES: [True, False]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0982
|
||||
CATEGORY: single_document_synthesis
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok, pri ktorom je na ZP Wiki uvedený záznam typu „diplomová práca“.
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť rok začiatku štúdia pre osobu Dárius Lindvai, hoci je uvedený záznam o diplomovom projekte (často používanom ako súčasť diplomovej práce) v roku 2020.
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
EXPECTED: ['2016', '2021']
|
||||
ANSWER MATCHES: [True, False]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q0990
|
||||
CATEGORY: single_document_synthesis
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: V dokumente osoby Tomáš Kuchárik, ktorý súvisí s témou „Question Answering“, uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca uvedená.
|
||||
ANSWER: Tomáš Kuchárik má zaznamenanú diplomovú prácu s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou crowdsourcingu" z roku 2021, ktorá sa zameriava na tému Question Answering.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik
|
||||
EXPECTED: ['Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu', '2022']
|
||||
ANSWER MATCHES: [False, False]
|
||||
ANSWER SCORE: 0.0
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q1022
|
||||
CATEGORY: single_document_synthesis
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: V dokumente osoby Dárius Lindvai, ktorý súvisí s témou „interpunkcia“, uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca uvedená.
|
||||
ANSWER: Dárius Lindvai vypracoval diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí“ v roku 2020.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
EXPECTED: ['Obnovenie interpunkcie pomocou hlbokých neurónových sietí', '2021']
|
||||
ANSWER MATCHES: [True, False]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai']
|
||||
SOURCE MATCHES: [True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q1092
|
||||
CATEGORY: multi_document
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri osobe Dárius Lindvai použi záznam typu „diplomová práca“ a pri osobe Ján Pták záznam typu „diplomový projekt“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.
|
||||
ANSWER: Z dostupných záznamov na ZP Wiki sa Dárius Lindvai uvádza s diplomovou prácou (konkrétne v kontexte diplomového projektu z roku 2020) s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí", zatiaľ čo pri Jánovi Ptákovi sa v zázname typu „diplomový projekt" uvádza téma „Agent pre manažment záverečných prác" pre rok 2026, no explicitný finálny názov práce nie je v poskytnutom kontexte uvedený.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak
|
||||
EXPECTED: ['Obnovenie interpunkcie pomocou hlbokých neurónových sietí', '2021', 'Agent pre manažment záverečných prác', '2026']
|
||||
ANSWER MATCHES: [True, False, True, True]
|
||||
ANSWER SCORE: 0.75
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai', 'https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak']
|
||||
SOURCE MATCHES: [True, True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q1109
|
||||
CATEGORY: multi_document
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.
|
||||
ANSWER: V dostupných dokumentoch ZP Wiki sa pre Martina Šarišského pri diplomovej práci z roku 2026 nenachádza uvedený názov práce, pretože sekcia „Názov práce“ je prázdna a obsahuje len odkaz na repozitár, zatiaľ čo pre Andriiho Pervashova je pri diplomovej práci z roku 2027 uvedený návrh na tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu“, pričom konkrétne pole „Názov“ nie je explicitne vyplnené.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
EXPECTED: ['Agentové systémy pomocou veľkého jazykového modelu', '2026', 'Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu', '2027']
|
||||
ANSWER MATCHES: [False, True, True, True]
|
||||
ANSWER SCORE: 0.75
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky', 'https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov']
|
||||
SOURCE MATCHES: [True, True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q1125
|
||||
CATEGORY: multi_document
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri osobe Dárius Lindvai použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.
|
||||
ANSWER: Dárius Lindvai vypracovával diplomový projekt s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí“ v roku 2020 a Andrii Pervashov má na ZP Wiki uvedený návrh na tému diplomovej práce s názvom „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu“ pre rok 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov
|
||||
EXPECTED: ['Obnovenie interpunkcie pomocou hlbokých neurónových sietí', '2021', 'Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu', '2027']
|
||||
ANSWER MATCHES: [True, False, True, True]
|
||||
ANSWER SCORE: 0.75
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai', 'https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov']
|
||||
SOURCE MATCHES: [True, True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
|
||||
==============================================================================================================
|
||||
ID: q1136
|
||||
CATEGORY: multi_document
|
||||
DIFFICULTY: hard
|
||||
SHOULD ANSWER: True
|
||||
QUESTION: Porovnaj záznamy osôb Ján Malinovský a Yevhenii Medushivskyi. Pri osobe Ján Malinovský použi záznam typu „diplomový projekt“ a pri osobe Yevhenii Medushivskyi záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.
|
||||
ANSWER: Ján Malinovský vypracováva diplomový projekt s názvom „Grafové vyhľadávanie pre podporu vzdelávania" a v ZP Wiki je uvedený v roku 2022, kedy študent začal štúdium. Yevhenii Medushivskyi vypracúva bakalársku prácu s názvom „Získavanie textu z webových stránok" a v ZP Wiki je uvedený v roku 2023, kedy študent začal štúdium.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi
|
||||
EXPECTED: ['Grafové vyhľadávanie pre podporu vzdelávania', '2026', 'Získavanie textu z webových stránok', '2026']
|
||||
ANSWER MATCHES: [True, False, True, False]
|
||||
ANSWER SCORE: 0.5
|
||||
EXPECTED SOURCES: ['https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky', 'https://zp.kemt.fei.tuke.sk/students/2023/yevhenii_medushivskyi']
|
||||
SOURCE MATCHES: [True, True]
|
||||
SOURCE SCORE: 1.0
|
||||
SHOULD ANSWER OK: True
|
||||
RETURNED NO ANSWER: False
|
||||
STRICT: False
|
||||
File diff suppressed because it is too large
Load Diff
48542
tools/rag/evaluation/results/full_dev_1148_dataset/rag_answers_dev.json
Normal file
48542
tools/rag/evaluation/results/full_dev_1148_dataset/rag_answers_dev.json
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
1156
tools/rag/evaluation/results/full_dev_review/rag_answers_dev.csv
Normal file
1156
tools/rag/evaluation/results/full_dev_review/rag_answers_dev.csv
Normal file
File diff suppressed because it is too large
Load Diff
27377
tools/rag/evaluation/results/full_dev_review/rag_answers_dev.json
Normal file
27377
tools/rag/evaluation/results/full_dev_review/rag_answers_dev.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,25 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0393,dev,specific_fact,medium,Akú databázu a backend už Ján Pták podľa stavu používal?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,1.0,0.75,False,"[""SQLite"", ""FastAPI""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o tom, akú konkrétnu databázu a backend Ján Pták už používal, keďže dostupný záznam o jeho diplomovom projekte s témou „Agent pre manažment záverečných prác"" obsahuje iba ciele na semester (napr. využitie OpenWebUI, Docker, GraphRAG) a úlohy, ale neuvádza použité technológie v štádiu, kde by boli implementované.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.956512,0.261738,22.794016,25.012961,26940,209,27149,
|
||||
q0575,dev,specific_fact,medium,Aké tri hlavné ciele mal Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""recognize unknown named entities"", ""manually annotated training set"", ""annotation schema""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"", ""https://zp.kemt.fei.tuke.sk/students/2020/pavol_hudak"", ""https://zp.kemt.fei.tuke.sk/students/2023/ronald_zalacko"", ""https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2022/samuel_vasil""]","Cesar Abascal Gutierrez mal počas svojej stáže v roku 2019 tri hlavné ciele: byť schopný rozpoznávať neznáme pomenované entity, manuálne vytvoriť trénovacie množiny z transkriptov reči a navrhnúť schému pre anotáciu.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.459015,0.348397,27.967248,29.775782,34053,143,34196,
|
||||
q0690,dev,multi_document,hard,"Pri osobách Dominik Nagy, Patrik Pavlišin a Tomáš Kuchárik uveď názvy prác, ktoré súvisia so strojovým alebo automatickým prekladom.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Neurónový strojový preklad pomocou knižnice Fairseq"", ""Metódy automatického prekladu"", ""Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu""]","[""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/question""]","Dominik Nagy vypracoval diplomovú prácu s názvom „Neurónový strojový preklad pomocou knižnice Fairseq"", Patrik Pavlišin má bakalársku prácu s názvom „Metódy automatického prekladu"" a Tomáš Kuchárik vypracoval diplomovú prácu s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"".
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",2.193351,0.593168,21.867701,24.654982,24629,271,24900,
|
||||
q0942,dev,single_document_synthesis,hard,"Pri osobe Dárius Lindvai uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","Dárius Lindvai má na ZP Wiki uvedenú diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ktorá je sprevádzaná sekciou pre rok 2020.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.795557,0.532373,22.129303,24.457964,26502,151,26653,
|
||||
q1094,dev,multi_document,hard,"Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin. Pri osobe Maroš Harahus použi záznam typu „diplomová práca“ a pri osobe Patrik Pavlišin záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Neurónová morfologická anotácia slovenského jazyka"", ""2021"", ""Metódy automatického prekladu"", ""2020""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin""]","[""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2019/david_stromp"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","Maroš Harahus vypracoval diplomovú prácu s názvom „Neurónová morfologická anotácia slovenského jazyka“ v roku 2021 a Patrik Pavlišin vypracoval bakalársku prácu s názvom „Metódy automatického prekladu“ v roku 2020.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",2.16103,0.854158,23.37353,26.389377,27225,211,27436,
|
||||
q1095,dev,multi_document,hard,"Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh. Pri osobe Filip Tomáš použi záznam typu „bakalárska práca“ a pri osobe Denis Le Thanh záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia"", ""2022"", ""Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh""]","[""https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Filip Tomáš vypracoval bakalársku prácu s názvom „Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia"" v roku 2022 a Denis Le Thanh má záznam o bakalárskej práci s témou „Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku"" uvedený pre rok 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",1.737561,0.509305,21.999032,24.246729,25104,211,25315,
|
||||
|
@ -0,0 +1,631 @@
|
||||
{
|
||||
"generated_at": "2026-09-27T13:04:55.299626+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "bcbf187780279264f5dcf6255c842547e2bc9e7b0f93cfc31f9695668547dbde",
|
||||
"selected_question_ids": [
|
||||
"q0393",
|
||||
"q0575",
|
||||
"q0690",
|
||||
"q0942",
|
||||
"q1094",
|
||||
"q1095"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.75,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.9375,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 25.756299,
|
||||
"prompt_tokens": 164453,
|
||||
"completion_tokens": 1196,
|
||||
"total_tokens": 165649,
|
||||
"by_category": {
|
||||
"multi_document": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 25.097029,
|
||||
"prompt_tokens": 76958,
|
||||
"completion_tokens": 693,
|
||||
"total_tokens": 77651
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 0,
|
||||
"strict_pass_rate": 0.0,
|
||||
"mean_latency_seconds": 24.457964,
|
||||
"prompt_tokens": 26502,
|
||||
"completion_tokens": 151,
|
||||
"total_tokens": 26653
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 27.394371,
|
||||
"prompt_tokens": 60993,
|
||||
"completion_tokens": 352,
|
||||
"total_tokens": 61345
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.875,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.96875,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.75,
|
||||
"mean_latency_seconds": 24.937263,
|
||||
"prompt_tokens": 103460,
|
||||
"completion_tokens": 844,
|
||||
"total_tokens": 104304
|
||||
},
|
||||
"medium": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.5,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 0.5,
|
||||
"mean_latency_seconds": 27.394371,
|
||||
"prompt_tokens": 60993,
|
||||
"completion_tokens": 352,
|
||||
"total_tokens": 61345
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0393",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"SQLite",
|
||||
"FastAPI"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa nepodarilo spoľahlivo nájsť informáciu o tom, akú konkrétnu databázu a backend Ján Pták už používal, keďže dostupný záznam o jeho diplomovom projekte s témou „Agent pre manažment záverečných prác\" obsahuje iba ciele na semester (napr. využitie OpenWebUI, Docker, GraphRAG) a úlohy, ale neuvádza použité technológie v štádiu, kde by boli implementované.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"limit": 5,
|
||||
"published_only": false
|
||||
},
|
||||
"latency_seconds": 0.261738,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.956512,
|
||||
"tool_latency_seconds": 0.261738,
|
||||
"final_model_latency_seconds": 22.794016,
|
||||
"total_latency_seconds": 25.012961,
|
||||
"usage": {
|
||||
"prompt_tokens": 26940,
|
||||
"completion_tokens": 209,
|
||||
"total_tokens": 27149
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
false,
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.75,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0575",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/cesar_gutierrez/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"recognize unknown named entities",
|
||||
"manually annotated training set",
|
||||
"annotation schema"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Cesar Abascal Gutierrez mal počas svojej stáže v roku 2019 tri hlavné ciele: byť schopný rozpoznávať neznáme pomenované entity, manuálne vytvoriť trénovacie množiny z transkriptov reči a navrhnúť schému pre anotáciu.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"limit": 10
|
||||
},
|
||||
"latency_seconds": 0.348397,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/pavol_hudak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/ronald_zalacko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/samuel_vasil"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/pavol_hudak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/ronald_zalacko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/oleh_poiasnik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/samuel_vasil"
|
||||
],
|
||||
"first_model_latency_seconds": 1.459015,
|
||||
"tool_latency_seconds": 0.348397,
|
||||
"final_model_latency_seconds": 27.967248,
|
||||
"total_latency_seconds": 29.775782,
|
||||
"usage": {
|
||||
"prompt_tokens": 34053,
|
||||
"completion_tokens": 143,
|
||||
"total_tokens": 34196
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0690",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Pri osobách Dominik Nagy, Patrik Pavlišin a Tomáš Kuchárik uveď názvy prác, ktoré súvisia so strojovým alebo automatickým prekladom.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/dominik_nagy/README.md",
|
||||
"pages/students/2016/patrik_pavlisin/README.md",
|
||||
"pages/students/2016/tomas_kucharik/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Neurónový strojový preklad pomocou knižnice Fairseq",
|
||||
"Metódy automatického prekladu",
|
||||
"Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Opravené placeholder source URL na reálne ZP Wiki URL.",
|
||||
"answer": "Dominik Nagy vypracoval diplomovú prácu s názvom „Neurónový strojový preklad pomocou knižnice Fairseq\", Patrik Pavlišin má bakalársku prácu s názvom „Metódy automatického prekladu\" a Tomáš Kuchárik vypracoval diplomovú prácu s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu\".\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy\nhttps://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin\nhttps://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Pri osobách Dominik Nagy, Patrik Pavlišin a Tomáš Kuchárik uveď názvy prác, ktoré súvisia so strojovým alebo automatickým prekladom.",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 2
|
||||
},
|
||||
"latency_seconds": 0.593168,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/translation",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/translation",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question"
|
||||
],
|
||||
"first_model_latency_seconds": 2.193351,
|
||||
"tool_latency_seconds": 0.593168,
|
||||
"final_model_latency_seconds": 21.867701,
|
||||
"total_latency_seconds": 24.654982,
|
||||
"usage": {
|
||||
"prompt_tokens": 24629,
|
||||
"completion_tokens": 271,
|
||||
"total_tokens": 24900
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 3,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0942",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Pri osobe Dárius Lindvai uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Dárius Lindvai má na ZP Wiki uvedenú diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\", ktorá je sprevádzaná sekciou pre rok 2020.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Pri osobe Dárius Lindvai uveď názov záznamu typu „diplomová práca“ a rok, pri ktorom je táto práca na ZP Wiki uvedená.",
|
||||
"published_only": true,
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.532373,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.795557,
|
||||
"tool_latency_seconds": 0.532373,
|
||||
"final_model_latency_seconds": 22.129303,
|
||||
"total_latency_seconds": 24.457964,
|
||||
"usage": {
|
||||
"prompt_tokens": 26502,
|
||||
"completion_tokens": 151,
|
||||
"total_tokens": 26653
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
false
|
||||
],
|
||||
"answer_contains_score": 0.5,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.875,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1094",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin. Pri osobe Maroš Harahus použi záznam typu „diplomová práca“ a pri osobe Patrik Pavlišin záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md",
|
||||
"pages/students/2016/patrik_pavlisin/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Neurónová morfologická anotácia slovenského jazyka",
|
||||
"2021",
|
||||
"Metódy automatického prekladu",
|
||||
"2020"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Maroš Harahus vypracoval diplomovú prácu s názvom „Neurónová morfologická anotácia slovenského jazyka“ v roku 2021 a Patrik Pavlišin vypracoval bakalársku prácu s názvom „Metódy automatického prekladu“ v roku 2020.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin. Pri osobe Maroš Harahus použi záznam typu „diplomová práca“ a pri osobe Patrik Pavlišin záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.854158,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 2.16103,
|
||||
"tool_latency_seconds": 0.854158,
|
||||
"final_model_latency_seconds": 23.37353,
|
||||
"total_latency_seconds": 26.389377,
|
||||
"usage": {
|
||||
"prompt_tokens": 27225,
|
||||
"completion_tokens": 211,
|
||||
"total_tokens": 27436
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1095",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh. Pri osobe Filip Tomáš použi záznam typu „bakalárska práca“ a pri osobe Denis Le Thanh záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2019/filip_tomas/README.md",
|
||||
"pages/students/2023/denis_le_thanh/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia",
|
||||
"2022",
|
||||
"Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Filip Tomáš vypracoval bakalársku prácu s názvom „Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia\" v roku 2022 a Denis Le Thanh má záznam o bakalárskej práci s témou „Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku\" uvedený pre rok 2026.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/filip_tomas\nhttps://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh. Pri osobe Filip Tomáš použi záznam typu „bakalárska práca“ a pri osobe Denis Le Thanh záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.509305,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 1.737561,
|
||||
"tool_latency_seconds": 0.509305,
|
||||
"final_model_latency_seconds": 21.999032,
|
||||
"total_latency_seconds": 24.246729,
|
||||
"usage": {
|
||||
"prompt_tokens": 25104,
|
||||
"completion_tokens": 211,
|
||||
"total_tokens": 25315
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,26 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0393,dev,unanswerable_missing_fact,medium,Akú databázu a backend už Ján Pták podľa stavu používal?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,[],"[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.46631,0.232988,21.335165,23.035265,26914,87,27001,
|
||||
q0575,dev,specific_fact,medium,Aké tri hlavné ciele mal Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""recognize unknown named entities"", ""manually annotated training set"", ""annotation schema""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Cesar Abascal Gutierrez si stanovil tri hlavné ciele počas svojej stáže: byť schopný rozpoznávať neznáme pomenované entity, vytvoriť manuálne anotovanú trénovaciu sadu z prepisov reči a navrhnúť schému anotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.616364,0.10319,10.597929,12.318236,12063,152,12215,
|
||||
q0690,dev,multi_document,hard,"Pri osobách Dominik Nagy, Patrik Pavlišin a Tomáš Kuchárik uveď názvy prác, ktoré súvisia so strojovým alebo automatickým prekladom.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Neurónový strojový preklad pomocou knižnice Fairseq"", ""Metódy automatického prekladu"", ""Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu""]","[""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","[""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka/timovy_projekt"", ""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus/timovy_projekt"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","Dominik Nagy vypracoval diplomovú prácu s názvom „Neurónový strojový preklad pomocou knižnice Fairseq"", Patrik Pavlišin má bakalársku prácu s názvom „Metódy automatického prekladu"" a Tomáš Kuchárik vypracoval diplomovú prácu s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"".
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",2.095638,0.86741,34.819498,37.783597,40271,271,40542,
|
||||
q0942,dev,single_document_synthesis,hard,Pri osobe Dárius Lindvai uveď názov diplomovej práce patriacej do kategórie „dp2021“ a rok vyjadrený touto kategóriou.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/mark_feher"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" patriacu do kategórie dp2021, čo zodpovedá roku 2021.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.577307,0.850273,18.810862,21.239067,22122,144,22266,
|
||||
q1094,dev,multi_document,hard,"Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin. Pri osobe Maroš Harahus použi záznam typu „diplomová práca“ a pri osobe Patrik Pavlišin záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Neurónová morfologická anotácia slovenského jazyka"", ""2021"", ""Metódy automatického prekladu"", ""2020""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin""]","[""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/david_stromp"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela""]","Maroš Harahus vypracoval diplomovú prácu s názvom „Neurónová morfologická anotácia slovenského jazyka"" v roku 2021 a Patrik Pavlišin vypracoval bakalársku prácu s názvom „Metódy automatického prekladu"" v roku 2020.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",1.970918,1.251019,22.822001,26.044583,26504,208,26712,
|
||||
q1095,dev,multi_document,hard,"Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh. Pri osobe Filip Tomáš použi záznam typu „bakalárska práca“ a pri osobe Denis Le Thanh záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia"", ""2022"", ""Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh""]","[""https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Podľa záznamov na ZP Wiki má Filip Tomáš bakalársku prácu s názvom „Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia“ uvedenú v roku 2022, zatiaľ čo Denis Le Thanh má bakalársku prácu s témou „Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku“ uvedenú v roku 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",1.76637,0.505903,3.585043,5.858003,25104,220,25324,
|
||||
|
@ -0,0 +1,633 @@
|
||||
{
|
||||
"generated_at": "2026-09-27T13:15:55.674619+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "d2e82c40a2d657b58e9c8e6f983cfb7505db01f4d1decf934e5b20ad661428ba",
|
||||
"selected_question_ids": [
|
||||
"q0393",
|
||||
"q0575",
|
||||
"q0690",
|
||||
"q0942",
|
||||
"q1094",
|
||||
"q1095"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 6,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 6,
|
||||
"completed": 6,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 6,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 21.046458,
|
||||
"prompt_tokens": 152978,
|
||||
"completion_tokens": 1082,
|
||||
"total_tokens": 154060,
|
||||
"by_category": {
|
||||
"multi_document": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 23.228728,
|
||||
"prompt_tokens": 91879,
|
||||
"completion_tokens": 699,
|
||||
"total_tokens": 92578
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 21.239067,
|
||||
"prompt_tokens": 22122,
|
||||
"completion_tokens": 144,
|
||||
"total_tokens": 22266
|
||||
},
|
||||
"specific_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 12.318236,
|
||||
"prompt_tokens": 12063,
|
||||
"completion_tokens": 152,
|
||||
"total_tokens": 12215
|
||||
},
|
||||
"unanswerable_missing_fact": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 23.035265,
|
||||
"prompt_tokens": 26914,
|
||||
"completion_tokens": 87,
|
||||
"total_tokens": 27001
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 22.731313,
|
||||
"prompt_tokens": 114001,
|
||||
"completion_tokens": 843,
|
||||
"total_tokens": 114844
|
||||
},
|
||||
"medium": {
|
||||
"total": 2,
|
||||
"completed": 2,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 17.676751,
|
||||
"prompt_tokens": 38977,
|
||||
"completion_tokens": 239,
|
||||
"total_tokens": 39216
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0393",
|
||||
"split": "dev",
|
||||
"category": "unanswerable_missing_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Akú databázu a backend už Ján Pták podľa stavu používal?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [],
|
||||
"should_answer": false,
|
||||
"note": "Zdrojová stránka Jána Ptáka neuvádza, akú konkrétnu databázu a backend už používal. SQLite a FastAPI preto nemožno zo ZP Wiki spoľahlivo odvodiť.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Akú databázu a backend už Ján Pták podľa stavu používal?"
|
||||
},
|
||||
"latency_seconds": 0.232988,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 1.46631,
|
||||
"tool_latency_seconds": 0.232988,
|
||||
"final_model_latency_seconds": 21.335165,
|
||||
"total_latency_seconds": 23.035265,
|
||||
"usage": {
|
||||
"prompt_tokens": 26914,
|
||||
"completion_tokens": 87,
|
||||
"total_tokens": 27001
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": true,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0575",
|
||||
"split": "dev",
|
||||
"category": "specific_fact",
|
||||
"difficulty": "medium",
|
||||
"question": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/interns/cesar_gutierrez/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"recognize unknown named entities",
|
||||
"manually annotated training set",
|
||||
"annotation schema"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": null,
|
||||
"answer": "Cesar Abascal Gutierrez si stanovil tri hlavné ciele počas svojej stáže: byť schopný rozpoznávať neznáme pomenované entity, vytvoriť manuálne anotovanú trénovaciu sadu z prepisov reči a navrhnúť schému anotácie.\n\nZdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Aké tri hlavné ciele mal Cesar Abascal Gutierrez?",
|
||||
"limit": 5,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.10319,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"
|
||||
],
|
||||
"first_model_latency_seconds": 1.616364,
|
||||
"tool_latency_seconds": 0.10319,
|
||||
"final_model_latency_seconds": 10.597929,
|
||||
"total_latency_seconds": 12.318236,
|
||||
"usage": {
|
||||
"prompt_tokens": 12063,
|
||||
"completion_tokens": 152,
|
||||
"total_tokens": 12215
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0690",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Pri osobách Dominik Nagy, Patrik Pavlišin a Tomáš Kuchárik uveď názvy prác, ktoré súvisia so strojovým alebo automatickým prekladom.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/dominik_nagy/README.md",
|
||||
"pages/students/2016/patrik_pavlisin/README.md",
|
||||
"pages/students/2016/tomas_kucharik/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Neurónový strojový preklad pomocou knižnice Fairseq",
|
||||
"Metódy automatického prekladu",
|
||||
"Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Opravené placeholder source URL na reálne ZP Wiki URL.",
|
||||
"answer": "Dominik Nagy vypracoval diplomovú prácu s názvom „Neurónový strojový preklad pomocou knižnice Fairseq\", Patrik Pavlišin má bakalársku prácu s názvom „Metódy automatického prekladu\" a Tomáš Kuchárik vypracoval diplomovú prácu s názvom „Tvorba korpusu otázok a odpovedí v slovenskom jazyku pomocou strojového prekladu\".\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy\nhttps://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin\nhttps://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Pri osobách Dominik Nagy, Patrik Pavlišin a Tomáš Kuchárik uveď názvy prác, ktoré súvisia so strojovým alebo automatickým prekladom.",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.86741,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/translation",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka/timovy_projekt",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus/timovy_projekt",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/translation",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/question",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/lukas_pokryvka/timovy_projekt",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus/timovy_projekt",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"first_model_latency_seconds": 2.095638,
|
||||
"tool_latency_seconds": 0.86741,
|
||||
"final_model_latency_seconds": 34.819498,
|
||||
"total_latency_seconds": 37.783597,
|
||||
"usage": {
|
||||
"prompt_tokens": 40271,
|
||||
"completion_tokens": 271,
|
||||
"total_tokens": 40542
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 3,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q0942",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Pri osobe Dárius Lindvai uveď názov diplomovej práce patriacej do kategórie „dp2021“ a rok vyjadrený touto kategóriou.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Rok 2021 je odvodený explicitne z kategórie dp2021. Na stránke sa zároveň nachádzajú sekcie diplomového projektu z roku 2020, preto musí otázka rozlišovať rok kategórie od roku sekcie.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" patriacu do kategórie dp2021, čo zodpovedá roku 2021.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Pri osobe Dárius Lindvai uveď názov diplomovej práce patriacej do kategórie „dp2021“ a rok vyjadrený touto kategóriou.",
|
||||
"limit": 5
|
||||
},
|
||||
"latency_seconds": 0.850273,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/mark_feher",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/mark_feher",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"
|
||||
],
|
||||
"first_model_latency_seconds": 1.577307,
|
||||
"tool_latency_seconds": 0.850273,
|
||||
"final_model_latency_seconds": 18.810862,
|
||||
"total_latency_seconds": 21.239067,
|
||||
"usage": {
|
||||
"prompt_tokens": 22122,
|
||||
"completion_tokens": 144,
|
||||
"total_tokens": 22266
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1094",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin. Pri osobe Maroš Harahus použi záznam typu „diplomová práca“ a pri osobe Patrik Pavlišin záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/maros_harahus/README.md",
|
||||
"pages/students/2016/patrik_pavlisin/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Neurónová morfologická anotácia slovenského jazyka",
|
||||
"2021",
|
||||
"Metódy automatického prekladu",
|
||||
"2020"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Maroš Harahus vypracoval diplomovú prácu s názvom „Neurónová morfologická anotácia slovenského jazyka\" v roku 2021 a Patrik Pavlišin vypracoval bakalársku prácu s názvom „Metódy automatického prekladu\" v roku 2020.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/maros_harahus\nhttps://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin. Pri osobe Maroš Harahus použi záznam typu „diplomová práca“ a pri osobe Patrik Pavlišin záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 1.251019,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/david_stromp",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela"
|
||||
],
|
||||
"first_model_latency_seconds": 1.970918,
|
||||
"tool_latency_seconds": 1.251019,
|
||||
"final_model_latency_seconds": 22.822001,
|
||||
"total_latency_seconds": 26.044583,
|
||||
"usage": {
|
||||
"prompt_tokens": 26504,
|
||||
"completion_tokens": 208,
|
||||
"total_tokens": 26712
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1095",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh. Pri osobe Filip Tomáš použi záznam typu „bakalárska práca“ a pri osobe Denis Le Thanh záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2019/filip_tomas/README.md",
|
||||
"pages/students/2023/denis_le_thanh/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia",
|
||||
"2022",
|
||||
"Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Podľa záznamov na ZP Wiki má Filip Tomáš bakalársku prácu s názvom „Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia“ uvedenú v roku 2022, zatiaľ čo Denis Le Thanh má bakalársku prácu s témou „Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku“ uvedenú v roku 2026.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2019/filip_tomas\nhttps://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh. Pri osobe Filip Tomáš použi záznam typu „bakalárska práca“ a pri osobe Denis Le Thanh záznam typu „bakalárska práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.505903,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2018/ondrej_megela",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 1.76637,
|
||||
"tool_latency_seconds": 0.505903,
|
||||
"final_model_latency_seconds": 3.585043,
|
||||
"total_latency_seconds": 5.858003,
|
||||
"usage": {
|
||||
"prompt_tokens": 25104,
|
||||
"completion_tokens": 220,
|
||||
"total_tokens": 25324
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,53 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, keďže v metadátach žiadneho dokumentu o grafových neurónových sieťach nie je ako autor uvedený Maroš Harahus; v ZP Wiki je Maroš Harahus uvedený ako autor dokumentu o študentovi Dmytro Mural, ktorý pracoval na bakalárskej práci s témou grafových neurónových sietí pre vyhľadávanie na internete, pričom samotný dokument o grafových sieťach je priradený študentovi Dmytro Mural, nie Marošovi Harahusovi.
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.33284,0.561576,21.524717,24.419828,23811,258,24069,
|
||||
q0146,dev,paraphrase,medium,Aká téma diplomovej práce je uvedená pri osobe „Dávid Omasta“ pre rok 2024?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Téma diplomovej práce Dávida Omasta pre rok 2024 je „Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku“.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2018/david_omasta",2.031153,0.196664,7.053768,9.281848,8103,152,8255,
|
||||
q0393,dev,specific_fact,medium,Akú databázu a backend už Ján Pták podľa stavu používal?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.0,1.0,0.0,0.5,False,"[""SQLite"", ""FastAPI""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2022/vladyslav_yanchenko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.899589,0.225847,21.242679,23.36894,26956,129,27085,
|
||||
q0575,dev,specific_fact,medium,Aké tri hlavné ciele mal Cesar Abascal Gutierrez?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.6666666666666666,1.0,1.0,0.9166666666666666,False,"[""recognize unknown named entities"", ""manually annotated training set"", ""annotation schema""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez""]","[""https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Cesar Abascal Gutierrez mal počas svojej stáže v lete 2019 tri hlavné ciele, ktoré zahŕňali schopnosť rozoznávať neznáme pomenované entity, vytvorenie manuálne anotovanej trénovacej množiny z prepisov reči a navrhnutie schémy anotácie.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/interns/cesar_gutierrez",1.896965,0.106753,10.780092,12.7842,12078,179,12257,
|
||||
q0690,dev,multi_document,hard,Nájdi dokumenty súvisiace so strojovým alebo automatickým prekladom.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,0.5,1.0,0.875,False,"[""strojový preklad"", ""automatický preklad""]","[""https://zp.kemt.fei.tuke.sk/students/2016/dominik_nagy"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/topics/translation"", ""https://zp.kemt.fei.tuke.sk/interns/oliver_pejic""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/tp20"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura""]","V ZP Wiki sa nachádzajú dokumenty súvisiace so strojovým alebo automatickým prekladom, medzi ktorými je diplomová práca Patrika Pavlišina venovaná štatistickému strojovému prekladu a metódam jeho vyhodnotenia, bakalárska práca Martina Jancury s názvom „Webová aplikácia pre demonštráciu strojového prekladu"" a dizertačná práca Maroša Harahusa, ktorá sa zaoberá experimentmi so strojovým prekladom a automatickou opravou textu.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin/tp20
|
||||
https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",1.979928,1.374685,16.937544,20.292711,18153,286,18439,
|
||||
q0821,dev,unanswerable_missing_fact,medium,"Koľko GB RAM mal počítač alebo server, ktorý Ján Holp používal pri práci?",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,[],"[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/eduard_matovka"", ""https://zp.kemt.fei.tuke.sk/students/2020/vladyslav_krupko"", ""https://zp.kemt.fei.tuke.sk/students/2023/rostyslav_rodzhuk""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",1.730877,0.303268,19.872212,21.906982,24725,116,24841,
|
||||
q0822,dev,unanswerable_missing_fact,medium,Koľko konkrétnych používateľov testovalo výsledný systém osoby Ján Holp?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,[],"[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2020/matus_suchanic"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2022/valerii_kutsenko""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",1.831417,0.291126,16.714124,18.837299,20508,121,20629,
|
||||
q0823,dev,unanswerable_missing_fact,medium,Aký konkrétny model GPU používal Ján Holp pri práci?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,[],"[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","[""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",1.893773,0.270439,7.074277,9.238766,8167,127,8294,
|
||||
q0942,dev,single_document_synthesis,hard,"Zhrň pri osobe Dárius Lindvai názov práce a rok, pri ktorom je táto práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/manohar_gowdru_shridharu"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta"", ""https://zp.kemt.fei.tuke.sk/students/2023/oleksandr_dorybohov"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky""]","Dárius Lindvai vypracovával diplomový projekt s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ktorý je uvedený v roku 2020.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",1.957513,0.727306,17.91672,20.602221,22031,152,22183,
|
||||
q0992,dev,single_document_synthesis,hard,"V dokumente osoby Dávid Omasta, ktorý súvisí s témou „generatívne modely“, uveď názov práce a rok, pri ktorom je práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku"", ""2024""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","[""https://zp.kemt.fei.tuke.sk/students/2018/david_omasta""]","Dávid Omasta má v dokumente uvedený návrh na názov diplomovej práce „Generatívne modely pre automatické odpovede na otázky v slovenskom jazyku“ pre rok 2024 a návrh na názov „Generatívne modely slovenského jazyka“ pre rok 2023.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2018/david_omasta",2.052944,0.248936,8.017358,10.319502,8393,185,8578,
|
||||
q1008,dev,single_document_synthesis,hard,Rozlíš pri osobe Ján Pták rok začiatku štúdia a rok uvedenej záverečnej práce.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2022"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat"", ""https://zp.kemt.fei.tuke.sk/students/2020/jakub_kristian_lukas"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin""]","Pri osobe Ján Pták je uvedený rok začiatku štúdia ako 2022 a rok záverečnej práce (diplomový projekt) ako 2026.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",2.13916,0.385244,19.276293,21.801445,23663,157,23820,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták: uveď názov práce pri každej osobe a rok, pri ktorom je daná práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/students/2019/kristian_sopkovic"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","Dárius Lindvai vypracovával diplomový projekt s názvom Obnovenie interpunkcie pomocou hlbokých neurónových sietí v roku 2021 a Ján Pták pracuje na diplomovom projekte s názvom Agent pre manažment záverečných prác v roku 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",2.108859,0.728522,25.138187,27.976318,29287,210,29497,
|
||||
q1094,dev,multi_document,hard,"Porovnaj záznamy osôb Maroš Harahus a Patrik Pavlišin: uveď názov práce pri každej osobe a rok, pri ktorom je daná práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""Neurónová morfologická anotácia slovenského jazyka"", ""2021"", ""Metódy automatického prekladu"", ""2020""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/topics/legal"", ""https://zp.kemt.fei.tuke.sk/students/2020/matej_kobylan"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"", ""https://zp.kemt.fei.tuke.sk/topics/question"", ""https://zp.kemt.fei.tuke.sk/students/2019/patrik_pokrivcak""]","U Maroša Harahusa je v dostupných dokumentoch uvedený návrh na tézu dizertačnej práce s názvom „Automatické opravy textu pri spracovaní prirodzeného jazyka"" v roku 2016, kedy sa záznam nachádza v sekcii druhého roku doktorandského štúdia. Patrik Pavlišin má uvedenú bakalársku prácu s názvom „Metódy automatického prekladu"" z roku 2020 a diplomový projekt s predbežným názvom „Neurónový strojový preklad"" uvedený v roku 2022.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",2.045444,1.098829,26.537682,29.682742,29439,276,29715,
|
||||
q1095,dev,multi_document,hard,"Porovnaj záznamy osôb Filip Tomáš a Denis Le Thanh: uveď názov práce pri každej osobe a rok, pri ktorom je daná práca uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.5,1.0,1.0,0.875,False,"[""Aplikačné testy v systémoch kontinuálnej integrácie a nasadenia"", ""2022"", ""Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh""]","[""https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas"", ""https://zp.kemt.fei.tuke.sk/students/2017/than_trung_thanh"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/legal"", ""https://zp.kemt.fei.tuke.sk/students/2021/vladimir_ferko"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2018/martin_wencel"", ""https://zp.kemt.fei.tuke.sk/students/2023/simona_bobrovcanova""]","Filip Tomáš vypracováva diplomovú prácu s názvom Vektorové vyhľadávanie dokumentov v prostredí Kubernetes, ktorá je uvedená pre rok 2025, keďže pôvodný zápis pre rok 2024 sa prekladá na rok 2025. Denis Le Thanh má záznam o bakalárskej práci s témou Trénovanie jazykového modelu pre spracovanie inštrukcií v prirodzenom jazyku, ktorá je evidovaná pre rok 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2019/filip_tomas
|
||||
https://zp.kemt.fei.tuke.sk/students/2023/denis_le_thanh",2.083936,0.576704,29.585007,32.246665,33407,256,33663,
|
||||
|
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,19 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca bola vypracovaná v roku 2021, čo je uvedené v kategórii dp2021, pričom rok 2020 v dokumente označuje rok sekcie diplomového projektu, nie rok vyhotovenia práce.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",23.563541,0.885095,49.481139,73.93032,15722,194,15916,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021, čo zodpovedá kategórii dp2021. Ján Pták realizuje diplomový projekt s názvom „Agent pre manažment záverečných prác"" v roku 2026, ako je uvedené v názve jeho príslušnej sekcie Diplomový projekt 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",8.071076,0.755932,65.688046,74.515424,10680,246,10926,
|
||||
q1109,dev,multi_document,hard,"Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agentové systémy pomocou veľkého jazykového modelu"", ""2026"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Martin Šarišský vypracováva diplomovú prácu s názvom „Agentové systémy pomocou veľkého jazykového modelu"" v roku 2026 a Andrii Pervashov má v roku 2027 naplánovanú diplomovú prácu s návrhom témy „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"".
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",5.946048,0.648092,52.195363,58.790439,28805,225,29030,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021 a Andrii Pervashov má v zázname uvedenú diplomovú prácu s názvom „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" v roku 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",5.007061,0.6561,44.511413,50.175219,19835,220,20055,
|
||||
|
@ -0,0 +1,422 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T09:28:39.614527+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "dd653c07c3cc3677dabc7a8c72432f5dd1c46b74776eb9ae56f68cc86bc4c8f1",
|
||||
"selected_question_ids": [
|
||||
"q0982",
|
||||
"q1092",
|
||||
"q1109",
|
||||
"q1125"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 4,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 64.352851,
|
||||
"prompt_tokens": 75042,
|
||||
"completion_tokens": 885,
|
||||
"total_tokens": 75927,
|
||||
"by_category": {
|
||||
"multi_document": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 61.160361,
|
||||
"prompt_tokens": 59320,
|
||||
"completion_tokens": 691,
|
||||
"total_tokens": 60011
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 73.93032,
|
||||
"prompt_tokens": 15722,
|
||||
"completion_tokens": 194,
|
||||
"total_tokens": 15916
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 4,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 64.352851,
|
||||
"prompt_tokens": 75042,
|
||||
"completion_tokens": 885,
|
||||
"total_tokens": 75927
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0982",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2016",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca bola vypracovaná v roku 2021, čo je uvedené v kategórii dp2021, pričom rok 2020 v dokumente označuje rok sekcie diplomového projektu, nie rok vyhotovenia práce.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.885095,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 23.563541,
|
||||
"tool_latency_seconds": 0.885095,
|
||||
"final_model_latency_seconds": 49.481139,
|
||||
"total_latency_seconds": 73.93032,
|
||||
"usage": {
|
||||
"prompt_tokens": 15722,
|
||||
"completion_tokens": 194,
|
||||
"total_tokens": 15916
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1092",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Agent pre manažment záverečných prác",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" v roku 2021, čo zodpovedá kategórii dp2021. Ján Pták realizuje diplomový projekt s názvom „Agent pre manažment záverečných prác\" v roku 2026, ako je uvedené v názve jeho príslušnej sekcie Diplomový projekt 2026.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.755932,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"first_model_latency_seconds": 8.071076,
|
||||
"tool_latency_seconds": 0.755932,
|
||||
"final_model_latency_seconds": 65.688046,
|
||||
"total_latency_seconds": 74.515424,
|
||||
"usage": {
|
||||
"prompt_tokens": 10680,
|
||||
"completion_tokens": 246,
|
||||
"total_tokens": 10926
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1109",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2021/martin_sarissky/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Agentové systémy pomocou veľkého jazykového modelu",
|
||||
"2026",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Martin Šarišský vypracováva diplomovú prácu s názvom „Agentové systémy pomocou veľkého jazykového modelu\" v roku 2026 a Andrii Pervashov má v roku 2027 naplánovanú diplomovú prácu s návrhom témy „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\".\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": true,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.648092,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
],
|
||||
"first_model_latency_seconds": 5.946048,
|
||||
"tool_latency_seconds": 0.648092,
|
||||
"final_model_latency_seconds": 52.195363,
|
||||
"total_latency_seconds": 58.790439,
|
||||
"usage": {
|
||||
"prompt_tokens": 28805,
|
||||
"completion_tokens": 225,
|
||||
"total_tokens": 29030
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1125",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" v roku 2021 a Andrii Pervashov má v zázname uvedenú diplomovú prácu s názvom „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\" v roku 2027.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.6561,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat"
|
||||
],
|
||||
"first_model_latency_seconds": 5.007061,
|
||||
"tool_latency_seconds": 0.6561,
|
||||
"final_model_latency_seconds": 44.511413,
|
||||
"total_latency_seconds": 50.175219,
|
||||
"usage": {
|
||||
"prompt_tokens": 19835,
|
||||
"completion_tokens": 220,
|
||||
"total_tokens": 20055
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,18 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca je zaradená do kategórie, ktorá vyjadruje rok 2021 (dp2021).
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",2.677976,0.961402,12.348958,15.988898,15718,163,15881,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/topics/faq""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021, kým Ján Pták realizuje diplomový projekt s témou „Agent pre manažment záverečných prác"" v roku 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",1.939179,0.715975,19.303855,21.96002,23210,210,23420,
|
||||
q1109,dev,multi_document,hard,"Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agentové systémy pomocou veľkého jazykového modelu"", ""2026"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Martin Šarišský vypracováva diplomovú prácu s názvom „Agentové systémy pomocou veľkého jazykového modelu“ v roku 2026 a Andrii Pervashov má v roku 2027 ako diplomovú prácu tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu“.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.851683,1.005713,24.53852,27.396667,28805,219,29024,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.75,0.5,1.0,0.8125,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat""]","V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaia nenašla explicitne uvedená záverečná práca typu „diplomová práca"" s rokom vyjadreným v kategórii, nakoľko jeho záznam obsahuje „Diplomový projekt 2020"" a „Názov: Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" s kategóriou dp2021, čo nezodpovedá požiadavke na rok vyjadrený kategóriou diplomovej práce a nie projektu. Pre Andrii Pervashova je v sekcii „Diplomová práca 2027"" uvedený názov témy „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" a rok 2027.
|
||||
|
||||
Zdroj:
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",1.752968,0.688275,4.410374,6.852814,19835,291,20126,
|
||||
|
@ -0,0 +1,428 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T09:32:02.988618+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "dd653c07c3cc3677dabc7a8c72432f5dd1c46b74776eb9ae56f68cc86bc4c8f1",
|
||||
"selected_question_ids": [
|
||||
"q0982",
|
||||
"q1092",
|
||||
"q1109",
|
||||
"q1125"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 4,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.9375,
|
||||
"source_url_score": 0.875,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.953125,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.75,
|
||||
"mean_latency_seconds": 18.0496,
|
||||
"prompt_tokens": 87568,
|
||||
"completion_tokens": 883,
|
||||
"total_tokens": 88451,
|
||||
"by_category": {
|
||||
"multi_document": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.916667,
|
||||
"source_url_score": 0.833333,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.9375,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 18.7365,
|
||||
"prompt_tokens": 71850,
|
||||
"completion_tokens": 720,
|
||||
"total_tokens": 72570
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 15.988898,
|
||||
"prompt_tokens": 15718,
|
||||
"completion_tokens": 163,
|
||||
"total_tokens": 15881
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.9375,
|
||||
"source_url_score": 0.875,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.953125,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.75,
|
||||
"mean_latency_seconds": 18.0496,
|
||||
"prompt_tokens": 87568,
|
||||
"completion_tokens": 883,
|
||||
"total_tokens": 88451
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0982",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2016",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca je zaradená do kategórie, ktorá vyjadruje rok 2021 (dp2021).\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"limit": 10,
|
||||
"published_only": true,
|
||||
"max_per_document": 10
|
||||
},
|
||||
"latency_seconds": 0.961402,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 2.677976,
|
||||
"tool_latency_seconds": 0.961402,
|
||||
"final_model_latency_seconds": 12.348958,
|
||||
"total_latency_seconds": 15.988898,
|
||||
"usage": {
|
||||
"prompt_tokens": 15718,
|
||||
"completion_tokens": 163,
|
||||
"total_tokens": 15881
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1092",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Agent pre manažment záverečných prác",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" v roku 2021, kým Ján Pták realizuje diplomový projekt s témou „Agent pre manažment záverečných prác\" v roku 2026.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 0.715975,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_malinovsky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq"
|
||||
],
|
||||
"first_model_latency_seconds": 1.939179,
|
||||
"tool_latency_seconds": 0.715975,
|
||||
"final_model_latency_seconds": 19.303855,
|
||||
"total_latency_seconds": 21.96002,
|
||||
"usage": {
|
||||
"prompt_tokens": 23210,
|
||||
"completion_tokens": 210,
|
||||
"total_tokens": 23420
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1109",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2021/martin_sarissky/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Agentové systémy pomocou veľkého jazykového modelu",
|
||||
"2026",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Martin Šarišský vypracováva diplomovú prácu s názvom „Agentové systémy pomocou veľkého jazykového modelu“ v roku 2026 a Andrii Pervashov má v roku 2027 ako diplomovú prácu tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu“.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 1.005713,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
],
|
||||
"first_model_latency_seconds": 1.851683,
|
||||
"tool_latency_seconds": 1.005713,
|
||||
"final_model_latency_seconds": 24.53852,
|
||||
"total_latency_seconds": 27.396667,
|
||||
"usage": {
|
||||
"prompt_tokens": 28805,
|
||||
"completion_tokens": 219,
|
||||
"total_tokens": 29024
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1125",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaia nenašla explicitne uvedená záverečná práca typu „diplomová práca\" s rokom vyjadreným v kategórii, nakoľko jeho záznam obsahuje „Diplomový projekt 2020\" a „Názov: Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" s kategóriou dp2021, čo nezodpovedá požiadavke na rok vyjadrený kategóriou diplomovej práce a nie projektu. Pre Andrii Pervashova je v sekcii „Diplomová práca 2027\" uvedený názov témy „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\" a rok 2027.\n\nZdroj:\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.688275,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat"
|
||||
],
|
||||
"first_model_latency_seconds": 1.752968,
|
||||
"tool_latency_seconds": 0.688275,
|
||||
"final_model_latency_seconds": 4.410374,
|
||||
"total_latency_seconds": 6.852814,
|
||||
"usage": {
|
||||
"prompt_tokens": 19835,
|
||||
"completion_tokens": 291,
|
||||
"total_tokens": 20126
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 0.75,
|
||||
"source_matches": [
|
||||
false,
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 0.5,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.8125,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,19 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0982,dev,single_document_synthesis,hard,Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""2016"", ""2021""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura"", ""https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik"", ""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak""]","Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca, ktorej kategória je uvedená ako dp2021, je vyjadrená rokom 2021, pričom rok 2020 v kontexte sekcie diplomového projektu nesmie byť zamieňaný s rokom vyjadreným kategóriou práce.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",2.698268,1.132829,48.265575,52.098326,43139,189,43328,
|
||||
q1092,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Agent pre manažment záverečných prác"", ""2026""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak""]","Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" v roku 2021, zatiaľ čo Ján Pták pracuje na diplomovom projekte s názvom „Agent pre manažment záverečných prác"" v roku 2026.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",2.787441,1.878599,47.39079,52.057184,10676,211,10887,
|
||||
q1109,dev,multi_document,hard,"Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Agentové systémy pomocou veľkého jazykového modelu"", ""2026"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky"", ""https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp""]","Z dostupných záznamov na ZP Wiki má Martin Šarišský diplomovú prácu s názvom „Agentové systémy pomocou veľkého jazykového modelu"" z roku 2026 a Andrii Pervashov má diplomovú prácu s návrhom témy „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" z roku 2027.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",4.352311,1.347279,19.079527,24.779934,28805,230,29035,
|
||||
q1125,dev,multi_document,hard,"Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,0.75,1.0,1.0,0.9375,False,"[""Obnovenie interpunkcie pomocou hlbokých neurónových sietí"", ""2021"", ""Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"", ""2027""]","[""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov""]","[""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"", ""https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"", ""https://zp.kemt.fei.tuke.sk/topics/faq"", ""https://zp.kemt.fei.tuke.sk/topics/akopisat""]","V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaja nenašiel explicitný názov diplomovej práce, iba názov „Obnovenie interpunkcie pomocou hlbokých neurónových sietí"" je uvedený v súvislosti s diplomovým projektom, a pre Andriiho Pervashova je v sekcii Diplomová práca 2027 uvedený návrh na tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu"" bez finálneho názvu práce.
|
||||
|
||||
Zdroje:
|
||||
https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai
|
||||
https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",4.093054,0.664814,20.58326,25.341756,19832,245,20077,
|
||||
|
@ -0,0 +1,433 @@
|
||||
{
|
||||
"generated_at": "2026-09-28T09:33:24.197557+00:00",
|
||||
"status": "complete",
|
||||
"configuration": {
|
||||
"schema_version": 2,
|
||||
"questions_file": "/home/janko/DP/zp-agent/evaluation/json_files/questions.json",
|
||||
"overrides_file": "/home/janko/DP/zp-agent/evaluation/json_files/rag_answer_overrides.json",
|
||||
"questions_fingerprint": "dd653c07c3cc3677dabc7a8c72432f5dd1c46b74776eb9ae56f68cc86bc4c8f1",
|
||||
"selected_question_ids": [
|
||||
"q0982",
|
||||
"q1092",
|
||||
"q1109",
|
||||
"q1125"
|
||||
],
|
||||
"applied_override_ids": [],
|
||||
"split": "dev",
|
||||
"selected_question_count": 4,
|
||||
"requested_model": "model120-fast",
|
||||
"operation_id": "retrieve_zpwiki_context",
|
||||
"openwebui_url": "https://ui.tukekemt.xyz/api/chat/completions",
|
||||
"rag_url": "http://localhost:8000/rag",
|
||||
"timeout": 180,
|
||||
"max_attempts": 4,
|
||||
"backoff_base": 1.0,
|
||||
"backoff_max": 8.0,
|
||||
"delay": 0.0
|
||||
},
|
||||
"summary": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.9375,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.984375,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.75,
|
||||
"mean_latency_seconds": 38.5693,
|
||||
"prompt_tokens": 102452,
|
||||
"completion_tokens": 875,
|
||||
"total_tokens": 103327,
|
||||
"by_category": {
|
||||
"multi_document": {
|
||||
"total": 3,
|
||||
"completed": 3,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.916667,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.979167,
|
||||
"strict_pass_count": 2,
|
||||
"strict_pass_rate": 0.666667,
|
||||
"mean_latency_seconds": 34.059625,
|
||||
"prompt_tokens": 59313,
|
||||
"completion_tokens": 686,
|
||||
"total_tokens": 59999
|
||||
},
|
||||
"single_document_synthesis": {
|
||||
"total": 1,
|
||||
"completed": 1,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 1.0,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass_count": 1,
|
||||
"strict_pass_rate": 1.0,
|
||||
"mean_latency_seconds": 52.098326,
|
||||
"prompt_tokens": 43139,
|
||||
"completion_tokens": 189,
|
||||
"total_tokens": 43328
|
||||
}
|
||||
},
|
||||
"by_difficulty": {
|
||||
"hard": {
|
||||
"total": 4,
|
||||
"completed": 4,
|
||||
"errors": 0,
|
||||
"tool_call_rate": 1.0,
|
||||
"answer_contains_score": 0.9375,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_score": 1.0,
|
||||
"overall_score": 0.984375,
|
||||
"strict_pass_count": 3,
|
||||
"strict_pass_rate": 0.75,
|
||||
"mean_latency_seconds": 38.5693,
|
||||
"prompt_tokens": 102452,
|
||||
"completion_tokens": 875,
|
||||
"total_tokens": 103327
|
||||
}
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "q0982",
|
||||
"split": "dev",
|
||||
"category": "single_document_synthesis",
|
||||
"difficulty": "hard",
|
||||
"question": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"2016",
|
||||
"2021"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard test: odpoveď treba poskladať z viacerých faktov jedného dokumentu.",
|
||||
"answer": "Dárius Lindvai začal štúdium v roku 2016 a jeho diplomová práca, ktorej kategória je uvedená ako dp2021, je vyjadrená rokom 2021, pričom rok 2020 v kontexte sekcie diplomového projektu nesmie byť zamieňaný s rokom vyjadreným kategóriou práce.\n\nZdroj: https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Rozlíš pri osobe Dárius Lindvai rok začiatku štúdia a rok vyjadrený kategóriou jeho diplomovej práce. Nezamieňaj ho s rokom sekcie diplomového projektu.",
|
||||
"limit": 10,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 1.132829,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/martin_jancura",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2017/dmytro_ushatenko",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2023/samuel_dzurina",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/matej_scislak"
|
||||
],
|
||||
"first_model_latency_seconds": 2.698268,
|
||||
"tool_latency_seconds": 1.132829,
|
||||
"final_model_latency_seconds": 48.265575,
|
||||
"total_latency_seconds": 52.098326,
|
||||
"usage": {
|
||||
"prompt_tokens": 43139,
|
||||
"completion_tokens": 189,
|
||||
"total_tokens": 43328
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true
|
||||
],
|
||||
"source_match_count": 1,
|
||||
"source_required_count": 1,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1092",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/jan_ptak/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Agent pre manažment záverečných prác",
|
||||
"2026"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Dárius Lindvai vypracovával diplomovú prácu s názvom „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" v roku 2021, zatiaľ čo Ján Pták pracuje na diplomovom projekte s názvom „Agent pre manažment záverečných prác\" v roku 2026.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/jan_ptak",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Ján Pták. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Jánovi Ptákovi použi diplomový projekt a rok uvedený v názve jeho príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 5
|
||||
},
|
||||
"latency_seconds": 1.878599,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/jan_ptak"
|
||||
],
|
||||
"first_model_latency_seconds": 2.787441,
|
||||
"tool_latency_seconds": 1.878599,
|
||||
"final_model_latency_seconds": 47.39079,
|
||||
"total_latency_seconds": 52.057184,
|
||||
"usage": {
|
||||
"prompt_tokens": 10676,
|
||||
"completion_tokens": 211,
|
||||
"total_tokens": 10887
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1109",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2021/martin_sarissky/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Agentové systémy pomocou veľkého jazykového modelu",
|
||||
"2026",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "Z dostupných záznamov na ZP Wiki má Martin Šarišský diplomovú prácu s názvom „Agentové systémy pomocou veľkého jazykového modelu\" z roku 2026 a Andrii Pervashov má diplomovú prácu s návrhom témy „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\" z roku 2027.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Martin Šarišský a Andrii Pervashov. Pri osobe Martin Šarišský použi záznam typu „diplomová práca“ a pri osobe Andrii Pervashov záznam typu „diplomová práca“. Uveď názov každej práce a rok, pri ktorom je na ZP Wiki uvedená.",
|
||||
"limit": 5,
|
||||
"published_only": false,
|
||||
"max_per_document": 1
|
||||
},
|
||||
"latency_seconds": 1.347279,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2021/martin_sarissky",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/patrik_pavlisin",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp"
|
||||
],
|
||||
"first_model_latency_seconds": 4.352311,
|
||||
"tool_latency_seconds": 1.347279,
|
||||
"final_model_latency_seconds": 19.079527,
|
||||
"total_latency_seconds": 24.779934,
|
||||
"usage": {
|
||||
"prompt_tokens": 28805,
|
||||
"completion_tokens": 230,
|
||||
"total_tokens": 29035
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 1.0,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 1.0,
|
||||
"strict_pass": true,
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"id": "q1125",
|
||||
"split": "dev",
|
||||
"category": "multi_document",
|
||||
"difficulty": "hard",
|
||||
"question": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"override_applied": false,
|
||||
"requested_model": "model120-fast",
|
||||
"expected_documents": [
|
||||
"pages/students/2016/darius_lindvai/README.md",
|
||||
"pages/students/2022/andrii_pervashov/README.md"
|
||||
],
|
||||
"expected_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"
|
||||
],
|
||||
"expected_answer_contains": [
|
||||
"Obnovenie interpunkcie pomocou hlbokých neurónových sietí",
|
||||
"2021",
|
||||
"Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu",
|
||||
"2027"
|
||||
],
|
||||
"should_answer": true,
|
||||
"note": "Nový hard multi-document test vyžadujúci fakty z dvoch samostatných dokumentov.",
|
||||
"answer": "V dostupných dokumentoch ZP Wiki sa pre Dáriusa Lindvaja nenašiel explicitný názov diplomovej práce, iba názov „Obnovenie interpunkcie pomocou hlbokých neurónových sietí\" je uvedený v súvislosti s diplomovým projektom, a pre Andriiho Pervashova je v sekcii Diplomová práca 2027 uvedený návrh na tému „Prepis reči pre tvorbu štruktúrovaného zdravotného záznamu\" bez finálneho názvu práce.\n\nZdroje:\nhttps://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai\nhttps://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"tool_called": true,
|
||||
"tool_call_count": 1,
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "retrieve_zpwiki_context",
|
||||
"arguments": {
|
||||
"query": "Porovnaj záznamy osôb Dárius Lindvai a Andrii Pervashov. Pri Dáriusovi Lindvaiovi uveď názov diplomovej práce a rok vyjadrený kategóriou diplomovej práce, nie rok diplomového projektu. Pri Andrii Pervashovovi použi diplomovú prácu a rok uvedený v názve príslušnej sekcie. Uveď názov a rok pri oboch osobách.",
|
||||
"limit": 5,
|
||||
"published_only": true
|
||||
},
|
||||
"latency_seconds": 0.664814,
|
||||
"source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat"
|
||||
]
|
||||
}
|
||||
],
|
||||
"rag_source_urls": [
|
||||
"https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/darius_lindvai",
|
||||
"https://zp.kemt.fei.tuke.sk/students/2016/jan_holp",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/faq",
|
||||
"https://zp.kemt.fei.tuke.sk/topics/akopisat"
|
||||
],
|
||||
"first_model_latency_seconds": 4.093054,
|
||||
"tool_latency_seconds": 0.664814,
|
||||
"final_model_latency_seconds": 20.58326,
|
||||
"total_latency_seconds": 25.341756,
|
||||
"usage": {
|
||||
"prompt_tokens": 19832,
|
||||
"completion_tokens": 245,
|
||||
"total_tokens": 20077
|
||||
},
|
||||
"response_model": "Qwen3.5-122B-A10B-Q4_K_M",
|
||||
"answer_matches": [
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
],
|
||||
"answer_contains_score": 0.75,
|
||||
"source_matches": [
|
||||
true,
|
||||
true
|
||||
],
|
||||
"source_match_count": 2,
|
||||
"source_required_count": 2,
|
||||
"source_url_score": 1.0,
|
||||
"should_answer_ok": true,
|
||||
"should_answer_score": 1.0,
|
||||
"returned_no_answer": false,
|
||||
"tool_score": 1.0,
|
||||
"overall_score": 0.9375,
|
||||
"strict_pass": false,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,4 @@
|
||||
id,split,category,difficulty,question,requested_model,response_model,tool_called,tool_call_count,answer_contains_score,source_url_score,should_answer_score,overall_score,strict_pass,expected_answer_contains,expected_source_urls,rag_source_urls,answer,first_model_latency_seconds,tool_latency_seconds,final_model_latency_seconds,total_latency_seconds,prompt_tokens,completion_tokens,total_tokens,error
|
||||
q0024,dev,keyword_grounding,medium,Ktorý študentský dokument o grafových neurónových sieťach má v metadátach ako autora dokumentu uvedeného Maroša Harahusa?,model120-fast,Qwen3.5-122B-A10B-Q4_K_M,True,1,1.0,1.0,1.0,1.0,True,"[""Dmytro Mural"", ""Maroš Harahus"", ""grafové neurónové siete""]","[""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural""]","[""https://zp.kemt.fei.tuke.sk/students/2016/maros_harahus"", ""https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural"", ""https://zp.kemt.fei.tuke.sk/students/2021/oleksandr_hryshchenko"", ""https://zp.kemt.fei.tuke.sk/students/2022/andrii_pervashov"", ""https://zp.kemt.fei.tuke.sk/students/2016/tomas_kucharik""]","V dostupných dokumentoch ZP Wiki sa túto informáciu nepodarilo spoľahlivo nájsť, pretože v metadátach žiadneho dokumentu o grafových neurónových sieťach nie je ako autor uvedený Maroš Harahus; v nájdených záznamoch je Maroš Harahus uvedený ako autor iba dokumentu o študentovi Dmytro Mural (bakalárska práca 2022), ktorej autorom študenta je Dmytro Mural a téma je grafové neurónové siete, no v tomto prípade je Maroš Harahus autorom dokumentu (správcu stránky), nie priamo študentom vypracujúcim prácu, a otázka sa pýtala na dokument, kde je Maroš Harahus uvedený ako autor dokumentu v metadátach, čo sa týka práve tohto dokumentu, ale otázka sa pýtala na dokument, kde je Maroš Harahus autorom dokumentu a téma je grafové neurónové siete, čo je práve tento dokument.
|
||||
|
||||
Zdroj: https://zp.kemt.fei.tuke.sk/students/2019/dmytro_mural",2.12978,0.576961,22.413007,25.120476,24187,350,24537,
|
||||
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user