odpovede, stránkovanie

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ján Pták 2026-11-16 01:16:15 +01:00
parent e8fed0a960
commit 55d29ae167
27 changed files with 1079 additions and 167 deletions

View File

@ -43,6 +43,37 @@ má doslovný výňatok, aktéra, konkrétnu prácu, fázu a dátum. Neisté poz
fakty nepoužijú. Rovnaké pojmy v rôznych tvaroch sa automaticky zjednotia.
Extrakciu spúšťa `python -m tools.knowledge.build` (vyžaduje nakonfigurované LLM API).
**CRZP** – Centrálny register záverečných prác (všetky slovenské vysoké školy) je súčasťou
`/rag` aj `/graphrag`, nie samostatný nástroj. Každá otázka sa súbežne pýta ZP Wiki aj CRZP
(`tools/answers/combined.py`); CRZP pridá práce osoby na TUKE (vedúci, oponent, abstrakt),
práce na rovnakú tému alebo podobné práce z iných škôl a odpovedá sám na otázky o iných
školách, oponentoch, katedrách a obhájených prácach (zverejnená práca v CRZP = obhájená).
Číta sa len verejné HTML (metadáta a abstrakty, nie plné texty), s obmedzením počtu
požiadaviek a lokálnou cache (`data/crzp`). Použité práce sa ukladajú do Neo4j a prepájajú
s osobami a témami ZP Wiki (`PROBABLY_SAME`, `SAME_TOPIC`).
## Odpoveď
- Jedna odpoveď: zoznam ľudí je jeden očíslovaný zoznam (najprv ZP Wiki, potom CRZP,
s označením zdroja), zdroje sú v jednom riadku na konci.
- Dlhý zoznam ukáže 6 položiek a opýta sa „Chceš vidieť ďalšie?“; odpoveď „áno“ ukáže ďalšie.
- Doplňujúce otázky cez podpísaný token: „a z akých škôl sú?“, „a čo robil prvý z nich?“,
výber práce („2“, „tak 1“, „bakalársku“), výber menovca a „a o čom je?“.
- Súvisiace témy: téma, kto ju robil v ZP Wiki a koľko prác je v CRZP (vážené podľa toho,
ako špecifický je pojem); podobné práce: najšpecifickejšie spoločné pojmy v základnom tvare.
- Čas každej odpovede (kroky, počet a typ volaní modelu, bez textu otázky) sa zapisuje do
`data/logs/answer_timings.jsonl`.
## Testovanie
- `pytest` – testy funkcionality a regresie.
- `python -m tools.evaluation.gold_set run --label X` – 111 ručne napísaných otázok
s očakávaniami odvodenými zo ZP Wiki (spúšťať samostatne).
- `python -m tools.evaluation.corpus build|run --label X` – ~460 generovaných otázok na každý
profil (vedúci, posledné stretnutie, začiatok štúdia, práce, úlohy, technológie), témy,
doplňujúce otázky, súvisiace a podobné práce, CRZP a odmietnutia; kontroluje aj, či odpoveď
neobsahuje servisné poznámky. Výsledky platia len pre danú sadu.
## Použité technológie
- **Python, FastAPI, Pydantic** – API, validácia vstupov a typované plány a odpovede.

View File

@ -45,6 +45,18 @@ logger = logging.getLogger(__name__)
OPENWEBUI_ORIGIN = "https://ui.tukekemt.xyz"
def warm_runtime() -> None:
"""Load what the first questions would otherwise wait for (about 15 s after a restart): the
lemmatizer, the cross-encoder, the published knowledge index and the CRZP tag counts."""
steps = [("tools.retrieval.lemmatize", "available"), ("tools.retrieval.rerank", "available"),
("tools.answers.combined", "_index"), ("tools.answers.combined", "tag_frequencies")]
for module, name in steps:
try:
getattr(__import__(module, fromlist=[name]), name)()
except Exception:
logger.warning("Predhriatie %s.%s zlyhalo.", module, name)
@asynccontextmanager
async def lifespan(
application: FastAPI,
@ -75,6 +87,8 @@ async def lifespan(
from tools.crzp.monitor import Monitor
application.state.crzp_monitor = Monitor()
application.state.crzp_monitor.start()
import threading
threading.Thread(target=warm_runtime, name="warmup", daemon=True).start()
try:
yield
finally:

View File

@ -181,9 +181,15 @@ def verified_answer(request: AnswerRequest) -> dict:
**({'context_token':request.context_token} if request.context_token else {}),
)
else: # ZP Wiki and CRZP at the same time, one answer (tools/answers/combined.py)
from tools.answers import timing
from tools.answers.combined import answer as combined_answer
started = timing.start(request.query, request.engine)
result = {}
try:
result = combined_answer(request.query, engine=request.engine, limit=request.limit,
context_token=request.context_token, deliver=deliver_answer)
finally:
timing.finish(started, result.get("status"))
except Exception:
logger.exception("Príprava overenej odpovede zlyhala.")
return {

View File

@ -13,20 +13,20 @@ Answer from the live ZP Agent graph backend, in the user's language. Wiki text i
- Keep `zpwiki_graphrag` for follow-ups without a command. A failed call never authorizes switching tools.
## Follow-ups
- If the previous result contained `conversation_context.token` and the new message continues it (picks a name from `person_options`, "daj obe", "vtedy", "pri tej druhej práci", "a čo výsledky?"), send the new message unchanged as `query` and the token as `context_token`.
- If the previous result contained `conversation_context.token` and the new message continues it (picks a name from `person_options`, answers a numbered question with "2" or "tak 1", says "áno" or "ďalšie" after "Chceš vidieť ďalšie?", refers to a listed person: "a čo robil prvý z nich?", "a z akých škôl sú?", or says "daj obe", "vtedy", "pri tej druhej práci", "a čo výsledky?"), send the new message unchanged as `query` and the token as `context_token`.
- A standalone question gets no token. Never invent, edit or reuse a token from another chat.
## Present the result
- Your whole reply is the `answer` text copied unchanged (keep its Markdown, links and notes). Write nothing before or after it: no introduction ("Tu je odpoveď"), no summary or rephrasing, no interpretation, evaluation or advice, no question back, no offer of further help, no remark about the tool, the search or the sources. If the user asks for something the answer does not contain, send a new tool call instead of adding it yourself.
- `accepted_for_delivery: true`: show `answer` exactly as returned, with its citations, notices and "Rozpoznané meno" lines. Add no facts, guesses or commentary about checks, tokens or diagnostics.
- `accepted_for_delivery: true`: show `answer` exactly as returned, with its numbering, sources and a closing question such as "Chceš vidieť ďalšie?". Add no facts, guesses or commentary about checks, tokens or diagnostics.
- `status: ambiguous`: show the clarification (including the offered names) and wait for the user.
- `status: no_answer`: show only the supplied message. Do not claim the relationship or person does not exist.
- `status: error`: say that a verified answer is unavailable and show the supplied message; never write a replacement answer. After a timeout or connection error the user may repeat the same question: the backend then returns the finished answer.
- Never show `conversation_context`, `delivery_check` or other technical fields.
- Keep lists complete: do not shorten a list of students or drop its notes (for example "téma je v zadaní…" or "Vynechaní sú študenti…").
- Keep lists as returned: do not shorten or renumber a list. A long list shows its first items and asks "Chceš vidieť ďalšie?"; the next items come from a new call with "áno" (or the user's reply) and the token.
## Meaning already applied by the backend
- Every answer also uses CRZP (Centrálny register záverečných prác): a person's works at TUKE, works on the same topic or similar works at other schools. The CRZP part is marked "V CRZP" and keeps its CRZP links; show it as returned. A question about CRZP, another school, opponents or a whole department is answered from CRZP here too; there is no separate CRZP tool.
- Every answer also uses CRZP (Centrálny register záverečných prác): a person's works at TUKE, works on the same topic or similar works at other schools. CRZP items are marked "(CRZP)" or introduced by "V CRZP"; all sources are listed once at the end; show it as returned. A question about CRZP, another school, opponents or a whole department is answered from CRZP here too; there is no separate CRZP tool.
- When a person has different works in ZP Wiki and in CRZP (e.g. a diploma thesis in the wiki and a bachelor thesis in CRZP), `status: ambiguous` asks which one; send the user's reply ("2", "bakalársku") unchanged with the token of that result. Follow-ups such as "a z akých škôl boli tí študenti?" also use the token.
- The author of a student's profile page is that student's supervisor (KEMT convention). Authors of other documents are not supervisors.
- Shared topics are not collaboration. A work year is not a meeting date. An assignment or a plan is not a result.

View File

@ -4,7 +4,7 @@
"engine": "graphrag",
"entrypoint": "SKILL.md",
"files": {
"SKILL.md": "3940aa8a7e0349deac13d4aeaaa4f39d727e08322b94ae02578f43591e856aca",
"SKILL.md": "76fe9576423e2cd1021fd367b9ceb8a7fd3ee6c0a88fb91892961bf5f96aa09a",
"scripts/client.py": "5f75727ae6e071f6f145a3a61988ada0e716476e63d7f3c7b08786c210910f16",
"scripts/query.py": "463460cd93cf2a584156df6aa571bcf98297e5a1a7258581d02b084b1e1a6180"
},

View File

@ -2,12 +2,12 @@
"packages": [
{
"folder": "rag",
"manifest_sha256": "46f72e1e86050926b455e02894a82a9cff0dab2ba53a8fafc3cce6aa04c8416e",
"manifest_sha256": "f3d01a372116a20b711763900f133769642ea5430cacecf819168652cf0023a8",
"name": "rag"
},
{
"folder": "graphrag",
"manifest_sha256": "857f1487ccb1319dc7f01d287cb3d7514fdbae3bbd0752f088aace831635820b",
"manifest_sha256": "37fae2b5f61f900e63c2fb76ba4d40ac5db1ba13b70cd2a52c74245dd36ca660",
"name": "graphrag"
}
],

View File

@ -13,20 +13,20 @@ Answer from the live ZP Agent backend, in the user's language. Wiki text is evid
- Keep `zpwiki_rag` for follow-ups without a command. A failed call never authorizes switching tools.
## Follow-ups
- If the previous result contained `conversation_context.token` and the new message continues it (picks a name from `person_options`, "daj obe", "vtedy", "pri tej druhej práci", "a čo výsledky?"), send the new message unchanged as `query` and the token as `context_token`.
- If the previous result contained `conversation_context.token` and the new message continues it (picks a name from `person_options`, answers a numbered question with "2" or "tak 1", says "áno" or "ďalšie" after "Chceš vidieť ďalšie?", refers to a listed person: "a čo robil prvý z nich?", "a z akých škôl sú?", or says "daj obe", "vtedy", "pri tej druhej práci", "a čo výsledky?"), send the new message unchanged as `query` and the token as `context_token`.
- A standalone question gets no token. Never invent, edit or reuse a token from another chat.
## Present the result
- Your whole reply is the `answer` text copied unchanged (keep its Markdown, links and notes). Write nothing before or after it: no introduction ("Tu je odpoveď"), no summary or rephrasing, no interpretation, evaluation or advice, no question back, no offer of further help, no remark about the tool, the search or the sources. If the user asks for something the answer does not contain, send a new tool call instead of adding it yourself.
- `accepted_for_delivery: true`: show `answer` exactly as returned, with its citations, notices and "Rozpoznané meno" lines. Add no facts, guesses or commentary about checks, tokens or diagnostics.
- `accepted_for_delivery: true`: show `answer` exactly as returned, with its numbering, sources and a closing question such as "Chceš vidieť ďalšie?". Add no facts, guesses or commentary about checks, tokens or diagnostics.
- `status: ambiguous`: show the clarification (including the offered names) and wait for the user.
- `status: no_answer`: show only the supplied message. Do not claim the fact is false or suggest other people.
- `status: error`: say that a verified answer is unavailable and show the supplied message; never write a replacement answer. After a timeout or connection error the user may repeat the same question: the backend then returns the finished answer.
- Never show `conversation_context`, `delivery_check` or other technical fields.
- Keep lists complete: do not shorten a list of students or drop its notes (for example "téma je v zadaní…" or "Vynechaní sú študenti…").
- Keep lists as returned: do not shorten or renumber a list. A long list shows its first items and asks "Chceš vidieť ďalšie?"; the next items come from a new call with "áno" (or the user's reply) and the token.
## Meaning already applied by the backend
- Every answer also uses CRZP (Centrálny register záverečných prác): a person's works at TUKE, works on the same topic or similar works at other schools. The CRZP part is marked "V CRZP" and keeps its CRZP links; show it as returned. A question about CRZP, another school, opponents or a whole department is answered from CRZP here too; there is no separate CRZP tool.
- Every answer also uses CRZP (Centrálny register záverečných prác): a person's works at TUKE, works on the same topic or similar works at other schools. CRZP items are marked "(CRZP)" or introduced by "V CRZP"; all sources are listed once at the end; show it as returned. A question about CRZP, another school, opponents or a whole department is answered from CRZP here too; there is no separate CRZP tool.
- When a person has different works in ZP Wiki and in CRZP (e.g. a diploma thesis in the wiki and a bachelor thesis in CRZP), `status: ambiguous` asks which one; send the user's reply ("2", "bakalársku") unchanged with the token of that result. Follow-ups such as "a z akých škôl boli tí študenti?" also use the token.
- The author of a student's profile page is that student's supervisor (KEMT convention). Authors of other documents are not supervisors.
- A work year is not a meeting date or the start of studies. An assignment or a plan is not a result.

View File

@ -4,7 +4,7 @@
"engine": "rag",
"entrypoint": "SKILL.md",
"files": {
"SKILL.md": "b64bd4b88857f0fc03bb1a3fbffc4cac8ef4ca6f00eacea4f9b4064ff3d55ee8",
"SKILL.md": "0900c6f27619ec874af13dba0adbdc1d8347e79b9dda55bcb9a182e11ccd51a0",
"scripts/client.py": "5f75727ae6e071f6f145a3a61988ada0e716476e63d7f3c7b08786c210910f16",
"scripts/query.py": "f482f2dc53430cbb383bce4fcc1cccaf580fc7b5a0c885a217fc40cf53713277"
},

View File

@ -27,14 +27,17 @@ QUESTION_POLICY = (
"paraphrase, translate, correct names or add/drop conditions; the backend resolves misspelled and partial names. "
"One call per question; several explicit questions in one message are separate calls in order. "
"If the previous result contained conversation_context.token and the new message continues it (a name chosen "
"from conversation_context.person_options, 'daj obe', 'vtedy', 'a čo výsledky?'), send the message unchanged "
"from conversation_context.person_options, a number answering a question, 'áno' or 'ďalšie' after 'Chceš "
"vidieť ďalšie?', 'a čo robil prvý z nich?', 'a z akých škôl sú?', 'daj obe', 'vtedy', 'a čo výsledky?'), "
"send the message unchanged "
"with that token as context_token. A standalone question gets no token. Never invent, edit or reuse another "
"chat's token. Respect the tool selected by the user's command and keep it for follow-ups; a failed call "
"never authorizes switching tools. "
)
ANSWER_PRESENTATION_POLICY = (
"When accepted_for_delivery is true, present answer exactly as returned, with its citations, notices and "
"recognized-name lines; add no facts, guesses or commentary about checks, tokens or diagnostics. "
"When accepted_for_delivery is true, present answer exactly as returned, with its numbering, sources and a "
"closing question such as 'Chceš vidieť ďalšie?'; add no facts, guesses or commentary about checks, tokens or "
"diagnostics. "
"The whole reply is that answer text unchanged: nothing before or after it (no introduction, summary, "
"rephrasing, evaluation, advice, question back or offer of further help); for anything the answer lacks, "
"call the tool again instead of adding it. "

View File

@ -98,7 +98,7 @@ def test_thesis_topic_counts_its_recorded_work_and_reading_is_not_doing(monkeypa
rows = {r["id"] for r in found["rows"]}
assert {"u3", "u4"} <= rows # Horáni: thesis about a chatbot + its RASA result
notes = " ".join(found["notes"])
assert "Samuel Horáni — téma práce" in notes
assert "Samuel Horáni" not in notes # recorded work of the thesis needs no explanatory note
assert "Peter Plán — téma je v zadaní" in notes
assert "Ján Pták — záznamy dokladajú len štúdium" in notes
order = [r["person"] for r in found["rows"]]
@ -302,4 +302,5 @@ def test_literal_safety_net_and_other_activity_bucket(monkeypatch):
# Recorded work of another kind is not labelled as a mere plan.
found = A.discover(graph, None, topic="RAG", canonical="rag", action="annotated")
notes = " ".join(found["notes"])
assert "nie však priamo požadovanú činnosť" in notes and "téma je v zadaní" not in notes
assert "téma je v zadaní" not in notes and "nie však priamo" not in notes
assert found["rows"] # shown as recorded work, without a note about the kind of activity

View File

@ -60,7 +60,7 @@ def test_meeting_prose_retains_planned_vs_executed_without_copying_imperatives()
'text':'Má vyhodnotiť systém.' if r['phase']=='planned' else 'Funguje agent RAG.'} for r in payload['evidence']]}
return {'choices':[{'message':{'content':json.dumps(value)}}]}
out=s.synthesize(result,plan,complete=model,diagnostics=True)
assert out['synthesis']['status']=='accepted' and 'Úlohy: Má vyhodnotiť systém.' in out['answer']
assert out['synthesis']['status']=='accepted' and 'Úlohy:\n- Má vyhodnotiť systém.' in out['answer']
assert 'Vyhodnoťte' not in out['answer'] and 'Vyhodnotil' not in out['answer']

View File

@ -23,8 +23,18 @@ def combo(monkeypatch):
return state
def bound(result, question):
"""Like the real wiki answer: bound to the question it was asked."""
from tools.answers.delivery_contract import fingerprint
check = result.get("delivery_check")
if isinstance(check, dict) and check.get("request_fingerprint") == "x":
result = {**result, "delivery_check": {**check, "request_fingerprint": fingerprint(question, "rag")}}
return result
def ask(combo, question, token=None):
return C.answer(question, engine="rag", context_token=token, deliver=lambda q, **kw: combo["deliver"](q, **kw))
return C.answer(question, engine="rag", context_token=token,
deliver=lambda q, **kw: bound(combo["deliver"](q, **kw), q))
def test_wiki_first_then_the_marked_crzp_part(combo):
@ -32,8 +42,9 @@ def test_wiki_first_then_the_marked_crzp_part(combo):
"rows": [], "urls": ["https://opac.test/1"], "people": [], "works": []}
result = ask(combo, "Kto vedie Jána Ptáka?")
text = result["answer"]
assert text.index("Vedúci práce: Daniel Hladek") < text.index("**V CRZP**") < text.index("Pavol Galajda")
assert text.index("Vedúci práce: Daniel Hladek") < text.index("V CRZP") < text.index("Pavol Galajda")
assert result["source_urls"] == ["https://zp.test/jan", "https://opac.test/1"] and result["status"] == "ok"
assert text.count("Zdroje:") == 1 and text.rstrip().endswith("https://zp.test/jan, https://opac.test/1")
import packages.claude.client as client
client.validate_metadata(result)
assert client.validated_delivery_check(result, "Kto vedie Jána Ptáka?", "rag")
@ -61,14 +72,79 @@ def test_a_work_only_crzp_has_is_asked_for_and_the_choice_continues(combo, monke
assert chosen[0]["source"] == "crzp"
def test_schools_of_the_people_of_the_previous_answer(combo):
combo["part"] = {"mode": "similar", "text": "- Maksym Zatirka — Technická univerzita v Košiciach: „X“", "rows": [],
"urls": [], "people": [("Eva Malá", "Slovenská technická univerzita v Bratislave")]}
first = ask(combo, "Kto pracoval na podobnej téme ako Ján Pták?")
LIST = {**WIKI, "answer": "**Ján Pták**\n\n- Diplomová práca 2027: GraphRAG.\n\n**Maksym Zatirka**\n\n- Bakalárska "
"práca 2026: Neo4J.\n\nZdroje: https://zp.test/students/2022/jan_ptak, https://zp.test/students/2023/maksym_zatirka",
"source_urls": ["https://zp.test/students/2022/jan_ptak", "https://zp.test/students/2023/maksym_zatirka"]}
def crzp_items(count):
return [{"name": f"Autor {chr(65 + i)}", "school": "Slovenská technická univerzita v Bratislave", "source": "crzp",
"text": f"Slovenská technická univerzita v Bratislave: „Práca {i}“ (bakalárska práca, 2025)",
"urls": [f"https://opac.test/{i}"], "sid": f"S{i}", "title": f"Práca {i}"} for i in range(count)]
def test_one_numbered_list_with_sources_at_the_end_and_the_rest_offered(combo):
combo["match"] = None
combo["deliver"] = lambda q, **kw: dict(LIST)
combo["part"] = {"mode": "topic", "items": crzp_items(7), "rows": [], "urls": [], "text": ""}
first = ask(combo, "Kto pracoval s GraphRAG?")
text = first["answer"]
assert "**1. Ján Pták** (ZP Wiki)" in text and "**2. Maksym Zatirka** (ZP Wiki)" in text
assert "**3. Autor A** (CRZP) — Slovenská technická" in text and "**6. Autor D**" in text and "Autor E" not in text
assert text.index("Zdroje:") < text.index("Našli sa ešte 3 ďalšie osoby (3 v CRZP). Chceš vidieť ďalšie?")
assert "https://opac.test/3" in text and "https://opac.test/4" not in text
more = ask(combo, "áno", first["conversation_context"]["token"])
assert "**7. Autor E** (CRZP)" in more["answer"] and "**9. Autor G**" in more["answer"]
assert "Chceš vidieť" not in more["answer"]
import packages.claude.client as client
assert client.validated_delivery_check(more, "áno", "rag")
def test_schools_of_the_people_of_the_previous_answer(combo):
combo["match"] = None
combo["deliver"] = lambda q, **kw: dict(LIST)
combo["part"] = {"mode": "similar", "items": crzp_items(1), "rows": [], "urls": [], "text": ""}
first = ask(combo, "Kto pracoval na podobnej téme ako Ján Pták?")
schools = ask(combo, "a z akých škôl boli tí študenti?", first["conversation_context"]["token"])
assert "- Ján Pták — Technická univerzita v Košiciach (KEMT FEI)" in schools["answer"]
assert "- Eva Malá — Slovenská technická univerzita v Bratislave" in schools["answer"]
assert "1. Ján Pták — Technická univerzita v Košiciach (KEMT FEI)" in schools["answer"]
assert "3. Autor A — Slovenská technická univerzita v Bratislave" in schools["answer"]
def test_an_ordinal_points_at_the_shown_person(combo):
combo["match"] = None
combo["deliver"] = lambda q, **kw: dict(LIST)
combo["part"] = None
first = ask(combo, "Kto pracoval s Neo4j?")
seen = []
combo["deliver"] = lambda q, **kw: (seen.append(q), dict(WIKI))[1]
combo["match"] = {"name": "Maksym Zatirka", "student": True}
result = ask(combo, "a čo robil druhý z nich?", first["conversation_context"]["token"])
assert seen == ["a čo robil Maksym Zatirka?"]
import packages.claude.client as client
assert client.validated_delivery_check(result, "a čo robil druhý z nich?", "rag")
def test_a_foreign_fingerprint_is_never_rebound(combo):
combo["deliver"] = lambda q, **kw: {**WIKI, "delivery_check": {**WIKI["delivery_check"], "request_fingerprint": "0" * 64}}
result = ask(combo, "Kto vedie Jána Ptáka?")
assert result["delivery_check"]["request_fingerprint"] == "0" * 64 # the client rejects it
def test_the_reply_to_a_work_choice_is_bound_to_the_reply(combo, monkeypatch):
combo["part"] = {"mode": "person", "text": "x", "rows": [], "urls": [],
"works": [{"source": "crzp", "label": "bakalárska práca 2025 „IoT“", "type": "bachelor_thesis", "key": "k"}]}
monkeypatch.setattr(C, "wiki_works", lambda person: {"Diplomová práca 2027": {"type": "diploma_thesis", "topic": "Agent"}})
asked = ask(combo, "Aké technológie využíva Ján Pták vo svojej práci?")
token = asked["conversation_context"]["token"]
import packages.claude.client as client
for reply in ("1", "tak 1", "jednotku", "tú diplomovú"):
result = ask(combo, reply, token)
assert result["status"] == "ok" and client.validated_delivery_check(result, reply, "rag"), reply
after = ask(combo, "2", token) # the CRZP work; the offer stays for the other one
assert after["conversation_context"]["token"] and ask(combo, "tak 1", after["conversation_context"]["token"])["status"] == "ok"
combo["match"] = None
again = ask(combo, "neviem", token)
assert again["status"] == "ambiguous" and "1 – Diplomová práca 2027" in again["answer"]
def test_a_slow_crzp_does_not_hold_the_wiki_answer(combo, monkeypatch):
@ -98,7 +174,7 @@ def test_crzp_people_and_keywords_link_to_the_wiki_graph_by_code():
def test_the_wiki_question_which_person_wins_over_crzp(combo):
combo["match"] = None
combo["part"] = {"mode": "explicit", "status": "ambiguous", "text": "Menu „Ján“ zodpovedá v CRZP…", "rows": [],
"urls": [], "people": []}
"urls": []}
combo["deliver"] = lambda q, **kw: {**WIKI, "status": "ambiguous", "reason": "ambiguous_person",
"answer": "Ktorého Jána myslíš? Ján Holp, Ján Pták."}
result = ask(combo, "Na čom pracoval Ján?")
@ -113,3 +189,65 @@ def test_an_unknown_person_goes_to_crzp_only_for_a_thesis_question():
assert C.THESIS.search(C.normalized("O čom je bakalárska práca Jána Nováka?"))
assert not C.THESIS.search(C.normalized("Na čom pracuje Peter Novák?"))
assert not C.ABOUT.search(C.normalized("Čomu sa venuje Jakub Schwarc?"))
def test_a_person_in_both_sources_is_listed_once():
wiki = [{"name": "Jakub Maruniak", "school": C.WIKI_SCHOOL, "source": "wiki", "text": "**Jakub Maruniak**", "urls": []}]
crzp = [{"name": "Jakub Maruniak", "school": C.TUKE, "source": "crzp", "text": f"{C.TUKE}: „Anotácia“ (diplomová práca, 2021)",
"urls": ["https://opac.test/m"]},
{"name": "Eva Malá", "school": "Iná", "source": "crzp", "text": "Iná: „X“", "urls": []}]
rest = C.merge_people(wiki, crzp)
assert [i["name"] for i in rest] == ["Eva Malá"] and "V CRZP: „Anotácia“" in wiki[0]["text"]
assert wiki[0]["urls"] == ["https://opac.test/m"]
def test_a_crzp_work_only_found_through_its_abstract_is_no_topic_answer(monkeypatch):
from tools.crzp.answering.understand import CrzpQuery
monkeypatch.setattr("tools.retrieval.rerank.scores", lambda q, texts: [-3.0] * len(texts))
rows = [{"title": "Fairy Tale Intertexts in Shakespeare's Tragedies", "keywords": ["intertext"]},
{"title": "Anotácia a rozpoznávanie pomenovaných entít", "keywords": ["anotácia"]}]
kept = C.relevant("Kto robil anotácie v Prodigy?", CrzpQuery(topics=["anotácie"]), rows)
assert [r["title"] for r in kept] == ["Anotácia a rozpoznávanie pomenovaných entít"]
def test_related_topics_skip_the_topic_its_synonyms_and_generic_tags(monkeypatch):
from tools.crzp.answering.understand import CrzpQuery
rows = [{"title": f"T{i}", "year": 2020 + i, "authors": ["Malá, Eva"], "school": "Technická univerzita v Košiciach",
"url": f"https://opac.test/{i}",
"keywords": ["rozpoznávanie reči", "speech recognition", "MFCC", "python", "syntéza reči"]} for i in range(3)]
monkeypatch.setattr("tools.crzp.answering.answer.find", lambda query: ({}, rows))
monkeypatch.setattr(C, "tag_frequencies", lambda: ({"mfcc": 20, "synteza reci": 30}, 30000))
monkeypatch.setattr(C, "wiki_people_for", lambda topic: ["Ján Pták"] if topic == "MFCC" else [])
part = C.related_part(CrzpQuery(topics=["rozpoznávanie reči"], intent="keywords"))
names = [i["name"] for i in part["items"]]
assert names == ["MFCC", "Syntéza reči"] and part["topic"] == "rozpoznávanie reči"
assert part["items"][0]["text"].startswith("ZP Wiki: Ján Pták; CRZP: 3 práce")
def test_synonym_groups_match_case_forms_quickly():
import time as _time
from tools.retrieval.synonyms import base_name, group_of
started = _time.monotonic()
for i in range(20000):
base_name(f"tag {i}")
assert _time.monotonic() - started < 2.0
assert base_name("skrytých Markovových modelov") == "skryté Markovove modely" and base_name("HMM") == base_name("hidden Markov model")
assert group_of("python") == ["python"]
def test_a_crzp_follow_up_is_bound_to_the_engine_asked(combo, monkeypatch):
from tools.crzp.answering import answer as crzp
from tools.answers.delivery_contract import fingerprint
combo["match"] = None
combo["part"] = {"mode": "explicit", "status": "ambiguous", "text": "Ktorú osobu myslíš?\n\n1. A\n2. B",
"rows": [], "urls": [], "crzp_token": "crzp-token"}
combo["deliver"] = lambda q, **kw: {**WIKI, "status": "ambiguous", "reason": "missing_subject", "answer": "?"}
first = ask(combo, "Kto bol oponent práce Novák?")
monkeypatch.setattr(crzp, "_answer", lambda q, complete=None, context_token=None: {
"status": "ok", "reason": None, "accepted_for_delivery": True, "answer": "Oponent: X\n\nZdroj: https://opac.test/1",
"source_urls": ["https://opac.test/1"], "claims": [], "index_version": None, "complete": True, "truncated": False,
"truncation_reasons": [], "delivery_check": {"layer": "delivery", "code": "complete",
"request_fingerprint": fingerprint(q, "crzp")}})
chosen = ask(combo, "3", first["conversation_context"]["token"])
import packages.claude.client as client
assert chosen["status"] == "ok" and client.validated_delivery_check(chosen, "3", "rag")

View File

@ -298,7 +298,8 @@ def test_a_meeting_question_is_not_narrowed_to_a_guessed_aspect():
from tools.answers.question_plan import direct_spec
from tools.retrieval.person_names import resolve_person_names
catalog = [{"name": "Jakub Schwarc", "kind": "Person", "source_document": "pages/students/2022/jakub_schwarc/README.md"}]
for question, facets in [("Kedy mal Jakub Schwarc posledné stretnutie a čo na ňom riešil?", []),
for question, facets in [("Kedy mal Jakub Schwarc posledné stretnutie a čo na ňom riešil?", ["progress"]),
("Čo bolo na poslednom stretnutí Jakuba Schwarca?", []),
("Aké výsledky mal Jakub Schwarc na poslednom stretnutí?", ["results"])]:
route = Route(type="meeting", mode="meeting", facets=["results"], meeting="latest")
spec = direct_spec(question, resolve_person_names(question, catalog), route=route)

View File

@ -5,15 +5,16 @@ at the same time, from CRZP when CRZP can add something (see crzp_plan):
- a person ZP Wiki knows: that person's works in CRZP at Technická univerzita
v Košiciach (supervisor, opponent, abstract, keywords the wiki may lack);
- a topic or "similar works": CRZP works on the same topic, with their schools;
- a question naming CRZP, another school or opponents: CRZP is the main source.
Meetings, tasks, progress and results stay ZP Wiki only. The wiki part comes
first, the CRZP part is marked with its source; nothing is mixed into one fact.
- a question naming CRZP, another school, opponents or defended works of a
department: CRZP is the main source.
Meetings, tasks, progress and results stay ZP Wiki only.
When a person has different works in the two sources (a diploma thesis in the
wiki, a bachelor thesis in CRZP) and the question does not say which one, the
answer asks; the reply ("2", "bakalársku") continues through a signed token
that also carries the wiki's own token and the people of the last answer
(for "a z akých škôl boli tí študenti?").
The answer is one text: a list of people is one numbered list (ZP Wiki people
first, then CRZP works, each marked with its source), the first PAGE items are
shown and the rest is offered ("Chceš vidieť ďalších?"); all sources are in one
line at the end. A signed token carries the wiki's own token, the shown people
(for "a z akých škôl sú?", "a čo robil prvý z nich?") and the id of the not
yet shown items (tools/answers/pages.py).
"""
from __future__ import annotations
@ -47,7 +48,13 @@ ABOUT = re.compile(r"\b(?:o com|zhrn\w*|abstrakt\w*|ciel\w*)\b") # not "čomu s
# A question about a thesis (not "na čom pracuje", which CRZP's submitted works cannot answer).
THESIS = re.compile(r"\b(?:prac(?:a|e|u|i|ou|ach|ami)?|bakalar\w*|diplom\w*|dizerta\w*|habilita\w*|tem(?:a|u|y|ou)|"
r"oponent\w*|vedu\w*|skolitel\w*|abstrakt\w*|klucov\w*|tagy?)\b")
WIKI_CLARIFICATION_YIELDS = {"missing_subject", "unsupported_content_question"}
WIKI_CLARIFICATION_YIELDS = {"missing_subject", "unsupported_content_question", "unsupported_constraint"}
# "Koľko prác obhájili na KEMT v roku 2024?": defended works of a school or department are CRZP's
# published works, not a wiki question about one student's defence.
DEFENDED = re.compile(r"\b(?:obhaj\w*|obhajen\w*|obhajil\w*|uspesn\w*)\b")
TOPIC_MIN_SCORE = float(os.getenv("CRZP_TOPIC_MIN_SCORE", "0"))
PAGE = int(os.getenv("ANSWER_PAGE_SIZE", "6"))
WIKI_FIRST = 4 # a mixed list shows at most this many ZP Wiki people before the CRZP works
REFUSED = {"request_out_of_scope", "irrelevant", "unsupported_request", "out_of_scope"}
RELATED = re.compile(r"\b(?:suvis\w*|pribuzn\w*)\b")
SIMILAR = re.compile(r"\b(?:podobn\w*|suvisiac\w*|pribuzn\w*|spolupra\w*|rovnak\w* tem\w*)\b")
@ -191,14 +198,21 @@ def wiki_profile(person: str) -> str:
def crzp_plan(question: str, person: str | None, complete=None, student: bool = True):
"""(mode, CrzpQuery) or None when CRZP adds nothing to this question."""
from tools.crzp.answering.understand import Person, understand
from tools.crzp.answering.units import unit_in_question
low = normalized(question)
explicit = bool(EXPLICIT.search(low))
if WIKI_ONLY.search(low) and not explicit:
defended_scope = bool(DEFENDED.search(low) and not person and (
unit_in_question(question) or re.search(r"\b(?:univerzit\w*|fakult\w*|katedr\w*|skol\w*|tuke|stu|kemt|fei)\b", low)))
if WIKI_ONLY.search(low) and not explicit and not defended_scope:
return None
query = understand(question, complete=complete)
# A person ZP Wiki knows needs no model reading of the question (the person and school are
# fixed below; rules give the intent): one model call less, the wiki part has the slots.
query = understand(question, complete=complete, use_model=not person or explicit)
if query.intent == "refused":
return None
other_school = query.school not in (None, "tuke") or explicit
if person and other_school and not query.from_model:
query = understand(question, complete=complete)
if person and not other_school:
if SIMILAR.search(low) and student:
return "similar", query
@ -213,8 +227,11 @@ def crzp_plan(question: str, person: str | None, complete=None, student: bool =
return "person", query
if query.people and not person and query.search_text() and THESIS.search(low):
return "explicit", query # a person ZP Wiki does not know: CRZP knows the theses of all schools
if query.topics and not query.people and RELATED.search(low) and not re.search(r"\b(?:tag\w*|klucov\w*)\b", low):
query.intent = "keywords" # "Ktoré témy súvisia s X?": topics of works about X, and who did them
return "related", query
if query.topics and not query.people and RELATED.search(low):
query.intent = "keywords" # "Ktoré témy súvisia s X?": the tags of CRZP works about X
query.intent = "keywords" # "Ktoré tagy súvisia s X?": the CRZP tags themselves
return "explicit", query
if query.topics and not query.people and query.intent in ("relation", "count", "keywords", "schools"):
return "explicit", query # "Kto viedol najviac prác o X?", "Koľko prác o X…": computed over CRZP
@ -243,9 +260,9 @@ def explicit_part(question: str, result: dict) -> dict | None:
if asked is not None:
issued = crzp_context.issue(question, asked, rows, options)
token = issued["token"] if issued else None
entities = result.pop("_entities", None) or []
return {"mode": "explicit", "status": result["status"], "text": clean_crzp(result["answer"]), "rows": rows,
"urls": result.get("source_urls", []), "crzp_token": token,
"people": [(_names(r.get("authors", [])), crzp.university(r)) for r in rows]}
"urls": result.get("source_urls", []), "crzp_token": token, "entities": entities}
def _kind(row) -> str:
@ -255,7 +272,7 @@ def _kind(row) -> str:
def _names(values) -> str:
from tools.crzp.source.parse import display_name
return ", ".join(display_name(v) for v in values) or "neuvedené"
return ", ".join(display_name(v) for v in values) or "neuvedený"
def _work(row) -> str:
@ -313,19 +330,20 @@ def _crzp_part(question: str, person: str | None, *, complete=None, student: boo
low = normalized(question)
if mode == "similar":
return crzp_similar(person, complete=complete)
if mode == "related":
return related_part(query)
if mode == "explicit":
return explicit_part(question, crzp._answer_query(question, query, complete))
found, rows = crzp.find(query)
if not rows:
return None
rows = crzp.ranked(question, query, rows)
if mode == "topic":
rows = rows[:5]
lines = [f"- {_names(r.get('authors', []))} — {crzp.university(r)}: „{r['title']}“ ({_kind(r)}, {r.get('year')})"
for r in rows]
return {"mode": mode, "text": "Práce na túto tému v CRZP (najbližšie podľa obsahu):\n\n" + "\n".join(lines),
"rows": rows, "urls": [r["url"] for r in rows if r.get("url")],
"people": [(_names(r.get("authors", [])), crzp.university(r)) for r in rows]}
rows = relevant(question, query, rows)[:12]
if not rows:
return None # nothing in CRZP is really about the topic: no filler works
return {"mode": mode, "items": [crzp_item(r) for r in rows], "rows": rows, "text": "",
"urls": [r["url"] for r in rows if r.get("url")]}
rows = crzp.ranked(question, query, rows)
# person: that person's works at TUKE
rows = [r for r in rows if crzp.university(r) == TUKE]
if not rows:
@ -335,8 +353,7 @@ def _crzp_part(question: str, person: str | None, *, complete=None, student: boo
lines = [f"- {_names(r.get('authors', []))} — „{r['title']}“ ({_kind(r)}, {r.get('year')})" for r in rows[:8]]
more = f"\n\nSpolu {len(rows)} prác v CRZP na TUKE; zobrazených je 8 najnovších." if len(rows) > 8 else ""
return {"mode": mode, "text": f"Práce, ktoré {person} viedol alebo oponoval, v CRZP (TUKE):\n\n"
+ "\n".join(lines) + more, "rows": rows[:8], "urls": [r["url"] for r in rows[:8] if r.get("url")],
"people": [(_names(r.get("authors", [])), crzp.university(r)) for r in rows[:8]]}
+ "\n".join(lines) + more, "rows": rows[:8], "urls": [r["url"] for r in rows[:8] if r.get("url")]}
lines, details = [], {}
for row in rows[:3]:
if ABOUT.search(low) or METHODS.search(low):
@ -351,7 +368,7 @@ def _crzp_part(question: str, person: str | None, *, complete=None, student: boo
lines.append(f"- {_work(row)} — vedúci {_names(row.get('supervisors', []))}; "
f"oponent {_names(row.get('opponents', []))}")
return {"mode": mode, "text": "\n\n".join(lines), "rows": rows[:3],
"urls": [r["url"] for r in rows[:3] if r.get("url")], "people": [],
"urls": [r["url"] for r in rows[:3] if r.get("url")],
"works": [{"source": "crzp", "label": _work(r), "type": r.get("work_type"), "key": crzp.work_key(r)}
for r in rows[:3]]}
except CrzpUnavailable:
@ -384,27 +401,262 @@ def crzp_similar(person: str, *, complete=None) -> dict | None:
order = rerank.scores(profile, [r["title"] + ". " + ", ".join(r.get("keywords", [])) for r, _ in candidates]) \
or [0] * len(candidates)
# Closest by content first; more shared topics/tools break ties.
best = [c for _, c in sorted(zip(order, candidates), key=lambda i: (-i[0], -len(i[1][1])))][:5]
lines = [f"- {_names(r.get('authors', []))} — {crzp.university(r)}: „{r['title']}“ ({_kind(r)}, {r.get('year')}); "
f"spoločné: {', '.join(shared)}" for r, shared in best]
return {"mode": "similar", "text": "Podobné práce v CRZP (spoločné témy a nástroje):\n\n" + "\n".join(lines),
"rows": [r for r, _ in best], "urls": [r["url"] for r, _ in best if r.get("url")],
"people": [(_names(r.get("authors", [])), crzp.university(r)) for r, _ in best]}
best = [c for _, c in sorted(zip(order, candidates), key=lambda i: (-i[0], -len(i[1][1])))][:12]
return {"mode": "similar", "items": [crzp_item(r, shared) for r, shared in best], "text": "",
"rows": [r for r, _ in best], "urls": [r["url"] for r, _ in best if r.get("url")]}
_tag_cache: dict = {}
_tag_lock = threading.Lock()
def tag_frequencies() -> tuple[dict, int]:
"""In how many cached CRZP works each tag (its reviewed base name) occurs, and of how many works:
a tag most works carry ("umelá inteligencia", "python") names no specific related topic."""
from collections import Counter
from tools.crzp.fetch import store
stamp = int(time.time() // 1800) # recounted at most every half hour
with _tag_lock:
if _tag_cache.get("stamp") == stamp:
return _tag_cache["df"], _tag_cache["total"]
from tools.retrieval.synonyms import base_name
counts, seen = Counter(), set()
for row in store().cached_rows():
key = (row.get("title"), row.get("year"))
if key in seen:
continue
seen.add(key)
counts.update({normalized(base_name(k.strip())) for k in row.get("keywords", []) if k.strip()})
with _tag_lock:
_tag_cache.update(stamp=stamp, df=dict(counts), total=max(1, len(seen)))
return _tag_cache["df"], _tag_cache["total"]
def wiki_people_for(topic: str) -> list[str]:
"""ZP Wiki students whose own records (topic, assignment, progress, results) name the topic."""
from tools.retrieval.synonyms import variants_pattern
pattern = variants_pattern(topic)
people = {}
for unit in (_index() or {}).get("units", []):
if unit.get("kind") in {"reference", "meeting", "supervisor", "consultant", "lifecycle"}:
continue
if pattern.search(normalized(unit.get("text", ""))) or any(
pattern.fullmatch(normalized(c)) for c in unit.get("concepts", []) if isinstance(c, str)):
people.setdefault(unit["person"], unit.get("source_url"))
return list(people)
def related_part(query) -> dict | None:
"""Topics related to the asked one: the CRZP keywords of works about it, weighted by how specific
they are, each with the ZP Wiki students and CRZP works that have it."""
import math
from tools.crzp.answering import answer as crzp
from tools.retrieval.synonyms import base_name, group_of, variants_pattern
found, rows = crzp.find(query)
if not rows:
return None
asked = [t for t in [*query.topics, *query.phrases] if t]
patterns = [variants_pattern(t) for t in asked]
asked_words = {w for t in asked for x in group_of(t) for w in normalized(x).split()}
generic = {normalized(g) for g in crzp.GENERIC_TAGS} | {"umela inteligencia", "strojove ucenie", "python"}
df, total = tag_frequencies()
stats: dict = {}
for row in rows:
for keyword in {base_name(k.strip()) for k in row.get("keywords", []) if k.strip()}:
key = normalized(keyword)
if (len(key) < 3 or key in generic or set(key.split()) <= asked_words
or any(p.fullmatch(key) for p in patterns)):
continue # the asked topic itself, a synonym, a part of it or a generic word
entry = stats.setdefault(key, {"label": keyword, "rows": []})
if keyword[:1].isupper() and not entry["label"][:1].isupper():
entry["label"] = keyword
entry["rows"].append(row)
ranked = sorted((e for e in stats.values() if len(e["rows"]) >= 2),
key=lambda e: -len(e["rows"]) * math.log(total / (1 + df.get(normalized(e["label"]), 0))))[:18]
if not ranked:
return None
items = []
for entry in ranked:
label = entry["label"][:1].upper() + entry["label"][1:]
example = max(entry["rows"], key=lambda r: r.get("year") or 0)
authors = example.get("authors") or []
from tools.crzp.source.parse import display_name
wiki = wiki_people_for(entry["label"])
text = (f"CRZP: {crzp.plural(len(entry['rows']))}, napr. {display_name(authors[0]) if authors else 'autor neuvedený'}"
f" — {crzp.university(example)}: „{example['title']}“ ({example.get('year') or 'rok neuvedený'})")
if wiki:
text = "ZP Wiki: " + ", ".join(wiki[:4]) + (f" a ďalší ({len(wiki) - 4})" if len(wiki) > 4 else "") + "; " + text
items.append({"name": label, "source": "topic", "text": text,
"urls": [example["url"]] if example.get("url") else []})
topic = base_name(asked[0]) if asked else ""
return {"mode": "related", "items": items, "topic": topic, "rows": [e["rows"][0] for e in ranked], "text": "",
"urls": [i["urls"][0] for i in items if i["urls"]]}
def crzp_item(row: dict, shared: list[str] | None = None) -> dict:
"""One CRZP work as an item of a combined list."""
from tools.crzp.answering import answer as crzp
from tools.crzp.source.parse import display_name
authors = row.get("authors") or []
name = display_name(authors[0]) if authors else "autor neuvedený"
school = crzp.university(row)
line = f"{school}: „{row['title']}“ ({_kind(row)}, {row.get('year') or 'rok neuvedený'})"
if shared:
line += "; spoločné: " + ", ".join(shared)
return {"name": name, "school": school, "source": "crzp", "text": line, "urls": [row["url"]] if row.get("url") else [],
"sid": row.get("sid"), "title": row["title"]}
def relevant(question: str, query, rows: list[dict]) -> list[dict]:
"""CRZP works really about the asked topic, most relevant first: the topic (or a reviewed
synonym, in any case form) in the title or the CRZP keywords, or a cross-encoder score
above CRZP_TOPIC_MIN_SCORE. A work found only through its abstract ("Fairy Tale
Intertexts" for "anotácie") is no answer to a topic question."""
from tools.retrieval import rerank
from tools.retrieval.synonyms import variants_pattern
if not rows:
return []
terms = [t for t in [*query.topics, *query.phrases] if t]
patterns = [variants_pattern(t) for t in terms]
texts = [r["title"] + ". " + ", ".join(r.get("keywords", [])) for r in rows]
asked = " ".join([question, *[t for t in [*query.topics, *query.alternatives] if normalized(t) not in normalized(question)]])
scores = rerank.scores(asked, texts)
if scores is not None: # the cross-encoder scores at most MAX_PAIRS texts; the rest only by name
scores = list(scores) + [float("-inf")] * (len(texts) - len(scores))
kept = []
for i, (row, text) in enumerate(zip(rows, texts)):
named = any(p.search(normalized(text)) for p in patterns)
score = scores[i] if scores is not None else (1.0 if named else -1.0)
if named or score >= TOPIC_MIN_SCORE:
kept.append((score + (1.0 if named else 0.0), i, row))
return [row for _, _, row in sorted(kept, key=lambda k: (-k[0], k[1]))]
# ------------------------------------------------------------------ items of a list answer
HEADING = re.compile(r"^\*\*([^*\n]+?)\*\*\s*$")
PERSON_NAME = re.compile(r"[A-ZÁ-Ž][\w'’.-]+(?:\s+[A-ZÁ-Ž][\w'’.-]+){1,3}")
# Blocks after the people of a wiki list (they belong to the whole list).
TAIL = re.compile(r"^(?:Téma je len v zadaní|Len štúdium alebo poznámky|Tému majú len|Podklady k porovnávanej osobe)")
SOURCE_LINE = re.compile(r"^Zdroj(?:e)?(?: CRZP)?:\s*(.+)$", re.M)
def split_sources(text: str) -> tuple[str, list[str]]:
"""The answer without its source lines, and their links."""
urls = []
for match in SOURCE_LINE.finditer(text or ""):
urls += re.findall(r"https?://[^\s,)\]]+", match.group(1))
body = SOURCE_LINE.sub("", text or "")
return re.sub(r"\n{3,}", "\n\n", body).strip(), list(dict.fromkeys(urls))
def _slug(name: str) -> str:
return "_".join(normalized(name).split())
def wiki_items(body: str, urls: list[str]) -> tuple[list[str], list[dict], list[str], list[str]]:
"""(head blocks, one item per person, tail blocks, links not tied to a person) of a wiki answer."""
blocks = [b.strip() for b in re.split(r"\n\s*\n", body) if b.strip()]
head, items, tail = [], [], []
current = None
for block in blocks:
first = block.split("\n", 1)[0]
heading = HEADING.match(first)
name = re.split(r"\s+—\s+", heading.group(1))[0].strip() if heading else ""
if tail or TAIL.match(block):
tail.append(block)
elif heading and PERSON_NAME.fullmatch(name) and not name.endswith(":"):
if current is not None and current["name"] == name:
current["blocks"].append(block) # the same person's next work
else:
current = {"name": name, "school": WIKI_SCHOOL, "source": "wiki", "blocks": [block]}
items.append(current)
elif current is None:
head.append(block)
else:
current["blocks"].append(block)
general = []
for url in urls:
owner = next((i for i in items if _slug(i["name"]) in url), None)
if owner is not None:
owner.setdefault("urls", []).append(url)
else:
general.append(url)
for item in items:
item["text"] = "\n\n".join(item.pop("blocks"))
item.setdefault("urls", [])
return head, items, tail, general
def render_item(number: int, item: dict, mixed: bool) -> str:
if item["source"] == "topic":
return f"**{number}. {item['name']}** — {item['text']}"
if item["source"] == "crzp":
return f"**{number}. {item['name']}** (CRZP) — {item['text']}"
first, _, rest = item["text"].partition("\n")
heading = HEADING.match(first)
title = heading.group(1) if heading else item["name"]
line = f"**{number}. {title}**" + (" (ZP Wiki)" if mixed else "")
return line + ("\n" + rest if rest else "")
def merge_people(wiki: list[dict], crzp_items: list[dict]) -> list[dict]:
"""A CRZP work of a person already in the ZP Wiki list joins that person's item (no person twice)."""
by_name = {normalized(item["name"]): item for item in wiki}
rest = []
for item in crzp_items:
owner = by_name.get(normalized(item["name"]))
if owner is not None and item.get("school") == TUKE:
owner["text"] += f"\n- V CRZP: {item['text'].split(': ', 1)[-1]}"
owner.setdefault("urls", []).extend(item.get("urls", []))
else:
rest.append(item)
return rest
def first_page(wiki: list[dict], crzp_items: list[dict]) -> tuple[list[dict], list[dict]]:
"""(shown, rest): ZP Wiki people first (at most WIKI_FIRST when CRZP has works too), CRZP fills the page."""
crzp_items = merge_people(wiki, crzp_items)
take_wiki = min(len(wiki), WIKI_FIRST if crzp_items else PAGE)
take_crzp = min(len(crzp_items), PAGE - take_wiki)
take_wiki = min(len(wiki), PAGE - take_crzp)
shown = wiki[:take_wiki] + crzp_items[:take_crzp]
return shown, wiki[take_wiki:] + crzp_items[take_crzp:]
def more_line(rest: list[dict]) -> str:
if all(i["source"] == "topic" for i in rest):
word = "ďalšia téma" if len(rest) == 1 else "ďalšie témy" if len(rest) < 5 else "ďalších tém"
return f"Našl{'a' if len(rest) == 1 else 'i' if len(rest) < 5 else 'o'} sa ešte {len(rest)} {word}. Chceš vidieť ďalšie?"
wiki = sum(1 for i in rest if i["source"] == "wiki")
crzp_count = len(rest) - wiki
parts = [p for p in (f"{wiki} v ZP Wiki" if wiki else "", f"{crzp_count} v CRZP" if crzp_count else "") if p]
word = "ďalšia osoba" if len(rest) == 1 else "ďalšie osoby" if len(rest) < 5 else "ďalších osôb"
return f"Našl{'a' if len(rest) == 1 else 'i' if len(rest) < 5 else 'o'} sa ešte {len(rest)} {word} ({', '.join(parts)}). Chceš vidieť ďalšie?"
def compose_list(head: list[str], shown: list[dict], rest: list[dict], tail: list[str], general_urls: list[str],
*, start: int = 1, mixed: bool) -> tuple[str, list[str], str | None]:
"""(text, sources, id of the stored rest)."""
blocks = list(head) + [render_item(start + i, item, mixed) for i, item in enumerate(shown)] + list(tail)
urls = list(dict.fromkeys([*general_urls, *[u for item in shown for u in item.get("urls", [])]]))
if urls:
blocks.append("Zdroje: " + ", ".join(urls))
from tools.answers import pages
page_id = pages.put(rest, {"mixed": mixed}) if rest else None
if rest and page_id:
blocks.append(more_line(rest))
return "\n\n".join(blocks), urls, page_id
def entity(item: dict, number: int) -> dict:
return {k: v for k, v in {"n": number, "name": item["name"], "school": item.get("school"),
"source": item.get("source"), "sid": item.get("sid"), "title": item.get("title")}.items()
if v is not None}
# ------------------------------------------------------------------ composition
def _wiki_people(result: dict) -> list[tuple[str, str]]:
names = re.findall(r"^\*\*([^*\n—]+?)(?:\s+—[^*\n]*)?\*\*\s*$", result.get("answer") or "", re.M)
return [(n.strip(), WIKI_SCHOOL) for n in dict.fromkeys(names) if len(n.split()) >= 2]
def _schools_answer(question: str, engine: str, people: list) -> dict:
lines = [f"- {name} — {school}" for name, school in people]
text = "Školy študentov z predchádzajúcej odpovede:\n\n" + "\n".join(lines)
return _ok(question, engine, text, [], {})
def _ok(question, engine, text, urls, base) -> dict:
keep = {k: v for k, v in base.items() if k not in ("answer", "status", "reason", "delivery_check",
"source_urls", "conversation_context")}
@ -415,11 +667,11 @@ def _ok(question, engine, text, urls, base) -> dict:
"delivery_check": {"layer": "delivery", "code": "complete", "request_fingerprint": fingerprint(question, engine)}}
def _with_token(result: dict, *, wiki_context=None, people=(), options=(), question="", crzp=None,
primary="wiki") -> dict:
def _with_token(result: dict, *, wiki_context=None, entities=(), options=(), question="", crzp=None,
primary="wiki", page=None) -> dict:
wiki_context = wiki_context or {}
token = seal({"wiki": wiki_context.get("token"), "people": list(people)[:30], "options": list(options)[:6],
"question": question, "crzp": crzp, "primary": primary})
token = seal({"wiki": wiki_context.get("token"), "entities": list(entities)[:40], "options": list(options)[:6],
"question": question, "crzp": crzp, "primary": primary, "page": page})
if token:
result["conversation_context"] = {**{k: v for k, v in wiki_context.items() if k != "token"},
"token": token, "work_options": wiki_context.get("work_options", []),
@ -427,14 +679,118 @@ def _with_token(result: dict, *, wiki_context=None, people=(), options=(), quest
return result
NEXT_PAGE = re.compile(r"\b(?:dalsi\w*|dalej|viac|ostatn\w*|zvysn\w*|pokracuj\w*|zvysok)\b")
YES = re.compile(r"^\W*(?:ano|jasne|ok|okej|hej|chcem|daj|ukaz|prosim|urcite|samozrejme)(?:\s+(?:ano|prosim|daj|ukaz|ich|mi|ho|aj))*\W*$")
ORDINAL_WORDS = {"prv": 1, "druh": 2, "tret": 3, "stvrt": 4, "piat": 5, "siest": 6, "siedm": 7, "osm": 8, "deviat": 9,
"desiat": 10}
REFERENCE = re.compile(
r"(?i)\b(?:(?:t[eé]n|t[aá]|t[uú]|toho|tej|tomu|tom|tou)\s+)?"
r"(prv|druh|tret|štvrt|stvrt|piat|šiest|siest|siedm|ôsm|osm|deviat|desiat|posledn)\w*"
r"(?:\s+(?:z\s+nich|zo\s+zoznamu|v\s+poradí|v\s+poradi|študent\w*|student\w*|osob\w*|človek\w*|clovek\w*))"
r"|\b(?:číslo|cislo|č\.)\s*(\d{1,2})\b")
def _reference(question: str, entities: list[dict]) -> tuple[re.Match, dict] | None:
""""prvý z nich", "toho tretieho študenta", "číslo 2" -> the shown person it points at."""
match = REFERENCE.search(question)
if not match or not entities:
return None
if match.group(2):
number = int(match.group(2))
else:
word = normalized(match.group(1))
number = len(entities) if word.startswith("posledn") else next(
(n for stem, n in ORDINAL_WORDS.items() if word.startswith(stem)), None)
chosen = next((e for e in entities if e.get("n") == number), None)
return (match, chosen) if chosen else None
def _stamp(result: dict, question: str, engine: str) -> dict:
"""An answer prepared for a question this layer rewrote ("prvý z nich" -> the name, "2" -> the
pending question about the chosen work) is bound to the message the user sent. Any other
fingerprint stays as it is, so the client still rejects an answer that belongs elsewhere."""
internal = result.pop("_asked_as", None) or []
check = result.get("delivery_check")
if isinstance(check, dict) and check.get("request_fingerprint") in {fingerprint(q, engine) for q in internal}:
result["delivery_check"] = {**check, "request_fingerprint": fingerprint(question, engine)}
return result
_warmed = threading.Event()
def answer(question: str, *, engine: str, limit: int = 5, context_token: str | None = None, complete=None,
deliver=None) -> dict:
"""deliver: the ZP Wiki answer function (the API passes its own, tests replace it)."""
from tools.answers.answer_delivery import deliver_answer as default_deliver
from tools.answers.finalize import finalize
deliver_answer = deliver or default_deliver
from tools.crzp.answering.wiki_bridge import wiki_person
question = question.strip()
if enabled() and not _warmed.is_set(): # tag rarity for related topics, counted once in the background
_warmed.set()
_pool().submit(tag_frequencies)
return _stamp(_answer(question, engine=engine, limit=limit, context_token=context_token, complete=complete,
deliver=deliver), question, engine)
def _ask_choice(question: str, engine: str, previous: dict) -> dict:
options = previous["options"]
text = ("Odpovedz prosím číslom: " + ", ".join(f"{i} – {o['label']}" for i, o in enumerate(options, 1)) + ".")
result = _ok(question, engine, text, [], {})
result.update(status="ambiguous", reason="clarification_required",
delivery_check={"layer": "planning", "code": "clarification_required",
"request_fingerprint": fingerprint(question, engine)})
return _with_token(result, options=options, question=previous["question"])
def _next_page(question: str, engine: str, previous: dict) -> dict:
from tools.answers import pages
stored = pages.get(previous.get("page"))
entities = previous.get("entities") or []
if not stored:
return _ok(question, engine, "Ďalšie položky predchádzajúcej odpovede už nie sú k dispozícii; zopakuj otázku.",
[], {})
items, mixed = stored["items"], stored["meta"].get("mixed", False)
start = max((e.get("n", 0) for e in entities), default=0) + 1
text, urls, page_id = compose_list([], items[:PAGE], items[PAGE:], [], [], start=start, mixed=mixed)
shown = entities + [entity(item, start + i) for i, item in enumerate(items[:PAGE])]
return _with_token(_ok(question, engine, text, urls, {}), entities=shown, question=question, page=page_id,
crzp=previous.get("crzp"), primary=previous.get("primary", "wiki"))
def _schools_answer(question: str, engine: str, previous: dict) -> dict:
entities = previous["entities"]
lines = [f"{e.get('n', i)}. {e['name']} — {e.get('school') or 'škola neuvedená'}"
for i, e in enumerate(entities, 1)]
text = "Školy ľudí z predchádzajúcej odpovede:\n\n" + "\n".join(lines)
return _with_token(_ok(question, engine, text, [], {}), entities=entities, question=question,
page=previous.get("page"), crzp=previous.get("crzp"), primary=previous.get("primary", "wiki"))
def _crzp_work_answer(question: str, engine: str, chosen: dict, complete=None) -> dict:
"""A follow-up about a CRZP work of the previous list ("o čom je práca tretieho z nich?")."""
from tools.crzp.answering import answer as crzp
from tools.crzp.answering.understand import Person, understand
query = understand(question, complete=complete)
query.people = [Person(name=chosen["name"], role="author", typed=chosen["name"])]
query.sids, query.title, query.topics, query.alternatives = [chosen["sid"]], None, [], []
query.school = query.unit = None
query.from_model = True
if query.intent in ("list", "similar", "schools"):
query.intent = "about"
result = crzp._answer_query(question, query, complete)
for key in ("_shown", "_asked", "_options", "_entities"):
result.pop(key, None)
if result.get("accepted_for_delivery") is not True:
return result
body, urls = split_sources(clean_crzp(result["answer"]))
urls = list(dict.fromkeys([*urls, *(result.get("source_urls") or [])]))
text = body + ("\n\nZdroje: " + ", ".join(urls) if urls else "")
return {**result, "answer": text, "source_urls": urls,
"delivery_check": {**result["delivery_check"], "request_fingerprint": fingerprint(question, engine)}}
def _answer(question: str, *, engine: str, limit: int, context_token: str | None, complete, deliver) -> dict:
from tools.answers.answer_delivery import deliver_answer as default_deliver
deliver_answer = deliver or default_deliver
from tools.crzp.answering.wiki_bridge import wiki_match, wiki_person
previous = unseal(context_token)
if previous and previous.get("error"):
from tools.answers.answer_errors import public_failure
@ -444,15 +800,39 @@ def answer(question: str, *, engine: str, limit: int = 5, context_token: str | N
return failure
wiki_token = previous.get("wiki") if previous else context_token
low = normalized(question)
if previous and previous.get("people") and SCHOOLS.search(low) and not wiki_person(question):
return _with_token(_schools_answer(question, engine, previous["people"]), people=previous["people"],
question=question)
if previous and previous.get("options"):
if previous and previous.get("options"): # "Ktorú prácu myslíš?" was asked
chosen = _choice(low, previous["options"])
if chosen is not None:
return _answer_choice(previous["question"], chosen, engine=engine, limit=limit, complete=complete,
result = _answer_choice(previous["question"], chosen, engine=engine, limit=limit, complete=complete,
deliver=deliver_answer)
from tools.crzp.answering.wiki_bridge import wiki_match
result["_asked_as"] = [previous["question"], *result.pop("_asked_as", [])]
if result.get("accepted_for_delivery") is True and result.get("status") == "ok":
# the offer stays: "a tú druhú?" / "1" after "2" answers the same question for the other work
result = _with_token(result, wiki_context=result.get("conversation_context"),
options=previous["options"], question=previous["question"])
return result
if len(low.split()) <= 3 and not wiki_person(question):
return _ask_choice(question, engine, previous)
wants_more = bool(YES.match(low) or NEXT_PAGE.search(low) and len(low.split()) <= 6)
if previous and previous.get("page") and wants_more:
return _next_page(question, engine, previous)
if previous and previous.get("entities") and wants_more: # nothing more was offered
return _with_token(_ok(question, engine, "Všetko nájdené už je v predchádzajúcej odpovedi.", [], {}),
entities=previous["entities"], question=question, crzp=previous.get("crzp"),
primary=previous.get("primary", "wiki"))
entities = (previous or {}).get("entities") or []
if entities and SCHOOLS.search(low) and not wiki_person(question):
return _schools_answer(question, engine, previous)
asked = question
reference = _reference(question, entities) if len(entities) > 1 else None
if reference:
match, chosen = reference
if chosen.get("source") == "crzp" and chosen.get("sid"):
return _crzp_work_answer(question, engine, chosen, complete)
question = (question[:match.start()] + chosen["name"] + question[match.end():]).strip()
low, wiki_token, previous = normalized(question), None, None # a question about that person now
return {**_answer(question, engine=engine, limit=limit, context_token=None, complete=complete,
deliver=deliver), "_asked_as": [question]}
match = wiki_match(question)
if previous and previous.get("primary") == "crzp" and previous.get("crzp") and not match:
# The last answer came from CRZP ("O čom je práca Jána Nováka?" -> "2", "a kto bol oponent?").
@ -460,22 +840,35 @@ def answer(question: str, *, engine: str, limit: int = 5, context_token: str | N
followed = crzp._answer(question, complete, context_token=previous["crzp"])
rows = followed.pop("_shown", None) or []
followed.pop("_asked", None), followed.pop("_options", None)
shown = followed.pop("_entities", None) or []
if followed.get("accepted_for_delivery") is True:
urls = followed.get("source_urls") or []
text = clean_crzp(followed["answer"]) + ("\n\nZdroje CRZP: " + ", ".join(urls[:8]) if urls else "")
result = {**followed, "answer": text, "delivery_check": {**followed["delivery_check"],
"request_fingerprint": fingerprint(question, engine)}}
body, urls = split_sources(clean_crzp(followed["answer"]))
urls = list(dict.fromkeys([*urls, *(followed.get("source_urls") or [])]))[:12]
result = {**followed, "answer": body + ("\n\nZdroje: " + ", ".join(urls) if urls else ""),
"delivery_check": {**followed["delivery_check"], "request_fingerprint": fingerprint(question, engine)}}
token = (followed.get("conversation_context") or {}).get("token")
result.pop("conversation_context", None)
return _with_token(result, people=previous.get("people", []), question=question,
crzp=token or previous["crzp"], primary="crzp")
entities = [{**e, "n": i} for i, e in enumerate(shown, 1)] or previous.get("entities", [])
return _with_token(result, entities=entities, question=question, crzp=token or previous["crzp"],
primary="crzp")
person, student = (match["name"], match["student"]) if match else (None, True)
started = time.monotonic()
pool = _pool()
from tools.answers import timing
import contextvars
def timed(name, function, *args, **kwargs):
began = time.monotonic()
try:
wiki_future = pool.submit(deliver_answer, question, engine=engine, limit=limit,
return function(*args, **kwargs)
finally:
timing.step(name, time.monotonic() - began)
wiki_future = pool.submit(contextvars.copy_context().run, timed, "wiki", deliver_answer, question, engine=engine,
limit=limit,
**({"context_token": wiki_token} if wiki_token else {}))
crzp_future = pool.submit(crzp_part, question, person, complete=complete, student=student) if enabled() else None
crzp_future = pool.submit(contextvars.copy_context().run, timed, "crzp", crzp_part, question, person,
complete=complete, student=student) if enabled() else None
wiki = wiki_future.result()
try:
part = crzp_future.result(timeout=max(1.0, CRZP_SECONDS - (time.monotonic() - started))) if crzp_future else None
@ -483,8 +876,13 @@ def answer(question: str, *, engine: str, limit: int = 5, context_token: str | N
part = None # CRZP too slow this time: the wiki answer is not held back (its search stays cached)
except Exception:
part = None
finally:
pass # the shared pool stays; a slow CRZP part finishes in it and its search stays cached
return compose(asked, question, engine, wiki, part, person)
def compose(asked: str, question: str, engine: str, wiki: dict, part: dict | None, person: str | None) -> dict:
"""One answer from the ZP Wiki result and the CRZP part."""
from tools.answers.finalize import finalize
low = normalized(question)
if wiki.get("reason") in REFUSED or wiki.get("status") == "ambiguous" and (
person or wiki.get("reason") not in WIKI_CLARIFICATION_YIELDS):
part = None # refused/out of scope, or the wiki asks first (which person, which work): no CRZP
@ -496,9 +894,18 @@ def answer(question: str, *, engine: str, limit: int = 5, context_token: str | N
wiki_context = wiki.get("conversation_context")
if part is None or wiki.get("status") == "ambiguous" and part["mode"] != "explicit":
# (a wiki clarification such as "Uveď meno študenta" yields to a question CRZP answers as a whole)
people = _wiki_people(wiki) if wiki_ok else []
return _with_token(wiki, wiki_context=wiki_context, people=people, question=question) if wiki.get(
"accepted_for_delivery") is True else wiki
if wiki.get("accepted_for_delivery") is not True:
return wiki
if wiki_ok:
body, urls = split_sources(wiki["answer"])
head, items, tail, general = wiki_items(body, urls or wiki.get("source_urls") or [])
if len(items) >= 2:
shown, rest = first_page(items, [])
text, shown_urls, page_id = compose_list(head, shown, rest, tail, general, mixed=False)
result = {**wiki, "answer": text, "source_urls": shown_urls}
return _with_token(result, wiki_context=wiki_context, question=question, page=page_id,
entities=[entity(item, i) for i, item in enumerate(shown, 1)])
return _with_token(wiki, wiki_context=wiki_context, question=question)
# Different works in the two sources and the question names none: ask which one.
if person and part["mode"] == "person" and (METHODS.search(low) or ABOUT.search(low)):
options = _work_options(person, part, low)
@ -517,19 +924,43 @@ def answer(question: str, *, engine: str, limit: int = 5, context_token: str | N
delivery_check={"layer": "planning", "code": "clarification_required",
"request_fingerprint": fingerprint(question, engine)})
return _with_token(result, question=question, crzp=part.get("crzp_token"), primary="crzp")
crzp_text = "**V CRZP** (Centrálny register záverečných prác):\n\n" + part["text"]
base = wiki if wiki_ok else {"index_version": wiki.get("index_version")}
wiki_body, wiki_urls = split_sources(wiki["answer"]) if wiki_ok else ("", [])
wiki_urls = wiki_urls or (wiki.get("source_urls") or [] if wiki_ok else [])
if part["mode"] == "related": # "Ktoré témy súvisia s X?": the topics and who did them
own = wiki_people_for(part["topic"]) if part.get("topic") else []
head = [f"Témy súvisiace s témou „{part['topic']}“ (vyskytujú sa v tých istých prácach):"]
if own:
head.append(f"Samotnej téme „{part['topic']}“ sa v ZP Wiki venujú: {', '.join(own[:8])}"
+ (f" a ďalší ({len(own) - 8})." if len(own) > 8 else "."))
shown, rest = part["items"][:PAGE], part["items"][PAGE:]
text, urls, page_id = compose_list(head, shown, rest, [], [], mixed=False)
return _with_token(_ok(question, engine, text, urls, base), wiki_context=wiki_context, question=question,
entities=[entity(item, i) for i, item in enumerate(shown, 1)], page=page_id)
if part.get("items") is not None: # topic or similar works: one numbered list with the wiki people
head, items, tail, general = wiki_items(wiki_body, wiki_urls) if wiki_ok else ([], [], [], [])
if wiki_ok and not items: # a wiki answer that is no list stays whole, the CRZP works follow it
head, tail, general = [wiki_body], [], wiki_urls
if not wiki_ok:
head = ["V ZP Wiki k tomu nie je záznam. Práce na túto tému v CRZP (Centrálny register záverečných prác):"]
shown, rest = first_page(items, part["items"])
text, urls, page_id = compose_list(head, shown, rest, tail, general, mixed=True)
return _with_token(_ok(question, engine, text, urls, base), wiki_context=wiki_context, question=question,
entities=[entity(item, i) for i, item in enumerate(shown, 1)], page=page_id)
crzp_body, crzp_urls = split_sources(part["text"])
crzp_urls = list(dict.fromkeys([*crzp_urls, *part.get("urls", [])]))[:12]
entities = [{**e, "n": i} for i, e in enumerate(part.get("entities") or [], 1)]
if not wiki_ok and (part["mode"] == "explicit" or wiki.get("status") == "error"):
text, urls = part["text"], part["urls"]
text = crzp_body
elif wiki_ok:
text = wiki["answer"] + "\n\n" + crzp_text
urls = [*(wiki.get("source_urls") or []), *part["urls"]]
text = wiki_body + "\n\nV CRZP (Centrálny register záverečných prác):\n\n" + crzp_body
else:
text, urls = "V ZP Wiki k tomu nie je záznam.\n\n" + crzp_text, part["urls"]
if part["urls"]:
text += "\n\nZdroje CRZP: " + ", ".join(part["urls"][:8])
result = _ok(question, engine, text, urls, wiki if wiki_ok else {"index_version": wiki.get("index_version")})
people = (_wiki_people(wiki) if wiki_ok else []) + list(part.get("people") or [])
return _with_token(result, wiki_context=wiki_context, people=people, question=question,
text = "V ZP Wiki k tomu nie je záznam.\n\nV CRZP (Centrálny register záverečných prác):\n\n" + crzp_body
urls = list(dict.fromkeys([*wiki_urls, *crzp_urls]))
if urls:
text += "\n\nZdroje: " + ", ".join(urls)
result = _ok(question, engine, text, urls, base)
return _with_token(result, wiki_context=wiki_context, entities=entities, question=question,
crzp=part.get("crzp_token"), primary="crzp" if part["mode"] == "explicit" and not wiki_ok else "wiki")
@ -548,25 +979,36 @@ def _work_options(person: str, part: dict, low: str) -> list[dict]:
return wiki_options + crzp_only if wiki_options else []
CHOICE_FILLERS = {"tak", "teda", "ta", "tu", "ten", "to", "tej", "chcem", "daj", "prosim", "cislo", "moznost", "a", "ano",
"no", "hej", "mi", "tu", "pracu", "praca", "prace", "o", "tej", "tom", "volba", "ok", "jasne", "tú"}
NUMBER_WORDS = {"jedna": 1, "jednotku": 1, "jednotka": 1, "prvu": 1, "prva": 1, "prvy": 1, "prve": 1, "prvej": 1,
"dva": 2, "dve": 2, "dvojku": 2, "dvojka": 2, "druhu": 2, "druha": 2, "druhy": 2, "druhe": 2, "druhej": 2,
"tri": 3, "trojku": 3, "trojka": 3, "tretiu": 3, "tretia": 3, "treti": 3, "tretej": 3}
def _choice(low: str, options: list[dict]) -> dict | None:
number = re.fullmatch(r"\s*(?:cislo\s+)?(\d)\s*[.)]?\s*", low)
if number and 1 <= int(number.group(1)) <= len(options):
return options[int(number.group(1)) - 1]
if CHOICE_CRZP.search(low):
""""2", "tak 1", "tú prvú", "jednotku", "bakalársku", "z wiki" -> the chosen option."""
words = [w for w in re.findall(r"[a-z0-9]+", low) if w not in CHOICE_FILLERS]
if len(words) == 1:
word = words[0]
number = int(word) if word.isdigit() else NUMBER_WORDS.get(word)
if number is not None:
return options[number - 1] if 1 <= number <= len(options) else None
if len(words) <= 4 and CHOICE_CRZP.search(low):
return next((o for o in options if o["source"] == "crzp"), None)
if CHOICE_WIKI.search(low):
if len(words) <= 4 and CHOICE_WIKI.search(low):
return next((o for o in options if o["source"] == "wiki"), None)
return None
def _answer_choice(question: str, chosen: dict, *, engine: str, limit: int, complete=None, deliver=None) -> dict:
"""The answer to the pending question about the chosen work (the caller binds it to the reply)."""
from tools.answers.finalize import finalize
deliver_answer = deliver
from tools.crzp.answering.wiki_bridge import wiki_person
if chosen["source"] == "wiki":
adjective = {"diplom": "diplomovej", "bakal": "bakalárskej"}.get(normalized(chosen["work"])[:6].rstrip("ao"), "")
adjective = "diplomovej" if normalized(chosen["work"]).startswith("diplom") else \
"bakalárskej" if normalized(chosen["work"]).startswith("bakal") else adjective
work = normalized(chosen["work"])
adjective = "diplomovej" if work.startswith("diplom") else "bakalárskej" if work.startswith("bakal") else ""
asked = re.sub(r"\b(svojej\s+|jeho\s+|jej\s+)?(práci|práce|prácu)\b", lambda m: (m.group(1) or "") + adjective + " " + m.group(2),
question, count=1) if adjective else question
if asked == question:
@ -574,9 +1016,7 @@ def _answer_choice(question: str, chosen: dict, *, engine: str, limit: int, comp
result = deliver_answer(asked, engine=engine, limit=limit)
if result.get("accepted_for_delivery") is True and isinstance(result.get("answer"), str):
result = {**result, "answer": finalize(result["answer"], result.get("source_urls"))}
result["delivery_check"] = {**(result.get("delivery_check") or {}),
"request_fingerprint": fingerprint(question, engine)}
return result
return {**result, "_asked_as": [asked]}
person = wiki_person(question)
part = crzp_part(question, person, complete=complete)
rows = [r for r in (part or {}).get("rows", []) if chosen.get("key") and _key_of(r) == chosen["key"]]

View File

@ -20,6 +20,15 @@ DROP = [
r"Údaje sú z indexovan[^\n]*",
r"Indexovan[éý][^\n]*",
r"Zobrazený je obmedzený výber[^\n]*",
r"Zobrazený je výber podkladov[^\n]*",
r"Rozpoznané men[oá]:[^\n]*",
r"Do hľadania boli zahrnuté pojmy[^\n]*",
r"Súvislosť znamená spoločný výskyt[^\n]*",
r"Ide o obsahovú podobnosť[^\n]*",
r"Upozornenie: záverečná kontrola úplnosti sa pre časový limit nevykonala[^\n]*",
r"Počet zobrazených osôb bol obmedzený[^\n]*",
r"Zobrazené sú príklady z obmedzeného výberu[^\n]*",
r"Upozornenie: niektorú časť otázky dostupné podklady nemusia pokrývať úplne[^\n]*",
]
PHASE_TAG = re.compile(r"\s+\((?:zaznamenaný výsledok|priebežný stav|plán|zadanie|podklad|navrhovaný cieľ|téma alebo názov|"
r"úloha označená ako splnená)\)(?=\s*$)", re.M)

View File

@ -4,6 +4,7 @@ from __future__ import annotations
import contextvars
import json
import re
import os
import urllib.parse
import urllib.request
@ -266,7 +267,14 @@ def complete_with_configured_model(messages: list[dict], *, max_tokens: int | No
# Enable only for a provider/model whose capability was configured.
if output_schema is not None and os.getenv('ANSWER_STRUCTURED_OUTPUT','prompt')=='json_schema':
options['output_schema']=strict_output_schema(output_schema)
from tools.answers import timing
started=time.monotonic()
try:
result = _complete_request(messages,config,**options)
finally:
system=next((m.get('content') or '' for m in messages if m.get('role')=='system'),'')
kind=re.sub(r'^Reply with exactly ONE line of minified JSON[^\n]*\n','',system).strip().split('\n')[0]
timing.model_call(time.monotonic()-started,kind,sum(len(str(m.get('content') or '')) for m in messages))
except DeadlineExceeded:
raise # The answer budget ended; this says nothing about model health.
except Exception:

57
tools/answers/pages.py Normal file
View File

@ -0,0 +1,57 @@
"""The rest of a long list answer, for "áno, ukáž ďalších" (kept 24 hours).
A list answer shows its first items and asks whether to show more; the not yet
shown items are stored here under a random id that the signed conversation
token carries. Only rendered answer lines with their source links are stored
(the same text the user would see), never chat history. The file stays small:
expired pages are deleted on every write.
"""
from __future__ import annotations
import json
import secrets
import sqlite3
import threading
import time
from pathlib import Path
LIFETIME = 24 * 3600
_lock = threading.Lock()
def _file() -> Path:
from tools.rag.common import DB_FILE
return DB_FILE.parent / "conversation_pages.sqlite"
def _connect() -> sqlite3.Connection:
connection = sqlite3.connect(_file(), timeout=10)
connection.execute("CREATE TABLE IF NOT EXISTS pages (id TEXT PRIMARY KEY, expires REAL, payload TEXT)")
return connection
def put(items: list[dict], meta: dict | None = None) -> str | None:
"""Store the remaining items; returns their id (None when storing is impossible)."""
if not items:
return None
page_id = secrets.token_urlsafe(12)
payload = json.dumps({"items": items, "meta": meta or {}}, ensure_ascii=False)
try:
with _lock, _connect() as connection:
connection.execute("DELETE FROM pages WHERE expires < ?", (time.time(),))
connection.execute("INSERT INTO pages VALUES (?, ?, ?)", (page_id, time.time() + LIFETIME, payload))
return page_id
except (sqlite3.Error, OSError):
return None
def get(page_id: str | None) -> dict | None:
if not isinstance(page_id, str) or not page_id:
return None
try:
with _lock, _connect() as connection:
row = connection.execute("SELECT payload FROM pages WHERE id = ? AND expires >= ?",
(page_id, time.time())).fetchone()
except (sqlite3.Error, OSError):
return None
return json.loads(row[0]) if row else None

View File

@ -174,10 +174,13 @@ def direct_spec(question, resolution, *, meeting=None, route=None, relative_year
meeting=meeting.model_copy(update={'count':count})
facets=requested_facets(q)
if route is not None and route.facets:
# A meeting question asks for the whole meeting ("čo na ňom riešil") unless it names an
# A meeting question ("čo bolo na stretnutí") asks for the whole meeting unless it names an
# aspect itself; the classifier must not narrow it to e.g. results it guessed.
# Aspects the question names itself ("aké úlohy dostal" -> plans) win over the classifier's guess.
facets=[f for f in route.facets if f in facets] if route.type=='meeting' else (facets or list(route.facets))
if not facets and (mode=='meeting' or route is not None and route.type=='meeting') and re.search(
r'\b(?:riesil\w*|riesi|robil\w*|urobil\w*|spravil\w*|stihol\w*|stihla|pracoval\w*|dosiahol\w*|dosiahla|dokoncil\w*)\b',n):
facets=['progress'] # "Čo riešil na stretnutí?": what was done (status, results), not the new tasks
global_discovery=discovery_question(q)
if route is None and mode not in {'similar','supervisor','advisors','completion','meeting','period','proposal'} and global_discovery:
mode='discovery'

View File

@ -486,7 +486,7 @@ def render(draft, rows, result, plan):
for phase in phase_order:
claims=groups.get((scope,label,phase,stamp),[])
if not claims:continue
bullets=bool(re.search(r'\bbod\w*\b',normalized(plan.question))) or plan.presentation=='list' and plan.intent!='meeting'
bullets=bool(re.search(r'\bbod\w*\b',normalized(plan.question))) or plan.presentation=='list' or plan.intent=='meeting'
text=('\n' if bullets else ' ').join(dict.fromkeys(('- ' if bullets else '')+c.text+' '+cites(c) for c in claims))
dates=' (dátum záznamu: '+displayed+')' if displayed and plan.intent!='meeting' else ''
title=({'result':'Výsledky','progress':'Stav',

60
tools/answers/timing.py Normal file
View File

@ -0,0 +1,60 @@
"""Where the time of one answer goes: steps and model calls (local log only).
Every /answer writes one JSON line to data/logs/answer_timings.jsonl: the
total time, the time of the ZP Wiki and CRZP parts and the number and time of
model calls. The question itself is stored only as a short hash, so the log
holds no user text. ANSWER_TIMING_LOG=0 turns it off.
"""
from __future__ import annotations
import contextvars
import hashlib
import json
import os
import threading
import time
_current: contextvars.ContextVar[dict | None] = contextvars.ContextVar("answer_timing", default=None)
_write_lock = threading.Lock()
def start(question: str, engine: str) -> contextvars.Token:
record = {"q": hashlib.sha256(question.encode()).hexdigest()[:12], "engine": engine, "started": time.monotonic(),
"steps": {}, "model_calls": 0, "model_seconds": 0.0, "_lock": threading.Lock()}
return _current.set(record)
def step(name: str, seconds: float) -> None:
record = _current.get()
if record is not None:
with record["_lock"]:
record["steps"][name] = round(record["steps"].get(name, 0.0) + seconds, 2)
def model_call(seconds: float, kind: str = "", prompt_chars: int = 0) -> None:
record = _current.get()
if record is not None:
with record["_lock"]:
record["model_calls"] += 1
record["model_seconds"] = round(record["model_seconds"] + seconds, 2)
record.setdefault("calls", []).append({"s": round(seconds, 2), "kind": kind[:60], "chars": prompt_chars})
def finish(token: contextvars.Token, status: str | None) -> dict | None:
record = _current.get()
_current.reset(token)
if record is None:
return None
out = {k: v for k, v in record.items() if not k.startswith("_") and k != "started"}
out.update(total=round(time.monotonic() - record["started"], 2), status=status,
at=time.strftime("%Y-%m-%dT%H:%M:%S"))
if os.getenv("ANSWER_TIMING_LOG", "1") != "0":
try:
from tools.rag.common import DB_FILE
path = DB_FILE.parent / "logs" / "answer_timings.jsonl"
path.parent.mkdir(parents=True, exist_ok=True)
with _write_lock, path.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(out, ensure_ascii=False) + "\n")
except OSError:
pass
return out

View File

@ -73,6 +73,8 @@ def keep(row: dict, query: CrzpQuery) -> bool:
return False
if query.unit and not in_unit(row, query.unit):
return False
if query.published and not row.get("published"):
return False # "obhájili": an unpublished work may be undefended or have its publication deferred
return True
@ -243,7 +245,7 @@ def clarification(rows: list[dict], query: CrzpQuery) -> tuple[str, list[dict]]
groups = identities(rows, query.people[0])
if len(groups) < 2:
return None
groups = groups[:8]
groups = sorted(groups, key=lambda g: g["school"] != "Technická univerzita v Košiciach")[:8] # ZP Wiki's school first
text = ("Menu „" + query.people[0].name + "“ zodpovedá v CRZP viac pravdepodobných osôb (CRZP ich nerozlišuje "
"identifikátorom). Ktorú myslíš?\n\n" + "\n".join(f"{i}. {_identity_line(g)}" for i, g in
enumerate(groups, 1)) +
@ -292,7 +294,7 @@ def by_facets(query: CrzpQuery) -> bool:
CRZP total and the full year/type/supervisor facets are exact without reading rows."""
counted = query.intent == "count" or query.intent == "relation" and query.target_role == "supervisor"
return (counted and not query.people and not query.title and not query.sids and not query.school_name
and not query.alternatives and len(query.topics) <= 1)
and not query.alternatives and len(query.topics) <= 1 and not query.published)
def reader(query: CrzpQuery):
@ -379,7 +381,7 @@ def ranked(question: str, query: CrzpQuery, rows: list[dict]) -> list[dict]:
# ------------------------------------------------------------------ rendering
def _names(values: list[str]) -> str:
return ", ".join(display_name(v) for v in values) or "neuvedené"
return ", ".join(display_name(v) for v in values) or "neuvedený"
def _kind(row: dict) -> str:
@ -438,9 +440,29 @@ def describe(query: CrzpQuery) -> str:
parts.append(UNITS[query.unit]["label"] + " " + SCHOOLS[query.school][0])
elif query.school:
parts.append(SCHOOLS[query.school][0])
if query.published:
parts.append("zverejnené, teda obhájené")
return "; ".join(parts)
def plural(count: int) -> str:
return f"{count} {'práca' if count == 1 else 'práce' if 2 <= count <= 4 else 'prác'}"
def unpublished_note(found: dict, query: CrzpQuery) -> str:
"""How many works meeting the other conditions are not published (defence not confirmed)."""
if not query.published:
return ""
other = query.model_copy(update={"published": False})
hidden = {work_key(r) for r in found["rows"] if keep(r, other) and not r.get("published")}
if not hidden:
return ""
n = len(hidden)
lead = (f"Ďalšia {n} práca je" if n == 1 else f"Ďalšie {n} práce sú" if n <= 4 else f"Ďalších {n} prác je")
return (f"\n\n{lead} v CRZP {'nezverejnená' if n == 1 else 'nezverejnené' if n <= 4 else 'nezverejnených'}; "
"CRZP neuvádza, či ide o odklad zverejnenia alebo o neobhájenú prácu.")
# ------------------------------------------------------------------ model steps
SUMMARY_PROMPT = ("Si odborník, ktorý stručne vysvetľuje záverečné práce. Z abstraktu a kľúčových slov napíš po "
@ -536,10 +558,12 @@ def _count(question, query, found, rows) -> tuple[str, list[dict]]:
count, note = sum(values), " (súhrnný počet z CRZP)"
else:
note = f" (aspoň; prehľadaných {len(found['rows'])} z {found['total']} výsledkov hľadania)"
text = f"Počet prác v CRZP ({describe(query)}): **{count}**{note}."
if count:
text = f"Počet prác v CRZP ({describe(query)}): **{count}**{note}." + unpublished_note(found, query)
if count and len(types) > 1: # one asked type or year needs no breakdown
text += "\n\n- Podľa typu: " + ", ".join(f"{k} {v}" for k, v in types.most_common())
text += "\n- Podľa roku: " + ", ".join(f"{y}: {n}" for y, n in sorted(years.items(), reverse=True))
if count and len(years) > 1:
text += ("\n" if len(types) > 1 else "\n\n") + "- Podľa roku: " + ", ".join(
f"{y}: {n}" for y, n in sorted(years.items(), reverse=True))
return text, rows
@ -591,11 +615,12 @@ def _relation(question, query, found, rows) -> tuple[str, list[dict]]:
if query.title: # the asked work itself, if CRZP has that exact title
exact = [r for r in rows if fold(r["title"]).rstrip(" .") == fold(query.title).rstrip(" .")]
rows = exact or rows
if len(rows) <= 3:
lines = [f"- **{r['title']}** ({_kind(r)}, {r.get('year') or '?'}): {ROLE_LABEL[target]} "
f"{_names(r.get(ROLE_FIELD[target], []))} · {_link(r)}" for r in rows]
aggregate = re.search(r"najcast|najviac|najcastejs|kolk|pocet", fold(question))
if len(rows) <= 3 or query.people and len(rows) <= 8 and not aggregate: # a person's few works: each with its role
rows = sorted(rows, key=lambda r: -(r.get("year") or 0))
lines = [f"{i}. **{r['title']}** ({_kind(r)}, {r.get('year') or '?'}): {ROLE_LABEL[target]} "
f"{_names(r.get(ROLE_FIELD[target], []))}" for i, r in enumerate(rows, 1)]
return f"{ROLE_PLURAL[target]} prác ({describe(query)}):\n\n" + "\n".join(lines), rows
plural = lambda c: f"{c} {'práca' if c == 1 else 'práce' if c < 5 else 'prác'}"
if not query.people: # "Kto viedol práce o X?": the target role over the found works
counts = Counter(n for r in rows for n in r.get(ROLE_FIELD[target], []))
if (found.get("facet_only") and target == "supervisor" and not found["complete"]
@ -605,10 +630,10 @@ def _relation(question, query, found, rows) -> tuple[str, list[dict]]:
counts.update({f["label"]: f["count"] for f in part["facets"].get("Školiteľ", [])})
total = found["total"]
lines = [f"- {display_name(n)} ({plural(c)})" for n, c in sorted(counts.items(), key=lambda i: (-i[1], i[0]))[:15]]
return (f"{ROLE_PLURAL[target]} prác ({describe(query)}; {total} prác v CRZP, súhrn CRZP):\n\n" +
return (f"{ROLE_PLURAL[target]} prác ({describe(query)}; súhrn CRZP z {plural(total)}):\n\n" +
("\n".join(lines) or "- CRZP ich neuvádza")), ranked(question, query, rows)[:10]
lines = [f"- {display_name(n)} ({plural(c)})" for n, c in sorted(counts.items(), key=lambda i: (-i[1], i[0]))[:15]]
return (f"{ROLE_PLURAL[target]} prác ({describe(query)}; {len(rows)} prác v CRZP):\n\n" +
return (f"{ROLE_PLURAL[target]} prác ({describe(query)}; spočítané z {plural(len(rows))} v CRZP):\n\n" +
("\n".join(lines) or "- CRZP ich neuvádza")), ranked(question, query, rows)[:10]
person = query.people[0]
source = person.role if person.role in ROLE_FIELD else max(
@ -618,7 +643,7 @@ def _relation(question, query, found, rows) -> tuple[str, list[dict]]:
lines = [f"- {display_name(n)} ({plural(c)})" for n, c in pairs[:15]]
conditions = describe(query.model_copy(update={"people": []}))
text = (f"{ROLE_PLURAL[target]} prác, pri ktorých je {display_name(names[0]) if names else person.name} "
f"{ROLE_LABEL[source]} ({conditions + '; ' if conditions else ''}{len(rows)} prác v CRZP):\n\n" + ("\n".join(lines) or "- CRZP ich neuvádza"))
f"{ROLE_LABEL[source]} ({conditions + '; ' if conditions else ''}{plural(len(rows))} v CRZP):\n\n" + ("\n".join(lines) or "- CRZP ich neuvádza"))
return text, rows
@ -774,6 +799,12 @@ def _answer(question: str, complete=None, context_token: str | None = None) -> d
query, clarification = context.follow_up(question, query, previous)
if clarification:
result = _result(question, "ambiguous", clarification, reason="clarification_required")
if clarification.startswith("Ktorú prácu"): # the reply ("2") answers this question
if query.intent == "list" and re.search(r"\bo com\b|\bo com je\b", fold(question)):
query.intent = "about"
result["conversation_context"] = context.reissue(question, query, previous) or {
"token": context_token, "work_options": [], "meeting_options": []}
else:
result["conversation_context"] = {"token": context_token, "work_options": [], "meeting_options": []}
return result
result = _answer_query(question, query, complete)
@ -829,15 +860,33 @@ def _answer_query(question: str, query: CrzpQuery, complete=None) -> dict:
if query.unverified_only:
body = (f"Výraz „{query.topic_key()}“ sa v CRZP nenašiel; uvedené práce sú nájdené pod navrhnutým "
f"príbuzným výrazom „{'“, „'.join(query.unverified_only)}“ (návrh, nie overené synonymum).\n\n" + body)
terms = _without_suffix(query, found["texts"])
if len(terms) > 1 and query.topics:
body += "\n\nHľadané výrazy v CRZP: „" + "“, „".join(terms) + "“ (vrátane príbuzných výrazov témy)."
truncated = not found["complete"] and not found.get("facet_only") # a CRZP total needs no read rows
reasons = ["crzp_result_limit"] if truncated else []
for text, total, read in (found["incomplete"] if truncated else []):
body += (f"\n\nPoznámka: hľadanie „{text}“ vrátilo v CRZP {total} záznamov; prehľadaných bolo prvých {read}.")
body += "\n\nZdroj: Centrálny register záverečných prác (CRZP), metadáta záznamov."
result = _result(question, "ok", body, rows=used, complete=not truncated, truncated=truncated, reasons=reasons,
count=len(rows) if query.intent == "count" else None)
result["_shown"], result["_asked"] = used, asked
result["_entities"] = displayed_entities(query, body, used, rows)
return result
def _school_of(name: str, rows: list[dict], field: str) -> str:
schools = Counter(university(r) for r in rows if any(display_name(n) == name for n in r.get(field, [])))
return schools.most_common(1)[0][0] if schools else "škola neuvedená"
def displayed_entities(query: CrzpQuery, body: str, used: list[dict], rows: list[dict]) -> list[dict]:
"""The people the answer shows, in its order, with their school ("a z akých škôl sú?",
"a čo robil prvý z nich?"): the counted supervisors/opponents, else the authors of the shown works."""
if query.intent == "relation" and query.target_role in ROLE_FIELD and not query.people:
field = ROLE_FIELD[query.target_role]
names = re.findall(r"^- (.+?) \(\d+ pr[aá]c\w*\)$", body, re.M)
if names:
return [{"name": n, "school": _school_of(n, rows, field), "source": "crzp", "role": query.target_role}
for n in names]
entities = []
for row in used:
for author in row.get("authors", [])[:1]:
entities.append({"name": display_name(author), "school": university(row), "source": "crzp",
"role": "author", "sid": row.get("sid"), "title": row["title"]})
return entities

View File

@ -64,7 +64,8 @@ def issue(question: str, query: CrzpQuery, shown: list[dict], options: list[dict
rows = [r for r in shown if r.get("sid")][:MAX_SIDS]
payload = {"schema": SCHEMA, "expires": int(time.time()) + LIFETIME, "question": question,
"query": query.model_dump(mode="json", include=CARRIED - {"sids"}),
"sids": [r["sid"] for r in rows], "options": (options or [])[:8]}
"sids": [r["sid"] for r in rows], "options": (options or [])[:8],
"titles": [f"{r['title'][:120]} ({r.get('year') or '?'})" for r in rows]}
body = _b64(json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode())
token = body + "." + _b64(hmac.digest(key, body.encode(), "sha256"))
if len(token) > MAX_TOKEN:
@ -73,6 +74,28 @@ def issue(question: str, query: CrzpQuery, shown: list[dict], options: list[dict
"meeting_options": []}
def reissue(question: str, query: CrzpQuery, previous: dict) -> dict | None:
"""A token for the pending question ("a o čom je?" -> "Ktorú prácu myslíš?" -> "2"): the
reply then continues this query over the same works."""
key = _key()
if key is None:
return None
payload = {"schema": SCHEMA, "expires": int(time.time()) + LIFETIME, "question": question,
"query": query.model_dump(mode="json", include=CARRIED - {"sids"}),
"sids": previous.get("sids") or [], "options": [], "titles": previous.get("titles") or [],
"pending": True}
body = _b64(json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode())
token = body + "." + _b64(hmac.digest(key, body.encode(), "sha256"))
return {"token": token, "work_options": [], "meeting_options": []} if len(token) <= MAX_TOKEN else None
def _which(sids: list[str], titles: list[str]) -> str:
if titles and len(titles) == len(sids):
return ("Ktorú prácu myslíš?\n\n" + "\n".join(f"{i}. {t}" for i, t in enumerate(titles, 1))
+ "\n\nOdpovedz číslom.")
return "Predchádzajúca odpoveď mala viac prác; ktorú myslíš? Napíš napr. „tá druhá“ alebo názov práce."
def restore(token: str) -> dict:
key = _key()
try:
@ -103,6 +126,14 @@ def follow_up(question: str, new: CrzpQuery, previous: dict) -> tuple[CrzpQuery,
if own_subject and not CONTINUES.search(low):
return new, None # "Koľko prác viedol Staš?" stands on its own even with a token
old: CrzpQuery = previous["query"]
pending = re.fullmatch(r"\s*(?:(?:tak|ta|tu|tú|cislo|prace?|pracu)\s+)*(\d{1,2})\s*[.)]?\s*", low)
if previous.get("pending") and pending: # the answer to "Ktorú prácu myslíš?": the pending question
sids = previous.get("sids") or []
index = int(pending.group(1)) - 1
if 0 <= index < len(sids):
merged = old.model_copy(deep=True)
merged.sids, merged.searched, merged.relaxed_author, merged.unverified_only = [sids[index]], [], False, []
return merged, None
merged = old.model_copy(deep=True)
if new.people:
merged.people = new.people
@ -151,11 +182,10 @@ def follow_up(question: str, new: CrzpQuery, previous: dict) -> tuple[CrzpQuery,
if not sids:
return merged, "Predchádzajúca odpoveď nemala žiadnu prácu; uveď názov práce alebo osobu."
if ordinal >= len(sids):
return merged, f"Predchádzajúca odpoveď mala {len(sids)} {'práce' if len(sids) < 5 else 'prác'}; ktorú myslíš?"
return merged, _which(sids, previous.get("titles") or [])
merged.sids = [sids[ordinal]]
elif SINGLE.search(low) and not own_subject:
if len(sids) != 1:
return merged, ("Predchádzajúca odpoveď mala viac prác; ktorú myslíš? Napíš napr. „tá druhá“ "
"alebo názov práce.")
return merged, _which(sids, previous.get("titles") or [])
merged.sids = list(sids)
return merged, None

View File

@ -78,8 +78,11 @@ bolo boli bude budú mali majú mal mala existuje existovali koľkokrát ktorým
tí tie tých tieto tejto tento nich toho prvá prvej prvú prvý druhá druhej druhú druhý tretia tretej tretiu
posledná poslednej poslednú čo tak
zaoberali venoval venovala venovali venuje venujú písali robili skúmal skúmala skúmali riešil riešila riešili
obhájili obhájil obhájila obhájené obhájených obhajoba obhajobe úspešne úspešných úspešné zverejnené zverejnených
""".split())
STOP_FOLDED = {fold(w) for w in STOP}
# "Koľko prác obhájili", "úspešne obhájené": CRZP marks a work "Zverejnená" after a successful defence.
DEFENDED = re.compile(r"\b(?:obhaj\w*|obhajen\w*|obhajil\w*|uspesn\w*|zverejnen\w*)\b")
class Person(BaseModel):
@ -108,6 +111,7 @@ class CrzpQuery(BaseModel):
works: list[str] = Field(default_factory=list, max_length=30) # follow-up: the chosen namesake's works
searched: list[str] = Field(default_factory=list) # texts actually sent to CRZP (set by answer.find)
unverified_only: list[str] = Field(default_factory=list) # found only under unverified proposed terms
published: bool = False # "obhájili": only published works (CRZP publishes a work after its defence)
def topic_key(self) -> str:
"""The topic the alternative terms belong to (the rule phrase without the model)."""
@ -207,6 +211,7 @@ def _rules(question: str) -> CrzpQuery:
school = UNITS[unit]["school"]
query = CrzpQuery(intent=intent, years=years, work_types=work_types, school=school, unit=unit,
title=quoted.group(1).strip() if quoted else None)
query.published = bool(DEFENDED.search(fold(question)))
asked = re.search(r"\bkto (?:bol|bola|je|boli|sú|su)\b[^?]*?\b(ved[uú]c|[sš]kolite|oponent|konzultant)", low)
if asked and not query.target_role:
query.target_role = {"v": "supervisor", "š": "supervisor", "s": "supervisor", "o": "opponent"}.get(

View File

@ -345,7 +345,7 @@ def test_topic_is_searched_with_alternatives_and_the_answer_names_them(crzp):
result = A._answer("Kto pracoval na LLM v roku 2024?", complete=crzp["no_model"])
assert "LLM agent" in result["answer"] and "Jazykový model pre slovenčinu" in result["answer"]
assert result["answer"].count("**LLM agent**") == 1
assert "Hľadané výrazy v CRZP: „llm“, „jazykový model“, „GPT“" in result["answer"]
assert "Hľadané výrazy" not in result["answer"] # the searched synonyms are no part of the answer
assert {f["years"] for _, f in crzp["searches"]} == {(2024, 2024)}
@ -584,7 +584,9 @@ def test_follow_up_about_one_of_the_shown_works(crzp, signed):
assert second["status"] == "ok" and second["source_urls"] == ["https://opac.crzp.sk/?fn=detailBiblioForm&sid=AAAA0001"]
unclear = follow(crzp, "o čom je tá práca?", first)
assert unclear["status"] == "ambiguous" and unclear["accepted_for_delivery"] is True
assert unclear["conversation_context"]["token"] == first["conversation_context"]["token"]
assert "1. " in unclear["answer"] and "2. " in unclear["answer"] # the shown works, numbered
chosen = follow(crzp, "2", unclear) # the reply answers the pending question about work 2
assert chosen["status"] == "ok" and chosen["source_urls"] == ["https://opac.crzp.sk/?fn=detailBiblioForm&sid=AAAA0001"]
assert_client_accepts(unclear, "o čom je tá práca?")
opponents = follow(crzp, "a kto boli oponenti?", first)
assert "Oponenti" in opponents["answer"] and "Igor Malý" in opponents["answer"]

View File

@ -686,6 +686,28 @@ class ActivityGraph:
return clean
return concept.get("label", gid)
def display_label(self, gid):
"""The concept as a reader names it, in its base form: "Agent", "Git", "GraphRAG" (not
"agenta", "Gite"); a reviewed synonym group gives its Slovak base name."""
from tools.retrieval.synonyms import base_name, group_of
concept = self.concepts.get(gid) or {}
lemmas = set(concept.get("lemma_keys") or []) | set(concept.get("canonicals") or [])
def nominative(words, lemma): # "grafová databáza" ~ lemma "grafovy databaza" (the adjective agrees)
return len(words) == len(lemma) and all(
w == l or l.endswith("y") and len(w) == len(l) and w[:-1] == l[:-1] and w[-1] in "aey"
for w, l in zip(words, lemma))
cleaned = [re.sub(r"\s+", " ", re.sub(r"(?:https?://|www\.)\S+|\[[^\]]*\]\([^)]*\)", " ", text)).strip(" .,:;-")
for text in concept.get("literals") or []]
cleaned = [c for c in cleaned if c and len(c) <= 40]
canonicals = set(concept.get("canonicals") or [])
chosen = next((c for c in cleaned if normalized(c) in canonicals), None) or next(
(c for c in cleaned if normalized(c) in lemmas or any(nominative(normalized(c).split(), l.split()) for l in lemmas)),
None)
if chosen:
return base_name(chosen) if len(group_of(chosen)) > 1 else chosen
label = self.concept_label(gid)
return base_name(label) if len(group_of(label)) > 1 else label
def resolve(self, literal: str | None, canonical: str | None = None, *, complete=None) -> tuple[list[str], str]:
"""Concept groups for a question topic and how they were matched.
@ -895,6 +917,23 @@ _ACTION_WORDS = [
]
# The asked activity written as a noun or verb in the record itself ("Rozbehané Prodigy
# anotácie" documents annotation even when the extraction labelled it "deployed").
ACTION_TEXT = {
"annotated": r"\b(?:anot\w*|oznacova\w*|oznacen\w* dat\w*|label\w*)",
"prepared_data": r"\b(?:anot\w*|zber\w* dat|zbiera\w* dat|priprav\w* dat|cisten\w* dat|dataset\w*|korpus\w*)",
"trained": r"\b(?:trenova\w*|natrenova\w*|trening\w*|training)",
"fine_tuned": r"\b(?:dotren\w*|fine.?tun\w*|doladen\w*)",
"evaluated": r"\b(?:vyhodnot\w*|vyhodnoc\w*|evaluac\w*|evaluat\w*|testova\w*|porovna\w*|meran\w*)",
"deployed": r"\b(?:nasaden\w*|nasadil\w*|deploy\w*)",
}
def names_action(action: str, text: str) -> bool:
pattern = ACTION_TEXT.get(action)
return bool(pattern and re.search(pattern, normalized(text)))
def question_action(question: str) -> str:
"""The action class asked for; "robil/pracoval/venoval sa" is any activity."""
text = normalized(question)
@ -1011,7 +1050,7 @@ def discover(graph: ActivityGraph, spec, *, topic: str | None, canonical: str |
if not admitted(unit):
continue
performed = unit.get("phase") in {"result", "progress"}
if performed and activity["action"] in allowed:
if performed and (activity["action"] in allowed or names_action(action, unit["text"])):
bucket = "primary"
elif performed and activity["action"] in DOING:
bucket = "other" # recorded work on the topic, another kind of activity
@ -1039,6 +1078,8 @@ def discover(graph: ActivityGraph, spec, *, topic: str | None, canonical: str |
if any(e[1]["id"] == unit["id"] for b in people[key].values() for e in b):
continue
best = next((a for a in acts if a["action"] in allowed), None)
if best is None and names_action(action, unit["text"]):
best = acts[0]
if best is not None:
people[key]["primary"].append((best, unit))
elif any(a["action"] in DOING for a in acts):
@ -1102,14 +1143,12 @@ def discover(graph: ActivityGraph, spec, *, topic: str | None, canonical: str |
for activity, unit, _ in entries:
if unit["id"] not in {c["id"] for c in chosen} and len(chosen) < 3:
chosen.append({**unit, "_bucket": "work"})
notes.append(f"{person} — téma práce zodpovedá otázke; uvedené sú zaznamenané výsledky tej istej práce.")
rows.extend({k: v for k, v in row.items() if k != "_bucket"} for row in chosen)
continue
if not buckets["primary"] and buckets["other"]:
entries = sorted(buckets["other"], key=pick)
for activity, unit in entries[:2]:
chosen.append({**unit, "_bucket": "other"})
notes.append(f"{person} — záznamy dokladajú prácu s touto témou, nie však priamo požadovanú činnosť.")
rows.extend({k: v for k, v in row.items() if k != "_bucket"} for row in chosen)
continue
for bucket, take in (("primary", 3), ("secondary", 1), ("studied", 1)):
@ -1166,6 +1205,7 @@ def similar_people(graph: ActivityGraph, anchor_paths: set, *, limit=6):
concept = graph.concepts.get(gid) or {}
label = concept.get("label", "")
return (len(students[gid]) <= max(3, total * 0.2) and label not in GENERIC_KEYS
and not re.search(r"https?:|www\.|\.(?:sk|com|xyz|org|io|cz)\b", label + " " + " ".join(concept.get("literals") or []))
and concept.get("type") != "programming_language" and len(label) >= 3)
def best_by_concept(paths):
@ -1199,11 +1239,12 @@ def similar_people(graph: ActivityGraph, anchor_paths: set, *, limit=6):
# A connection resting only on the anchor's plans/readings is weak.
if max(anchor[g][0] * w for g, (w, _) in shared.items()) < 0.5:
continue
top = sorted(shared, key=lambda g: -(anchor[g][0] * shared[g][0]))[:3]
# The most specific shared concepts name the similarity ("GraphRAG, Neo4J", not "deployment").
top = sorted(shared, key=lambda g: -(anchor[g][0] * shared[g][0] * math.log(total / len(students[g]))))[:3]
ranked.append((score, path, top, [anchor[g][1] for g in top], [shared[g][1] for g in top]))
ranked.sort(key=lambda item: -item[0])
return [{"path": path, "person": rows[0]["person"], "score": round(score, 2),
"concepts": [graph.concept_label(g) for g in top], "anchor_rows": anchor_rows, "rows": rows}
"concepts": [graph.display_label(g) for g in top], "anchor_rows": anchor_rows, "rows": rows}
for score, path, top, anchor_rows, rows in ranked[:limit]]

View File

@ -1082,6 +1082,7 @@ def answer_from_index(
)
][:4]
)
anchor_evidence = anchor_evidence[:4] # the compared person's records behind the shared concepts
lines.append("Podklady k porovnávanej osobe:")
for row in anchor_evidence:
lines.append(f'- {row["person"]}; {row.get("work_label") or "profil"}: {row["text"]}')

View File

@ -8,6 +8,7 @@ all find the same topic. Groups are reviewed by hand; no synonym is guessed.
from __future__ import annotations
import re
from functools import lru_cache
from tools.answers.context_policy import normalized
@ -36,6 +37,9 @@ GROUPS = [
["oprava gramatiky", "grammatical error correction", "korekcia textu", "oprava interpunkcie"],
["generovanie otázok", "question generation"],
["rozpoznávanie emócií", "emotion recognition"],
["skryté Markovove modely", "HMM", "hidden Markov model", "skrytý Markovov model"],
["konvolučná neurónová sieť", "CNN", "convolutional neural network"],
["rekurentná neurónová sieť", "RNN", "recurrent neural network"],
]
_ENDING = r"\w{0,4}"
@ -64,13 +68,22 @@ def phrase_pattern(text: str) -> str:
return r"(?<![\w-])" + r"\s+".join(parts) + r"(?![\w-])"
_COMPILED = [(group, [re.compile(phrase_pattern(item)) for item in group]) for group in GROUPS]
# The first letters of every member: most phrases share none and need no regex at all.
_PREFIXES = {_key(item).split()[0][:3] for group in GROUPS for item in group}
@lru_cache(maxsize=65536)
def _group_index(key: str) -> int:
if not key or key.split()[0][:3] not in _PREFIXES:
return -1
return next((i for i, (_, patterns) in enumerate(_COMPILED) if any(p.fullmatch(key) for p in patterns)), -1)
def group_of(text: str) -> list[str]:
"""The reviewed group the phrase belongs to (any case form), else [text]."""
key = _key(text)
for group in GROUPS:
if any(re.fullmatch(phrase_pattern(item), key) for item in group):
return group
return [text]
index = _group_index(_key(text))
return GROUPS[index] if index >= 0 else [text]
def base_name(text: str) -> str: