Balíky nástrojov

This commit is contained in:
Ján Pták 2026-10-26 23:43:10 +01:00
parent aaa0a65f93
commit ed403e7490
9 changed files with 856 additions and 19 deletions

124
README.md
View File

@ -272,9 +272,15 @@ zp-agent/
│ ├── test_configuration.py
│ └── test_security.py
│
├── packages/
│ └── openwebui/
│ ├── rag/tool.py
│ └── graphrag/tool.py
│
├── scripts/
│ ├── start_project.sh
│ ├── compose.sh
│ ├── build_tool_packages.py
│ └── verify_project.py
│
├── data/
@ -295,6 +301,9 @@ zp-agent/
`retrieval/` zdieľa dátový kontrakt, publikačnú politiku a delenie podľa rozpočtu.
`runtime/` pripravuje model/tokenizér a kontroluje štart a dostupnosť služieb.
`sync/` spravuje Git, frontu a spoločné publikovanie SQLite/Neo4j indexov.
`packages/openwebui/` obsahuje samostatne inštalovateľné Tools. Každý
`tool.py` potrebuje iba štandardnú knižnicu a Pydantic dostupný v OpenWebUI;
neimportuje backend ani druhý balík. Je to klient existujúceho API.
V koreňovom `tools/` zostáva iba `__init__.py`.
Na Gite je z dokumentácie iba tento hlavný README. Vedľajšie návody,
@ -1048,6 +1057,110 @@ Výsledná odpoveď má byť grounded v ZP Wiki a má používať relevantný
`source_url`. Táto pôvodná cesta s voľným následným generovaním sama
nezaručuje kontroly doručenia pripraveného Pipe.
## Samostatné RAG a GraphRAG Tools pre OpenWebUI
| Nástroj | Zdroj na Gite | Funkcia | Backend |
|---|---|---|---|
| ZP Wiki RAG | [rag/tool.py](packages/openwebui/rag/tool.py) | `retrieve_zpwiki_context(query)` | `POST /rag` |
| ZP Wiki GraphRAG | [graphrag/tool.py](packages/openwebui/graphrag/tool.py) | `retrieve_zpwiki_graph(query)` | `POST /graphrag` |
Oba sú natívne **Tools**, ktoré model môže zavolať počas odpovedania.
Pipe opísaný vyššie je samostatná funkcia na riadené doručenie odpovede.
Nový autorizovaný `/graphrag` používa rovnaký API kľúč a chybový kontrakt
ako `/rag`. Zostáva mimo pôvodnej OpenAPI schémy, ktorú existujúci RAG
klient používa bez zmeny.
### Vytvorenie inštalovateľných balíkov
```bash
.venv/bin/python scripts/build_tool_packages.py
```
Vzniknú `data/tool_packages/zpwiki_rag.json`, `zpwiki_graphrag.json`
a príslušné ZIP archívy. JSON je zoznam objektov vo formáte importu
Tools v OpenWebUI 0.9.4: ID, názov, Python kód, metadáta a prázdne
prístupové granty. Neobsahuje kľúče ani používateľské údaje.
ZIP obsahuje tento importný JSON ako `openwebui-tool.json`, `tool.py`,
`config.example.json`, `valves.schema.json`, návod `INSTALL.txt` a manifest
SHA-256 jednotlivých súborov. Opakovaný build v rovnakom prostredí je
deterministický. Zdrojové súbory a builder sú na Gite; vygenerované
archívy, importné JSON a kontrolné výstupy ostávajú ignorované v `data/`.
### Inštalácia každého nástroja
1. V OpenWebUI otvorte **Workspace → Tools → Import** a vyberte
`zpwiki_rag.json` alebo `zpwiki_graphrag.json`. ZIP sa najprv rozbaľuje;
samotný ZIP ani `.py` nie sú súbory pre toto importné tlačidlo.
2. Alternatívne cez **Create** vložte celý obsah príslušného `tool.py`.
Pri načítaní z Gitu otvorte súbor v repozitári, kliknite na **Raw**
a použite túto URL cez import/načítanie z URL. Každý nástroj sa
inštaluje samostatne; backendový projekt netreba inštalovať do OpenWebUI.
3. Vo **Valves** daného nástroja nastavte hodnoty z tabuľky nižšie.
4. Nástroj zapnite v novom chate alebo ho pripojte k vlastnému modelu.
Použite model s podporou volania nástrojov. Pre RAG skúste otázku na
text alebo názov práce; pre GraphRAG porovnanie osôb a rokov.
5. Prístup sprístupnite iba požadovaným účtom. Importné balíky neurčujú
verejné zdieľanie; granty sa nastavujú až v školskom UI.
Import vyžaduje povolenie Tools Import. Možnosť vytvárať kód a meniť
Valves závisí od práv účtu. Na školskom UI sa zatiaľ nič neinštaluje;
čaká sa na profesora a sieťové sprístupnenie backendu. Postupy zodpovedajú
[oficiálnej dokumentácii Tools](https://docs.openwebui.com/features/extensibility/plugin/tools/).
### Konfigurácia Valves
| Nastavenie | RAG | GraphRAG |
|---|---|---|
| `BACKEND_URL` | Adresa API bez `/rag` | Rovnaká adresa API bez `/graphrag` |
| `BACKEND_API_KEY` | Hodnota `SEARCH_API_KEY` | Rovnaký vyhľadávací kľúč |
| `TIMEOUT_SECONDS` | Predvolene 30, rozsah 1–120 | Predvolene 30, rozsah 1–120 |
| `LIMIT` | Predvolene 5, rozsah 1–20 | Predvolene 40, rozsah 1–100 |
| `PUBLISHED_ONLY` | Predvolene true | Predvolene true |
| `MAX_PER_DOCUMENT` | Predvolene 1, rozsah 0–10 | Nepoužíva sa |
| `MAX_ENTITIES` | Nepoužíva sa | Predvolene 8, rozsah 1–20 |
Volanie zachová celú otázku a upraví iba vonkajšie medzery. Limit otázky
je 500 znakov. Kľúč nie je parametrom nástroja pre model ani súčasťou
importného JSON; nastavuje sa iba vo Valves. Nástroje samy negenerujú
odpovede a nepotrebujú kľúč k jazykovému modelu.
HTTP požiadavky idú zo servera OpenWebUI. `localhost` preto neoznačuje
vývojový počítač. V spoločnej Docker sieti môže byť adresa
`http://zp-agent-api:8000`; pri školskom serveri treba dostupnú adresu
dohodnutú s administrátorom. Cesty s prefixom reverzného proxy sú podporované.
`PUBLISHED_ONLY=true` skryje iba explicitné `published=false`; chýbajúca
hodnota sa zahrnie. Hodnota `MAX_PER_DOCUMENT=0` ruší toto obmedzenie.
### Výstup, chyby a kontroly
Výstup je JSON so stavom, dôvodom, zdrojmi, verziou dát a úplnosťou.
RAG prenáša úsporný kontext; GraphRAG fakty, cesty, obmedzenia a agregáciu.
Podklady sa v klientovi potichu neorezávajú ani neprepisujú. Zachovajú sa
aj `no_answer` a `ambiguous`; chyba spojenia sa nevydáva za prázdny výsledok.
HTTP chyby, timeout, neplatná konfigurácia, neplatný kontrakt a odpoveď
väčšia ako 2 MiB vracajú stav `error` bez čiastočných dôkazov. Presmerovania
sa neakceptujú, aby sa kľúč neposlal na inú adresu. Diagnostika a raw chyby
nie sú vstupom modelu.
`output_kind=retrieval_context` a `requires_final_answer_verification=true`
označujú, že nejde o schválený text odpovede. Priame zapnutie Tools pri
bežnom LLM nezaručuje faktickú správnosť jeho následnej parafrázy.
Existujúci Pipe a `/answer` naďalej vykonávajú kontrolu a renderer;
ich `accepted_for_delivery` sa nesmie obísť. Hlavný agent, ktorý bude
nástroje automaticky vyberať a riadiť finálne doručenie, je ďalší blok.
Lokálne testy:
```bash
.venv/bin/python -m pytest -q test/test_openwebui_packages.py
.venv/bin/python scripts/verify_project.py --git-network
```
Testy používajú skutočný dočasný HTTP server na loopback rozhraní a
FastAPI s riadenými dátami. Overujú autorizáciu, zachovanie otázky,
zdrojov a úplnosti, chyby a timeouty, zákaz presmerovaní aj import JSON/ZIP
v izolovanom interpreteri bez backendového projektu. Externý LLM sa nevolá.
## Retrieval evaluácia
Frozen retrieval benchmark sa spúšťa napríklad:
@ -1257,13 +1370,14 @@ Kompletná lokálna kontrola s bežiacim projektom a `.env`:
.venv/bin/python scripts/verify_project.py --git-network
```
Skript skompiluje všetky Python súbory v aplikácii, nástrojoch, testoch
Skript skompiluje všetky Python súbory v aplikácii, nástrojoch, balíkoch, testoch
a skriptoch, overí interné importy a syntax shell skriptov, Git,
Compose, API, spoločnú verziu indexov, integritu SQLite a kompletnú
regresiu vrátane lokálneho Neo4j. Prepínač `--git-network` navyše iba
načíta hlavičku nastavenej wiki vetvy cez existujúce SSH pripojenie.
Počas pytestu sú externé spojenia zablokované a kľúč generujúceho modelu
odstránený. Výsledky a log ostávajú lokálne v `data/`.
odstránený. HTTP transport testy povoľujú iba svoj dočasný loopback port.
Výsledky a log ostávajú lokálne v `data/`.
Kontrola syntaxe najdôležitejších upravovaných modulov:
@ -1325,8 +1439,10 @@ nové otázky; opakované meranie odhalených otázok je regresia.
### 3. OpenWebUI nástroje
Pripraviť samostatne inštalovateľný RAG nástroj a GraphRAG nástroj.
Každý komponent bude mať vlastný adresár, rozhranie a testy.
Samostatné zdroje sú v `packages/openwebui/rag/` a `graphrag/`.
Builder vytvára importné JSON a ZIP s konfiguráciou, schémou a odtlačkami;
komunikácia s backendom má lokálne HTTP testy. Inštalácia na školskom
OpenWebUI čaká na oprávnenia a dostupnosť backendu.
### 4. Hlavný agent

View File

@ -99,7 +99,7 @@ app = FastAPI(
@app.exception_handler(StarletteHTTPException)
async def retrieval_http_error(request: Request, error: StarletteHTTPException):
if request.url.path not in {'/rag', '/search'}:
if request.url.path not in {'/rag', '/search', '/graphrag'}:
return await http_exception_handler(request, error)
reason = {400: 'invalid_query', 401: 'unauthorized', 403: 'forbidden',
404: 'not_found', 405: 'method_not_allowed', 422: 'invalid_request',
@ -110,7 +110,7 @@ async def retrieval_http_error(request: Request, error: StarletteHTTPException):
@app.exception_handler(RequestValidationError)
async def retrieval_validation_error(request: Request, error: RequestValidationError):
if request.url.path not in {'/rag', '/search'}:
if request.url.path not in {'/rag', '/search', '/graphrag'}:
return await request_validation_exception_handler(request, error)
# Retain useful validation locations/types without echoing submitted secrets.
details = [{'loc': row['loc'], 'type': row['type'], 'msg': row['msg']} for row in error.errors()]
@ -159,7 +159,7 @@ async def add_security_headers(
try:
response = await call_next(request)
except Exception:
if request.url.path not in {'/rag', '/search'}:
if request.url.path not in {'/rag', '/search', '/graphrag'}:
raise
# Serialization failures must obey the same contract as backend errors.
logger.exception('Neočakávaná chyba výstupu retrieval API.')

View File

@ -42,11 +42,12 @@ from tools.rag.common import (
)
from tools.rag.rag_utils import build_rag_context
from tools.rag.llm_context import compact_rag_payload
from tools.graphrag.graph_search import search_graph, DEFAULT_LIMIT, DEFAULT_MAX_ENTITIES
from tools.rag.search_utils import search_database
from tools.sync.wiki_git import expected_branch
from tools.retrieval.publication import DEFAULT_PUBLISHED_ONLY
from tools.runtime.tool_health import tools_readiness
from tools.retrieval.retrieval_contract import RetrievalResponse, RetrievalSource, RetrievalErrorResponse
from tools.retrieval.retrieval_contract import RetrievalResponse, RetrievalSource, RetrievalErrorResponse, error_reason
from tools.answers.answer_delivery import deliver_answer
@ -166,6 +167,13 @@ class SyncRequest(
)
class GraphRequest(StrictRequestModel):
query: str = Field(min_length=1, max_length=500)
limit: int = Field(default=DEFAULT_LIMIT, ge=1, le=100)
max_entities: int = Field(default=DEFAULT_MAX_ENTITIES, ge=1, le=20)
published_only: bool = DEFAULT_PUBLISHED_ONLY
class AnswerRequest(StrictRequestModel):
query: str = Field(min_length=1, max_length=500)
engine: Literal['rag', 'graphrag'] = 'rag'
@ -329,14 +337,7 @@ def raise_retrieval_error(
),
) from error
if isinstance(
error,
(
FileNotFoundError,
sqlite3.Error,
RuntimeError,
),
):
if error_reason(error) == 'retrieval_unavailable':
logger.exception(
"Retrieval nie je dostupný."
)
@ -615,6 +616,16 @@ def rag(
)
@router.post('/graphrag', dependencies=[Depends(require_search_api_key)], include_in_schema=False)
def graph_retrieval(request: GraphRequest) -> dict[str, Any]:
"""Native GraphRAG Tool transport; the legacy OpenAPI schema stays stable."""
try:
return search_graph(request.query, limit=request.limit, max_entities=request.max_entities,
published_only=request.published_only)
except Exception as error:
raise_retrieval_error(error)
@router.post(
"/search",
dependencies=[

View File

@ -0,0 +1,143 @@
"""
title: ZP Wiki GraphRAG
description: Grafové vyhľadávanie vzťahov, prác a agregácií v ZP Wiki.
version: 1.0.0
required_open_webui_version: 0.9.4
"""
import asyncio
import json
import urllib.error
import urllib.parse
import urllib.request
from pydantic import BaseModel, Field
TOOL_ID = 'zpwiki_graphrag'
ENDPOINT = '/graphrag'
MAX_RESPONSE_BYTES = 2 * 1024 * 1024
class _NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
# Never forward the backend credential to a redirected host.
return None
class Tools:
class Valves(BaseModel):
BACKEND_URL: str = Field(default='', description='Adresa ZP Agent API dostupná zo servera OpenWebUI, bez /graphrag')
BACKEND_API_KEY: str = Field(default='', description='SEARCH_API_KEY backendu',
json_schema_extra={'input': {'type': 'password'}})
TIMEOUT_SECONDS: int = Field(default=30, ge=1, le=120)
LIMIT: int = Field(default=40, ge=1, le=100)
PUBLISHED_ONLY: bool = Field(default=True, description='Skryť iba explicitné published=false; chýbajúca hodnota sa zahrnie')
MAX_ENTITIES: int = Field(default=8, ge=1, le=20)
def __init__(self):
self.valves = self.Valves()
def _error(self, reason, http_status=None):
result = {'status': 'error', 'reason': reason, 'sources': [], 'index_version': None,
'complete': False, 'truncated': False, 'truncation_reasons': [],
'tool_id': TOOL_ID, 'output_kind': 'retrieval_context',
'requires_final_answer_verification': True}
if http_status is not None:
result['http_status'] = http_status
return result
def _url(self):
url = self.valves.BACKEND_URL.strip().rstrip('/')
parsed = urllib.parse.urlsplit(url)
if (parsed.scheme not in {'http', 'https'} or not parsed.hostname or
parsed.username is not None or parsed.password is not None or
parsed.query or parsed.fragment or any(char.isspace() for char in url)):
raise ValueError('Invalid backend URL')
parsed.port # Validate the port without exposing the configured URL.
key = self.valves.BACKEND_API_KEY.strip()
if not key or '\r' in key or '\n' in key:
raise ValueError('Missing or invalid backend key')
return url + ENDPOINT, key
def _validate(self, result):
required = {'status', 'reason', 'sources', 'index_version', 'complete', 'truncated', 'truncation_reasons'}
if not isinstance(result, dict) or not required <= result.keys():
raise ValueError('Missing response contract')
if result['status'] not in {'ok', 'no_answer', 'ambiguous', 'error'}:
raise ValueError('Invalid status')
if result['reason'] is not None and not isinstance(result['reason'], str):
raise ValueError('Invalid reason')
if result['index_version'] is not None and not isinstance(result['index_version'], dict):
raise ValueError('Invalid version')
if result['complete'] is not None and type(result['complete']) is not bool:
raise ValueError('Invalid completeness')
if type(result['truncated']) is not bool or (result['truncated'] and result['complete'] is not False):
raise ValueError('Contradictory completeness')
if not isinstance(result['truncation_reasons'], list) or not all(isinstance(x, str) for x in result['truncation_reasons']):
raise ValueError('Invalid truncation reasons')
if not isinstance(result['sources'], list):
raise ValueError('Invalid sources')
for source in result['sources']:
if (not isinstance(source, dict) or not isinstance(source.get('source_id'), str) or
not isinstance(source.get('document_path'), str) or
(source.get('source_url') is not None and not isinstance(source['source_url'], str))):
raise ValueError('Invalid source reference')
for field in ['facts', 'paths']:
if result['status'] == 'ok' and (not isinstance(result.get(field), list) or
not all(isinstance(row, dict) for row in result[field])):
raise ValueError('Missing graph evidence')
if result['truncation_reasons'] and not result['truncated']:
raise ValueError('Unreported truncation')
# Diagnostics can contain repeated evidence and are never model input.
clean = {key: value for key, value in result.items() if key not in {'diagnostics', 'detail', 'answer', 'accepted_for_delivery', 'attempts'}}
return {**clean, 'tool_id': TOOL_ID, 'output_kind': 'retrieval_context',
'requires_final_answer_verification': True}
def _request(self, query):
try:
url, key = self._url()
except (ValueError, TypeError):
return self._error('invalid_configuration')
body = {'query': query, 'limit': self.valves.LIMIT,
'published_only': self.valves.PUBLISHED_ONLY,
'max_entities': self.valves.MAX_ENTITIES}
request = urllib.request.Request(url, data=json.dumps(body, ensure_ascii=False).encode(),
headers={'Content-Type': 'application/json', 'Accept': 'application/json',
'Authorization': 'Bearer ' + key}, method='POST')
try:
opener = urllib.request.build_opener(_NoRedirect())
with opener.open(request, timeout=self.valves.TIMEOUT_SECONDS) as response:
raw = response.read(MAX_RESPONSE_BYTES + 1)
except urllib.error.HTTPError as error:
code = error.code
error.close()
reason = {401: 'unauthorized', 403: 'forbidden', 404: 'backend_endpoint_not_found',
400: 'invalid_query', 422: 'invalid_request', 429: 'backend_busy',
503: 'retrieval_unavailable'}.get(code, 'backend_http_error')
return self._error('redirect_not_allowed' if 300 <= code < 400 else reason, code)
except (TimeoutError, urllib.error.URLError, OSError) as error:
timeout = isinstance(error, TimeoutError) or isinstance(getattr(error, 'reason', None), TimeoutError)
return self._error('backend_timeout' if timeout else 'backend_unavailable')
if len(raw) > MAX_RESPONSE_BYTES:
return self._error('response_too_large')
try:
return self._validate(json.loads(raw))
except (ValueError, TypeError, KeyError):
return self._error('invalid_backend_response')
async def retrieve_zpwiki_graph(self, query: str) -> str:
"""Vyhľadaj grafové vzťahy, spoločné atribúty, práce a počty v ZP Wiki.
Zachovaj celú otázku vrátane mien, typu práce, roku a ďalších podmienok.
Výstup je JSON podkladov, nie overená finálna odpoveď. Rešpektuj status,
complete, truncated, zdroje a verziu dát. Pri chybe nič nedomýšľaj.
:param query: Celá vyhľadávacia otázka, najviac 500 znakov.
:return: Fakty, grafové cesty, agregácia, zdroje a metadáta alebo označená chyba.
"""
if not isinstance(query, str) or not 1 <= len(query.strip()) <= 500:
result = self._error('invalid_query')
else:
try:
result = await asyncio.to_thread(self._request, query.strip())
except Exception:
result = self._error('backend_unavailable')
return json.dumps(result, ensure_ascii=False, separators=(',', ':'))

View File

@ -0,0 +1,141 @@
"""
title: ZP Wiki RAG
description: Hybridné vyhľadávanie textových podkladov v ZP Wiki.
version: 1.0.0
required_open_webui_version: 0.9.4
"""
import asyncio
import json
import urllib.error
import urllib.parse
import urllib.request
from pydantic import BaseModel, Field
TOOL_ID = 'zpwiki_rag'
ENDPOINT = '/rag'
MAX_RESPONSE_BYTES = 2 * 1024 * 1024
class _NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
# Never forward the backend credential to a redirected host.
return None
class Tools:
class Valves(BaseModel):
BACKEND_URL: str = Field(default='', description='Adresa ZP Agent API dostupná zo servera OpenWebUI, bez /rag')
BACKEND_API_KEY: str = Field(default='', description='SEARCH_API_KEY backendu',
json_schema_extra={'input': {'type': 'password'}})
TIMEOUT_SECONDS: int = Field(default=30, ge=1, le=120)
LIMIT: int = Field(default=5, ge=1, le=20)
PUBLISHED_ONLY: bool = Field(default=True, description='Skryť iba explicitné published=false; chýbajúca hodnota sa zahrnie')
MAX_PER_DOCUMENT: int = Field(default=1, ge=0, le=10)
def __init__(self):
self.valves = self.Valves()
def _error(self, reason, http_status=None):
result = {'status': 'error', 'reason': reason, 'sources': [], 'index_version': None,
'complete': False, 'truncated': False, 'truncation_reasons': [],
'tool_id': TOOL_ID, 'output_kind': 'retrieval_context',
'requires_final_answer_verification': True}
if http_status is not None:
result['http_status'] = http_status
return result
def _url(self):
url = self.valves.BACKEND_URL.strip().rstrip('/')
parsed = urllib.parse.urlsplit(url)
if (parsed.scheme not in {'http', 'https'} or not parsed.hostname or
parsed.username is not None or parsed.password is not None or
parsed.query or parsed.fragment or any(char.isspace() for char in url)):
raise ValueError('Invalid backend URL')
parsed.port # Validate the port without exposing the configured URL.
key = self.valves.BACKEND_API_KEY.strip()
if not key or '\r' in key or '\n' in key:
raise ValueError('Missing or invalid backend key')
return url + ENDPOINT, key
def _validate(self, result):
required = {'status', 'reason', 'sources', 'index_version', 'complete', 'truncated', 'truncation_reasons'}
if not isinstance(result, dict) or not required <= result.keys():
raise ValueError('Missing response contract')
if result['status'] not in {'ok', 'no_answer', 'ambiguous', 'error'}:
raise ValueError('Invalid status')
if result['reason'] is not None and not isinstance(result['reason'], str):
raise ValueError('Invalid reason')
if result['index_version'] is not None and not isinstance(result['index_version'], dict):
raise ValueError('Invalid version')
if result['complete'] is not None and type(result['complete']) is not bool:
raise ValueError('Invalid completeness')
if type(result['truncated']) is not bool or (result['truncated'] and result['complete'] is not False):
raise ValueError('Contradictory completeness')
if not isinstance(result['truncation_reasons'], list) or not all(isinstance(x, str) for x in result['truncation_reasons']):
raise ValueError('Invalid truncation reasons')
if not isinstance(result['sources'], list):
raise ValueError('Invalid sources')
for source in result['sources']:
if (not isinstance(source, dict) or not isinstance(source.get('source_id'), str) or
not isinstance(source.get('document_path'), str) or
(source.get('source_url') is not None and not isinstance(source['source_url'], str))):
raise ValueError('Invalid source reference')
if not isinstance(result.get('context'), str) and result['status'] == 'ok':
raise ValueError('Missing text context')
if result['truncation_reasons'] and not result['truncated']:
raise ValueError('Unreported truncation')
# Diagnostics can contain repeated evidence and are never model input.
clean = {key: value for key, value in result.items() if key not in {'diagnostics', 'detail', 'answer', 'accepted_for_delivery', 'attempts'}}
return {**clean, 'tool_id': TOOL_ID, 'output_kind': 'retrieval_context',
'requires_final_answer_verification': True}
def _request(self, query):
try:
url, key = self._url()
except (ValueError, TypeError):
return self._error('invalid_configuration')
body = {'query': query, 'limit': self.valves.LIMIT,
'published_only': self.valves.PUBLISHED_ONLY,
'max_per_document': self.valves.MAX_PER_DOCUMENT, 'include_diagnostics': False}
request = urllib.request.Request(url, data=json.dumps(body, ensure_ascii=False).encode(),
headers={'Content-Type': 'application/json', 'Accept': 'application/json',
'Authorization': 'Bearer ' + key}, method='POST')
try:
opener = urllib.request.build_opener(_NoRedirect())
with opener.open(request, timeout=self.valves.TIMEOUT_SECONDS) as response:
raw = response.read(MAX_RESPONSE_BYTES + 1)
except urllib.error.HTTPError as error:
code = error.code
error.close()
reason = {401: 'unauthorized', 403: 'forbidden', 404: 'backend_endpoint_not_found',
400: 'invalid_query', 422: 'invalid_request', 429: 'backend_busy',
503: 'retrieval_unavailable'}.get(code, 'backend_http_error')
return self._error('redirect_not_allowed' if 300 <= code < 400 else reason, code)
except (TimeoutError, urllib.error.URLError, OSError) as error:
timeout = isinstance(error, TimeoutError) or isinstance(getattr(error, 'reason', None), TimeoutError)
return self._error('backend_timeout' if timeout else 'backend_unavailable')
if len(raw) > MAX_RESPONSE_BYTES:
return self._error('response_too_large')
try:
return self._validate(json.loads(raw))
except (ValueError, TypeError, KeyError):
return self._error('invalid_backend_response')
async def retrieve_zpwiki_context(self, query: str) -> str:
"""Vyhľadaj textové podklady o prácach, témach a osobách v ZP Wiki.
Zachovaj celú otázku vrátane mien, typu práce, roku a ďalších podmienok.
Výstup je JSON podkladov, nie overená finálna odpoveď. Rešpektuj status,
complete, truncated, zdroje a verziu dát. Pri chybe nič nedomýšľaj.
:param query: Celá vyhľadávacia otázka, najviac 500 znakov.
:return: Kontext, zdroje a spoločné metadáta alebo bezpečne označená chyba.
"""
if not isinstance(query, str) or not 1 <= len(query.strip()) <= 500:
result = self._error('invalid_query')
else:
try:
result = await asyncio.to_thread(self._request, query.strip())
except Exception:
result = self._error('backend_unavailable')
return json.dumps(result, ensure_ascii=False, separators=(',', ':'))

View File

@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""Build standalone OpenWebUI Tools archives; generated files stay in data/."""
from __future__ import annotations
import argparse
import ast
import hashlib
import importlib.util
import json
from pathlib import Path
import sys
import zipfile
ROOT = Path(__file__).resolve().parents[1]
ENGINES = ('rag', 'graphrag')
def json_bytes(value) -> bytes:
return (json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + '\n').encode()
def build_package(engine: str, output: Path) -> dict:
if engine not in ENGINES:
raise ValueError('Unknown tool package')
source = ROOT / 'packages/openwebui' / engine / 'tool.py'
content = source.read_bytes()
header = ast.get_docstring(ast.parse(content.decode()))
metadata = dict(line.strip().split(': ', 1) for line in header.splitlines() if ': ' in line)
name = 'zpwiki_package_' + engine
spec = importlib.util.spec_from_file_location(name, source)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
spec.loader.exec_module(module)
config = module.Tools.Valves().model_dump()
schema = module.Tools.Valves.model_json_schema()
finally:
sys.modules.pop(name, None)
instructions = (
f'ZP Wiki {engine.upper()} – OpenWebUI Tool\n\n'
'ZIP najprv rozbal. Vo Workspace > Tools > Import vyber openwebui-tool.json.\n'
'Alebo cez Create vytvor nástroj a vlož celý obsah tool.py do editora.\n'
'Pri importe cez URL použi raw URL súboru tool.py, nie ZIP ani stránku repozitára.\n'
'Vo Valves nastav BACKEND_URL a BACKEND_API_KEY = SEARCH_API_KEY backendu.\n'
'Adresa musí byť dostupná zo servera OpenWebUI; neobsahuje koncový endpoint.\n'
'Backend aj oba nástroje používajú rovnaký dátový kontrakt.\n'
'Výstup sú vyhľadané podklady, nie schválený finálny text.\n'
'Pre kontrolované doručenie používaj existujúci Pipe a /answer.\n'
'Žiadny ďalší nástroj ani lokálny Python balík zp-agent nie je potrebný.\n'
'Podrobný postup, testy a obmedzenia sú v hlavnom README projektu.\n')
import_form = [{'id': module.TOOL_ID, 'name': metadata['title'], 'content': content.decode(),
'meta': {'description': metadata['description'], 'manifest': metadata},
'access_grants': []}]
files = {'tool.py': content, 'openwebui-tool.json': json_bytes(import_form), 'config.example.json': json_bytes(config),
'valves.schema.json': json_bytes(schema), 'INSTALL.txt': instructions.encode()}
manifest = {'schema_version': 1, 'kind': 'openwebui_tool', 'package_id': module.TOOL_ID,
'version': metadata['version'], 'minimum_openwebui_version': metadata['required_open_webui_version'],
'backend_endpoint': module.ENDPOINT, 'entrypoint': 'tool.py', 'output_kind': 'retrieval_context',
'files': {name: hashlib.sha256(data).hexdigest() for name, data in files.items()}}
files['manifest.json'] = json_bytes(manifest)
output.mkdir(parents=True, exist_ok=True)
destination = output / (module.TOOL_ID + '.zip')
import_file = output / (module.TOOL_ID + '.json')
import_file.write_bytes(files['openwebui-tool.json'])
# Fixed timestamps and sorted entries make a rebuild byte-for-byte identical.
with zipfile.ZipFile(destination, 'w', compression=zipfile.ZIP_DEFLATED) as archive:
for name, data in sorted(files.items()):
info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
info.create_system = 3
info.external_attr = 0o100644 << 16
archive.writestr(info, data)
return {'package_id': module.TOOL_ID, 'archive': str(destination), 'import_file': str(import_file),
'sha256': hashlib.sha256(destination.read_bytes()).hexdigest(), 'manifest': manifest}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output', type=Path, default=ROOT / 'data/tool_packages')
args = parser.parse_args()
built = [build_package(engine, args.output) for engine in ENGINES]
(args.output / 'packages.json').write_bytes(json_bytes(built))
print(json.dumps([{key: row[key] for key in ['package_id', 'archive', 'import_file', 'sha256']} for row in built], ensure_ascii=False))
if __name__ == '__main__':
main()

View File

@ -4,6 +4,7 @@ from __future__ import annotations
import argparse
import ast
import asyncio
import ipaddress
import json
import os
@ -15,7 +16,9 @@ import sqlite3
import subprocess
import sys
import time
import types
import urllib.request
import zipfile
ROOT = Path(__file__).resolve().parents[1]
@ -34,9 +37,14 @@ def local_test_worker() -> int:
original_connect, original_connect_ex = socket.socket.connect, socket.socket.connect_ex
original_dns = socket.getaddrinfo
def local_ports():
# HTTP transport tests register only their ephemeral loopback listener.
extra = os.getenv('ZP_TEST_HTTP_PORT')
return {7687, 8000} | ({int(extra)} if extra else set())
def permitted(address):
try:
return address[1] in {7687, 8000} and ipaddress.ip_address(address[0]).is_loopback
return address[1] in local_ports() and ipaddress.ip_address(address[0]).is_loopback
except (ValueError, TypeError, IndexError):
return False
@ -51,7 +59,7 @@ def local_test_worker() -> int:
return original_connect_ex(self, address)
def guarded_dns(host, port, *args, **kwargs):
if host not in {'localhost', '127.0.0.1', '::1'} or int(port) not in {7687, 8000}:
if host not in {'localhost', '127.0.0.1', '::1'} or int(port) not in local_ports():
raise RuntimeError('External DNS disabled during local verification')
return original_dns(host, port, *args, **kwargs)
@ -62,7 +70,7 @@ def local_test_worker() -> int:
def verify_code() -> dict:
files = [path for name in ['app', 'tools', 'test', 'scripts'] for path in (ROOT / name).rglob('*.py')]
files = [path for name in ['app', 'tools', 'packages', 'test', 'scripts'] for path in (ROOT / name).rglob('*.py')]
for path in files:
text = path.read_text()
compile(text, str(path), 'exec')
@ -152,6 +160,53 @@ def main() -> int:
"assert p.returncode==0,'Wiki origin unavailable'; print(p.stdout.strip())")
check('wiki_git_ssh', lambda: command(*compose, 'exec', '-T', 'zp-agent-api', 'python', '-c', probe))
def tool_packages():
from scripts.build_tool_packages import build_package
key = os.getenv('SEARCH_API_KEY', '').strip()
if not key:
for line in (ROOT / '.env').read_text().splitlines():
name, separator, value = line.partition('=')
if separator and name == 'SEARCH_API_KEY':
key = shlex.split(value)[0]
assert key, 'Search API key is missing'
built = []
async def run():
graph_tool = None
versions = []
for engine in ('rag', 'graphrag'):
package = build_package(engine, ROOT / 'data/tool_packages')
with zipfile.ZipFile(package['archive']) as archive:
form = json.loads(archive.read('openwebui-tool.json'))[0]
module = types.ModuleType('verify_packaged_' + engine)
sys.modules[module.__name__] = module
try:
exec(compile(form['content'], '<packaged-tool>', 'exec'), module.__dict__)
tool = module.Tools()
tool.valves.BACKEND_URL = 'http://127.0.0.1:8000'
tool.valves.BACKEND_API_KEY = key
method = getattr(tool, 'retrieve_zpwiki_context' if engine == 'rag' else 'retrieve_zpwiki_graph')
result = json.loads(await method('Aký je názov diplomovej práce osoby Ján Holp?'))
assert result['status'] == 'ok' and result['sources'], 'Packaged tool cannot retrieve evidence'
assert result['requires_final_answer_verification'] is True
versions.append(result['index_version'])
if engine == 'graphrag':
graph_tool = tool
built.append({name: package[name] for name in ['package_id', 'archive', 'sha256']})
finally:
sys.modules.pop(module.__name__, None)
assert versions[0] == versions[1], 'Tools use different data versions'
negative = json.loads(await graph_tool.retrieve_zpwiki_graph(
'Aký je názov diplomovej práce osoby Ján Holp v roku 1999?'))
assert negative['status'] == 'no_answer' and not negative['facts']
graph_tool.valves.BACKEND_API_KEY = 'invalid-test-key'
assert json.loads(await graph_tool.retrieve_zpwiki_graph('Práca?'))['reason'] == 'unauthorized'
asyncio.run(run())
return {'packages': built, 'live_retrieval': True, 'same_data_version': True,
'negative_query': True, 'authorization': True, 'external_llm_calls': False}
check('openwebui_tool_packages', tool_packages)
def tests():
log = args.output.with_suffix('.pytest.log')
environment = {**os.environ}

View File

@ -84,6 +84,7 @@ def test_all_application_python_files_compile() -> None:
targets = [
PROJECT_ROOT / "app",
PROJECT_ROOT / "tools",
PROJECT_ROOT / "packages",
PROJECT_ROOT / "test",
PROJECT_ROOT / "scripts",
]

View File

@ -0,0 +1,283 @@
"""Standalone installation and real loopback HTTP transport, without an LLM."""
import asyncio
import copy
import hashlib
import importlib.util
import json
from pathlib import Path
import subprocess
import sys
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import zipfile
import pytest
from fastapi.testclient import TestClient
from neo4j.exceptions import ServiceUnavailable
import app.main as main
import app.routes as routes
from scripts.build_tool_packages import build_package
KEY = 'a' * 64
ROOT = Path(__file__).resolve().parents[1]
def load_tool(engine, path=None):
path = path or ROOT / 'packages/openwebui' / engine / 'tool.py'
spec = importlib.util.spec_from_file_location('standalone_' + engine, path)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
@pytest.fixture(params=['rag', 'graphrag'])
def engine(request):
return request.param
@pytest.fixture
def evidence():
return {'status': 'ok', 'reason': None, 'complete': False, 'truncated': True,
'truncation_reasons': ['result_limit'], 'index_version': {'version': 'fixture'},
'sources': [{'source_id': 'S1', 'document_path': 'pages/profile/README.md',
'source_url': 'https://example.test/profile', 'title': 'Eva Nová'}],
'context': 'Doložený text práce.', 'facts': [{'person': 'Eva Nová', 'year': 2024}],
'paths': [{'nodes': ['Eva Nová', 'Práca'], 'relationships': ['AUTHORED']}],
'aggregation': {'count': 1, 'complete': True}, 'constraints': {'years': [2024]}}
@pytest.fixture
def backend(monkeypatch, evidence):
monkeypatch.setattr(main, 'embed_query', lambda _: None)
calls = []
def retrieved(*args, **kwargs):
calls.append((args, kwargs))
return copy.deepcopy(evidence)
def rag_retrieved(*args, **kwargs):
calls.append((args, kwargs))
result = {key: copy.deepcopy(value) for key, value in evidence.items()
if key in routes.RagResponse.model_fields}
return {**result, 'query': args[1], 'engine': 'hybrid_fts5_embeddings',
'source_count': len(evidence['sources']), 'strategies': ['fixture'],
'instructions': [], 'answer_format': {}}
monkeypatch.setattr(routes, 'build_rag_context', rag_retrieved)
monkeypatch.setattr(routes, 'compact_rag_payload', lambda result, **kwargs: result)
monkeypatch.setattr(routes, 'search_graph', retrieved)
state = {'requests': [], 'mode': 'api', 'code': 200, 'raw': b''}
with TestClient(main.app) as client:
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_POST(self):
body = self.rfile.read(int(self.headers.get('Content-Length', '0')))
state['requests'].append({'path': self.path, 'body': json.loads(body),
'authorization': self.headers.get('Authorization')})
if state['mode'] == 'timeout':
time.sleep(1.25)
if state['mode'] == 'disconnect':
self.close_connection = True
return
if state['mode'] == 'api':
response = client.post(self.path.removeprefix('/prefix'), content=body,
headers=dict(self.headers))
code, data = response.status_code, response.content
else:
code, data = state['code'], state['raw']
self.send_response(code)
self.send_header('Content-Type', 'application/json')
if code == 302:
self.send_header('Location', state['base'] + '/redirected')
self.end_headers()
try:
self.wfile.write(data)
except OSError:
pass
server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
monkeypatch.setenv('ZP_TEST_HTTP_PORT', str(server.server_port))
state['base'] = f'http://127.0.0.1:{server.server_port}'
thread = threading.Thread(target=server.serve_forever, kwargs={'poll_interval': 0.01}, daemon=True)
thread.start()
try:
yield state, calls
finally:
server.shutdown()
server.server_close()
thread.join(timeout=3)
def configured(engine, backend, path=None):
module = load_tool(engine, path)
tool = module.Tools()
tool.valves.BACKEND_URL = backend[0]['base'] + '/prefix/'
tool.valves.BACKEND_API_KEY = KEY
return module, tool
def invoke(tool, query='Akú prácu má Eva Nová v roku 2024?'):
methods = [getattr(tool, name) for name in dir(tool) if name.startswith('retrieve_zpwiki_')]
assert len(methods) == 1
return json.loads(asyncio.run(methods[0](query)))
def test_actual_http_auth_query_and_all_evidence_survive(engine, backend, evidence):
_, tool = configured(engine, backend)
query = 'Akú prácu má Eva Nová v roku 2024?'
result = invoke(tool, query)
request = backend[0]['requests'][0]
assert request['path'] == '/prefix/' + engine
assert request['authorization'] == 'Bearer ' + KEY
assert request['body']['query'] == query and request['body']['published_only'] is True
assert result['status'] == 'ok'
for key in ['sources', 'index_version', 'complete', 'truncated', 'truncation_reasons']:
assert result[key] == evidence[key]
assert result['requires_final_answer_verification'] is True
if engine == 'rag':
assert result['context'] == evidence['context']
assert request['body']['include_diagnostics'] is False
else:
assert result['facts'] == evidence['facts'] and result['paths'] == evidence['paths']
assert result['aggregation'] == evidence['aggregation']
assert backend[1][0][1] == {'limit': 40, 'max_entities': 8, 'published_only': True}
def test_configuration_controls_scope_without_mutating_question(engine, backend):
_, tool = configured(engine, backend)
tool.valves.LIMIT = 2
tool.valves.PUBLISHED_ONLY = False
invoke(tool, ' Osoba, rok a tag ')
body = backend[0]['requests'][0]['body']
assert body['limit'] == 2 and body['published_only'] is False
assert body['query'] == 'Osoba, rok a tag'
@pytest.mark.parametrize('status', ['no_answer', 'ambiguous'])
def test_no_answer_and_ambiguity_are_not_relabelled_as_positive(engine, backend, evidence, status):
evidence.update(status=status, reason='no_matching_facts', sources=[], facts=[], paths=[], context='')
_, tool = configured(engine, backend)
result = invoke(tool)
assert result['status'] == status and result['reason'] == 'no_matching_facts'
assert result['sources'] == []
@pytest.mark.parametrize('query', ['', ' ', 'x' * 501, None])
def test_invalid_query_does_not_make_a_network_request(engine, backend, query):
_, tool = configured(engine, backend)
assert invoke(tool, query)['reason'] == 'invalid_query'
assert backend[0]['requests'] == []
@pytest.mark.parametrize('url', ['', 'file:///tmp/file', 'https://user:password@example.test',
'http://example.test?query=1', 'http://example.test/#fragment'])
def test_invalid_configuration_never_sends_credentials(engine, backend, url):
_, tool = configured(engine, backend)
tool.valves.BACKEND_URL = url
assert invoke(tool)['reason'] == 'invalid_configuration'
assert backend[0]['requests'] == []
@pytest.mark.parametrize('code,reason', [(401, 'unauthorized'), (403, 'forbidden'),
(404, 'backend_endpoint_not_found'), (429, 'backend_busy'), (503, 'retrieval_unavailable'),
(500, 'backend_http_error'), (302, 'redirect_not_allowed')])
def test_http_errors_do_not_expose_body_or_follow_redirect(engine, backend, code, reason):
_, tool = configured(engine, backend)
backend[0].update(mode='raw', code=code, raw=(KEY + ' PRIVATE_ERROR').encode())
result = invoke(tool)
assert result['status'] == 'error' and result['reason'] == reason
assert result['http_status'] == code and result['sources'] == []
assert KEY not in json.dumps(result) and 'PRIVATE_ERROR' not in json.dumps(result)
assert len(backend[0]['requests']) == 1
def test_real_unauthorized_backend_rejects_wrong_key(engine, backend):
_, tool = configured(engine, backend)
tool.valves.BACKEND_API_KEY = 'wrong'
assert invoke(tool)['reason'] == 'unauthorized'
assert backend[1] == []
@pytest.mark.parametrize('raw', [b'<html>not JSON</html>', b'[]', b'{"status":"ok"}'])
def test_invalid_backend_data_fails_closed(engine, backend, raw):
_, tool = configured(engine, backend)
backend[0].update(mode='raw', raw=raw)
assert invoke(tool)['reason'] == 'invalid_backend_response'
def test_contradictory_completeness_is_rejected(engine, backend, evidence):
_, tool = configured(engine, backend)
evidence.update(complete=True, truncated=True)
assert invoke(tool)['reason'] == 'invalid_backend_response'
def test_diagnostics_and_unverified_answer_are_not_passed_to_model(engine, backend, evidence):
_, tool = configured(engine, backend)
evidence.update(diagnostics={'private': 'PRIVATE'}, answer='UNVERIFIED', accepted_for_delivery=True)
backend[0].update(mode='raw', raw=json.dumps(evidence).encode())
result = invoke(tool)
assert 'diagnostics' not in result and 'answer' not in result and 'accepted_for_delivery' not in result
def test_large_responses_are_rejected_without_partial_evidence(engine, backend, monkeypatch):
module, tool = configured(engine, backend)
monkeypatch.setattr(module, 'MAX_RESPONSE_BYTES', 100)
backend[0].update(mode='raw', raw=b'x' * 101)
result = invoke(tool)
assert result['reason'] == 'response_too_large' and result['sources'] == []
def test_timeout_is_reported_without_raw_exception(engine, backend):
_, tool = configured(engine, backend)
tool.valves.TIMEOUT_SECONDS = 1
backend[0].update(mode='timeout')
assert invoke(tool)['reason'] == 'backend_timeout'
def test_connection_closed_before_response_returns_safe_error(engine, backend):
_, tool = configured(engine, backend)
backend[0]['mode'] = 'disconnect'
assert invoke(tool)['reason'] == 'backend_unavailable'
def test_graph_serialization_errors_have_shared_contract(backend, monkeypatch):
monkeypatch.setattr(routes, 'search_graph', lambda *args, **kwargs: {'invalid': object()})
with TestClient(main.app) as client:
response = client.post('/graphrag', json={'query': 'Práca?'}, headers={'X-API-Key': KEY})
assert response.status_code == 500 and response.json()['reason'] == 'internal_error'
def test_graph_api_uses_shared_error_contract_and_validates_query(backend, monkeypatch):
def unavailable(*args, **kwargs):
raise ServiceUnavailable('PRIVATE_DATABASE_ERROR')
monkeypatch.setattr(routes, 'search_graph', unavailable)
with TestClient(main.app) as client:
response = client.post('/graphrag', json={'query': 'Práca?'}, headers={'X-API-Key': KEY})
assert response.status_code == 503 and response.json()['reason'] == 'retrieval_unavailable'
assert 'PRIVATE_DATABASE_ERROR' not in response.text
response = client.post('/graphrag', json={'query': ' ', 'limit': 0}, headers={'X-API-Key': KEY})
assert response.status_code == 422 and response.json()['status'] == 'error'
assert client.post('/graphrag', json={'query': 'Práca?'}).status_code == 401
def test_zip_and_json_installation_are_standalone_and_reproducible(engine, backend, tmp_path):
built = build_package(engine, tmp_path / 'first')
again = build_package(engine, tmp_path / 'again')
assert built['sha256'] == again['sha256']
with zipfile.ZipFile(built['archive']) as archive:
manifest = json.loads(archive.read('manifest.json'))
for name, digest in manifest['files'].items():
assert hashlib.sha256(archive.read(name)).hexdigest() == digest
form = json.loads(archive.read('openwebui-tool.json'))
assert len(form) == 1 and form[0]['id'] == 'zpwiki_' + engine
assert form[0]['access_grants'] == []
assert form[0]['content'].encode() == archive.read('tool.py')
assert Path(built['import_file']).read_bytes() == archive.read('openwebui-tool.json')
extracted = tmp_path / 'isolated.py'
extracted.write_bytes(archive.read('tool.py'))
code = "import pathlib; source=pathlib.Path(__import__('sys').argv[1]).read_text(); scope={'__name__':'standalone_tool'}; exec(source,scope); tool=scope['Tools'](); assert len([n for n in dir(tool) if n.startswith('retrieve_zpwiki_')])==1"
result = subprocess.run([sys.executable, '-I', '-c', code, str(extracted)],
cwd=tmp_path, capture_output=True, text=True, timeout=20)
assert result.returncode == 0, result.stderr
_, tool = configured(engine, backend, extracted)
assert invoke(tool)['status'] == 'ok'