Hybrid vyhladavanie s embedding

This commit is contained in:
Ján Pták 2026-08-12 00:53:26 +02:00
parent f3b0f413d2
commit 32004e89bf
14 changed files with 2555 additions and 358 deletions

4
.gitignore vendored
View File

@ -1,9 +1,9 @@
.env
data/
.venv/ .venv/
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
.env
*.log *.log
.pytest_cache/ .pytest_cache/
.coverage .coverage
htmlcov/ htmlcov/
data/

View File

@ -2,7 +2,7 @@
Backend pre indexovanie a vyhľadávanie v repozitári záverečných prác `zpwiki`. Backend pre indexovanie a vyhľadávanie v repozitári záverečných prác `zpwiki`.
Projekt načítava Markdown dokumenty, spracuje YAML metadata, rozdelí obsah na tokenové chunky a vytvorí SQLite FTS5 index. Vyhľadávanie je dostupné cez FastAPI a systém podporuje manuálnu aj webhookovú synchronizáciu. Projekt načítava Markdown dokumenty, spracuje YAML metadata, rozdelí obsah na tokenové chunky a vytvorí SQLite index kombinujúci FTS5 fulltextové vyhľadávanie a embeddingy. Vyhľadávanie je dostupné cez FastAPI a systém podporuje manuálnu aj webhookovú synchronizáciu.
## Implementované ## Implementované
@ -12,13 +12,19 @@ Projekt načítava Markdown dokumenty, spracuje YAML metadata, rozdelí obsah na
- zachovanie názvu dokumentu a hierarchie nadpisov v chunku, - zachovanie názvu dokumentu a hierarchie nadpisov v chunku,
- SQLite databáza a FTS5 fulltextový index, - SQLite databáza a FTS5 fulltextový index,
- BM25 vyhľadávanie s podporou diakritiky a prefixových výrazov, - BM25 vyhľadávanie s podporou diakritiky a prefixových výrazov,
- embeddingy pre každý chunk pomocou modelu `intfloat/multilingual-e5-small`,
- uloženie embeddingov priamo v SQLite,
- vektorové vyhľadávanie pomocou cosine similarity,
- hybridné vyhľadávanie FTS5 + embeddings pomocou RRF,
- nižšia váha pre slabú `any_term` FTS stratégiu,
- zachovanie presných `all_terms` a `prefix_terms` výsledkov bez vektorového šumu,
- filtrovanie publikovaných dokumentov, - filtrovanie publikovaných dokumentov,
- FastAPI endpointy `/health`, `/search`, `/sync` a `/webhook/gitea`, - FastAPI endpointy `/health`, `/search`, `/sync` a `/webhook/gitea`,
- autorizácia `/sync` pomocou API kľúča, - autorizácia `/search` a `/sync` pomocou API kľúča,
- Gitea webhook s HMAC-SHA256 podpisom a kontrolou udalosti a repozitára, - Gitea webhook s HMAC-SHA256 podpisom a kontrolou udalosti a repozitára,
- zámok proti súbežnému reindexovaniu, - zámok proti súbežnému reindexovaniu,
- atomická výmena databázy po úspešnom reindexovaní, - atomická výmena databázy po úspešnom reindexovaní,
- automatizované a integračné testy nad reálnymi dátami. - automatizované a integračné testy.
## Štruktúra ## Štruktúra
@ -31,6 +37,7 @@ zp-agent/
│ ├── scan_zpwiki.py │ ├── scan_zpwiki.py
│ ├── build_chunks.py │ ├── build_chunks.py
│ ├── build_sqlite_index.py │ ├── build_sqlite_index.py
│ ├── embedding_utils.py
│ ├── rebuild_index.py │ ├── rebuild_index.py
│ ├── search_db.py │ ├── search_db.py
│ └── search_utils.py │ └── search_utils.py
@ -58,8 +65,13 @@ V koreňovom priečinku vytvor `.env`:
```dotenv ```dotenv
WEBHOOK_SECRET=<náhodná hodnota s minimálne 32 znakmi> WEBHOOK_SECRET=<náhodná hodnota s minimálne 32 znakmi>
SYNC_API_KEY=<iná náhodná hodnota s minimálne 32 znakmi> SYNC_API_KEY=<iná náhodná hodnota s minimálne 32 znakmi>
SEARCH_API_KEY=<ďalšia náhodná hodnota s minimálne 32 znakmi>
EXPECTED_GITEA_REPOSITORY=KEMT/zpwiki EXPECTED_GITEA_REPOSITORY=KEMT/zpwiki
WEBHOOK_PULL_GIT=false WEBHOOK_PULL_GIT=false
# Voliteľné
EMBEDDING_MODEL=intfloat/multilingual-e5-small
EMBEDDING_BATCH_SIZE=32
``` ```
Tajomstvá je možné vygenerovať príkazom: Tajomstvá je možné vygenerovať príkazom:
@ -73,7 +85,7 @@ Súbor `.env` sa nesmie commitovať.
## Spustenie cez Docker ## Spustenie cez Docker
```bash ```bash
docker compose build --no-cache docker compose build
docker compose up -d docker compose up -d
``` ```
@ -97,7 +109,7 @@ docker compose down
## Reindexovanie ## Reindexovanie
Celý proces načíta dokumenty, vytvorí chunky a obnoví SQLite FTS5 index: Celý proces načíta dokumenty, vytvorí chunky, obnoví FTS5 index a vytvorí embedding pre každý chunk:
```bash ```bash
docker compose run --rm zp-agent-api python scripts/rebuild_index.py docker compose run --rm zp-agent-api python scripts/rebuild_index.py
@ -111,18 +123,44 @@ data/chunks.json
data/zp_index.sqlite data/zp_index.sqlite
``` ```
Databáza obsahuje dokumenty, chunky, FTS5 index, metadata a embeddingy.
## Vyhľadávanie ## Vyhľadávanie
Vyhľadávanie kombinuje:
```text
dotaz
├── FTS5 / BM25
└── embeddingové vyhľadávanie
RRF fusion
výsledky
```
Test z terminálu: Test z terminálu:
```bash ```bash
docker compose run --rm zp-agent-api python scripts/search_db.py "rag agent" --limit 5 docker compose run --rm zp-agent-api \
python scripts/search_db.py "rag agent" --limit 5
``` ```
Vyhľadávanie cez API: Pred volaním API načítaj premenné z `.env`:
```bash ```bash
curl -X POST http://127.0.0.1:8000/search -H "Content-Type: application/json" -d '{ set -a
source .env
set +a
```
Vyhľadávanie cez zabezpečené API:
```bash
curl -X POST http://127.0.0.1:8000/search \
-H "Content-Type: application/json" \
-H "X-API-Key: $SEARCH_API_KEY" \
-d '{
"query": "rag agent", "query": "rag agent",
"limit": 5, "limit": 5,
"published_only": false, "published_only": false,
@ -130,10 +168,19 @@ curl -X POST http://127.0.0.1:8000/search -H "Content-Type: application/json"
}' }'
``` ```
API vracia hybridný engine:
```text
hybrid_fts5_embeddings
```
Manuálne reindexovanie cez zabezpečený endpoint: Manuálne reindexovanie cez zabezpečený endpoint:
```bash ```bash
curl -X POST http://127.0.0.1:8000/sync -H "Content-Type: application/json" -H "X-API-Key: $SYNC_API_KEY" -d '{"pull_git": false}' curl -X POST http://127.0.0.1:8000/sync \
-H "Content-Type: application/json" \
-H "X-API-Key: $SYNC_API_KEY" \
-d '{"pull_git": false}'
``` ```
## Testy ## Testy
@ -150,14 +197,18 @@ Bežné automatizované testy:
pytest -q test pytest -q test
``` ```
Aktuálna testovacia sada:
```text
65 passed, 2 skipped
```
Testy vrátane kontroly reálne vygenerovaných dát a databázy: Testy vrátane kontroly reálne vygenerovaných dát a databázy:
```bash ```bash
RUN_LIVE_TESTS=1 pytest -q test RUN_LIVE_TESTS=1 pytest -q test
``` ```
Aktuálna implementácia prešla všetkými 65 testami vrátane live testov.
## Ďalší krok ## Ďalší krok
Najbližšia etapa je integrácia s OpenWebUI a vytvorenie agentového rozhrania. Následne sa doplnia embeddingy, hybridné vyhľadávanie a RAG odpovede s citáciami. Najbližšia etapa je integrácia s OpenWebUI a vytvorenie agentového rozhrania. Následne sa doplnia RAG odpovede so zdrojmi a citáciami.

258
a nachadza v staged obsahu" Normal file
View File

@ -0,0 +1,258 @@
SSUUMMMMAARRYY OOFF LLEESSSS CCOOMMMMAANNDDSS
Commands marked with * may be preceded by a number, _N.
Notes in parentheses indicate the behavior if _N is given.
A key preceded by a caret indicates the Ctrl key; thus ^K is ctrl-K.
h H Display this help.
q :q Q :Q ZZ Exit.
---------------------------------------------------------------------------
MMOOVVIINNGG
e ^E j ^N CR * Forward one line (or _N lines).
y ^Y k ^K ^P * Backward one line (or _N lines).
f ^F ^V SPACE * Forward one window (or _N lines).
b ^B ESC-v * Backward one window (or _N lines).
z * Forward one window (and set window to _N).
w * Backward one window (and set window to _N).
ESC-SPACE * Forward one window, but don't stop at end-of-file.
d ^D * Forward one half-window (and set half-window to _N).
u ^U * Backward one half-window (and set half-window to _N).
ESC-) RightArrow * Right one half screen width (or _N positions).
ESC-( LeftArrow * Left one half screen width (or _N positions).
ESC-} ^RightArrow Right to last column displayed.
ESC-{ ^LeftArrow Left to first column.
F Forward forever; like "tail -f".
ESC-F Like F but stop when search pattern is found.
r ^R ^L Repaint screen.
R Repaint screen, discarding buffered input.
---------------------------------------------------
Default "window" is the screen height.
Default "half-window" is half of the screen height.
---------------------------------------------------------------------------
SSEEAARRCCHHIINNGG
/_p_a_t_t_e_r_n * Search forward for (_N-th) matching line.
?_p_a_t_t_e_r_n * Search backward for (_N-th) matching line.
n * Repeat previous search (for _N-th occurrence).
N * Repeat previous search in reverse direction.
ESC-n * Repeat previous search, spanning files.
ESC-N * Repeat previous search, reverse dir. & spanning files.
ESC-u Undo (toggle) search highlighting.
ESC-U Clear search highlighting.
&_p_a_t_t_e_r_n * Display only matching lines.
---------------------------------------------------
A search pattern may begin with one or more of:
^N or ! Search for NON-matching lines.
^E or * Search multiple files (pass thru END OF FILE).
^F or @ Start search at FIRST file (for /) or last file (for ?).
^K Highlight matches, but don't move (KEEP position).
^R Don't use REGULAR EXPRESSIONS.
^W WRAP search if no match found.
---------------------------------------------------------------------------
JJUUMMPPIINNGG
g < ESC-< * Go to first line in file (or line _N).
G > ESC-> * Go to last line in file (or line _N).
p % * Go to beginning of file (or _N percent into file).
t * Go to the (_N-th) next tag.
T * Go to the (_N-th) previous tag.
{ ( [ * Find close bracket } ) ].
} ) ] * Find open bracket { ( [.
ESC-^F _<_c_1_> _<_c_2_> * Find close bracket _<_c_2_>.
ESC-^B _<_c_1_> _<_c_2_> * Find open bracket _<_c_1_>.
---------------------------------------------------
Each "find close bracket" command goes forward to the close bracket
matching the (_N-th) open bracket in the top line.
Each "find open bracket" command goes backward to the open bracket
matching the (_N-th) close bracket in the bottom line.
m_<_l_e_t_t_e_r_> Mark the current top line with <letter>.
M_<_l_e_t_t_e_r_> Mark the current bottom line with <letter>.
'_<_l_e_t_t_e_r_> Go to a previously marked position.
'' Go to the previous position.
^X^X Same as '.
ESC-M_<_l_e_t_t_e_r_> Clear a mark.
---------------------------------------------------
A mark is any upper-case or lower-case letter.
Certain marks are predefined:
^ means beginning of the file
$ means end of the file
---------------------------------------------------------------------------
CCHHAANNGGIINNGG FFIILLEESS
:e [_f_i_l_e] Examine a new file.
^X^V Same as :e.
:n * Examine the (_N-th) next file from the command line.
:p * Examine the (_N-th) previous file from the command line.
:x * Examine the first (or _N-th) file from the command line.
:d Delete the current file from the command line list.
= ^G :f Print current file name.
---------------------------------------------------------------------------
MMIISSCCEELLLLAANNEEOOUUSS CCOOMMMMAANNDDSS
-_<_f_l_a_g_> Toggle a command line option [see OPTIONS below].
--_<_n_a_m_e_> Toggle a command line option, by name.
__<_f_l_a_g_> Display the setting of a command line option.
___<_n_a_m_e_> Display the setting of an option, by name.
+_c_m_d Execute the less cmd each time a new file is examined.
!_c_o_m_m_a_n_d Execute the shell command with $SHELL.
|XX_c_o_m_m_a_n_d Pipe file between current pos & mark XX to shell command.
s _f_i_l_e Save input to a file.
v Edit the current file with $VISUAL or $EDITOR.
V Print version number of "less".
---------------------------------------------------------------------------
OOPPTTIIOONNSS
Most options may be changed either on the command line,
or from within less by using the - or -- command.
Options may be given in one of two forms: either a single
character preceded by a -, or a name preceded by --.
-? ........ --help
Display help (from command line).
-a ........ --search-skip-screen
Search skips current screen.
-A ........ --SEARCH-SKIP-SCREEN
Search starts just after target line.
-b [_N] .... --buffers=[_N]
Number of buffers.
-B ........ --auto-buffers
Don't automatically allocate buffers for pipes.
-c ........ --clear-screen
Repaint by clearing rather than scrolling.
-d ........ --dumb
Dumb terminal.
-D xx_c_o_l_o_r . --color=xx_c_o_l_o_r
Set screen colors.
-e -E .... --quit-at-eof --QUIT-AT-EOF
Quit at end of file.
-f ........ --force
Force open non-regular files.
-F ........ --quit-if-one-screen
Quit if entire file fits on first screen.
-g ........ --hilite-search
Highlight only last match for searches.
-G ........ --HILITE-SEARCH
Don't highlight any matches for searches.
-h [_N] .... --max-back-scroll=[_N]
Backward scroll limit.
-i ........ --ignore-case
Ignore case in searches that do not contain uppercase.
-I ........ --IGNORE-CASE
Ignore case in all searches.
-j [_N] .... --jump-target=[_N]
Screen position of target lines.
-J ........ --status-column
Display a status column at left edge of screen.
-k [_f_i_l_e] . --lesskey-file=[_f_i_l_e]
Use a lesskey file.
-K ........ --quit-on-intr
Exit less in response to ctrl-C.
-L ........ --no-lessopen
Ignore the LESSOPEN environment variable.
-m -M .... --long-prompt --LONG-PROMPT
Set prompt style.
-n -N .... --line-numbers --LINE-NUMBERS
Don't use line numbers.
-o [_f_i_l_e] . --log-file=[_f_i_l_e]
Copy to log file (standard input only).
-O [_f_i_l_e] . --LOG-FILE=[_f_i_l_e]
Copy to log file (unconditionally overwrite).
-p [_p_a_t_t_e_r_n] --pattern=[_p_a_t_t_e_r_n]
Start at pattern (from command line).
-P [_p_r_o_m_p_t] --prompt=[_p_r_o_m_p_t]
Define new prompt.
-q -Q .... --quiet --QUIET --silent --SILENT
Quiet the terminal bell.
-r -R .... --raw-control-chars --RAW-CONTROL-CHARS
Output "raw" control characters.
-s ........ --squeeze-blank-lines
Squeeze multiple blank lines.
-S ........ --chop-long-lines
Chop (truncate) long lines rather than wrapping.
-t [_t_a_g] .. --tag=[_t_a_g]
Find a tag.
-T [_t_a_g_s_f_i_l_e] --tag-file=[_t_a_g_s_f_i_l_e]
Use an alternate tags file.
-u -U .... --underline-special --UNDERLINE-SPECIAL
Change handling of backspaces.
-V ........ --version
Display the version number of "less".
-w ........ --hilite-unread
Highlight first new line after forward-screen.
-W ........ --HILITE-UNREAD
Highlight first new line after any forward movement.
-x [_N[,...]] --tabs=[_N[,...]]
Set tab stops.
-X ........ --no-init
Don't use termcap init/deinit strings.
-y [_N] .... --max-forw-scroll=[_N]
Forward scroll limit.
-z [_N] .... --window=[_N]
Set size of window.
-" [_c[_c]] . --quotes=[_c[_c]]
Set shell quote characters.
-~ ........ --tilde
Don't display tildes after end of file.
-# [_N] .... --shift=[_N]
Set horizontal scroll amount (0 = one half screen width).
--file-size
Automatically determine the size of the input file.
--follow-name
The F command changes files if the input file is renamed.
--incsearch
Search file as each pattern character is typed in.
--line-num-width=N
Set the width of the -N line number field to N characters.
--mouse
Enable mouse input.
--no-keypad
Don't send termcap keypad init/deinit strings.
--no-histdups
Remove duplicates from command history.
--rscroll=C
Set the character used to mark truncated lines.
--save-marks
Retain marks across invocations of less.
--status-col-width=N
Set the width of the -J status column to N characters.
--use-backslash
Subsequent options use backslash as escape char.
--use-color
Enables colored text.
--wheel-lines=N
Each click of the mouse wheel moves N lines.
---------------------------------------------------------------------------
LLIINNEE EEDDIITTIINNGG
These keys can be used to edit text being entered
on the "command line" at the bottom of the screen.
RightArrow ..................... ESC-l ... Move cursor right one character.
LeftArrow ...................... ESC-h ... Move cursor left one character.
ctrl-RightArrow ESC-RightArrow ESC-w ... Move cursor right one word.
ctrl-LeftArrow ESC-LeftArrow ESC-b ... Move cursor left one word.
HOME ........................... ESC-0 ... Move cursor to start of line.
END ............................ ESC-$ ... Move cursor to end of line.
BACKSPACE ................................ Delete char to left of cursor.
DELETE ......................... ESC-x ... Delete char under cursor.
ctrl-BACKSPACE ESC-BACKSPACE ........... Delete word to left of cursor.
ctrl-DELETE .... ESC-DELETE .... ESC-X ... Delete word under cursor.
ctrl-U ......... ESC (MS-DOS only) ....... Delete entire line.
UpArrow ........................ ESC-k ... Retrieve previous command line.
DownArrow ...................... ESC-j ... Retrieve next command line.
TAB ...................................... Complete filename & cycle.
SHIFT-TAB ...................... ESC-TAB Complete filename & reverse cycle.
ctrl-L ................................... Complete filename, list all.

View File

@ -10,26 +10,44 @@ from contextlib import asynccontextmanager
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from fastapi import Depends, FastAPI, Header, HTTPException, Request, Security, status from fastapi import (
Depends,
FastAPI,
Header,
HTTPException,
Request,
Security,
status,
)
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from fastapi.security import APIKeyHeader from fastapi.security import APIKeyHeader
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
PROJECT_ROOT = Path(__file__).resolve().parents[1] PROJECT_ROOT = Path(__file__).resolve().parents[1]
if str(PROJECT_ROOT) not in sys.path: if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT)) sys.path.insert(0, str(PROJECT_ROOT))
from scripts.common import DB_FILE, ZPWIKI_ROOT from scripts.common import DB_FILE, ZPWIKI_ROOT
from scripts.rebuild_index import ReindexInProgressError, rebuild_index from scripts.rebuild_index import (
ReindexInProgressError,
rebuild_index,
)
from scripts.search_utils import search_database from scripts.search_utils import search_database
MIN_SECRET_LENGTH = 32 MIN_SECRET_LENGTH = 32
SEARCH_API_KEY_HEADER = "X-API-Key"
SYNC_API_KEY_HEADER = "X-API-Key" SYNC_API_KEY_HEADER = "X-API-Key"
search_api_key_scheme = APIKeyHeader(
name=SEARCH_API_KEY_HEADER,
auto_error=False,
description="API kľúč pre vyhľadávanie v zpwiki.",
)
sync_api_key_scheme = APIKeyHeader( sync_api_key_scheme = APIKeyHeader(
name=SYNC_API_KEY_HEADER, name=SYNC_API_KEY_HEADER,
auto_error=False, auto_error=False,
@ -38,10 +56,22 @@ sync_api_key_scheme = APIKeyHeader(
class SearchRequest(BaseModel): class SearchRequest(BaseModel):
query: str = Field(..., min_length=1, max_length=500) query: str = Field(
limit: int = Field(default=10, ge=1, le=50) ...,
min_length=1,
max_length=500,
)
limit: int = Field(
default=10,
ge=1,
le=50,
)
published_only: bool = False published_only: bool = False
max_per_document: int = Field(default=3, ge=0, le=10) max_per_document: int = Field(
default=3,
ge=0,
le=10,
)
class SyncRequest(BaseModel): class SyncRequest(BaseModel):
@ -55,7 +85,9 @@ def required_environment_value(name: str) -> str:
value = os.getenv(name, "").strip() value = os.getenv(name, "").strip()
if not value: if not value:
raise RuntimeError(f"Chýba povinná environment premenná {name}") raise RuntimeError(
f"Chýba povinná environment premenná {name}"
)
return value return value
@ -72,53 +104,104 @@ def validate_secret(name: str) -> str:
def expected_gitea_repository() -> str: def expected_gitea_repository() -> str:
value = required_environment_value("EXPECTED_GITEA_REPOSITORY") value = required_environment_value(
"EXPECTED_GITEA_REPOSITORY"
)
if "/" not in value: if "/" not in value:
raise RuntimeError( raise RuntimeError(
"EXPECTED_GITEA_REPOSITORY musí mať tvar vlastník/repozitár" "EXPECTED_GITEA_REPOSITORY musí mať tvar "
"vlastník/repozitár"
) )
return value return value
def webhook_should_pull_git() -> bool: def webhook_should_pull_git() -> bool:
value = os.getenv("WEBHOOK_PULL_GIT", "false").strip().casefold() value = os.getenv(
"WEBHOOK_PULL_GIT",
"false",
).strip().casefold()
return value in {"1", "true", "yes", "on"} return value in {
"1",
"true",
"yes",
"on",
}
def validate_security_configuration() -> None: def validate_security_configuration() -> None:
validate_secret("WEBHOOK_SECRET") validate_secret("WEBHOOK_SECRET")
validate_secret("SYNC_API_KEY") validate_secret("SYNC_API_KEY")
validate_secret("SEARCH_API_KEY")
expected_gitea_repository() expected_gitea_repository()
@asynccontextmanager @asynccontextmanager
async def lifespan(_: FastAPI): async def lifespan(_: FastAPI):
# Aplikácia sa nespustí s chýbajúcim alebo slabým tajomstvom.
validate_security_configuration() validate_security_configuration()
yield yield
app = FastAPI( app = FastAPI(
title="ZP Agent API", title="ZP Agent API",
description="API pre vyhľadávanie v repozitári záverečných prác zpwiki.", description=(
"API pre vyhľadávanie v repozitári "
"záverečných prác zpwiki."
),
version="0.6.0", version="0.6.0",
lifespan=lifespan, lifespan=lifespan,
) )
def require_sync_api_key( def require_search_api_key(
api_key: str | None = Security(sync_api_key_scheme), api_key: str | None = Security(
search_api_key_scheme
),
) -> None: ) -> None:
expected = validate_secret("SYNC_API_KEY") expected = validate_secret(
"SEARCH_API_KEY"
)
if not api_key or not hmac.compare_digest(api_key, expected): if (
not api_key
or not hmac.compare_digest(
api_key,
expected,
)
):
raise HTTPException( raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, status_code=status.HTTP_401_UNAUTHORIZED,
detail="Neplatný alebo chýbajúci API kľúč", detail="Neplatný alebo chýbajúci API kľúč",
headers={"WWW-Authenticate": "ApiKey"}, headers={
"WWW-Authenticate": "ApiKey",
},
)
def require_sync_api_key(
api_key: str | None = Security(
sync_api_key_scheme
),
) -> None:
expected = validate_secret(
"SYNC_API_KEY"
)
if (
not api_key
or not hmac.compare_digest(
api_key,
expected,
)
):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Neplatný alebo chýbajúci API kľúč",
headers={
"WWW-Authenticate": "ApiKey",
},
) )
@ -132,16 +215,18 @@ def verify_gitea_signature(
supplied = signature.strip().casefold() supplied = signature.strip().casefold()
# X-Gitea-Signature je čistý hex digest. Prefix prijímame iba # Kompatibilita podpisu.
# kvôli kompatibilite s X-Hub-Signature-256.
if supplied.startswith("sha256="): if supplied.startswith("sha256="):
supplied = supplied.removeprefix("sha256=") supplied = supplied.removeprefix(
"sha256="
)
if len(supplied) != 64: if len(supplied) != 64:
return False return False
try: try:
int(supplied, 16) int(supplied, 16)
except ValueError: except ValueError:
return False return False
@ -151,7 +236,10 @@ def verify_gitea_signature(
hashlib.sha256, hashlib.sha256,
).hexdigest() ).hexdigest()
return hmac.compare_digest(expected, supplied) return hmac.compare_digest(
expected,
supplied,
)
def repository_name_from_payload( def repository_name_from_payload(
@ -195,17 +283,28 @@ def health() -> dict[str, Any]:
"zpwiki_root": str(ZPWIKI_ROOT), "zpwiki_root": str(ZPWIKI_ROOT),
"zpwiki_exists": ZPWIKI_ROOT.exists(), "zpwiki_exists": ZPWIKI_ROOT.exists(),
"security_configured": all( "security_configured": all(
bool(os.getenv(name, "").strip()) bool(
os.getenv(
name,
"",
).strip()
)
for name in ( for name in (
"WEBHOOK_SECRET", "WEBHOOK_SECRET",
"SYNC_API_KEY", "SYNC_API_KEY",
"SEARCH_API_KEY",
"EXPECTED_GITEA_REPOSITORY", "EXPECTED_GITEA_REPOSITORY",
) )
), ),
} }
@app.post("/search") @app.post(
"/search",
dependencies=[
Depends(require_search_api_key)
],
)
def search( def search(
request: SearchRequest, request: SearchRequest,
) -> dict[str, Any]: ) -> dict[str, Any]:
@ -214,8 +313,12 @@ def search(
DB_FILE, DB_FILE,
request.query, request.query,
request.limit, request.limit,
published_only=request.published_only, published_only=(
max_per_document=request.max_per_document, request.published_only
),
max_per_document=(
request.max_per_document
),
) )
except FileNotFoundError as error: except FileNotFoundError as error:
@ -249,7 +352,9 @@ def search(
@app.post( @app.post(
"/sync", "/sync",
dependencies=[Depends(require_sync_api_key)], dependencies=[
Depends(require_sync_api_key)
],
) )
def sync( def sync(
request: SyncRequest, request: SyncRequest,
@ -274,14 +379,16 @@ def sync(
return { return {
"status": "ok", "status": "ok",
"pull_git": request.pull_git, "pull_git": request.pull_git,
"duration_seconds": result["duration_seconds"], "duration_seconds": (
result["duration_seconds"]
),
"counts": result["counts"], "counts": result["counts"],
} }
@app.post( @app.post(
"/webhook/gitea", "/webhook/gitea",
response_model = None, response_model=None,
) )
async def gitea_webhook( async def gitea_webhook(
request: Request, request: Request,
@ -295,7 +402,10 @@ async def gitea_webhook(
), ),
) -> dict[str, Any] | JSONResponse: ) -> dict[str, Any] | JSONResponse:
raw_body = await request.body() raw_body = await request.body()
secret = validate_secret("WEBHOOK_SECRET")
secret = validate_secret(
"WEBHOOK_SECRET"
)
if not verify_gitea_signature( if not verify_gitea_signature(
raw_body, raw_body,
@ -303,7 +413,9 @@ async def gitea_webhook(
secret, secret,
): ):
raise HTTPException( raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, status_code=(
status.HTTP_401_UNAUTHORIZED
),
detail="Neplatný webhook podpis", detail="Neplatný webhook podpis",
) )
@ -318,24 +430,35 @@ async def gitea_webhook(
) as error: ) as error:
raise HTTPException( raise HTTPException(
status_code=400, status_code=400,
detail="Webhook payload nie je platný JSON", detail=(
"Webhook payload nie je "
"platný JSON"
),
) from error ) from error
if not isinstance(payload, dict): if not isinstance(payload, dict):
raise HTTPException( raise HTTPException(
status_code=400, status_code=400,
detail="Webhook payload musí byť JSON objekt", detail=(
"Webhook payload musí byť "
"JSON objekt"
),
) )
if not x_gitea_event: if not x_gitea_event:
raise HTTPException( raise HTTPException(
status_code=400, status_code=400,
detail="Chýba hlavička X-Gitea-Event", detail=(
"Chýba hlavička "
"X-Gitea-Event"
),
) )
if x_gitea_event.casefold() != "push": if x_gitea_event.casefold() != "push":
return JSONResponse( return JSONResponse(
status_code=status.HTTP_202_ACCEPTED, status_code=(
status.HTTP_202_ACCEPTED
),
content={ content={
"status": "ignored", "status": "ignored",
"reason": "unsupported_event", "reason": "unsupported_event",
@ -343,8 +466,10 @@ async def gitea_webhook(
}, },
) )
repository_name = repository_name_from_payload( repository_name = (
payload repository_name_from_payload(
payload
)
) )
if repository_name is None: if repository_name is None:
@ -356,7 +481,9 @@ async def gitea_webhook(
), ),
) )
expected_repository = expected_gitea_repository() expected_repository = (
expected_gitea_repository()
)
if not same_repository( if not same_repository(
repository_name, repository_name,
@ -373,7 +500,9 @@ async def gitea_webhook(
try: try:
result = await asyncio.to_thread( result = await asyncio.to_thread(
rebuild_index, rebuild_index,
pull_git=webhook_should_pull_git(), pull_git=(
webhook_should_pull_git()
),
) )
except ReindexInProgressError as error: except ReindexInProgressError as error:
@ -393,6 +522,8 @@ async def gitea_webhook(
"event": x_gitea_event, "event": x_gitea_event,
"repository": repository_name, "repository": repository_name,
"verified_by": "hmac_sha256", "verified_by": "hmac_sha256",
"duration_seconds": result["duration_seconds"], "duration_seconds": (
result["duration_seconds"]
),
"counts": result["counts"], "counts": result["counts"],
} }

40
et -a Normal file
View File

@ -0,0 +1,40 @@
README.md:66:WEBHOOK_SECRET=<náhodná hodnota s minimálne 32 znakmi>
README.md:67:SYNC_API_KEY=<iná náhodná hodnota s minimálne 32 znakmi>
README.md:68:SEARCH_API_KEY=<ďalšia náhodná hodnota s minimálne 32 znakmi>
README.md:162: -H "X-API-Key: $SEARCH_API_KEY" \
README.md:182: -H "X-API-Key: $SYNC_API_KEY" \
app/main.py:41:SEARCH_API_KEY_HEADER = "X-API-Key"
app/main.py:42:SYNC_API_KEY_HEADER = "X-API-Key"
app/main.py:45:search_api_key_scheme = APIKeyHeader(
app/main.py:46: name=SEARCH_API_KEY_HEADER,
app/main.py:51:sync_api_key_scheme = APIKeyHeader(
app/main.py:52: name=SYNC_API_KEY_HEADER,
app/main.py:135: validate_secret("WEBHOOK_SECRET")
app/main.py:136: validate_secret("SYNC_API_KEY")
app/main.py:137: validate_secret("SEARCH_API_KEY")
app/main.py:158:def require_search_api_key(
app/main.py:160: search_api_key_scheme
app/main.py:164: "SEARCH_API_KEY"
app/main.py:183:def require_sync_api_key(
app/main.py:185: sync_api_key_scheme
app/main.py:189: "SYNC_API_KEY"
app/main.py:293: "WEBHOOK_SECRET",
app/main.py:294: "SYNC_API_KEY",
app/main.py:295: "SEARCH_API_KEY",
app/main.py:305: Depends(require_search_api_key)
app/main.py:356: Depends(require_sync_api_key)
app/main.py:407: "WEBHOOK_SECRET"
test/conftest.py:21: "WEBHOOK_SECRET",
test/conftest.py:26: "SYNC_API_KEY",
test/conftest.py:31: "SEARCH_API_KEY",
test/test_api.py:14:WEBHOOK_SECRET = "w" * 64
test/test_api.py:15:SYNC_API_KEY = "s" * 64
test/test_api.py:16:SEARCH_API_KEY = "a" * 64
test/test_api.py:34: WEBHOOK_SECRET.encode("utf-8"),
test/test_api.py:49: monkeypatch.delenv("WEBHOOK_SECRET")
test/test_api.py:53: match="WEBHOOK_SECRET",
test/test_api.py:63: "SYNC_API_KEY",
test/test_api.py:111: "X-API-Key": SEARCH_API_KEY,
test/test_api.py:133: "X-API-Key": SEARCH_API_KEY,
test/test_api.py:216: "X-API-Key": SYNC_API_KEY,
test/test_api.py:245: "X-API-Key": SYNC_API_KEY,

View File

@ -4,3 +4,5 @@ python-frontmatter==1.3.0
rich==15.0.0 rich==15.0.0
tiktoken>=0.8,<1 tiktoken>=0.8,<1
uvicorn[standard]==0.48.0 uvicorn[standard]==0.48.0
numpy==2.2.6
sentence-transformers==5.7.0

View File

@ -16,7 +16,18 @@ if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT)) sys.path.insert(0, str(PROJECT_ROOT))
from scripts.common import CHUNKS_FILE, DB_FILE, DOCUMENTS_FILE, read_json from scripts.common import (
CHUNKS_FILE,
DB_FILE,
DOCUMENTS_FILE,
read_json,
)
from scripts.embedding_utils import (
build_embedding_text,
embed_passages,
embedding_model_name,
vector_to_blob,
)
FTS_TOKENIZER = "unicode61 remove_diacritics 2" FTS_TOKENIZER = "unicode61 remove_diacritics 2"
@ -33,21 +44,29 @@ def published_to_db(value: Any) -> int | None:
return None return None
def verify_fts5(conn: sqlite3.Connection) -> None: def verify_fts5(
"""Overí, či aktuálna SQLite knižnica podporuje FTS5.""" conn: sqlite3.Connection,
) -> None:
try: try:
conn.execute( conn.execute(
"CREATE VIRTUAL TABLE temp.fts5_check USING fts5(value)" "CREATE VIRTUAL TABLE temp.fts5_check "
"USING fts5(value)"
) )
conn.execute("DROP TABLE temp.fts5_check") conn.execute(
"DROP TABLE temp.fts5_check"
)
except sqlite3.OperationalError as error: except sqlite3.OperationalError as error:
raise RuntimeError( raise RuntimeError(
"Táto inštalácia SQLite nemá dostupné FTS5. " "Táto inštalácia SQLite nemá dostupné FTS5. "
"Použi Python/SQLite zostavenie s podporou SQLITE_ENABLE_FTS5." "Použi Python/SQLite zostavenie s podporou "
"SQLITE_ENABLE_FTS5."
) from error ) from error
def create_tables(conn: sqlite3.Connection) -> None: def create_tables(
conn: sqlite3.Connection,
) -> None:
conn.executescript( conn.executescript(
f""" f"""
PRAGMA foreign_keys = ON; PRAGMA foreign_keys = ON;
@ -58,9 +77,14 @@ def create_tables(conn: sqlite3.Connection) -> None:
title TEXT, title TEXT,
author TEXT, author TEXT,
published INTEGER published INTEGER
CHECK (published IN (0, 1) OR published IS NULL), CHECK (
content_length INTEGER NOT NULL DEFAULT 0, published IN (0, 1)
metadata_json TEXT NOT NULL DEFAULT '{{}}' OR published IS NULL
),
content_length INTEGER
NOT NULL DEFAULT 0,
metadata_json TEXT
NOT NULL DEFAULT '{{}}'
); );
CREATE TABLE chunks ( CREATE TABLE chunks (
@ -70,11 +94,16 @@ def create_tables(conn: sqlite3.Connection) -> None:
title TEXT, title TEXT,
author TEXT, author TEXT,
published INTEGER published INTEGER
CHECK (published IN (0, 1) OR published IS NULL), CHECK (
published IN (0, 1)
OR published IS NULL
),
chunk_index INTEGER NOT NULL, chunk_index INTEGER NOT NULL,
heading_paths_json TEXT NOT NULL DEFAULT '[]', heading_paths_json TEXT
NOT NULL DEFAULT '[]',
text TEXT NOT NULL, text TEXT NOT NULL,
text_length INTEGER NOT NULL DEFAULT 0, text_length INTEGER
NOT NULL DEFAULT 0,
token_count INTEGER, token_count INTEGER,
content_hash TEXT, content_hash TEXT,
FOREIGN KEY(document_path) FOREIGN KEY(document_path)
@ -103,6 +132,17 @@ def create_tables(conn: sqlite3.Connection) -> None:
ON DELETE CASCADE ON DELETE CASCADE
); );
CREATE TABLE chunk_embeddings (
chunk_id TEXT PRIMARY KEY,
model TEXT NOT NULL,
dimensions INTEGER NOT NULL,
embedding BLOB NOT NULL,
FOREIGN KEY(chunk_id)
REFERENCES chunks(chunk_id)
ON UPDATE CASCADE
ON DELETE CASCADE
);
CREATE INDEX idx_documents_path CREATE INDEX idx_documents_path
ON documents(path); ON documents(path);
@ -127,6 +167,9 @@ def create_tables(conn: sqlite3.Connection) -> None:
CREATE INDEX idx_chunk_categories_category CREATE INDEX idx_chunk_categories_category
ON chunk_categories(category); ON chunk_categories(category);
CREATE INDEX idx_chunk_embeddings_model
ON chunk_embeddings(model);
CREATE VIRTUAL TABLE chunks_fts USING fts5( CREATE VIRTUAL TABLE chunks_fts USING fts5(
chunk_id UNINDEXED, chunk_id UNINDEXED,
title, title,
@ -151,8 +194,13 @@ def insert_documents(
document.get("path"), document.get("path"),
document.get("title"), document.get("title"),
document.get("author"), document.get("author"),
published_to_db(document.get("published")), published_to_db(
int(document.get("content_length") or 0), document.get("published")
),
int(
document.get("content_length")
or 0
),
json.dumps( json.dumps(
document.get("metadata") or {}, document.get("metadata") or {},
ensure_ascii=False, ensure_ascii=False,
@ -184,14 +232,19 @@ def insert_chunks(
) -> None: ) -> None:
chunk_rows: list[tuple] = [] chunk_rows: list[tuple] = []
tag_rows: list[tuple[str, str]] = [] tag_rows: list[tuple[str, str]] = []
category_rows: list[tuple[str, str]] = [] category_rows: list[
tuple[str, str]
] = []
for chunk in chunks: for chunk in chunks:
chunk_id = str(chunk.get("chunk_id") or "").strip() chunk_id = str(
chunk.get("chunk_id") or ""
).strip()
if not chunk_id: if not chunk_id:
raise ValueError( raise ValueError(
"Chunk bez chunk_id nie je možné indexovať" "Chunk bez chunk_id nie je "
"možné indexovať"
) )
text = chunk.get("text") or "" text = chunk.get("text") or ""
@ -202,14 +255,25 @@ def insert_chunks(
chunk.get("document_path"), chunk.get("document_path"),
chunk.get("title"), chunk.get("title"),
chunk.get("author"), chunk.get("author"),
published_to_db(chunk.get("published")), published_to_db(
int(chunk.get("chunk_index") or 0), chunk.get("published")
),
int(
chunk.get("chunk_index")
or 0
),
json.dumps( json.dumps(
chunk.get("heading_paths") or [], chunk.get(
"heading_paths"
)
or [],
ensure_ascii=False, ensure_ascii=False,
), ),
text, text,
int(chunk.get("text_length") or len(text)), int(
chunk.get("text_length")
or len(text)
),
chunk.get("token_count"), chunk.get("token_count"),
chunk.get("content_hash"), chunk.get("content_hash"),
) )
@ -219,13 +283,27 @@ def insert_chunks(
value = str(tag).strip() value = str(tag).strip()
if value: if value:
tag_rows.append((chunk_id, value)) tag_rows.append(
(
chunk_id,
value,
)
)
for category in chunk.get("categories") or []: for category in (
value = str(category).strip() chunk.get("categories") or []
):
value = str(
category
).strip()
if value: if value:
category_rows.append((chunk_id, value)) category_rows.append(
(
chunk_id,
value,
)
)
conn.executemany( conn.executemany(
""" """
@ -270,8 +348,83 @@ def insert_chunks(
) )
def build_fts_index(conn: sqlite3.Connection) -> None: def insert_embeddings(
"""Vytvorí FTS5 index nad chunkmi a ich metadátami.""" conn: sqlite3.Connection,
chunks: list[dict],
) -> None:
if not chunks:
return
chunk_ids: list[str] = []
texts: list[str] = []
for chunk in chunks:
chunk_id = str(
chunk.get("chunk_id") or ""
).strip()
if not chunk_id:
raise ValueError(
"Chunk bez chunk_id nemôže "
"mať embedding"
)
chunk_ids.append(
chunk_id
)
texts.append(
build_embedding_text(
chunk
)
)
vectors = embed_passages(
texts
)
if len(vectors) != len(chunk_ids):
raise RuntimeError(
"Počet embeddingov sa "
"nezhoduje s počtom chunkov"
)
model_name = (
embedding_model_name()
)
rows: list[tuple] = []
for chunk_id, vector in zip(
chunk_ids,
vectors,
):
rows.append(
(
chunk_id,
model_name,
int(vector.shape[0]),
vector_to_blob(vector),
)
)
conn.executemany(
"""
INSERT INTO chunk_embeddings (
chunk_id,
model,
dimensions,
embedding
)
VALUES (?, ?, ?, ?)
""",
rows,
)
def build_fts_index(
conn: sqlite3.Connection,
) -> None:
conn.execute( conn.execute(
""" """
INSERT INTO chunks_fts ( INSERT INTO chunks_fts (
@ -287,60 +440,89 @@ def build_fts_index(conn: sqlite3.Connection) -> None:
SELECT SELECT
chunks.id, chunks.id,
chunks.chunk_id, chunks.chunk_id,
COALESCE(chunks.title, ''), COALESCE(
COALESCE(chunks.author, ''), chunks.title,
''
),
COALESCE(
chunks.author,
''
),
chunks.document_path, chunks.document_path,
COALESCE(tags.values_text, ''), COALESCE(
COALESCE(categories.values_text, ''), tags.values_text,
''
),
COALESCE(
categories.values_text,
''
),
chunks.text chunks.text
FROM chunks FROM chunks
LEFT JOIN ( LEFT JOIN (
SELECT SELECT
chunk_id, chunk_id,
GROUP_CONCAT(tag, ' ') AS values_text GROUP_CONCAT(
tag,
' '
) AS values_text
FROM chunk_tags FROM chunk_tags
GROUP BY chunk_id GROUP BY chunk_id
) AS tags ) AS tags
ON tags.chunk_id = chunks.chunk_id ON tags.chunk_id
= chunks.chunk_id
LEFT JOIN ( LEFT JOIN (
SELECT SELECT
chunk_id, chunk_id,
GROUP_CONCAT(category, ' ') AS values_text GROUP_CONCAT(
category,
' '
) AS values_text
FROM chunk_categories FROM chunk_categories
GROUP BY chunk_id GROUP BY chunk_id
) AS categories ) AS categories
ON categories.chunk_id = chunks.chunk_id ON categories.chunk_id
= chunks.chunk_id
ORDER BY chunks.id ORDER BY chunks.id
""" """
) )
conn.execute( conn.execute(
"INSERT INTO chunks_fts(chunks_fts) VALUES('optimize')" "INSERT INTO chunks_fts"
"(chunks_fts) "
"VALUES('optimize')"
) )
def validate_database(conn: sqlite3.Connection) -> None: def validate_database(
conn: sqlite3.Connection,
) -> None:
integrity = conn.execute( integrity = conn.execute(
"PRAGMA integrity_check" "PRAGMA integrity_check"
).fetchone()[0] ).fetchone()[0]
if integrity != "ok": if integrity != "ok":
raise RuntimeError( raise RuntimeError(
f"SQLite integrity check zlyhal: {integrity}" "SQLite integrity check "
f"zlyhal: {integrity}"
) )
foreign_key_errors = conn.execute( foreign_key_errors = (
"PRAGMA foreign_key_check" conn.execute(
).fetchall() "PRAGMA foreign_key_check"
).fetchall()
)
if foreign_key_errors: if foreign_key_errors:
raise RuntimeError( raise RuntimeError(
"Databáza obsahuje chyby cudzích kľúčov: " "Databáza obsahuje chyby "
"cudzích kľúčov: "
f"{foreign_key_errors[:5]}" f"{foreign_key_errors[:5]}"
) )
conn.execute( conn.execute(
"INSERT INTO chunks_fts(chunks_fts) " "INSERT INTO chunks_fts"
"(chunks_fts) "
"VALUES('integrity-check')" "VALUES('integrity-check')"
) )
@ -352,72 +534,168 @@ def validate_database(conn: sqlite3.Connection) -> None:
"SELECT COUNT(*) FROM chunks_fts" "SELECT COUNT(*) FROM chunks_fts"
).fetchone()[0] ).fetchone()[0]
embedding_count = conn.execute(
"""
SELECT COUNT(*)
FROM chunk_embeddings
"""
).fetchone()[0]
if chunk_count != fts_count: if chunk_count != fts_count:
raise RuntimeError( raise RuntimeError(
"Počet záznamov v chunks a chunks_fts sa nezhoduje: " "Počet záznamov v chunks "
"a chunks_fts sa nezhoduje: "
f"{chunk_count} != {fts_count}" f"{chunk_count} != {fts_count}"
) )
if chunk_count != embedding_count:
raise RuntimeError(
"Počet chunkov a embeddingov "
"sa nezhoduje: "
f"{chunk_count} != "
f"{embedding_count}"
)
model_count = conn.execute(
"""
SELECT COUNT(
DISTINCT model
)
FROM chunk_embeddings
"""
).fetchone()[0]
if (
embedding_count > 0
and model_count != 1
):
raise RuntimeError(
"Embedding index obsahuje "
"viac modelov"
)
def get_counts( def get_counts(
conn: sqlite3.Connection, conn: sqlite3.Connection,
) -> dict[str, int]: ) -> dict[str, int]:
return { return {
"documents": conn.execute( "documents": conn.execute(
"SELECT COUNT(*) FROM documents" """
SELECT COUNT(*)
FROM documents
"""
).fetchone()[0], ).fetchone()[0],
"chunks": conn.execute( "chunks": conn.execute(
"SELECT COUNT(*) FROM chunks" """
SELECT COUNT(*)
FROM chunks
"""
).fetchone()[0], ).fetchone()[0],
"fts_chunks": conn.execute( "fts_chunks": conn.execute(
"SELECT COUNT(*) FROM chunks_fts" """
SELECT COUNT(*)
FROM chunks_fts
"""
).fetchone()[0],
"embedding_chunks": conn.execute(
"""
SELECT COUNT(*)
FROM chunk_embeddings
"""
).fetchone()[0], ).fetchone()[0],
"tags": conn.execute( "tags": conn.execute(
"SELECT COUNT(*) FROM chunk_tags" """
SELECT COUNT(*)
FROM chunk_tags
"""
).fetchone()[0], ).fetchone()[0],
"categories": conn.execute( "categories": conn.execute(
"SELECT COUNT(*) FROM chunk_categories" """
SELECT COUNT(*)
FROM chunk_categories
"""
).fetchone()[0], ).fetchone()[0],
} }
def temporary_database_path(db_file: Path) -> Path: def temporary_database_path(
db_file: Path,
) -> Path:
return db_file.with_name( return db_file.with_name(
f".{db_file.name}.tmp" f".{db_file.name}.tmp"
) )
def build_database() -> dict[str, int]: def build_database() -> dict[str, int]:
documents = read_json(DOCUMENTS_FILE) documents = read_json(
chunks = read_json(CHUNKS_FILE) DOCUMENTS_FILE
)
chunks = read_json(
CHUNKS_FILE
)
DB_FILE.parent.mkdir( DB_FILE.parent.mkdir(
parents=True, parents=True,
exist_ok=True, exist_ok=True,
) )
temporary_file = temporary_database_path(DB_FILE) temporary_file = (
temporary_database_path(
DB_FILE
)
)
if temporary_file.exists(): if temporary_file.exists():
temporary_file.unlink() temporary_file.unlink()
try: try:
with sqlite3.connect(temporary_file) as conn: with sqlite3.connect(
conn.execute("PRAGMA foreign_keys = ON") temporary_file
conn.execute("PRAGMA temp_store = MEMORY") ) as conn:
conn.execute(
"PRAGMA foreign_keys = ON"
)
conn.execute(
"PRAGMA temp_store = MEMORY"
)
verify_fts5(conn) verify_fts5(
conn
)
with conn: with conn:
create_tables(conn) create_tables(
insert_documents(conn, documents) conn
insert_chunks(conn, chunks) )
build_fts_index(conn)
validate_database(conn) insert_documents(
counts = get_counts(conn) conn,
documents,
)
insert_chunks(
conn,
chunks,
)
build_fts_index(
conn
)
insert_embeddings(
conn,
chunks,
)
validate_database(
conn
)
counts = get_counts(
conn
)
# Nová databáza nahradí starú až po úspešnom vytvorení.
os.replace( os.replace(
temporary_file, temporary_file,
DB_FILE, DB_FILE,
@ -430,23 +708,38 @@ def build_database() -> dict[str, int]:
raise raise
print( print(
f"[green]SQLite index vytvorený:[/green] " "[green]SQLite index "
"vytvorený:[/green] "
f"{DB_FILE}" f"{DB_FILE}"
) )
print( print(
f"Dokumentov: {counts['documents']}" f"Dokumentov: "
f"{counts['documents']}"
) )
print( print(
f"Chunkov: {counts['chunks']}" f"Chunkov: "
f"{counts['chunks']}"
) )
print( print(
f"FTS5 chunkov: {counts['fts_chunks']}" f"FTS5 chunkov: "
f"{counts['fts_chunks']}"
) )
print( print(
f"Tag záznamov: {counts['tags']}" f"Embedding chunkov: "
f"{counts['embedding_chunks']}"
) )
print( print(
f"Kategória záznamov: " f"Tag záznamov: "
f"{counts['tags']}"
)
print(
"Kategória záznamov: "
f"{counts['categories']}" f"{counts['categories']}"
) )

289
scripts/embedding_utils.py Normal file
View File

@ -0,0 +1,289 @@
from __future__ import annotations
import os
from functools import lru_cache
from typing import Any
import numpy as np
DEFAULT_EMBEDDING_MODEL = (
"intfloat/multilingual-e5-small"
)
DEFAULT_BATCH_SIZE = 32
def embedding_model_name() -> str:
return (
os.getenv(
"EMBEDDING_MODEL",
DEFAULT_EMBEDDING_MODEL,
).strip()
or DEFAULT_EMBEDDING_MODEL
)
def embedding_batch_size() -> int:
raw_value = os.getenv(
"EMBEDDING_BATCH_SIZE",
str(DEFAULT_BATCH_SIZE),
).strip()
try:
value = int(
raw_value
)
except ValueError:
return DEFAULT_BATCH_SIZE
return max(
1,
value,
)
@lru_cache(maxsize=2)
def get_embedding_model(
model_name: str,
):
from sentence_transformers import (
SentenceTransformer,
)
return SentenceTransformer(
model_name
)
def build_embedding_text(
chunk: dict[str, Any],
) -> str:
parts: list[str] = []
title = str(
chunk.get("title")
or ""
).strip()
author = str(
chunk.get("author")
or ""
).strip()
tags = [
str(value).strip()
for value in (
chunk.get("tags")
or []
)
if str(value).strip()
]
categories = [
str(value).strip()
for value in (
chunk.get("categories")
or []
)
if str(value).strip()
]
text = str(
chunk.get("text")
or ""
).strip()
if title:
parts.append(
f"Názov: {title}"
)
if author:
parts.append(
f"Autor: {author}"
)
if tags:
parts.append(
"Tagy: "
+ ", ".join(
tags
)
)
if categories:
parts.append(
"Kategórie: "
+ ", ".join(
categories
)
)
if text:
parts.append(
text
)
return "\n".join(
parts
)
def embed_passages(
texts: list[str],
*,
model_name: str | None = None,
) -> np.ndarray:
if not texts:
return np.empty(
(
0,
0,
),
dtype=np.float32,
)
selected_model = (
model_name
or embedding_model_name()
)
model = get_embedding_model(
selected_model
)
prepared = [
"passage: "
+ text.strip()
for text in texts
]
vectors = model.encode(
prepared,
batch_size=(
embedding_batch_size()
),
show_progress_bar=False,
convert_to_numpy=True,
normalize_embeddings=True,
)
return np.asarray(
vectors,
dtype=np.float32,
)
def embed_query(
query: str,
*,
model_name: str | None = None,
) -> np.ndarray:
clean_query = query.strip()
if not clean_query:
raise ValueError(
"Query nesmie byť prázdny"
)
selected_model = (
model_name
or embedding_model_name()
)
model = get_embedding_model(
selected_model
)
vector = model.encode(
[
"query: "
+ clean_query
],
batch_size=1,
show_progress_bar=False,
convert_to_numpy=True,
normalize_embeddings=True,
)[0]
return np.asarray(
vector,
dtype=np.float32,
)
def vector_to_blob(
vector: np.ndarray,
) -> bytes:
normalized = np.asarray(
vector,
dtype=np.float32,
)
return normalized.tobytes()
def blob_to_vector(
blob: bytes,
dimensions: int,
) -> np.ndarray:
vector = np.frombuffer(
blob,
dtype=np.float32,
)
if (
vector.shape[0]
!= dimensions
):
raise RuntimeError(
"Neplatný rozmer "
"uloženého embeddingu"
)
return vector
def cosine_similarity(
first: np.ndarray,
second: np.ndarray,
) -> float:
if (
first.shape
!= second.shape
):
raise ValueError(
"Embeddingy majú "
"rozdielny rozmer"
)
first_norm = float(
np.linalg.norm(
first
)
)
second_norm = float(
np.linalg.norm(
second
)
)
if (
first_norm == 0.0
or second_norm == 0.0
):
return 0.0
return float(
np.dot(
first,
second,
)
/ (
first_norm
* second_norm
)
)

File diff suppressed because it is too large Load Diff

40
tatus Normal file
View File

@ -0,0 +1,40 @@
README.md:66:WEBHOOK_SECRET=<náhodná hodnota s minimálne 32 znakmi>
README.md:67:SYNC_API_KEY=<iná náhodná hodnota s minimálne 32 znakmi>
README.md:68:SEARCH_API_KEY=<ďalšia náhodná hodnota s minimálne 32 znakmi>
README.md:162: -H "X-API-Key: $SEARCH_API_KEY" \
README.md:182: -H "X-API-Key: $SYNC_API_KEY" \
app/main.py:41:SEARCH_API_KEY_HEADER = "X-API-Key"
app/main.py:42:SYNC_API_KEY_HEADER = "X-API-Key"
app/main.py:45:search_api_key_scheme = APIKeyHeader(
app/main.py:46: name=SEARCH_API_KEY_HEADER,
app/main.py:51:sync_api_key_scheme = APIKeyHeader(
app/main.py:52: name=SYNC_API_KEY_HEADER,
app/main.py:135: validate_secret("WEBHOOK_SECRET")
app/main.py:136: validate_secret("SYNC_API_KEY")
app/main.py:137: validate_secret("SEARCH_API_KEY")
app/main.py:158:def require_search_api_key(
app/main.py:160: search_api_key_scheme
app/main.py:164: "SEARCH_API_KEY"
app/main.py:183:def require_sync_api_key(
app/main.py:185: sync_api_key_scheme
app/main.py:189: "SYNC_API_KEY"
app/main.py:293: "WEBHOOK_SECRET",
app/main.py:294: "SYNC_API_KEY",
app/main.py:295: "SEARCH_API_KEY",
app/main.py:305: Depends(require_search_api_key)
app/main.py:356: Depends(require_sync_api_key)
app/main.py:407: "WEBHOOK_SECRET"
test/conftest.py:21: "WEBHOOK_SECRET",
test/conftest.py:26: "SYNC_API_KEY",
test/conftest.py:31: "SEARCH_API_KEY",
test/test_api.py:14:WEBHOOK_SECRET = "w" * 64
test/test_api.py:15:SYNC_API_KEY = "s" * 64
test/test_api.py:16:SEARCH_API_KEY = "a" * 64
test/test_api.py:34: WEBHOOK_SECRET.encode("utf-8"),
test/test_api.py:49: monkeypatch.delenv("WEBHOOK_SECRET")
test/test_api.py:53: match="WEBHOOK_SECRET",
test/test_api.py:63: "SYNC_API_KEY",
test/test_api.py:111: "X-API-Key": SEARCH_API_KEY,
test/test_api.py:133: "X-API-Key": SEARCH_API_KEY,
test/test_api.py:216: "X-API-Key": SYNC_API_KEY,
test/test_api.py:245: "X-API-Key": SYNC_API_KEY,

View File

@ -5,7 +5,6 @@ from pathlib import Path
import pytest import pytest
PROJECT_ROOT = Path(__file__).resolve().parents[1] PROJECT_ROOT = Path(__file__).resolve().parents[1]
if str(PROJECT_ROOT) not in sys.path: if str(PROJECT_ROOT) not in sys.path:
@ -17,10 +16,28 @@ def security_environment(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
"""Každý test dostane platnú bezpečnostnú konfiguráciu.""" """Každý test dostane platnú bezpečnostnú konfiguráciu."""
monkeypatch.setenv("WEBHOOK_SECRET", "w" * 64)
monkeypatch.setenv("SYNC_API_KEY", "s" * 64) monkeypatch.setenv(
"WEBHOOK_SECRET",
"w" * 64,
)
monkeypatch.setenv(
"SYNC_API_KEY",
"s" * 64,
)
monkeypatch.setenv(
"SEARCH_API_KEY",
"a" * 64,
)
monkeypatch.setenv( monkeypatch.setenv(
"EXPECTED_GITEA_REPOSITORY", "EXPECTED_GITEA_REPOSITORY",
"KEMT/zpwiki", "KEMT/zpwiki",
) )
monkeypatch.setenv("WEBHOOK_PULL_GIT", "false")
monkeypatch.setenv(
"WEBHOOK_PULL_GIT",
"false",
)

View File

@ -13,6 +13,7 @@ from scripts.rebuild_index import ReindexInProgressError
WEBHOOK_SECRET = "w" * 64 WEBHOOK_SECRET = "w" * 64
SYNC_API_KEY = "s" * 64 SYNC_API_KEY = "s" * 64
SEARCH_API_KEY = "a" * 64
def fake_rebuild_result() -> dict: def fake_rebuild_result() -> dict:
@ -47,7 +48,10 @@ def test_startup_rejects_missing_secret(
) -> None: ) -> None:
monkeypatch.delenv("WEBHOOK_SECRET") monkeypatch.delenv("WEBHOOK_SECRET")
with pytest.raises(RuntimeError, match="WEBHOOK_SECRET"): with pytest.raises(
RuntimeError,
match="WEBHOOK_SECRET",
):
with TestClient(main.app): with TestClient(main.app):
pass pass
@ -55,18 +59,28 @@ def test_startup_rejects_missing_secret(
def test_startup_rejects_short_secret( def test_startup_rejects_short_secret(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
monkeypatch.setenv("SYNC_API_KEY", "short") monkeypatch.setenv(
"SYNC_API_KEY",
"short",
)
with pytest.raises(RuntimeError, match="aspoň 32"): with pytest.raises(
RuntimeError,
match="aspoň 32",
):
with TestClient(main.app): with TestClient(main.app):
pass pass
def test_health_endpoint(client: TestClient) -> None: def test_health_endpoint(
client: TestClient,
) -> None:
response = client.get("/health") response = client.get("/health")
assert response.status_code == 200 assert response.status_code == 200
payload = response.json() payload = response.json()
assert payload["status"] == "ok" assert payload["status"] == "ok"
assert payload["search_engine"] == "sqlite_fts5" assert payload["search_engine"] == "sqlite_fts5"
assert payload["security_configured"] is True assert payload["security_configured"] is True
@ -82,36 +96,107 @@ def test_search_endpoint_uses_shared_search_logic(
lambda *args, **kwargs: { lambda *args, **kwargs: {
"engine": "sqlite_fts5", "engine": "sqlite_fts5",
"strategies": ["all_terms"], "strategies": ["all_terms"],
"results": [{"chunk_id": "test::0", "published": True}], "results": [
{
"chunk_id": "test::0",
"published": True,
}
],
}, },
) )
response = client.post( response = client.post(
"/search", "/search",
json={"query": "jan ptak", "limit": 5}, headers={
"X-API-Key": SEARCH_API_KEY,
},
json={
"query": "jan ptak",
"limit": 5,
},
) )
assert response.status_code == 200 assert response.status_code == 200
assert response.json()["count"] == 1 assert response.json()["count"] == 1
assert response.json()["results"][0]["chunk_id"] == "test::0" assert (
response.json()["results"][0]["chunk_id"]
== "test::0"
)
def test_search_rejects_empty_query(client: TestClient) -> None: def test_search_rejects_empty_query(
response = client.post("/search", json={"query": ""}) client: TestClient,
) -> None:
response = client.post(
"/search",
headers={
"X-API-Key": SEARCH_API_KEY,
},
json={
"query": "",
},
)
assert response.status_code == 422 assert response.status_code == 422
def test_sync_rejects_missing_api_key(client: TestClient) -> None: def test_search_rejects_missing_api_key(
response = client.post("/sync", json={"pull_git": False}) client: TestClient,
) -> None:
response = client.post(
"/search",
json={
"query": "jan ptak",
"limit": 5,
},
)
assert response.status_code == 401 assert response.status_code == 401
def test_sync_rejects_wrong_api_key(client: TestClient) -> None: def test_search_rejects_wrong_api_key(
client: TestClient,
) -> None:
response = client.post(
"/search",
headers={
"X-API-Key": "x" * 64,
},
json={
"query": "jan ptak",
"limit": 5,
},
)
assert response.status_code == 401
def test_sync_rejects_missing_api_key(
client: TestClient,
) -> None:
response = client.post( response = client.post(
"/sync", "/sync",
headers={"X-API-Key": "x" * 64}, json={
json={"pull_git": False}, "pull_git": False,
},
) )
assert response.status_code == 401
def test_sync_rejects_wrong_api_key(
client: TestClient,
) -> None:
response = client.post(
"/sync",
headers={
"X-API-Key": "x" * 64,
},
json={
"pull_git": False,
},
)
assert response.status_code == 401 assert response.status_code == 401
@ -127,8 +212,12 @@ def test_sync_accepts_valid_api_key(
response = client.post( response = client.post(
"/sync", "/sync",
headers={"X-API-Key": SYNC_API_KEY}, headers={
json={"pull_git": False}, "X-API-Key": SYNC_API_KEY,
},
json={
"pull_git": False,
},
) )
assert response.status_code == 200 assert response.status_code == 200
@ -140,22 +229,38 @@ def test_sync_returns_conflict_when_reindex_is_running(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
def busy(*args, **kwargs): def busy(*args, **kwargs):
raise ReindexInProgressError("Reindexovanie už prebieha") raise ReindexInProgressError(
"Reindexovanie už prebieha"
)
monkeypatch.setattr(main, "rebuild_index", busy) monkeypatch.setattr(
main,
"rebuild_index",
busy,
)
response = client.post( response = client.post(
"/sync", "/sync",
headers={"X-API-Key": SYNC_API_KEY}, headers={
json={"pull_git": False}, "X-API-Key": SYNC_API_KEY,
},
json={
"pull_git": False,
},
) )
assert response.status_code == 409 assert response.status_code == 409
def test_webhook_rejects_invalid_signature(client: TestClient) -> None: def test_webhook_rejects_invalid_signature(
client: TestClient,
) -> None:
body = json.dumps( body = json.dumps(
{"repository": {"full_name": "KEMT/zpwiki"}} {
"repository": {
"full_name": "KEMT/zpwiki",
}
}
).encode("utf-8") ).encode("utf-8")
response = client.post( response = client.post(
@ -189,9 +294,15 @@ def test_webhook_rejects_invalid_json_with_valid_signature(
assert response.status_code == 400 assert response.status_code == 400
def test_webhook_requires_event_header(client: TestClient) -> None: def test_webhook_requires_event_header(
client: TestClient,
) -> None:
body = json.dumps( body = json.dumps(
{"repository": {"full_name": "KEMT/zpwiki"}} {
"repository": {
"full_name": "KEMT/zpwiki",
}
}
).encode("utf-8") ).encode("utf-8")
response = client.post( response = client.post(
@ -217,9 +328,18 @@ def test_webhook_ignores_non_push_event(
calls += 1 calls += 1
return fake_rebuild_result() return fake_rebuild_result()
monkeypatch.setattr(main, "rebuild_index", fake_rebuild) monkeypatch.setattr(
main,
"rebuild_index",
fake_rebuild,
)
body = json.dumps( body = json.dumps(
{"repository": {"full_name": "KEMT/zpwiki"}} {
"repository": {
"full_name": "KEMT/zpwiki",
}
}
).encode("utf-8") ).encode("utf-8")
response = client.post( response = client.post(
@ -237,9 +357,15 @@ def test_webhook_ignores_non_push_event(
assert calls == 0 assert calls == 0
def test_webhook_rejects_unexpected_repository(client: TestClient) -> None: def test_webhook_rejects_unexpected_repository(
client: TestClient,
) -> None:
body = json.dumps( body = json.dumps(
{"repository": {"full_name": "OTHER/repository"}} {
"repository": {
"full_name": "OTHER/repository",
}
}
).encode("utf-8") ).encode("utf-8")
response = client.post( response = client.post(
@ -264,8 +390,13 @@ def test_webhook_accepts_signed_push(
"rebuild_index", "rebuild_index",
lambda pull_git=False: fake_rebuild_result(), lambda pull_git=False: fake_rebuild_result(),
) )
body = json.dumps( body = json.dumps(
{"repository": {"full_name": "KEMT/zpwiki"}} {
"repository": {
"full_name": "KEMT/zpwiki",
}
}
).encode("utf-8") ).encode("utf-8")
response = client.post( response = client.post(
@ -279,8 +410,14 @@ def test_webhook_accepts_signed_push(
) )
assert response.status_code == 200 assert response.status_code == 200
assert response.json()["verified_by"] == "hmac_sha256" assert (
assert response.json()["repository"] == "KEMT/zpwiki" response.json()["verified_by"]
== "hmac_sha256"
)
assert (
response.json()["repository"]
== "KEMT/zpwiki"
)
def test_webhook_returns_conflict_when_reindex_is_running( def test_webhook_returns_conflict_when_reindex_is_running(
@ -288,11 +425,22 @@ def test_webhook_returns_conflict_when_reindex_is_running(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
def busy(*args, **kwargs): def busy(*args, **kwargs):
raise ReindexInProgressError("Reindexovanie už prebieha") raise ReindexInProgressError(
"Reindexovanie už prebieha"
)
monkeypatch.setattr(
main,
"rebuild_index",
busy,
)
monkeypatch.setattr(main, "rebuild_index", busy)
body = json.dumps( body = json.dumps(
{"repository": {"full_name": "KEMT/zpwiki"}} {
"repository": {
"full_name": "KEMT/zpwiki",
}
}
).encode("utf-8") ).encode("utf-8")
response = client.post( response = client.post(

View File

@ -10,9 +10,16 @@ import scripts.build_sqlite_index as indexer
def write_json(path: Path, value) -> None: def write_json(path: Path, value) -> None:
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(
parents=True,
exist_ok=True,
)
path.write_text( path.write_text(
json.dumps(value, ensure_ascii=False), json.dumps(
value,
ensure_ascii=False,
),
encoding="utf-8", encoding="utf-8",
) )
@ -25,7 +32,9 @@ def sample_documents() -> list[dict]:
"author": "Autor", "author": "Autor",
"published": True, "published": True,
"content_length": 100, "content_length": 100,
"metadata": {"published": True}, "metadata": {
"published": True,
},
} }
] ]
@ -33,19 +42,33 @@ def sample_documents() -> list[dict]:
def sample_chunks() -> list[dict]: def sample_chunks() -> list[dict]:
return [ return [
{ {
"chunk_id": "pages/test/README.md::chunk-0", "chunk_id": (
"document_path": "pages/test/README.md", "pages/test/README.md::chunk-0"
),
"document_path": (
"pages/test/README.md"
),
"title": "Strojový preklad", "title": "Strojový preklad",
"author": "Autor", "author": "Autor",
"published": True, "published": True,
"chunk_index": 0, "chunk_index": 0,
"heading_paths": [["Úvod"]], "heading_paths": [
"text": "Dokument: Strojový preklad. Neurónový preklad textu.", ["Úvod"]
],
"text": (
"Dokument: Strojový preklad. "
"Neurónový preklad textu."
),
"text_length": 58, "text_length": 58,
"token_count": 16, "token_count": 16,
"content_hash": "abc", "content_hash": "abc",
"tags": ["translation", "nlp"], "tags": [
"categories": ["project"], "translation",
"nlp",
],
"categories": [
"project",
],
} }
] ]
@ -53,26 +76,66 @@ def sample_chunks() -> list[dict]:
def configure_indexer( def configure_indexer(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
tmp_path: Path, tmp_path: Path,
) -> tuple[Path, Path, Path]: ) -> tuple[
documents_file = tmp_path / "documents.json" Path,
chunks_file = tmp_path / "chunks.json" Path,
db_file = tmp_path / "zp_index.sqlite" Path,
]:
documents_file = (
tmp_path / "documents.json"
)
write_json(documents_file, sample_documents()) chunks_file = (
write_json(chunks_file, sample_chunks()) tmp_path / "chunks.json"
)
monkeypatch.setattr(indexer, "DOCUMENTS_FILE", documents_file) db_file = (
monkeypatch.setattr(indexer, "CHUNKS_FILE", chunks_file) tmp_path / "zp_index.sqlite"
monkeypatch.setattr(indexer, "DB_FILE", db_file) )
return documents_file, chunks_file, db_file write_json(
documents_file,
sample_documents(),
)
write_json(
chunks_file,
sample_chunks(),
)
monkeypatch.setattr(
indexer,
"DOCUMENTS_FILE",
documents_file,
)
monkeypatch.setattr(
indexer,
"CHUNKS_FILE",
chunks_file,
)
monkeypatch.setattr(
indexer,
"DB_FILE",
db_file,
)
return (
documents_file,
chunks_file,
db_file,
)
def test_database_contains_documents_chunks_metadata_and_fts( def test_database_contains_documents_chunks_metadata_and_fts(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
_, _, db_file = configure_indexer(monkeypatch, tmp_path) _, _, db_file = configure_indexer(
monkeypatch,
tmp_path,
)
counts = indexer.build_database() counts = indexer.build_database()
@ -80,55 +143,205 @@ def test_database_contains_documents_chunks_metadata_and_fts(
"documents": 1, "documents": 1,
"chunks": 1, "chunks": 1,
"fts_chunks": 1, "fts_chunks": 1,
"embedding_chunks": 1,
"tags": 2, "tags": 2,
"categories": 1, "categories": 1,
} }
with sqlite3.connect(db_file) as conn: with sqlite3.connect(
assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" db_file
assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] ) as conn:
assert conn.execute("SELECT published FROM chunks").fetchone()[0] == 1 assert (
assert conn.execute("SELECT COUNT(*) FROM chunk_tags").fetchone()[0] == 2 conn.execute(
assert conn.execute("SELECT COUNT(*) FROM chunk_categories").fetchone()[0] == 1 "PRAGMA integrity_check"
assert conn.execute( ).fetchone()[0]
"SELECT COUNT(*) FROM chunks_fts WHERE chunks_fts MATCH 'strojovy'" == "ok"
).fetchone()[0] == 1 )
assert (
conn.execute(
"PRAGMA foreign_key_check"
).fetchall()
== []
)
assert (
conn.execute(
"""
SELECT published
FROM chunks
"""
).fetchone()[0]
== 1
)
assert (
conn.execute(
"""
SELECT COUNT(*)
FROM chunk_tags
"""
).fetchone()[0]
== 2
)
assert (
conn.execute(
"""
SELECT COUNT(*)
FROM chunk_categories
"""
).fetchone()[0]
== 1
)
assert (
conn.execute(
"""
SELECT COUNT(*)
FROM chunks_fts
WHERE chunks_fts
MATCH 'strojovy'
"""
).fetchone()[0]
== 1
)
assert (
conn.execute(
"""
SELECT COUNT(*)
FROM chunk_embeddings
"""
).fetchone()[0]
== 1
)
embedding_row = conn.execute(
"""
SELECT
model,
dimensions,
LENGTH(embedding)
FROM chunk_embeddings
"""
).fetchone()
assert embedding_row is not None
model, dimensions, blob_size = (
embedding_row
)
assert model
assert dimensions > 0
# float32 = 4 bajty.
assert (
blob_size
== dimensions * 4
)
def test_database_rebuild_replaces_old_database_only_after_success( def test_database_rebuild_replaces_old_database_only_after_success(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
_, _, db_file = configure_indexer(monkeypatch, tmp_path) _, _, db_file = configure_indexer(
monkeypatch,
tmp_path,
)
with sqlite3.connect(
db_file
) as conn:
conn.execute(
"""
CREATE TABLE marker(
value TEXT
)
"""
)
conn.execute(
"""
INSERT INTO marker
VALUES ('old database')
"""
)
with sqlite3.connect(db_file) as conn:
conn.execute("CREATE TABLE marker(value TEXT)")
conn.execute("INSERT INTO marker VALUES ('old database')")
conn.commit() conn.commit()
def fail_validation(conn: sqlite3.Connection) -> None: def fail_validation(
raise RuntimeError("úmyselná chyba validácie") conn: sqlite3.Connection,
) -> None:
raise RuntimeError(
"úmyselná chyba validácie"
)
monkeypatch.setattr(indexer, "validate_database", fail_validation) monkeypatch.setattr(
indexer,
"validate_database",
fail_validation,
)
with pytest.raises(RuntimeError, match="úmyselná chyba"): with pytest.raises(
RuntimeError,
match="úmyselná chyba",
):
indexer.build_database() indexer.build_database()
with sqlite3.connect(db_file) as conn: with sqlite3.connect(
assert conn.execute("SELECT value FROM marker").fetchone()[0] == "old database" db_file
) as conn:
assert (
conn.execute(
"""
SELECT value
FROM marker
"""
).fetchone()[0]
== "old database"
)
assert not indexer.temporary_database_path(db_file).exists() assert not (
indexer.temporary_database_path(
db_file
).exists()
)
def test_database_rejects_chunk_without_chunk_id( def test_database_rejects_chunk_without_chunk_id(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
documents_file, chunks_file, _ = configure_indexer(monkeypatch, tmp_path) (
broken = sample_chunks() documents_file,
broken[0]["chunk_id"] = "" chunks_file,
write_json(documents_file, sample_documents()) _,
write_json(chunks_file, broken) ) = configure_indexer(
monkeypatch,
tmp_path,
)
with pytest.raises(ValueError, match="chunk_id"): broken = sample_chunks()
broken[0][
"chunk_id"
] = ""
write_json(
documents_file,
sample_documents(),
)
write_json(
chunks_file,
broken,
)
with pytest.raises(
ValueError,
match="chunk_id",
):
indexer.build_database() indexer.build_database()

View File

@ -11,7 +11,13 @@ from scripts.search_utils import search_database
def write_json(path: Path, value) -> None: def write_json(path: Path, value) -> None:
path.write_text(json.dumps(value, ensure_ascii=False), encoding="utf-8") path.write_text(
json.dumps(
value,
ensure_ascii=False,
),
encoding="utf-8",
)
def build_search_database( def build_search_database(
@ -141,18 +147,43 @@ def build_search_database(
"deep::0", "deep::0",
"pages/topics/deep/README.md", "pages/topics/deep/README.md",
"Neurónové siete", "Neurónové siete",
"Modely hlbokého učenia a trénovanie hlbokých neurónových sietí.", (
"Modely hlbokého učenia a trénovanie "
"hlbokých neurónových sietí."
),
False, False,
tags=["nn"], tags=["nn"],
), ),
] ]
write_json(documents_file, documents) write_json(
write_json(chunks_file, chunks) documents_file,
documents,
)
write_json(
chunks_file,
chunks,
)
monkeypatch.setattr(
indexer,
"DOCUMENTS_FILE",
documents_file,
)
monkeypatch.setattr(
indexer,
"CHUNKS_FILE",
chunks_file,
)
monkeypatch.setattr(
indexer,
"DB_FILE",
db_file,
)
monkeypatch.setattr(indexer, "DOCUMENTS_FILE", documents_file)
monkeypatch.setattr(indexer, "CHUNKS_FILE", chunks_file)
monkeypatch.setattr(indexer, "DB_FILE", db_file)
indexer.build_database() indexer.build_database()
return db_file return db_file
@ -162,58 +193,124 @@ def test_person_query_does_not_add_any_term_noise(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
response = search_database(db_file, "jan ptak", limit=5) response = search_database(
db_file,
"jan ptak",
limit=5,
)
assert response["strategies"] == ["all_terms"] assert response["strategies"] == [
assert [item["title"] for item in response["results"]] == ["Ján Pták"] "all_terms"
]
assert [
item["title"]
for item in response["results"]
] == [
"Ján Pták"
]
def test_topic_query_returns_best_topic_first( def test_topic_query_returns_best_topic_first(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
response = search_database(db_file, "strojovy preklad", limit=5) response = search_database(
db_file,
"strojovy preklad",
limit=5,
)
assert response["strategies"] == ["all_terms"] assert response["strategies"] == [
assert response["results"][0]["title"] == "Strojový preklad" "all_terms"
]
assert (
response["results"][0]["title"]
== "Strojový preklad"
)
def test_prefix_fallback_handles_slovak_inflection( def test_prefix_fallback_handles_slovak_inflection(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
response = search_database(db_file, "hlboke ucenie", limit=5) response = search_database(
db_file,
"hlboke ucenie",
limit=5,
)
assert response["strategies"] == ["prefix_terms"] assert response["strategies"] == [
assert response["results"][0]["title"] == "Neurónové siete" "prefix_terms"
]
assert (
response["results"][0]["title"]
== "Neurónové siete"
)
def test_any_term_is_used_only_when_stricter_queries_find_nothing( def test_any_term_is_used_only_when_stricter_queries_find_nothing(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
response = search_database(db_file, "nezmysel preklad", limit=5) response = search_database(
db_file,
"nezmysel preklad",
limit=5,
)
assert response["strategies"] == [
"any_term"
]
assert response["strategies"] == ["any_term"]
assert response["results"] assert response["results"]
assert all("preklad" in item["text"].casefold() for item in response["results"])
assert any(
item.get("match_strategy")
== "any_term"
and "preklad"
in item["text"].casefold()
for item in response["results"]
)
def test_published_only_excludes_unpublished_chunks( def test_published_only_excludes_unpublished_chunks(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
all_results = search_database(
db_file,
"hlboke ucenie",
limit=5,
)
all_results = search_database(db_file, "hlboke ucenie", limit=5)
public_results = search_database( public_results = search_database(
db_file, db_file,
"hlboke ucenie", "hlboke ucenie",
@ -222,14 +319,26 @@ def test_published_only_excludes_unpublished_chunks(
) )
assert all_results["results"] assert all_results["results"]
assert public_results["results"] == []
assert any(
item["published"] is False
for item in all_results["results"]
)
assert all(
item["published"] is True
for item in public_results["results"]
)
def test_results_are_diversified_by_document( def test_results_are_diversified_by_document(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
response = search_database( response = search_database(
db_file, db_file,
@ -238,48 +347,97 @@ def test_results_are_diversified_by_document(
max_per_document=1, max_per_document=1,
) )
paths = [item["document_path"] for item in response["results"]] paths = [
assert len(paths) == len(set(paths)) item["document_path"]
for item in response["results"]
]
assert len(paths) == len(
set(paths)
)
def test_public_result_format_has_no_internal_id_and_uses_boolean( def test_public_result_format_has_no_internal_id_and_uses_boolean(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
result = search_database(db_file, "jan ptak", limit=1)["results"][0] result = search_database(
db_file,
"jan ptak",
limit=1,
)["results"][0]
assert "id" not in result assert "id" not in result
assert result["published"] is True assert result["published"] is True
assert result["chunk_id"] == "jan-ptak::0" assert result["chunk_id"] == "jan-ptak::0"
assert result["source_url"].endswith("students/jan_ptak")
assert result["source_url"].endswith(
"students/jan_ptak"
)
def test_empty_query_returns_empty_result( def test_empty_query_returns_empty_result(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
db_file = build_search_database(tmp_path, monkeypatch) db_file = build_search_database(
tmp_path,
monkeypatch,
)
response = search_database(db_file, " ", limit=5) response = search_database(
db_file,
" ",
limit=5,
)
assert response == { assert response == {
"engine": "sqlite_fts5", "engine": "hybrid_fts5_embeddings",
"strategies": [], "strategies": [],
"results": [], "results": [],
} }
def test_missing_database_is_reported(tmp_path: Path) -> None: def test_missing_database_is_reported(
with pytest.raises(FileNotFoundError): tmp_path: Path,
search_database(tmp_path / "missing.sqlite", "rag") ) -> None:
with pytest.raises(
FileNotFoundError
):
search_database(
tmp_path / "missing.sqlite",
"rag",
)
def test_missing_fts_schema_is_reported(tmp_path: Path) -> None: def test_missing_fts_schema_is_reported(
db_file = tmp_path / "broken.sqlite" tmp_path: Path,
with sqlite3.connect(db_file) as conn: ) -> None:
conn.execute("CREATE TABLE chunks(id INTEGER PRIMARY KEY)") db_file = (
tmp_path / "broken.sqlite"
)
with pytest.raises(RuntimeError, match="FTS5 index"): with sqlite3.connect(
search_database(db_file, "rag") db_file
) as conn:
conn.execute(
"""
CREATE TABLE chunks(
id INTEGER PRIMARY KEY
)
"""
)
with pytest.raises(
RuntimeError,
match="FTS5 index",
):
search_database(
db_file,
"rag",
)