Hybrid vyhladavanie s embedding
This commit is contained in:
parent
f3b0f413d2
commit
32004e89bf
4
.gitignore
vendored
4
.gitignore
vendored
@ -1,9 +1,9 @@
|
|||||||
|
.env
|
||||||
|
data/
|
||||||
.venv/
|
.venv/
|
||||||
__pycache__/
|
__pycache__/
|
||||||
*.py[cod]
|
*.py[cod]
|
||||||
.env
|
|
||||||
*.log
|
*.log
|
||||||
.pytest_cache/
|
.pytest_cache/
|
||||||
.coverage
|
.coverage
|
||||||
htmlcov/
|
htmlcov/
|
||||||
data/
|
|
||||||
|
|||||||
75
README.md
75
README.md
@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Backend pre indexovanie a vyhľadávanie v repozitári záverečných prác `zpwiki`.
|
Backend pre indexovanie a vyhľadávanie v repozitári záverečných prác `zpwiki`.
|
||||||
|
|
||||||
Projekt načítava Markdown dokumenty, spracuje YAML metadata, rozdelí obsah na tokenové chunky a vytvorí SQLite FTS5 index. Vyhľadávanie je dostupné cez FastAPI a systém podporuje manuálnu aj webhookovú synchronizáciu.
|
Projekt načítava Markdown dokumenty, spracuje YAML metadata, rozdelí obsah na tokenové chunky a vytvorí SQLite index kombinujúci FTS5 fulltextové vyhľadávanie a embeddingy. Vyhľadávanie je dostupné cez FastAPI a systém podporuje manuálnu aj webhookovú synchronizáciu.
|
||||||
|
|
||||||
## Implementované
|
## Implementované
|
||||||
|
|
||||||
@ -12,13 +12,19 @@ Projekt načítava Markdown dokumenty, spracuje YAML metadata, rozdelí obsah na
|
|||||||
- zachovanie názvu dokumentu a hierarchie nadpisov v chunku,
|
- zachovanie názvu dokumentu a hierarchie nadpisov v chunku,
|
||||||
- SQLite databáza a FTS5 fulltextový index,
|
- SQLite databáza a FTS5 fulltextový index,
|
||||||
- BM25 vyhľadávanie s podporou diakritiky a prefixových výrazov,
|
- BM25 vyhľadávanie s podporou diakritiky a prefixových výrazov,
|
||||||
|
- embeddingy pre každý chunk pomocou modelu `intfloat/multilingual-e5-small`,
|
||||||
|
- uloženie embeddingov priamo v SQLite,
|
||||||
|
- vektorové vyhľadávanie pomocou cosine similarity,
|
||||||
|
- hybridné vyhľadávanie FTS5 + embeddings pomocou RRF,
|
||||||
|
- nižšia váha pre slabú `any_term` FTS stratégiu,
|
||||||
|
- zachovanie presných `all_terms` a `prefix_terms` výsledkov bez vektorového šumu,
|
||||||
- filtrovanie publikovaných dokumentov,
|
- filtrovanie publikovaných dokumentov,
|
||||||
- FastAPI endpointy `/health`, `/search`, `/sync` a `/webhook/gitea`,
|
- FastAPI endpointy `/health`, `/search`, `/sync` a `/webhook/gitea`,
|
||||||
- autorizácia `/sync` pomocou API kľúča,
|
- autorizácia `/search` a `/sync` pomocou API kľúča,
|
||||||
- Gitea webhook s HMAC-SHA256 podpisom a kontrolou udalosti a repozitára,
|
- Gitea webhook s HMAC-SHA256 podpisom a kontrolou udalosti a repozitára,
|
||||||
- zámok proti súbežnému reindexovaniu,
|
- zámok proti súbežnému reindexovaniu,
|
||||||
- atomická výmena databázy po úspešnom reindexovaní,
|
- atomická výmena databázy po úspešnom reindexovaní,
|
||||||
- automatizované a integračné testy nad reálnymi dátami.
|
- automatizované a integračné testy.
|
||||||
|
|
||||||
## Štruktúra
|
## Štruktúra
|
||||||
|
|
||||||
@ -31,6 +37,7 @@ zp-agent/
|
|||||||
│ ├── scan_zpwiki.py
|
│ ├── scan_zpwiki.py
|
||||||
│ ├── build_chunks.py
|
│ ├── build_chunks.py
|
||||||
│ ├── build_sqlite_index.py
|
│ ├── build_sqlite_index.py
|
||||||
|
│ ├── embedding_utils.py
|
||||||
│ ├── rebuild_index.py
|
│ ├── rebuild_index.py
|
||||||
│ ├── search_db.py
|
│ ├── search_db.py
|
||||||
│ └── search_utils.py
|
│ └── search_utils.py
|
||||||
@ -58,8 +65,13 @@ V koreňovom priečinku vytvor `.env`:
|
|||||||
```dotenv
|
```dotenv
|
||||||
WEBHOOK_SECRET=<náhodná hodnota s minimálne 32 znakmi>
|
WEBHOOK_SECRET=<náhodná hodnota s minimálne 32 znakmi>
|
||||||
SYNC_API_KEY=<iná náhodná hodnota s minimálne 32 znakmi>
|
SYNC_API_KEY=<iná náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
SEARCH_API_KEY=<ďalšia náhodná hodnota s minimálne 32 znakmi>
|
||||||
EXPECTED_GITEA_REPOSITORY=KEMT/zpwiki
|
EXPECTED_GITEA_REPOSITORY=KEMT/zpwiki
|
||||||
WEBHOOK_PULL_GIT=false
|
WEBHOOK_PULL_GIT=false
|
||||||
|
|
||||||
|
# Voliteľné
|
||||||
|
EMBEDDING_MODEL=intfloat/multilingual-e5-small
|
||||||
|
EMBEDDING_BATCH_SIZE=32
|
||||||
```
|
```
|
||||||
|
|
||||||
Tajomstvá je možné vygenerovať príkazom:
|
Tajomstvá je možné vygenerovať príkazom:
|
||||||
@ -73,7 +85,7 @@ Súbor `.env` sa nesmie commitovať.
|
|||||||
## Spustenie cez Docker
|
## Spustenie cez Docker
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose build --no-cache
|
docker compose build
|
||||||
docker compose up -d
|
docker compose up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -97,7 +109,7 @@ docker compose down
|
|||||||
|
|
||||||
## Reindexovanie
|
## Reindexovanie
|
||||||
|
|
||||||
Celý proces načíta dokumenty, vytvorí chunky a obnoví SQLite FTS5 index:
|
Celý proces načíta dokumenty, vytvorí chunky, obnoví FTS5 index a vytvorí embedding pre každý chunk:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose run --rm zp-agent-api python scripts/rebuild_index.py
|
docker compose run --rm zp-agent-api python scripts/rebuild_index.py
|
||||||
@ -111,18 +123,44 @@ data/chunks.json
|
|||||||
data/zp_index.sqlite
|
data/zp_index.sqlite
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Databáza obsahuje dokumenty, chunky, FTS5 index, metadata a embeddingy.
|
||||||
|
|
||||||
## Vyhľadávanie
|
## Vyhľadávanie
|
||||||
|
|
||||||
|
Vyhľadávanie kombinuje:
|
||||||
|
|
||||||
|
```text
|
||||||
|
dotaz
|
||||||
|
├── FTS5 / BM25
|
||||||
|
└── embeddingové vyhľadávanie
|
||||||
|
↓
|
||||||
|
RRF fusion
|
||||||
|
↓
|
||||||
|
výsledky
|
||||||
|
```
|
||||||
|
|
||||||
Test z terminálu:
|
Test z terminálu:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose run --rm zp-agent-api python scripts/search_db.py "rag agent" --limit 5
|
docker compose run --rm zp-agent-api \
|
||||||
|
python scripts/search_db.py "rag agent" --limit 5
|
||||||
```
|
```
|
||||||
|
|
||||||
Vyhľadávanie cez API:
|
Pred volaním API načítaj premenné z `.env`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://127.0.0.1:8000/search -H "Content-Type: application/json" -d '{
|
set -a
|
||||||
|
source .env
|
||||||
|
set +a
|
||||||
|
```
|
||||||
|
|
||||||
|
Vyhľadávanie cez zabezpečené API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://127.0.0.1:8000/search \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "X-API-Key: $SEARCH_API_KEY" \
|
||||||
|
-d '{
|
||||||
"query": "rag agent",
|
"query": "rag agent",
|
||||||
"limit": 5,
|
"limit": 5,
|
||||||
"published_only": false,
|
"published_only": false,
|
||||||
@ -130,10 +168,19 @@ curl -X POST http://127.0.0.1:8000/search -H "Content-Type: application/json"
|
|||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
API vracia hybridný engine:
|
||||||
|
|
||||||
|
```text
|
||||||
|
hybrid_fts5_embeddings
|
||||||
|
```
|
||||||
|
|
||||||
Manuálne reindexovanie cez zabezpečený endpoint:
|
Manuálne reindexovanie cez zabezpečený endpoint:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://127.0.0.1:8000/sync -H "Content-Type: application/json" -H "X-API-Key: $SYNC_API_KEY" -d '{"pull_git": false}'
|
curl -X POST http://127.0.0.1:8000/sync \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "X-API-Key: $SYNC_API_KEY" \
|
||||||
|
-d '{"pull_git": false}'
|
||||||
```
|
```
|
||||||
|
|
||||||
## Testy
|
## Testy
|
||||||
@ -150,14 +197,18 @@ Bežné automatizované testy:
|
|||||||
pytest -q test
|
pytest -q test
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Aktuálna testovacia sada:
|
||||||
|
|
||||||
|
```text
|
||||||
|
65 passed, 2 skipped
|
||||||
|
```
|
||||||
|
|
||||||
Testy vrátane kontroly reálne vygenerovaných dát a databázy:
|
Testy vrátane kontroly reálne vygenerovaných dát a databázy:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
RUN_LIVE_TESTS=1 pytest -q test
|
RUN_LIVE_TESTS=1 pytest -q test
|
||||||
```
|
```
|
||||||
|
|
||||||
Aktuálna implementácia prešla všetkými 65 testami vrátane live testov.
|
|
||||||
|
|
||||||
## Ďalší krok
|
## Ďalší krok
|
||||||
|
|
||||||
Najbližšia etapa je integrácia s OpenWebUI a vytvorenie agentového rozhrania. Následne sa doplnia embeddingy, hybridné vyhľadávanie a RAG odpovede s citáciami.
|
Najbližšia etapa je integrácia s OpenWebUI a vytvorenie agentového rozhrania. Následne sa doplnia RAG odpovede so zdrojmi a citáciami.
|
||||||
|
|||||||
258
a nachadza v staged obsahu"
Normal file
258
a nachadza v staged obsahu"
Normal file
@ -0,0 +1,258 @@
|
|||||||
|
|
||||||
|
SSUUMMMMAARRYY OOFF LLEESSSS CCOOMMMMAANNDDSS
|
||||||
|
|
||||||
|
Commands marked with * may be preceded by a number, _N.
|
||||||
|
Notes in parentheses indicate the behavior if _N is given.
|
||||||
|
A key preceded by a caret indicates the Ctrl key; thus ^K is ctrl-K.
|
||||||
|
|
||||||
|
h H Display this help.
|
||||||
|
q :q Q :Q ZZ Exit.
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
MMOOVVIINNGG
|
||||||
|
|
||||||
|
e ^E j ^N CR * Forward one line (or _N lines).
|
||||||
|
y ^Y k ^K ^P * Backward one line (or _N lines).
|
||||||
|
f ^F ^V SPACE * Forward one window (or _N lines).
|
||||||
|
b ^B ESC-v * Backward one window (or _N lines).
|
||||||
|
z * Forward one window (and set window to _N).
|
||||||
|
w * Backward one window (and set window to _N).
|
||||||
|
ESC-SPACE * Forward one window, but don't stop at end-of-file.
|
||||||
|
d ^D * Forward one half-window (and set half-window to _N).
|
||||||
|
u ^U * Backward one half-window (and set half-window to _N).
|
||||||
|
ESC-) RightArrow * Right one half screen width (or _N positions).
|
||||||
|
ESC-( LeftArrow * Left one half screen width (or _N positions).
|
||||||
|
ESC-} ^RightArrow Right to last column displayed.
|
||||||
|
ESC-{ ^LeftArrow Left to first column.
|
||||||
|
F Forward forever; like "tail -f".
|
||||||
|
ESC-F Like F but stop when search pattern is found.
|
||||||
|
r ^R ^L Repaint screen.
|
||||||
|
R Repaint screen, discarding buffered input.
|
||||||
|
---------------------------------------------------
|
||||||
|
Default "window" is the screen height.
|
||||||
|
Default "half-window" is half of the screen height.
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SSEEAARRCCHHIINNGG
|
||||||
|
|
||||||
|
/_p_a_t_t_e_r_n * Search forward for (_N-th) matching line.
|
||||||
|
?_p_a_t_t_e_r_n * Search backward for (_N-th) matching line.
|
||||||
|
n * Repeat previous search (for _N-th occurrence).
|
||||||
|
N * Repeat previous search in reverse direction.
|
||||||
|
ESC-n * Repeat previous search, spanning files.
|
||||||
|
ESC-N * Repeat previous search, reverse dir. & spanning files.
|
||||||
|
ESC-u Undo (toggle) search highlighting.
|
||||||
|
ESC-U Clear search highlighting.
|
||||||
|
&_p_a_t_t_e_r_n * Display only matching lines.
|
||||||
|
---------------------------------------------------
|
||||||
|
A search pattern may begin with one or more of:
|
||||||
|
^N or ! Search for NON-matching lines.
|
||||||
|
^E or * Search multiple files (pass thru END OF FILE).
|
||||||
|
^F or @ Start search at FIRST file (for /) or last file (for ?).
|
||||||
|
^K Highlight matches, but don't move (KEEP position).
|
||||||
|
^R Don't use REGULAR EXPRESSIONS.
|
||||||
|
^W WRAP search if no match found.
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
JJUUMMPPIINNGG
|
||||||
|
|
||||||
|
g < ESC-< * Go to first line in file (or line _N).
|
||||||
|
G > ESC-> * Go to last line in file (or line _N).
|
||||||
|
p % * Go to beginning of file (or _N percent into file).
|
||||||
|
t * Go to the (_N-th) next tag.
|
||||||
|
T * Go to the (_N-th) previous tag.
|
||||||
|
{ ( [ * Find close bracket } ) ].
|
||||||
|
} ) ] * Find open bracket { ( [.
|
||||||
|
ESC-^F _<_c_1_> _<_c_2_> * Find close bracket _<_c_2_>.
|
||||||
|
ESC-^B _<_c_1_> _<_c_2_> * Find open bracket _<_c_1_>.
|
||||||
|
---------------------------------------------------
|
||||||
|
Each "find close bracket" command goes forward to the close bracket
|
||||||
|
matching the (_N-th) open bracket in the top line.
|
||||||
|
Each "find open bracket" command goes backward to the open bracket
|
||||||
|
matching the (_N-th) close bracket in the bottom line.
|
||||||
|
|
||||||
|
m_<_l_e_t_t_e_r_> Mark the current top line with <letter>.
|
||||||
|
M_<_l_e_t_t_e_r_> Mark the current bottom line with <letter>.
|
||||||
|
'_<_l_e_t_t_e_r_> Go to a previously marked position.
|
||||||
|
'' Go to the previous position.
|
||||||
|
^X^X Same as '.
|
||||||
|
ESC-M_<_l_e_t_t_e_r_> Clear a mark.
|
||||||
|
---------------------------------------------------
|
||||||
|
A mark is any upper-case or lower-case letter.
|
||||||
|
Certain marks are predefined:
|
||||||
|
^ means beginning of the file
|
||||||
|
$ means end of the file
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CCHHAANNGGIINNGG FFIILLEESS
|
||||||
|
|
||||||
|
:e [_f_i_l_e] Examine a new file.
|
||||||
|
^X^V Same as :e.
|
||||||
|
:n * Examine the (_N-th) next file from the command line.
|
||||||
|
:p * Examine the (_N-th) previous file from the command line.
|
||||||
|
:x * Examine the first (or _N-th) file from the command line.
|
||||||
|
:d Delete the current file from the command line list.
|
||||||
|
= ^G :f Print current file name.
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
MMIISSCCEELLLLAANNEEOOUUSS CCOOMMMMAANNDDSS
|
||||||
|
|
||||||
|
-_<_f_l_a_g_> Toggle a command line option [see OPTIONS below].
|
||||||
|
--_<_n_a_m_e_> Toggle a command line option, by name.
|
||||||
|
__<_f_l_a_g_> Display the setting of a command line option.
|
||||||
|
___<_n_a_m_e_> Display the setting of an option, by name.
|
||||||
|
+_c_m_d Execute the less cmd each time a new file is examined.
|
||||||
|
|
||||||
|
!_c_o_m_m_a_n_d Execute the shell command with $SHELL.
|
||||||
|
|XX_c_o_m_m_a_n_d Pipe file between current pos & mark XX to shell command.
|
||||||
|
s _f_i_l_e Save input to a file.
|
||||||
|
v Edit the current file with $VISUAL or $EDITOR.
|
||||||
|
V Print version number of "less".
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
OOPPTTIIOONNSS
|
||||||
|
|
||||||
|
Most options may be changed either on the command line,
|
||||||
|
or from within less by using the - or -- command.
|
||||||
|
Options may be given in one of two forms: either a single
|
||||||
|
character preceded by a -, or a name preceded by --.
|
||||||
|
|
||||||
|
-? ........ --help
|
||||||
|
Display help (from command line).
|
||||||
|
-a ........ --search-skip-screen
|
||||||
|
Search skips current screen.
|
||||||
|
-A ........ --SEARCH-SKIP-SCREEN
|
||||||
|
Search starts just after target line.
|
||||||
|
-b [_N] .... --buffers=[_N]
|
||||||
|
Number of buffers.
|
||||||
|
-B ........ --auto-buffers
|
||||||
|
Don't automatically allocate buffers for pipes.
|
||||||
|
-c ........ --clear-screen
|
||||||
|
Repaint by clearing rather than scrolling.
|
||||||
|
-d ........ --dumb
|
||||||
|
Dumb terminal.
|
||||||
|
-D xx_c_o_l_o_r . --color=xx_c_o_l_o_r
|
||||||
|
Set screen colors.
|
||||||
|
-e -E .... --quit-at-eof --QUIT-AT-EOF
|
||||||
|
Quit at end of file.
|
||||||
|
-f ........ --force
|
||||||
|
Force open non-regular files.
|
||||||
|
-F ........ --quit-if-one-screen
|
||||||
|
Quit if entire file fits on first screen.
|
||||||
|
-g ........ --hilite-search
|
||||||
|
Highlight only last match for searches.
|
||||||
|
-G ........ --HILITE-SEARCH
|
||||||
|
Don't highlight any matches for searches.
|
||||||
|
-h [_N] .... --max-back-scroll=[_N]
|
||||||
|
Backward scroll limit.
|
||||||
|
-i ........ --ignore-case
|
||||||
|
Ignore case in searches that do not contain uppercase.
|
||||||
|
-I ........ --IGNORE-CASE
|
||||||
|
Ignore case in all searches.
|
||||||
|
-j [_N] .... --jump-target=[_N]
|
||||||
|
Screen position of target lines.
|
||||||
|
-J ........ --status-column
|
||||||
|
Display a status column at left edge of screen.
|
||||||
|
-k [_f_i_l_e] . --lesskey-file=[_f_i_l_e]
|
||||||
|
Use a lesskey file.
|
||||||
|
-K ........ --quit-on-intr
|
||||||
|
Exit less in response to ctrl-C.
|
||||||
|
-L ........ --no-lessopen
|
||||||
|
Ignore the LESSOPEN environment variable.
|
||||||
|
-m -M .... --long-prompt --LONG-PROMPT
|
||||||
|
Set prompt style.
|
||||||
|
-n -N .... --line-numbers --LINE-NUMBERS
|
||||||
|
Don't use line numbers.
|
||||||
|
-o [_f_i_l_e] . --log-file=[_f_i_l_e]
|
||||||
|
Copy to log file (standard input only).
|
||||||
|
-O [_f_i_l_e] . --LOG-FILE=[_f_i_l_e]
|
||||||
|
Copy to log file (unconditionally overwrite).
|
||||||
|
-p [_p_a_t_t_e_r_n] --pattern=[_p_a_t_t_e_r_n]
|
||||||
|
Start at pattern (from command line).
|
||||||
|
-P [_p_r_o_m_p_t] --prompt=[_p_r_o_m_p_t]
|
||||||
|
Define new prompt.
|
||||||
|
-q -Q .... --quiet --QUIET --silent --SILENT
|
||||||
|
Quiet the terminal bell.
|
||||||
|
-r -R .... --raw-control-chars --RAW-CONTROL-CHARS
|
||||||
|
Output "raw" control characters.
|
||||||
|
-s ........ --squeeze-blank-lines
|
||||||
|
Squeeze multiple blank lines.
|
||||||
|
-S ........ --chop-long-lines
|
||||||
|
Chop (truncate) long lines rather than wrapping.
|
||||||
|
-t [_t_a_g] .. --tag=[_t_a_g]
|
||||||
|
Find a tag.
|
||||||
|
-T [_t_a_g_s_f_i_l_e] --tag-file=[_t_a_g_s_f_i_l_e]
|
||||||
|
Use an alternate tags file.
|
||||||
|
-u -U .... --underline-special --UNDERLINE-SPECIAL
|
||||||
|
Change handling of backspaces.
|
||||||
|
-V ........ --version
|
||||||
|
Display the version number of "less".
|
||||||
|
-w ........ --hilite-unread
|
||||||
|
Highlight first new line after forward-screen.
|
||||||
|
-W ........ --HILITE-UNREAD
|
||||||
|
Highlight first new line after any forward movement.
|
||||||
|
-x [_N[,...]] --tabs=[_N[,...]]
|
||||||
|
Set tab stops.
|
||||||
|
-X ........ --no-init
|
||||||
|
Don't use termcap init/deinit strings.
|
||||||
|
-y [_N] .... --max-forw-scroll=[_N]
|
||||||
|
Forward scroll limit.
|
||||||
|
-z [_N] .... --window=[_N]
|
||||||
|
Set size of window.
|
||||||
|
-" [_c[_c]] . --quotes=[_c[_c]]
|
||||||
|
Set shell quote characters.
|
||||||
|
-~ ........ --tilde
|
||||||
|
Don't display tildes after end of file.
|
||||||
|
-# [_N] .... --shift=[_N]
|
||||||
|
Set horizontal scroll amount (0 = one half screen width).
|
||||||
|
--file-size
|
||||||
|
Automatically determine the size of the input file.
|
||||||
|
--follow-name
|
||||||
|
The F command changes files if the input file is renamed.
|
||||||
|
--incsearch
|
||||||
|
Search file as each pattern character is typed in.
|
||||||
|
--line-num-width=N
|
||||||
|
Set the width of the -N line number field to N characters.
|
||||||
|
--mouse
|
||||||
|
Enable mouse input.
|
||||||
|
--no-keypad
|
||||||
|
Don't send termcap keypad init/deinit strings.
|
||||||
|
--no-histdups
|
||||||
|
Remove duplicates from command history.
|
||||||
|
--rscroll=C
|
||||||
|
Set the character used to mark truncated lines.
|
||||||
|
--save-marks
|
||||||
|
Retain marks across invocations of less.
|
||||||
|
--status-col-width=N
|
||||||
|
Set the width of the -J status column to N characters.
|
||||||
|
--use-backslash
|
||||||
|
Subsequent options use backslash as escape char.
|
||||||
|
--use-color
|
||||||
|
Enables colored text.
|
||||||
|
--wheel-lines=N
|
||||||
|
Each click of the mouse wheel moves N lines.
|
||||||
|
|
||||||
|
|
||||||
|
---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
LLIINNEE EEDDIITTIINNGG
|
||||||
|
|
||||||
|
These keys can be used to edit text being entered
|
||||||
|
on the "command line" at the bottom of the screen.
|
||||||
|
|
||||||
|
RightArrow ..................... ESC-l ... Move cursor right one character.
|
||||||
|
LeftArrow ...................... ESC-h ... Move cursor left one character.
|
||||||
|
ctrl-RightArrow ESC-RightArrow ESC-w ... Move cursor right one word.
|
||||||
|
ctrl-LeftArrow ESC-LeftArrow ESC-b ... Move cursor left one word.
|
||||||
|
HOME ........................... ESC-0 ... Move cursor to start of line.
|
||||||
|
END ............................ ESC-$ ... Move cursor to end of line.
|
||||||
|
BACKSPACE ................................ Delete char to left of cursor.
|
||||||
|
DELETE ......................... ESC-x ... Delete char under cursor.
|
||||||
|
ctrl-BACKSPACE ESC-BACKSPACE ........... Delete word to left of cursor.
|
||||||
|
ctrl-DELETE .... ESC-DELETE .... ESC-X ... Delete word under cursor.
|
||||||
|
ctrl-U ......... ESC (MS-DOS only) ....... Delete entire line.
|
||||||
|
UpArrow ........................ ESC-k ... Retrieve previous command line.
|
||||||
|
DownArrow ...................... ESC-j ... Retrieve next command line.
|
||||||
|
TAB ...................................... Complete filename & cycle.
|
||||||
|
SHIFT-TAB ...................... ESC-TAB Complete filename & reverse cycle.
|
||||||
|
ctrl-L ................................... Complete filename, list all.
|
||||||
209
app/main.py
209
app/main.py
@ -10,26 +10,44 @@ from contextlib import asynccontextmanager
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from fastapi import Depends, FastAPI, Header, HTTPException, Request, Security, status
|
from fastapi import (
|
||||||
|
Depends,
|
||||||
|
FastAPI,
|
||||||
|
Header,
|
||||||
|
HTTPException,
|
||||||
|
Request,
|
||||||
|
Security,
|
||||||
|
status,
|
||||||
|
)
|
||||||
from fastapi.responses import JSONResponse
|
from fastapi.responses import JSONResponse
|
||||||
from fastapi.security import APIKeyHeader
|
from fastapi.security import APIKeyHeader
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
if str(PROJECT_ROOT) not in sys.path:
|
if str(PROJECT_ROOT) not in sys.path:
|
||||||
sys.path.insert(0, str(PROJECT_ROOT))
|
sys.path.insert(0, str(PROJECT_ROOT))
|
||||||
|
|
||||||
|
|
||||||
from scripts.common import DB_FILE, ZPWIKI_ROOT
|
from scripts.common import DB_FILE, ZPWIKI_ROOT
|
||||||
from scripts.rebuild_index import ReindexInProgressError, rebuild_index
|
from scripts.rebuild_index import (
|
||||||
|
ReindexInProgressError,
|
||||||
|
rebuild_index,
|
||||||
|
)
|
||||||
from scripts.search_utils import search_database
|
from scripts.search_utils import search_database
|
||||||
|
|
||||||
|
|
||||||
MIN_SECRET_LENGTH = 32
|
MIN_SECRET_LENGTH = 32
|
||||||
|
|
||||||
|
SEARCH_API_KEY_HEADER = "X-API-Key"
|
||||||
SYNC_API_KEY_HEADER = "X-API-Key"
|
SYNC_API_KEY_HEADER = "X-API-Key"
|
||||||
|
|
||||||
|
|
||||||
|
search_api_key_scheme = APIKeyHeader(
|
||||||
|
name=SEARCH_API_KEY_HEADER,
|
||||||
|
auto_error=False,
|
||||||
|
description="API kľúč pre vyhľadávanie v zpwiki.",
|
||||||
|
)
|
||||||
|
|
||||||
sync_api_key_scheme = APIKeyHeader(
|
sync_api_key_scheme = APIKeyHeader(
|
||||||
name=SYNC_API_KEY_HEADER,
|
name=SYNC_API_KEY_HEADER,
|
||||||
auto_error=False,
|
auto_error=False,
|
||||||
@ -38,10 +56,22 @@ sync_api_key_scheme = APIKeyHeader(
|
|||||||
|
|
||||||
|
|
||||||
class SearchRequest(BaseModel):
|
class SearchRequest(BaseModel):
|
||||||
query: str = Field(..., min_length=1, max_length=500)
|
query: str = Field(
|
||||||
limit: int = Field(default=10, ge=1, le=50)
|
...,
|
||||||
|
min_length=1,
|
||||||
|
max_length=500,
|
||||||
|
)
|
||||||
|
limit: int = Field(
|
||||||
|
default=10,
|
||||||
|
ge=1,
|
||||||
|
le=50,
|
||||||
|
)
|
||||||
published_only: bool = False
|
published_only: bool = False
|
||||||
max_per_document: int = Field(default=3, ge=0, le=10)
|
max_per_document: int = Field(
|
||||||
|
default=3,
|
||||||
|
ge=0,
|
||||||
|
le=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class SyncRequest(BaseModel):
|
class SyncRequest(BaseModel):
|
||||||
@ -55,7 +85,9 @@ def required_environment_value(name: str) -> str:
|
|||||||
value = os.getenv(name, "").strip()
|
value = os.getenv(name, "").strip()
|
||||||
|
|
||||||
if not value:
|
if not value:
|
||||||
raise RuntimeError(f"Chýba povinná environment premenná {name}")
|
raise RuntimeError(
|
||||||
|
f"Chýba povinná environment premenná {name}"
|
||||||
|
)
|
||||||
|
|
||||||
return value
|
return value
|
||||||
|
|
||||||
@ -72,53 +104,104 @@ def validate_secret(name: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def expected_gitea_repository() -> str:
|
def expected_gitea_repository() -> str:
|
||||||
value = required_environment_value("EXPECTED_GITEA_REPOSITORY")
|
value = required_environment_value(
|
||||||
|
"EXPECTED_GITEA_REPOSITORY"
|
||||||
|
)
|
||||||
|
|
||||||
if "/" not in value:
|
if "/" not in value:
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
"EXPECTED_GITEA_REPOSITORY musí mať tvar vlastník/repozitár"
|
"EXPECTED_GITEA_REPOSITORY musí mať tvar "
|
||||||
|
"vlastník/repozitár"
|
||||||
)
|
)
|
||||||
|
|
||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
def webhook_should_pull_git() -> bool:
|
def webhook_should_pull_git() -> bool:
|
||||||
value = os.getenv("WEBHOOK_PULL_GIT", "false").strip().casefold()
|
value = os.getenv(
|
||||||
|
"WEBHOOK_PULL_GIT",
|
||||||
|
"false",
|
||||||
|
).strip().casefold()
|
||||||
|
|
||||||
return value in {"1", "true", "yes", "on"}
|
return value in {
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
"on",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def validate_security_configuration() -> None:
|
def validate_security_configuration() -> None:
|
||||||
validate_secret("WEBHOOK_SECRET")
|
validate_secret("WEBHOOK_SECRET")
|
||||||
validate_secret("SYNC_API_KEY")
|
validate_secret("SYNC_API_KEY")
|
||||||
|
validate_secret("SEARCH_API_KEY")
|
||||||
expected_gitea_repository()
|
expected_gitea_repository()
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def lifespan(_: FastAPI):
|
async def lifespan(_: FastAPI):
|
||||||
# Aplikácia sa nespustí s chýbajúcim alebo slabým tajomstvom.
|
|
||||||
validate_security_configuration()
|
validate_security_configuration()
|
||||||
yield
|
yield
|
||||||
|
|
||||||
|
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
title="ZP Agent API",
|
title="ZP Agent API",
|
||||||
description="API pre vyhľadávanie v repozitári záverečných prác zpwiki.",
|
description=(
|
||||||
|
"API pre vyhľadávanie v repozitári "
|
||||||
|
"záverečných prác zpwiki."
|
||||||
|
),
|
||||||
version="0.6.0",
|
version="0.6.0",
|
||||||
lifespan=lifespan,
|
lifespan=lifespan,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def require_sync_api_key(
|
def require_search_api_key(
|
||||||
api_key: str | None = Security(sync_api_key_scheme),
|
api_key: str | None = Security(
|
||||||
|
search_api_key_scheme
|
||||||
|
),
|
||||||
) -> None:
|
) -> None:
|
||||||
expected = validate_secret("SYNC_API_KEY")
|
expected = validate_secret(
|
||||||
|
"SEARCH_API_KEY"
|
||||||
|
)
|
||||||
|
|
||||||
if not api_key or not hmac.compare_digest(api_key, expected):
|
if (
|
||||||
|
not api_key
|
||||||
|
or not hmac.compare_digest(
|
||||||
|
api_key,
|
||||||
|
expected,
|
||||||
|
)
|
||||||
|
):
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
detail="Neplatný alebo chýbajúci API kľúč",
|
detail="Neplatný alebo chýbajúci API kľúč",
|
||||||
headers={"WWW-Authenticate": "ApiKey"},
|
headers={
|
||||||
|
"WWW-Authenticate": "ApiKey",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def require_sync_api_key(
|
||||||
|
api_key: str | None = Security(
|
||||||
|
sync_api_key_scheme
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
expected = validate_secret(
|
||||||
|
"SYNC_API_KEY"
|
||||||
|
)
|
||||||
|
|
||||||
|
if (
|
||||||
|
not api_key
|
||||||
|
or not hmac.compare_digest(
|
||||||
|
api_key,
|
||||||
|
expected,
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Neplatný alebo chýbajúci API kľúč",
|
||||||
|
headers={
|
||||||
|
"WWW-Authenticate": "ApiKey",
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@ -132,16 +215,18 @@ def verify_gitea_signature(
|
|||||||
|
|
||||||
supplied = signature.strip().casefold()
|
supplied = signature.strip().casefold()
|
||||||
|
|
||||||
# X-Gitea-Signature je čistý hex digest. Prefix prijímame iba
|
# Kompatibilita podpisu.
|
||||||
# kvôli kompatibilite s X-Hub-Signature-256.
|
|
||||||
if supplied.startswith("sha256="):
|
if supplied.startswith("sha256="):
|
||||||
supplied = supplied.removeprefix("sha256=")
|
supplied = supplied.removeprefix(
|
||||||
|
"sha256="
|
||||||
|
)
|
||||||
|
|
||||||
if len(supplied) != 64:
|
if len(supplied) != 64:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
try:
|
try:
|
||||||
int(supplied, 16)
|
int(supplied, 16)
|
||||||
|
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
@ -151,7 +236,10 @@ def verify_gitea_signature(
|
|||||||
hashlib.sha256,
|
hashlib.sha256,
|
||||||
).hexdigest()
|
).hexdigest()
|
||||||
|
|
||||||
return hmac.compare_digest(expected, supplied)
|
return hmac.compare_digest(
|
||||||
|
expected,
|
||||||
|
supplied,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def repository_name_from_payload(
|
def repository_name_from_payload(
|
||||||
@ -195,17 +283,28 @@ def health() -> dict[str, Any]:
|
|||||||
"zpwiki_root": str(ZPWIKI_ROOT),
|
"zpwiki_root": str(ZPWIKI_ROOT),
|
||||||
"zpwiki_exists": ZPWIKI_ROOT.exists(),
|
"zpwiki_exists": ZPWIKI_ROOT.exists(),
|
||||||
"security_configured": all(
|
"security_configured": all(
|
||||||
bool(os.getenv(name, "").strip())
|
bool(
|
||||||
|
os.getenv(
|
||||||
|
name,
|
||||||
|
"",
|
||||||
|
).strip()
|
||||||
|
)
|
||||||
for name in (
|
for name in (
|
||||||
"WEBHOOK_SECRET",
|
"WEBHOOK_SECRET",
|
||||||
"SYNC_API_KEY",
|
"SYNC_API_KEY",
|
||||||
|
"SEARCH_API_KEY",
|
||||||
"EXPECTED_GITEA_REPOSITORY",
|
"EXPECTED_GITEA_REPOSITORY",
|
||||||
)
|
)
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@app.post("/search")
|
@app.post(
|
||||||
|
"/search",
|
||||||
|
dependencies=[
|
||||||
|
Depends(require_search_api_key)
|
||||||
|
],
|
||||||
|
)
|
||||||
def search(
|
def search(
|
||||||
request: SearchRequest,
|
request: SearchRequest,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
@ -214,8 +313,12 @@ def search(
|
|||||||
DB_FILE,
|
DB_FILE,
|
||||||
request.query,
|
request.query,
|
||||||
request.limit,
|
request.limit,
|
||||||
published_only=request.published_only,
|
published_only=(
|
||||||
max_per_document=request.max_per_document,
|
request.published_only
|
||||||
|
),
|
||||||
|
max_per_document=(
|
||||||
|
request.max_per_document
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
except FileNotFoundError as error:
|
except FileNotFoundError as error:
|
||||||
@ -249,7 +352,9 @@ def search(
|
|||||||
|
|
||||||
@app.post(
|
@app.post(
|
||||||
"/sync",
|
"/sync",
|
||||||
dependencies=[Depends(require_sync_api_key)],
|
dependencies=[
|
||||||
|
Depends(require_sync_api_key)
|
||||||
|
],
|
||||||
)
|
)
|
||||||
def sync(
|
def sync(
|
||||||
request: SyncRequest,
|
request: SyncRequest,
|
||||||
@ -274,7 +379,9 @@ def sync(
|
|||||||
return {
|
return {
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"pull_git": request.pull_git,
|
"pull_git": request.pull_git,
|
||||||
"duration_seconds": result["duration_seconds"],
|
"duration_seconds": (
|
||||||
|
result["duration_seconds"]
|
||||||
|
),
|
||||||
"counts": result["counts"],
|
"counts": result["counts"],
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -295,7 +402,10 @@ async def gitea_webhook(
|
|||||||
),
|
),
|
||||||
) -> dict[str, Any] | JSONResponse:
|
) -> dict[str, Any] | JSONResponse:
|
||||||
raw_body = await request.body()
|
raw_body = await request.body()
|
||||||
secret = validate_secret("WEBHOOK_SECRET")
|
|
||||||
|
secret = validate_secret(
|
||||||
|
"WEBHOOK_SECRET"
|
||||||
|
)
|
||||||
|
|
||||||
if not verify_gitea_signature(
|
if not verify_gitea_signature(
|
||||||
raw_body,
|
raw_body,
|
||||||
@ -303,7 +413,9 @@ async def gitea_webhook(
|
|||||||
secret,
|
secret,
|
||||||
):
|
):
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
status_code=(
|
||||||
|
status.HTTP_401_UNAUTHORIZED
|
||||||
|
),
|
||||||
detail="Neplatný webhook podpis",
|
detail="Neplatný webhook podpis",
|
||||||
)
|
)
|
||||||
|
|
||||||
@ -318,24 +430,35 @@ async def gitea_webhook(
|
|||||||
) as error:
|
) as error:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=400,
|
status_code=400,
|
||||||
detail="Webhook payload nie je platný JSON",
|
detail=(
|
||||||
|
"Webhook payload nie je "
|
||||||
|
"platný JSON"
|
||||||
|
),
|
||||||
) from error
|
) from error
|
||||||
|
|
||||||
if not isinstance(payload, dict):
|
if not isinstance(payload, dict):
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=400,
|
status_code=400,
|
||||||
detail="Webhook payload musí byť JSON objekt",
|
detail=(
|
||||||
|
"Webhook payload musí byť "
|
||||||
|
"JSON objekt"
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
if not x_gitea_event:
|
if not x_gitea_event:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=400,
|
status_code=400,
|
||||||
detail="Chýba hlavička X-Gitea-Event",
|
detail=(
|
||||||
|
"Chýba hlavička "
|
||||||
|
"X-Gitea-Event"
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
if x_gitea_event.casefold() != "push":
|
if x_gitea_event.casefold() != "push":
|
||||||
return JSONResponse(
|
return JSONResponse(
|
||||||
status_code=status.HTTP_202_ACCEPTED,
|
status_code=(
|
||||||
|
status.HTTP_202_ACCEPTED
|
||||||
|
),
|
||||||
content={
|
content={
|
||||||
"status": "ignored",
|
"status": "ignored",
|
||||||
"reason": "unsupported_event",
|
"reason": "unsupported_event",
|
||||||
@ -343,9 +466,11 @@ async def gitea_webhook(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
repository_name = repository_name_from_payload(
|
repository_name = (
|
||||||
|
repository_name_from_payload(
|
||||||
payload
|
payload
|
||||||
)
|
)
|
||||||
|
)
|
||||||
|
|
||||||
if repository_name is None:
|
if repository_name is None:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
@ -356,7 +481,9 @@ async def gitea_webhook(
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
expected_repository = expected_gitea_repository()
|
expected_repository = (
|
||||||
|
expected_gitea_repository()
|
||||||
|
)
|
||||||
|
|
||||||
if not same_repository(
|
if not same_repository(
|
||||||
repository_name,
|
repository_name,
|
||||||
@ -373,7 +500,9 @@ async def gitea_webhook(
|
|||||||
try:
|
try:
|
||||||
result = await asyncio.to_thread(
|
result = await asyncio.to_thread(
|
||||||
rebuild_index,
|
rebuild_index,
|
||||||
pull_git=webhook_should_pull_git(),
|
pull_git=(
|
||||||
|
webhook_should_pull_git()
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
except ReindexInProgressError as error:
|
except ReindexInProgressError as error:
|
||||||
@ -393,6 +522,8 @@ async def gitea_webhook(
|
|||||||
"event": x_gitea_event,
|
"event": x_gitea_event,
|
||||||
"repository": repository_name,
|
"repository": repository_name,
|
||||||
"verified_by": "hmac_sha256",
|
"verified_by": "hmac_sha256",
|
||||||
"duration_seconds": result["duration_seconds"],
|
"duration_seconds": (
|
||||||
|
result["duration_seconds"]
|
||||||
|
),
|
||||||
"counts": result["counts"],
|
"counts": result["counts"],
|
||||||
}
|
}
|
||||||
|
|||||||
40
et -a
Normal file
40
et -a
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
README.md[36m:[m66[36m:[m[1;31mWEBHOOK_SECRET[m=<náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
README.md[36m:[m67[36m:[m[1;31mSYNC_API_KEY[m=<iná náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
README.md[36m:[m68[36m:[m[1;31mSEARCH_API_KEY[m=<ďalšia náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
README.md[36m:[m162[36m:[m -H "X-API-Key: $[1;31mSEARCH_API_KEY[m" \
|
||||||
|
README.md[36m:[m182[36m:[m -H "X-API-Key: $[1;31mSYNC_API_KEY[m" \
|
||||||
|
app/main.py[36m:[m41[36m:[m[1;31mSEARCH_API_KEY[m_HEADER = "X-API-Key"
|
||||||
|
app/main.py[36m:[m42[36m:[m[1;31mSYNC_API_KEY[m_HEADER = "X-API-Key"
|
||||||
|
app/main.py[36m:[m45[36m:[m[1;31msearch_api_key[m_scheme = APIKeyHeader(
|
||||||
|
app/main.py[36m:[m46[36m:[m name=[1;31mSEARCH_API_KEY[m_HEADER,
|
||||||
|
app/main.py[36m:[m51[36m:[m[1;31msync_api_key[m_scheme = APIKeyHeader(
|
||||||
|
app/main.py[36m:[m52[36m:[m name=[1;31mSYNC_API_KEY[m_HEADER,
|
||||||
|
app/main.py[36m:[m135[36m:[m validate_secret("[1;31mWEBHOOK_SECRET[m")
|
||||||
|
app/main.py[36m:[m136[36m:[m validate_secret("[1;31mSYNC_API_KEY[m")
|
||||||
|
app/main.py[36m:[m137[36m:[m validate_secret("[1;31mSEARCH_API_KEY[m")
|
||||||
|
app/main.py[36m:[m158[36m:[mdef require_[1;31msearch_api_key[m(
|
||||||
|
app/main.py[36m:[m160[36m:[m [1;31msearch_api_key[m_scheme
|
||||||
|
app/main.py[36m:[m164[36m:[m "[1;31mSEARCH_API_KEY[m"
|
||||||
|
app/main.py[36m:[m183[36m:[mdef require_[1;31msync_api_key[m(
|
||||||
|
app/main.py[36m:[m185[36m:[m [1;31msync_api_key[m_scheme
|
||||||
|
app/main.py[36m:[m189[36m:[m "[1;31mSYNC_API_KEY[m"
|
||||||
|
app/main.py[36m:[m293[36m:[m "[1;31mWEBHOOK_SECRET[m",
|
||||||
|
app/main.py[36m:[m294[36m:[m "[1;31mSYNC_API_KEY[m",
|
||||||
|
app/main.py[36m:[m295[36m:[m "[1;31mSEARCH_API_KEY[m",
|
||||||
|
app/main.py[36m:[m305[36m:[m Depends(require_[1;31msearch_api_key[m)
|
||||||
|
app/main.py[36m:[m356[36m:[m Depends(require_[1;31msync_api_key[m)
|
||||||
|
app/main.py[36m:[m407[36m:[m "[1;31mWEBHOOK_SECRET[m"
|
||||||
|
test/conftest.py[36m:[m21[36m:[m "[1;31mWEBHOOK_SECRET[m",
|
||||||
|
test/conftest.py[36m:[m26[36m:[m "[1;31mSYNC_API_KEY[m",
|
||||||
|
test/conftest.py[36m:[m31[36m:[m "[1;31mSEARCH_API_KEY[m",
|
||||||
|
test/test_api.py[36m:[m14[36m:[m[1;31mWEBHOOK_SECRET[m = "w" * 64
|
||||||
|
test/test_api.py[36m:[m15[36m:[m[1;31mSYNC_API_KEY[m = "s" * 64
|
||||||
|
test/test_api.py[36m:[m16[36m:[m[1;31mSEARCH_API_KEY[m = "a" * 64
|
||||||
|
test/test_api.py[36m:[m34[36m:[m [1;31mWEBHOOK_SECRET[m.encode("utf-8"),
|
||||||
|
test/test_api.py[36m:[m49[36m:[m monkeypatch.delenv("[1;31mWEBHOOK_SECRET[m")
|
||||||
|
test/test_api.py[36m:[m53[36m:[m match="[1;31mWEBHOOK_SECRET[m",
|
||||||
|
test/test_api.py[36m:[m63[36m:[m "[1;31mSYNC_API_KEY[m",
|
||||||
|
test/test_api.py[36m:[m111[36m:[m "X-API-Key": [1;31mSEARCH_API_KEY[m,
|
||||||
|
test/test_api.py[36m:[m133[36m:[m "X-API-Key": [1;31mSEARCH_API_KEY[m,
|
||||||
|
test/test_api.py[36m:[m216[36m:[m "X-API-Key": [1;31mSYNC_API_KEY[m,
|
||||||
|
test/test_api.py[36m:[m245[36m:[m "X-API-Key": [1;31mSYNC_API_KEY[m,
|
||||||
@ -4,3 +4,5 @@ python-frontmatter==1.3.0
|
|||||||
rich==15.0.0
|
rich==15.0.0
|
||||||
tiktoken>=0.8,<1
|
tiktoken>=0.8,<1
|
||||||
uvicorn[standard]==0.48.0
|
uvicorn[standard]==0.48.0
|
||||||
|
numpy==2.2.6
|
||||||
|
sentence-transformers==5.7.0
|
||||||
|
|||||||
@ -16,7 +16,18 @@ if str(PROJECT_ROOT) not in sys.path:
|
|||||||
sys.path.insert(0, str(PROJECT_ROOT))
|
sys.path.insert(0, str(PROJECT_ROOT))
|
||||||
|
|
||||||
|
|
||||||
from scripts.common import CHUNKS_FILE, DB_FILE, DOCUMENTS_FILE, read_json
|
from scripts.common import (
|
||||||
|
CHUNKS_FILE,
|
||||||
|
DB_FILE,
|
||||||
|
DOCUMENTS_FILE,
|
||||||
|
read_json,
|
||||||
|
)
|
||||||
|
from scripts.embedding_utils import (
|
||||||
|
build_embedding_text,
|
||||||
|
embed_passages,
|
||||||
|
embedding_model_name,
|
||||||
|
vector_to_blob,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
FTS_TOKENIZER = "unicode61 remove_diacritics 2"
|
FTS_TOKENIZER = "unicode61 remove_diacritics 2"
|
||||||
@ -33,21 +44,29 @@ def published_to_db(value: Any) -> int | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def verify_fts5(conn: sqlite3.Connection) -> None:
|
def verify_fts5(
|
||||||
"""Overí, či aktuálna SQLite knižnica podporuje FTS5."""
|
conn: sqlite3.Connection,
|
||||||
|
) -> None:
|
||||||
try:
|
try:
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"CREATE VIRTUAL TABLE temp.fts5_check USING fts5(value)"
|
"CREATE VIRTUAL TABLE temp.fts5_check "
|
||||||
|
"USING fts5(value)"
|
||||||
)
|
)
|
||||||
conn.execute("DROP TABLE temp.fts5_check")
|
conn.execute(
|
||||||
|
"DROP TABLE temp.fts5_check"
|
||||||
|
)
|
||||||
|
|
||||||
except sqlite3.OperationalError as error:
|
except sqlite3.OperationalError as error:
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
"Táto inštalácia SQLite nemá dostupné FTS5. "
|
"Táto inštalácia SQLite nemá dostupné FTS5. "
|
||||||
"Použi Python/SQLite zostavenie s podporou SQLITE_ENABLE_FTS5."
|
"Použi Python/SQLite zostavenie s podporou "
|
||||||
|
"SQLITE_ENABLE_FTS5."
|
||||||
) from error
|
) from error
|
||||||
|
|
||||||
|
|
||||||
def create_tables(conn: sqlite3.Connection) -> None:
|
def create_tables(
|
||||||
|
conn: sqlite3.Connection,
|
||||||
|
) -> None:
|
||||||
conn.executescript(
|
conn.executescript(
|
||||||
f"""
|
f"""
|
||||||
PRAGMA foreign_keys = ON;
|
PRAGMA foreign_keys = ON;
|
||||||
@ -58,9 +77,14 @@ def create_tables(conn: sqlite3.Connection) -> None:
|
|||||||
title TEXT,
|
title TEXT,
|
||||||
author TEXT,
|
author TEXT,
|
||||||
published INTEGER
|
published INTEGER
|
||||||
CHECK (published IN (0, 1) OR published IS NULL),
|
CHECK (
|
||||||
content_length INTEGER NOT NULL DEFAULT 0,
|
published IN (0, 1)
|
||||||
metadata_json TEXT NOT NULL DEFAULT '{{}}'
|
OR published IS NULL
|
||||||
|
),
|
||||||
|
content_length INTEGER
|
||||||
|
NOT NULL DEFAULT 0,
|
||||||
|
metadata_json TEXT
|
||||||
|
NOT NULL DEFAULT '{{}}'
|
||||||
);
|
);
|
||||||
|
|
||||||
CREATE TABLE chunks (
|
CREATE TABLE chunks (
|
||||||
@ -70,11 +94,16 @@ def create_tables(conn: sqlite3.Connection) -> None:
|
|||||||
title TEXT,
|
title TEXT,
|
||||||
author TEXT,
|
author TEXT,
|
||||||
published INTEGER
|
published INTEGER
|
||||||
CHECK (published IN (0, 1) OR published IS NULL),
|
CHECK (
|
||||||
|
published IN (0, 1)
|
||||||
|
OR published IS NULL
|
||||||
|
),
|
||||||
chunk_index INTEGER NOT NULL,
|
chunk_index INTEGER NOT NULL,
|
||||||
heading_paths_json TEXT NOT NULL DEFAULT '[]',
|
heading_paths_json TEXT
|
||||||
|
NOT NULL DEFAULT '[]',
|
||||||
text TEXT NOT NULL,
|
text TEXT NOT NULL,
|
||||||
text_length INTEGER NOT NULL DEFAULT 0,
|
text_length INTEGER
|
||||||
|
NOT NULL DEFAULT 0,
|
||||||
token_count INTEGER,
|
token_count INTEGER,
|
||||||
content_hash TEXT,
|
content_hash TEXT,
|
||||||
FOREIGN KEY(document_path)
|
FOREIGN KEY(document_path)
|
||||||
@ -103,6 +132,17 @@ def create_tables(conn: sqlite3.Connection) -> None:
|
|||||||
ON DELETE CASCADE
|
ON DELETE CASCADE
|
||||||
);
|
);
|
||||||
|
|
||||||
|
CREATE TABLE chunk_embeddings (
|
||||||
|
chunk_id TEXT PRIMARY KEY,
|
||||||
|
model TEXT NOT NULL,
|
||||||
|
dimensions INTEGER NOT NULL,
|
||||||
|
embedding BLOB NOT NULL,
|
||||||
|
FOREIGN KEY(chunk_id)
|
||||||
|
REFERENCES chunks(chunk_id)
|
||||||
|
ON UPDATE CASCADE
|
||||||
|
ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
CREATE INDEX idx_documents_path
|
CREATE INDEX idx_documents_path
|
||||||
ON documents(path);
|
ON documents(path);
|
||||||
|
|
||||||
@ -127,6 +167,9 @@ def create_tables(conn: sqlite3.Connection) -> None:
|
|||||||
CREATE INDEX idx_chunk_categories_category
|
CREATE INDEX idx_chunk_categories_category
|
||||||
ON chunk_categories(category);
|
ON chunk_categories(category);
|
||||||
|
|
||||||
|
CREATE INDEX idx_chunk_embeddings_model
|
||||||
|
ON chunk_embeddings(model);
|
||||||
|
|
||||||
CREATE VIRTUAL TABLE chunks_fts USING fts5(
|
CREATE VIRTUAL TABLE chunks_fts USING fts5(
|
||||||
chunk_id UNINDEXED,
|
chunk_id UNINDEXED,
|
||||||
title,
|
title,
|
||||||
@ -151,8 +194,13 @@ def insert_documents(
|
|||||||
document.get("path"),
|
document.get("path"),
|
||||||
document.get("title"),
|
document.get("title"),
|
||||||
document.get("author"),
|
document.get("author"),
|
||||||
published_to_db(document.get("published")),
|
published_to_db(
|
||||||
int(document.get("content_length") or 0),
|
document.get("published")
|
||||||
|
),
|
||||||
|
int(
|
||||||
|
document.get("content_length")
|
||||||
|
or 0
|
||||||
|
),
|
||||||
json.dumps(
|
json.dumps(
|
||||||
document.get("metadata") or {},
|
document.get("metadata") or {},
|
||||||
ensure_ascii=False,
|
ensure_ascii=False,
|
||||||
@ -184,14 +232,19 @@ def insert_chunks(
|
|||||||
) -> None:
|
) -> None:
|
||||||
chunk_rows: list[tuple] = []
|
chunk_rows: list[tuple] = []
|
||||||
tag_rows: list[tuple[str, str]] = []
|
tag_rows: list[tuple[str, str]] = []
|
||||||
category_rows: list[tuple[str, str]] = []
|
category_rows: list[
|
||||||
|
tuple[str, str]
|
||||||
|
] = []
|
||||||
|
|
||||||
for chunk in chunks:
|
for chunk in chunks:
|
||||||
chunk_id = str(chunk.get("chunk_id") or "").strip()
|
chunk_id = str(
|
||||||
|
chunk.get("chunk_id") or ""
|
||||||
|
).strip()
|
||||||
|
|
||||||
if not chunk_id:
|
if not chunk_id:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"Chunk bez chunk_id nie je možné indexovať"
|
"Chunk bez chunk_id nie je "
|
||||||
|
"možné indexovať"
|
||||||
)
|
)
|
||||||
|
|
||||||
text = chunk.get("text") or ""
|
text = chunk.get("text") or ""
|
||||||
@ -202,14 +255,25 @@ def insert_chunks(
|
|||||||
chunk.get("document_path"),
|
chunk.get("document_path"),
|
||||||
chunk.get("title"),
|
chunk.get("title"),
|
||||||
chunk.get("author"),
|
chunk.get("author"),
|
||||||
published_to_db(chunk.get("published")),
|
published_to_db(
|
||||||
int(chunk.get("chunk_index") or 0),
|
chunk.get("published")
|
||||||
|
),
|
||||||
|
int(
|
||||||
|
chunk.get("chunk_index")
|
||||||
|
or 0
|
||||||
|
),
|
||||||
json.dumps(
|
json.dumps(
|
||||||
chunk.get("heading_paths") or [],
|
chunk.get(
|
||||||
|
"heading_paths"
|
||||||
|
)
|
||||||
|
or [],
|
||||||
ensure_ascii=False,
|
ensure_ascii=False,
|
||||||
),
|
),
|
||||||
text,
|
text,
|
||||||
int(chunk.get("text_length") or len(text)),
|
int(
|
||||||
|
chunk.get("text_length")
|
||||||
|
or len(text)
|
||||||
|
),
|
||||||
chunk.get("token_count"),
|
chunk.get("token_count"),
|
||||||
chunk.get("content_hash"),
|
chunk.get("content_hash"),
|
||||||
)
|
)
|
||||||
@ -219,13 +283,27 @@ def insert_chunks(
|
|||||||
value = str(tag).strip()
|
value = str(tag).strip()
|
||||||
|
|
||||||
if value:
|
if value:
|
||||||
tag_rows.append((chunk_id, value))
|
tag_rows.append(
|
||||||
|
(
|
||||||
|
chunk_id,
|
||||||
|
value,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
for category in chunk.get("categories") or []:
|
for category in (
|
||||||
value = str(category).strip()
|
chunk.get("categories") or []
|
||||||
|
):
|
||||||
|
value = str(
|
||||||
|
category
|
||||||
|
).strip()
|
||||||
|
|
||||||
if value:
|
if value:
|
||||||
category_rows.append((chunk_id, value))
|
category_rows.append(
|
||||||
|
(
|
||||||
|
chunk_id,
|
||||||
|
value,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
conn.executemany(
|
conn.executemany(
|
||||||
"""
|
"""
|
||||||
@ -270,8 +348,83 @@ def insert_chunks(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_fts_index(conn: sqlite3.Connection) -> None:
|
def insert_embeddings(
|
||||||
"""Vytvorí FTS5 index nad chunkmi a ich metadátami."""
|
conn: sqlite3.Connection,
|
||||||
|
chunks: list[dict],
|
||||||
|
) -> None:
|
||||||
|
if not chunks:
|
||||||
|
return
|
||||||
|
|
||||||
|
chunk_ids: list[str] = []
|
||||||
|
texts: list[str] = []
|
||||||
|
|
||||||
|
for chunk in chunks:
|
||||||
|
chunk_id = str(
|
||||||
|
chunk.get("chunk_id") or ""
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
if not chunk_id:
|
||||||
|
raise ValueError(
|
||||||
|
"Chunk bez chunk_id nemôže "
|
||||||
|
"mať embedding"
|
||||||
|
)
|
||||||
|
|
||||||
|
chunk_ids.append(
|
||||||
|
chunk_id
|
||||||
|
)
|
||||||
|
|
||||||
|
texts.append(
|
||||||
|
build_embedding_text(
|
||||||
|
chunk
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
vectors = embed_passages(
|
||||||
|
texts
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(vectors) != len(chunk_ids):
|
||||||
|
raise RuntimeError(
|
||||||
|
"Počet embeddingov sa "
|
||||||
|
"nezhoduje s počtom chunkov"
|
||||||
|
)
|
||||||
|
|
||||||
|
model_name = (
|
||||||
|
embedding_model_name()
|
||||||
|
)
|
||||||
|
|
||||||
|
rows: list[tuple] = []
|
||||||
|
|
||||||
|
for chunk_id, vector in zip(
|
||||||
|
chunk_ids,
|
||||||
|
vectors,
|
||||||
|
):
|
||||||
|
rows.append(
|
||||||
|
(
|
||||||
|
chunk_id,
|
||||||
|
model_name,
|
||||||
|
int(vector.shape[0]),
|
||||||
|
vector_to_blob(vector),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
conn.executemany(
|
||||||
|
"""
|
||||||
|
INSERT INTO chunk_embeddings (
|
||||||
|
chunk_id,
|
||||||
|
model,
|
||||||
|
dimensions,
|
||||||
|
embedding
|
||||||
|
)
|
||||||
|
VALUES (?, ?, ?, ?)
|
||||||
|
""",
|
||||||
|
rows,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def build_fts_index(
|
||||||
|
conn: sqlite3.Connection,
|
||||||
|
) -> None:
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"""
|
"""
|
||||||
INSERT INTO chunks_fts (
|
INSERT INTO chunks_fts (
|
||||||
@ -287,60 +440,89 @@ def build_fts_index(conn: sqlite3.Connection) -> None:
|
|||||||
SELECT
|
SELECT
|
||||||
chunks.id,
|
chunks.id,
|
||||||
chunks.chunk_id,
|
chunks.chunk_id,
|
||||||
COALESCE(chunks.title, ''),
|
COALESCE(
|
||||||
COALESCE(chunks.author, ''),
|
chunks.title,
|
||||||
|
''
|
||||||
|
),
|
||||||
|
COALESCE(
|
||||||
|
chunks.author,
|
||||||
|
''
|
||||||
|
),
|
||||||
chunks.document_path,
|
chunks.document_path,
|
||||||
COALESCE(tags.values_text, ''),
|
COALESCE(
|
||||||
COALESCE(categories.values_text, ''),
|
tags.values_text,
|
||||||
|
''
|
||||||
|
),
|
||||||
|
COALESCE(
|
||||||
|
categories.values_text,
|
||||||
|
''
|
||||||
|
),
|
||||||
chunks.text
|
chunks.text
|
||||||
FROM chunks
|
FROM chunks
|
||||||
LEFT JOIN (
|
LEFT JOIN (
|
||||||
SELECT
|
SELECT
|
||||||
chunk_id,
|
chunk_id,
|
||||||
GROUP_CONCAT(tag, ' ') AS values_text
|
GROUP_CONCAT(
|
||||||
|
tag,
|
||||||
|
' '
|
||||||
|
) AS values_text
|
||||||
FROM chunk_tags
|
FROM chunk_tags
|
||||||
GROUP BY chunk_id
|
GROUP BY chunk_id
|
||||||
) AS tags
|
) AS tags
|
||||||
ON tags.chunk_id = chunks.chunk_id
|
ON tags.chunk_id
|
||||||
|
= chunks.chunk_id
|
||||||
LEFT JOIN (
|
LEFT JOIN (
|
||||||
SELECT
|
SELECT
|
||||||
chunk_id,
|
chunk_id,
|
||||||
GROUP_CONCAT(category, ' ') AS values_text
|
GROUP_CONCAT(
|
||||||
|
category,
|
||||||
|
' '
|
||||||
|
) AS values_text
|
||||||
FROM chunk_categories
|
FROM chunk_categories
|
||||||
GROUP BY chunk_id
|
GROUP BY chunk_id
|
||||||
) AS categories
|
) AS categories
|
||||||
ON categories.chunk_id = chunks.chunk_id
|
ON categories.chunk_id
|
||||||
|
= chunks.chunk_id
|
||||||
ORDER BY chunks.id
|
ORDER BY chunks.id
|
||||||
"""
|
"""
|
||||||
)
|
)
|
||||||
|
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO chunks_fts(chunks_fts) VALUES('optimize')"
|
"INSERT INTO chunks_fts"
|
||||||
|
"(chunks_fts) "
|
||||||
|
"VALUES('optimize')"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def validate_database(conn: sqlite3.Connection) -> None:
|
def validate_database(
|
||||||
|
conn: sqlite3.Connection,
|
||||||
|
) -> None:
|
||||||
integrity = conn.execute(
|
integrity = conn.execute(
|
||||||
"PRAGMA integrity_check"
|
"PRAGMA integrity_check"
|
||||||
).fetchone()[0]
|
).fetchone()[0]
|
||||||
|
|
||||||
if integrity != "ok":
|
if integrity != "ok":
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
f"SQLite integrity check zlyhal: {integrity}"
|
"SQLite integrity check "
|
||||||
|
f"zlyhal: {integrity}"
|
||||||
)
|
)
|
||||||
|
|
||||||
foreign_key_errors = conn.execute(
|
foreign_key_errors = (
|
||||||
|
conn.execute(
|
||||||
"PRAGMA foreign_key_check"
|
"PRAGMA foreign_key_check"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
|
)
|
||||||
|
|
||||||
if foreign_key_errors:
|
if foreign_key_errors:
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
"Databáza obsahuje chyby cudzích kľúčov: "
|
"Databáza obsahuje chyby "
|
||||||
|
"cudzích kľúčov: "
|
||||||
f"{foreign_key_errors[:5]}"
|
f"{foreign_key_errors[:5]}"
|
||||||
)
|
)
|
||||||
|
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO chunks_fts(chunks_fts) "
|
"INSERT INTO chunks_fts"
|
||||||
|
"(chunks_fts) "
|
||||||
"VALUES('integrity-check')"
|
"VALUES('integrity-check')"
|
||||||
)
|
)
|
||||||
|
|
||||||
@ -352,72 +534,168 @@ def validate_database(conn: sqlite3.Connection) -> None:
|
|||||||
"SELECT COUNT(*) FROM chunks_fts"
|
"SELECT COUNT(*) FROM chunks_fts"
|
||||||
).fetchone()[0]
|
).fetchone()[0]
|
||||||
|
|
||||||
|
embedding_count = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_embeddings
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
if chunk_count != fts_count:
|
if chunk_count != fts_count:
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
"Počet záznamov v chunks a chunks_fts sa nezhoduje: "
|
"Počet záznamov v chunks "
|
||||||
|
"a chunks_fts sa nezhoduje: "
|
||||||
f"{chunk_count} != {fts_count}"
|
f"{chunk_count} != {fts_count}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if chunk_count != embedding_count:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Počet chunkov a embeddingov "
|
||||||
|
"sa nezhoduje: "
|
||||||
|
f"{chunk_count} != "
|
||||||
|
f"{embedding_count}"
|
||||||
|
)
|
||||||
|
|
||||||
|
model_count = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(
|
||||||
|
DISTINCT model
|
||||||
|
)
|
||||||
|
FROM chunk_embeddings
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
|
if (
|
||||||
|
embedding_count > 0
|
||||||
|
and model_count != 1
|
||||||
|
):
|
||||||
|
raise RuntimeError(
|
||||||
|
"Embedding index obsahuje "
|
||||||
|
"viac modelov"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def get_counts(
|
def get_counts(
|
||||||
conn: sqlite3.Connection,
|
conn: sqlite3.Connection,
|
||||||
) -> dict[str, int]:
|
) -> dict[str, int]:
|
||||||
return {
|
return {
|
||||||
"documents": conn.execute(
|
"documents": conn.execute(
|
||||||
"SELECT COUNT(*) FROM documents"
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM documents
|
||||||
|
"""
|
||||||
).fetchone()[0],
|
).fetchone()[0],
|
||||||
"chunks": conn.execute(
|
"chunks": conn.execute(
|
||||||
"SELECT COUNT(*) FROM chunks"
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunks
|
||||||
|
"""
|
||||||
).fetchone()[0],
|
).fetchone()[0],
|
||||||
"fts_chunks": conn.execute(
|
"fts_chunks": conn.execute(
|
||||||
"SELECT COUNT(*) FROM chunks_fts"
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunks_fts
|
||||||
|
"""
|
||||||
|
).fetchone()[0],
|
||||||
|
"embedding_chunks": conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_embeddings
|
||||||
|
"""
|
||||||
).fetchone()[0],
|
).fetchone()[0],
|
||||||
"tags": conn.execute(
|
"tags": conn.execute(
|
||||||
"SELECT COUNT(*) FROM chunk_tags"
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_tags
|
||||||
|
"""
|
||||||
).fetchone()[0],
|
).fetchone()[0],
|
||||||
"categories": conn.execute(
|
"categories": conn.execute(
|
||||||
"SELECT COUNT(*) FROM chunk_categories"
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_categories
|
||||||
|
"""
|
||||||
).fetchone()[0],
|
).fetchone()[0],
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def temporary_database_path(db_file: Path) -> Path:
|
def temporary_database_path(
|
||||||
|
db_file: Path,
|
||||||
|
) -> Path:
|
||||||
return db_file.with_name(
|
return db_file.with_name(
|
||||||
f".{db_file.name}.tmp"
|
f".{db_file.name}.tmp"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_database() -> dict[str, int]:
|
def build_database() -> dict[str, int]:
|
||||||
documents = read_json(DOCUMENTS_FILE)
|
documents = read_json(
|
||||||
chunks = read_json(CHUNKS_FILE)
|
DOCUMENTS_FILE
|
||||||
|
)
|
||||||
|
|
||||||
|
chunks = read_json(
|
||||||
|
CHUNKS_FILE
|
||||||
|
)
|
||||||
|
|
||||||
DB_FILE.parent.mkdir(
|
DB_FILE.parent.mkdir(
|
||||||
parents=True,
|
parents=True,
|
||||||
exist_ok=True,
|
exist_ok=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
temporary_file = temporary_database_path(DB_FILE)
|
temporary_file = (
|
||||||
|
temporary_database_path(
|
||||||
|
DB_FILE
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
if temporary_file.exists():
|
if temporary_file.exists():
|
||||||
temporary_file.unlink()
|
temporary_file.unlink()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
with sqlite3.connect(temporary_file) as conn:
|
with sqlite3.connect(
|
||||||
conn.execute("PRAGMA foreign_keys = ON")
|
temporary_file
|
||||||
conn.execute("PRAGMA temp_store = MEMORY")
|
) as conn:
|
||||||
|
conn.execute(
|
||||||
|
"PRAGMA foreign_keys = ON"
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"PRAGMA temp_store = MEMORY"
|
||||||
|
)
|
||||||
|
|
||||||
verify_fts5(conn)
|
verify_fts5(
|
||||||
|
conn
|
||||||
|
)
|
||||||
|
|
||||||
with conn:
|
with conn:
|
||||||
create_tables(conn)
|
create_tables(
|
||||||
insert_documents(conn, documents)
|
conn
|
||||||
insert_chunks(conn, chunks)
|
)
|
||||||
build_fts_index(conn)
|
|
||||||
|
|
||||||
validate_database(conn)
|
insert_documents(
|
||||||
counts = get_counts(conn)
|
conn,
|
||||||
|
documents,
|
||||||
|
)
|
||||||
|
|
||||||
|
insert_chunks(
|
||||||
|
conn,
|
||||||
|
chunks,
|
||||||
|
)
|
||||||
|
|
||||||
|
build_fts_index(
|
||||||
|
conn
|
||||||
|
)
|
||||||
|
|
||||||
|
insert_embeddings(
|
||||||
|
conn,
|
||||||
|
chunks,
|
||||||
|
)
|
||||||
|
|
||||||
|
validate_database(
|
||||||
|
conn
|
||||||
|
)
|
||||||
|
|
||||||
|
counts = get_counts(
|
||||||
|
conn
|
||||||
|
)
|
||||||
|
|
||||||
# Nová databáza nahradí starú až po úspešnom vytvorení.
|
|
||||||
os.replace(
|
os.replace(
|
||||||
temporary_file,
|
temporary_file,
|
||||||
DB_FILE,
|
DB_FILE,
|
||||||
@ -430,23 +708,38 @@ def build_database() -> dict[str, int]:
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
print(
|
print(
|
||||||
f"[green]SQLite index vytvorený:[/green] "
|
"[green]SQLite index "
|
||||||
|
"vytvorený:[/green] "
|
||||||
f"{DB_FILE}"
|
f"{DB_FILE}"
|
||||||
)
|
)
|
||||||
|
|
||||||
print(
|
print(
|
||||||
f"Dokumentov: {counts['documents']}"
|
f"Dokumentov: "
|
||||||
|
f"{counts['documents']}"
|
||||||
)
|
)
|
||||||
|
|
||||||
print(
|
print(
|
||||||
f"Chunkov: {counts['chunks']}"
|
f"Chunkov: "
|
||||||
|
f"{counts['chunks']}"
|
||||||
)
|
)
|
||||||
|
|
||||||
print(
|
print(
|
||||||
f"FTS5 chunkov: {counts['fts_chunks']}"
|
f"FTS5 chunkov: "
|
||||||
|
f"{counts['fts_chunks']}"
|
||||||
)
|
)
|
||||||
|
|
||||||
print(
|
print(
|
||||||
f"Tag záznamov: {counts['tags']}"
|
f"Embedding chunkov: "
|
||||||
|
f"{counts['embedding_chunks']}"
|
||||||
)
|
)
|
||||||
|
|
||||||
print(
|
print(
|
||||||
f"Kategória záznamov: "
|
f"Tag záznamov: "
|
||||||
|
f"{counts['tags']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Kategória záznamov: "
|
||||||
f"{counts['categories']}"
|
f"{counts['categories']}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
289
scripts/embedding_utils.py
Normal file
289
scripts/embedding_utils.py
Normal file
@ -0,0 +1,289 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from functools import lru_cache
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_EMBEDDING_MODEL = (
|
||||||
|
"intfloat/multilingual-e5-small"
|
||||||
|
)
|
||||||
|
|
||||||
|
DEFAULT_BATCH_SIZE = 32
|
||||||
|
|
||||||
|
|
||||||
|
def embedding_model_name() -> str:
|
||||||
|
return (
|
||||||
|
os.getenv(
|
||||||
|
"EMBEDDING_MODEL",
|
||||||
|
DEFAULT_EMBEDDING_MODEL,
|
||||||
|
).strip()
|
||||||
|
or DEFAULT_EMBEDDING_MODEL
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def embedding_batch_size() -> int:
|
||||||
|
raw_value = os.getenv(
|
||||||
|
"EMBEDDING_BATCH_SIZE",
|
||||||
|
str(DEFAULT_BATCH_SIZE),
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
try:
|
||||||
|
value = int(
|
||||||
|
raw_value
|
||||||
|
)
|
||||||
|
|
||||||
|
except ValueError:
|
||||||
|
return DEFAULT_BATCH_SIZE
|
||||||
|
|
||||||
|
return max(
|
||||||
|
1,
|
||||||
|
value,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=2)
|
||||||
|
def get_embedding_model(
|
||||||
|
model_name: str,
|
||||||
|
):
|
||||||
|
from sentence_transformers import (
|
||||||
|
SentenceTransformer,
|
||||||
|
)
|
||||||
|
|
||||||
|
return SentenceTransformer(
|
||||||
|
model_name
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def build_embedding_text(
|
||||||
|
chunk: dict[str, Any],
|
||||||
|
) -> str:
|
||||||
|
parts: list[str] = []
|
||||||
|
|
||||||
|
title = str(
|
||||||
|
chunk.get("title")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
author = str(
|
||||||
|
chunk.get("author")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
tags = [
|
||||||
|
str(value).strip()
|
||||||
|
for value in (
|
||||||
|
chunk.get("tags")
|
||||||
|
or []
|
||||||
|
)
|
||||||
|
if str(value).strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
categories = [
|
||||||
|
str(value).strip()
|
||||||
|
for value in (
|
||||||
|
chunk.get("categories")
|
||||||
|
or []
|
||||||
|
)
|
||||||
|
if str(value).strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
text = str(
|
||||||
|
chunk.get("text")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
if title:
|
||||||
|
parts.append(
|
||||||
|
f"Názov: {title}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if author:
|
||||||
|
parts.append(
|
||||||
|
f"Autor: {author}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if tags:
|
||||||
|
parts.append(
|
||||||
|
"Tagy: "
|
||||||
|
+ ", ".join(
|
||||||
|
tags
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if categories:
|
||||||
|
parts.append(
|
||||||
|
"Kategórie: "
|
||||||
|
+ ", ".join(
|
||||||
|
categories
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if text:
|
||||||
|
parts.append(
|
||||||
|
text
|
||||||
|
)
|
||||||
|
|
||||||
|
return "\n".join(
|
||||||
|
parts
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def embed_passages(
|
||||||
|
texts: list[str],
|
||||||
|
*,
|
||||||
|
model_name: str | None = None,
|
||||||
|
) -> np.ndarray:
|
||||||
|
if not texts:
|
||||||
|
return np.empty(
|
||||||
|
(
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
),
|
||||||
|
dtype=np.float32,
|
||||||
|
)
|
||||||
|
|
||||||
|
selected_model = (
|
||||||
|
model_name
|
||||||
|
or embedding_model_name()
|
||||||
|
)
|
||||||
|
|
||||||
|
model = get_embedding_model(
|
||||||
|
selected_model
|
||||||
|
)
|
||||||
|
|
||||||
|
prepared = [
|
||||||
|
"passage: "
|
||||||
|
+ text.strip()
|
||||||
|
for text in texts
|
||||||
|
]
|
||||||
|
|
||||||
|
vectors = model.encode(
|
||||||
|
prepared,
|
||||||
|
batch_size=(
|
||||||
|
embedding_batch_size()
|
||||||
|
),
|
||||||
|
show_progress_bar=False,
|
||||||
|
convert_to_numpy=True,
|
||||||
|
normalize_embeddings=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
return np.asarray(
|
||||||
|
vectors,
|
||||||
|
dtype=np.float32,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def embed_query(
|
||||||
|
query: str,
|
||||||
|
*,
|
||||||
|
model_name: str | None = None,
|
||||||
|
) -> np.ndarray:
|
||||||
|
clean_query = query.strip()
|
||||||
|
|
||||||
|
if not clean_query:
|
||||||
|
raise ValueError(
|
||||||
|
"Query nesmie byť prázdny"
|
||||||
|
)
|
||||||
|
|
||||||
|
selected_model = (
|
||||||
|
model_name
|
||||||
|
or embedding_model_name()
|
||||||
|
)
|
||||||
|
|
||||||
|
model = get_embedding_model(
|
||||||
|
selected_model
|
||||||
|
)
|
||||||
|
|
||||||
|
vector = model.encode(
|
||||||
|
[
|
||||||
|
"query: "
|
||||||
|
+ clean_query
|
||||||
|
],
|
||||||
|
batch_size=1,
|
||||||
|
show_progress_bar=False,
|
||||||
|
convert_to_numpy=True,
|
||||||
|
normalize_embeddings=True,
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
return np.asarray(
|
||||||
|
vector,
|
||||||
|
dtype=np.float32,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def vector_to_blob(
|
||||||
|
vector: np.ndarray,
|
||||||
|
) -> bytes:
|
||||||
|
normalized = np.asarray(
|
||||||
|
vector,
|
||||||
|
dtype=np.float32,
|
||||||
|
)
|
||||||
|
|
||||||
|
return normalized.tobytes()
|
||||||
|
|
||||||
|
|
||||||
|
def blob_to_vector(
|
||||||
|
blob: bytes,
|
||||||
|
dimensions: int,
|
||||||
|
) -> np.ndarray:
|
||||||
|
vector = np.frombuffer(
|
||||||
|
blob,
|
||||||
|
dtype=np.float32,
|
||||||
|
)
|
||||||
|
|
||||||
|
if (
|
||||||
|
vector.shape[0]
|
||||||
|
!= dimensions
|
||||||
|
):
|
||||||
|
raise RuntimeError(
|
||||||
|
"Neplatný rozmer "
|
||||||
|
"uloženého embeddingu"
|
||||||
|
)
|
||||||
|
|
||||||
|
return vector
|
||||||
|
|
||||||
|
|
||||||
|
def cosine_similarity(
|
||||||
|
first: np.ndarray,
|
||||||
|
second: np.ndarray,
|
||||||
|
) -> float:
|
||||||
|
if (
|
||||||
|
first.shape
|
||||||
|
!= second.shape
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
"Embeddingy majú "
|
||||||
|
"rozdielny rozmer"
|
||||||
|
)
|
||||||
|
|
||||||
|
first_norm = float(
|
||||||
|
np.linalg.norm(
|
||||||
|
first
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
second_norm = float(
|
||||||
|
np.linalg.norm(
|
||||||
|
second
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if (
|
||||||
|
first_norm == 0.0
|
||||||
|
or second_norm == 0.0
|
||||||
|
):
|
||||||
|
return 0.0
|
||||||
|
|
||||||
|
return float(
|
||||||
|
np.dot(
|
||||||
|
first,
|
||||||
|
second,
|
||||||
|
)
|
||||||
|
/ (
|
||||||
|
first_norm
|
||||||
|
* second_norm
|
||||||
|
)
|
||||||
|
)
|
||||||
File diff suppressed because it is too large
Load Diff
40
tatus
Normal file
40
tatus
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
README.md[36m:[m66[36m:[m[1;31mWEBHOOK_SECRET[m=<náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
README.md[36m:[m67[36m:[m[1;31mSYNC_API_KEY[m=<iná náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
README.md[36m:[m68[36m:[m[1;31mSEARCH_API_KEY[m=<ďalšia náhodná hodnota s minimálne 32 znakmi>
|
||||||
|
README.md[36m:[m162[36m:[m -H "X-API-Key: $[1;31mSEARCH_API_KEY[m" \
|
||||||
|
README.md[36m:[m182[36m:[m -H "X-API-Key: $[1;31mSYNC_API_KEY[m" \
|
||||||
|
app/main.py[36m:[m41[36m:[m[1;31mSEARCH_API_KEY[m_HEADER = "X-API-Key"
|
||||||
|
app/main.py[36m:[m42[36m:[m[1;31mSYNC_API_KEY[m_HEADER = "X-API-Key"
|
||||||
|
app/main.py[36m:[m45[36m:[m[1;31msearch_api_key[m_scheme = APIKeyHeader(
|
||||||
|
app/main.py[36m:[m46[36m:[m name=[1;31mSEARCH_API_KEY[m_HEADER,
|
||||||
|
app/main.py[36m:[m51[36m:[m[1;31msync_api_key[m_scheme = APIKeyHeader(
|
||||||
|
app/main.py[36m:[m52[36m:[m name=[1;31mSYNC_API_KEY[m_HEADER,
|
||||||
|
app/main.py[36m:[m135[36m:[m validate_secret("[1;31mWEBHOOK_SECRET[m")
|
||||||
|
app/main.py[36m:[m136[36m:[m validate_secret("[1;31mSYNC_API_KEY[m")
|
||||||
|
app/main.py[36m:[m137[36m:[m validate_secret("[1;31mSEARCH_API_KEY[m")
|
||||||
|
app/main.py[36m:[m158[36m:[mdef require_[1;31msearch_api_key[m(
|
||||||
|
app/main.py[36m:[m160[36m:[m [1;31msearch_api_key[m_scheme
|
||||||
|
app/main.py[36m:[m164[36m:[m "[1;31mSEARCH_API_KEY[m"
|
||||||
|
app/main.py[36m:[m183[36m:[mdef require_[1;31msync_api_key[m(
|
||||||
|
app/main.py[36m:[m185[36m:[m [1;31msync_api_key[m_scheme
|
||||||
|
app/main.py[36m:[m189[36m:[m "[1;31mSYNC_API_KEY[m"
|
||||||
|
app/main.py[36m:[m293[36m:[m "[1;31mWEBHOOK_SECRET[m",
|
||||||
|
app/main.py[36m:[m294[36m:[m "[1;31mSYNC_API_KEY[m",
|
||||||
|
app/main.py[36m:[m295[36m:[m "[1;31mSEARCH_API_KEY[m",
|
||||||
|
app/main.py[36m:[m305[36m:[m Depends(require_[1;31msearch_api_key[m)
|
||||||
|
app/main.py[36m:[m356[36m:[m Depends(require_[1;31msync_api_key[m)
|
||||||
|
app/main.py[36m:[m407[36m:[m "[1;31mWEBHOOK_SECRET[m"
|
||||||
|
test/conftest.py[36m:[m21[36m:[m "[1;31mWEBHOOK_SECRET[m",
|
||||||
|
test/conftest.py[36m:[m26[36m:[m "[1;31mSYNC_API_KEY[m",
|
||||||
|
test/conftest.py[36m:[m31[36m:[m "[1;31mSEARCH_API_KEY[m",
|
||||||
|
test/test_api.py[36m:[m14[36m:[m[1;31mWEBHOOK_SECRET[m = "w" * 64
|
||||||
|
test/test_api.py[36m:[m15[36m:[m[1;31mSYNC_API_KEY[m = "s" * 64
|
||||||
|
test/test_api.py[36m:[m16[36m:[m[1;31mSEARCH_API_KEY[m = "a" * 64
|
||||||
|
test/test_api.py[36m:[m34[36m:[m [1;31mWEBHOOK_SECRET[m.encode("utf-8"),
|
||||||
|
test/test_api.py[36m:[m49[36m:[m monkeypatch.delenv("[1;31mWEBHOOK_SECRET[m")
|
||||||
|
test/test_api.py[36m:[m53[36m:[m match="[1;31mWEBHOOK_SECRET[m",
|
||||||
|
test/test_api.py[36m:[m63[36m:[m "[1;31mSYNC_API_KEY[m",
|
||||||
|
test/test_api.py[36m:[m111[36m:[m "X-API-Key": [1;31mSEARCH_API_KEY[m,
|
||||||
|
test/test_api.py[36m:[m133[36m:[m "X-API-Key": [1;31mSEARCH_API_KEY[m,
|
||||||
|
test/test_api.py[36m:[m216[36m:[m "X-API-Key": [1;31mSYNC_API_KEY[m,
|
||||||
|
test/test_api.py[36m:[m245[36m:[m "X-API-Key": [1;31mSYNC_API_KEY[m,
|
||||||
@ -5,7 +5,6 @@ from pathlib import Path
|
|||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
if str(PROJECT_ROOT) not in sys.path:
|
if str(PROJECT_ROOT) not in sys.path:
|
||||||
@ -17,10 +16,28 @@ def security_environment(
|
|||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Každý test dostane platnú bezpečnostnú konfiguráciu."""
|
"""Každý test dostane platnú bezpečnostnú konfiguráciu."""
|
||||||
monkeypatch.setenv("WEBHOOK_SECRET", "w" * 64)
|
|
||||||
monkeypatch.setenv("SYNC_API_KEY", "s" * 64)
|
monkeypatch.setenv(
|
||||||
|
"WEBHOOK_SECRET",
|
||||||
|
"w" * 64,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setenv(
|
||||||
|
"SYNC_API_KEY",
|
||||||
|
"s" * 64,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setenv(
|
||||||
|
"SEARCH_API_KEY",
|
||||||
|
"a" * 64,
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setenv(
|
monkeypatch.setenv(
|
||||||
"EXPECTED_GITEA_REPOSITORY",
|
"EXPECTED_GITEA_REPOSITORY",
|
||||||
"KEMT/zpwiki",
|
"KEMT/zpwiki",
|
||||||
)
|
)
|
||||||
monkeypatch.setenv("WEBHOOK_PULL_GIT", "false")
|
|
||||||
|
monkeypatch.setenv(
|
||||||
|
"WEBHOOK_PULL_GIT",
|
||||||
|
"false",
|
||||||
|
)
|
||||||
|
|||||||
216
test/test_api.py
216
test/test_api.py
@ -13,6 +13,7 @@ from scripts.rebuild_index import ReindexInProgressError
|
|||||||
|
|
||||||
WEBHOOK_SECRET = "w" * 64
|
WEBHOOK_SECRET = "w" * 64
|
||||||
SYNC_API_KEY = "s" * 64
|
SYNC_API_KEY = "s" * 64
|
||||||
|
SEARCH_API_KEY = "a" * 64
|
||||||
|
|
||||||
|
|
||||||
def fake_rebuild_result() -> dict:
|
def fake_rebuild_result() -> dict:
|
||||||
@ -47,7 +48,10 @@ def test_startup_rejects_missing_secret(
|
|||||||
) -> None:
|
) -> None:
|
||||||
monkeypatch.delenv("WEBHOOK_SECRET")
|
monkeypatch.delenv("WEBHOOK_SECRET")
|
||||||
|
|
||||||
with pytest.raises(RuntimeError, match="WEBHOOK_SECRET"):
|
with pytest.raises(
|
||||||
|
RuntimeError,
|
||||||
|
match="WEBHOOK_SECRET",
|
||||||
|
):
|
||||||
with TestClient(main.app):
|
with TestClient(main.app):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@ -55,18 +59,28 @@ def test_startup_rejects_missing_secret(
|
|||||||
def test_startup_rejects_short_secret(
|
def test_startup_rejects_short_secret(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
monkeypatch.setenv("SYNC_API_KEY", "short")
|
monkeypatch.setenv(
|
||||||
|
"SYNC_API_KEY",
|
||||||
|
"short",
|
||||||
|
)
|
||||||
|
|
||||||
with pytest.raises(RuntimeError, match="aspoň 32"):
|
with pytest.raises(
|
||||||
|
RuntimeError,
|
||||||
|
match="aspoň 32",
|
||||||
|
):
|
||||||
with TestClient(main.app):
|
with TestClient(main.app):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
def test_health_endpoint(client: TestClient) -> None:
|
def test_health_endpoint(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
response = client.get("/health")
|
response = client.get("/health")
|
||||||
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
|
|
||||||
payload = response.json()
|
payload = response.json()
|
||||||
|
|
||||||
assert payload["status"] == "ok"
|
assert payload["status"] == "ok"
|
||||||
assert payload["search_engine"] == "sqlite_fts5"
|
assert payload["search_engine"] == "sqlite_fts5"
|
||||||
assert payload["security_configured"] is True
|
assert payload["security_configured"] is True
|
||||||
@ -82,36 +96,107 @@ def test_search_endpoint_uses_shared_search_logic(
|
|||||||
lambda *args, **kwargs: {
|
lambda *args, **kwargs: {
|
||||||
"engine": "sqlite_fts5",
|
"engine": "sqlite_fts5",
|
||||||
"strategies": ["all_terms"],
|
"strategies": ["all_terms"],
|
||||||
"results": [{"chunk_id": "test::0", "published": True}],
|
"results": [
|
||||||
|
{
|
||||||
|
"chunk_id": "test::0",
|
||||||
|
"published": True,
|
||||||
|
}
|
||||||
|
],
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/search",
|
"/search",
|
||||||
json={"query": "jan ptak", "limit": 5},
|
headers={
|
||||||
|
"X-API-Key": SEARCH_API_KEY,
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"query": "jan ptak",
|
||||||
|
"limit": 5,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert response.json()["count"] == 1
|
assert response.json()["count"] == 1
|
||||||
assert response.json()["results"][0]["chunk_id"] == "test::0"
|
assert (
|
||||||
|
response.json()["results"][0]["chunk_id"]
|
||||||
|
== "test::0"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_search_rejects_empty_query(client: TestClient) -> None:
|
def test_search_rejects_empty_query(
|
||||||
response = client.post("/search", json={"query": ""})
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
|
response = client.post(
|
||||||
|
"/search",
|
||||||
|
headers={
|
||||||
|
"X-API-Key": SEARCH_API_KEY,
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"query": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
assert response.status_code == 422
|
assert response.status_code == 422
|
||||||
|
|
||||||
|
|
||||||
def test_sync_rejects_missing_api_key(client: TestClient) -> None:
|
def test_search_rejects_missing_api_key(
|
||||||
response = client.post("/sync", json={"pull_git": False})
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
|
response = client.post(
|
||||||
|
"/search",
|
||||||
|
json={
|
||||||
|
"query": "jan ptak",
|
||||||
|
"limit": 5,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
assert response.status_code == 401
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
def test_sync_rejects_wrong_api_key(client: TestClient) -> None:
|
def test_search_rejects_wrong_api_key(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
|
response = client.post(
|
||||||
|
"/search",
|
||||||
|
headers={
|
||||||
|
"X-API-Key": "x" * 64,
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"query": "jan ptak",
|
||||||
|
"limit": 5,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_rejects_missing_api_key(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/sync",
|
"/sync",
|
||||||
headers={"X-API-Key": "x" * 64},
|
json={
|
||||||
json={"pull_git": False},
|
"pull_git": False,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_rejects_wrong_api_key(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
|
response = client.post(
|
||||||
|
"/sync",
|
||||||
|
headers={
|
||||||
|
"X-API-Key": "x" * 64,
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"pull_git": False,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
assert response.status_code == 401
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
@ -127,8 +212,12 @@ def test_sync_accepts_valid_api_key(
|
|||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/sync",
|
"/sync",
|
||||||
headers={"X-API-Key": SYNC_API_KEY},
|
headers={
|
||||||
json={"pull_git": False},
|
"X-API-Key": SYNC_API_KEY,
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"pull_git": False,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
@ -140,22 +229,38 @@ def test_sync_returns_conflict_when_reindex_is_running(
|
|||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
def busy(*args, **kwargs):
|
def busy(*args, **kwargs):
|
||||||
raise ReindexInProgressError("Reindexovanie už prebieha")
|
raise ReindexInProgressError(
|
||||||
|
"Reindexovanie už prebieha"
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setattr(main, "rebuild_index", busy)
|
monkeypatch.setattr(
|
||||||
|
main,
|
||||||
|
"rebuild_index",
|
||||||
|
busy,
|
||||||
|
)
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/sync",
|
"/sync",
|
||||||
headers={"X-API-Key": SYNC_API_KEY},
|
headers={
|
||||||
json={"pull_git": False},
|
"X-API-Key": SYNC_API_KEY,
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"pull_git": False,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
assert response.status_code == 409
|
assert response.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
def test_webhook_rejects_invalid_signature(client: TestClient) -> None:
|
def test_webhook_rejects_invalid_signature(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
body = json.dumps(
|
body = json.dumps(
|
||||||
{"repository": {"full_name": "KEMT/zpwiki"}}
|
{
|
||||||
|
"repository": {
|
||||||
|
"full_name": "KEMT/zpwiki",
|
||||||
|
}
|
||||||
|
}
|
||||||
).encode("utf-8")
|
).encode("utf-8")
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
@ -189,9 +294,15 @@ def test_webhook_rejects_invalid_json_with_valid_signature(
|
|||||||
assert response.status_code == 400
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
def test_webhook_requires_event_header(client: TestClient) -> None:
|
def test_webhook_requires_event_header(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
body = json.dumps(
|
body = json.dumps(
|
||||||
{"repository": {"full_name": "KEMT/zpwiki"}}
|
{
|
||||||
|
"repository": {
|
||||||
|
"full_name": "KEMT/zpwiki",
|
||||||
|
}
|
||||||
|
}
|
||||||
).encode("utf-8")
|
).encode("utf-8")
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
@ -217,9 +328,18 @@ def test_webhook_ignores_non_push_event(
|
|||||||
calls += 1
|
calls += 1
|
||||||
return fake_rebuild_result()
|
return fake_rebuild_result()
|
||||||
|
|
||||||
monkeypatch.setattr(main, "rebuild_index", fake_rebuild)
|
monkeypatch.setattr(
|
||||||
|
main,
|
||||||
|
"rebuild_index",
|
||||||
|
fake_rebuild,
|
||||||
|
)
|
||||||
|
|
||||||
body = json.dumps(
|
body = json.dumps(
|
||||||
{"repository": {"full_name": "KEMT/zpwiki"}}
|
{
|
||||||
|
"repository": {
|
||||||
|
"full_name": "KEMT/zpwiki",
|
||||||
|
}
|
||||||
|
}
|
||||||
).encode("utf-8")
|
).encode("utf-8")
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
@ -237,9 +357,15 @@ def test_webhook_ignores_non_push_event(
|
|||||||
assert calls == 0
|
assert calls == 0
|
||||||
|
|
||||||
|
|
||||||
def test_webhook_rejects_unexpected_repository(client: TestClient) -> None:
|
def test_webhook_rejects_unexpected_repository(
|
||||||
|
client: TestClient,
|
||||||
|
) -> None:
|
||||||
body = json.dumps(
|
body = json.dumps(
|
||||||
{"repository": {"full_name": "OTHER/repository"}}
|
{
|
||||||
|
"repository": {
|
||||||
|
"full_name": "OTHER/repository",
|
||||||
|
}
|
||||||
|
}
|
||||||
).encode("utf-8")
|
).encode("utf-8")
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
@ -264,8 +390,13 @@ def test_webhook_accepts_signed_push(
|
|||||||
"rebuild_index",
|
"rebuild_index",
|
||||||
lambda pull_git=False: fake_rebuild_result(),
|
lambda pull_git=False: fake_rebuild_result(),
|
||||||
)
|
)
|
||||||
|
|
||||||
body = json.dumps(
|
body = json.dumps(
|
||||||
{"repository": {"full_name": "KEMT/zpwiki"}}
|
{
|
||||||
|
"repository": {
|
||||||
|
"full_name": "KEMT/zpwiki",
|
||||||
|
}
|
||||||
|
}
|
||||||
).encode("utf-8")
|
).encode("utf-8")
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
@ -279,8 +410,14 @@ def test_webhook_accepts_signed_push(
|
|||||||
)
|
)
|
||||||
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert response.json()["verified_by"] == "hmac_sha256"
|
assert (
|
||||||
assert response.json()["repository"] == "KEMT/zpwiki"
|
response.json()["verified_by"]
|
||||||
|
== "hmac_sha256"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
response.json()["repository"]
|
||||||
|
== "KEMT/zpwiki"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_webhook_returns_conflict_when_reindex_is_running(
|
def test_webhook_returns_conflict_when_reindex_is_running(
|
||||||
@ -288,11 +425,22 @@ def test_webhook_returns_conflict_when_reindex_is_running(
|
|||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
def busy(*args, **kwargs):
|
def busy(*args, **kwargs):
|
||||||
raise ReindexInProgressError("Reindexovanie už prebieha")
|
raise ReindexInProgressError(
|
||||||
|
"Reindexovanie už prebieha"
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
main,
|
||||||
|
"rebuild_index",
|
||||||
|
busy,
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setattr(main, "rebuild_index", busy)
|
|
||||||
body = json.dumps(
|
body = json.dumps(
|
||||||
{"repository": {"full_name": "KEMT/zpwiki"}}
|
{
|
||||||
|
"repository": {
|
||||||
|
"full_name": "KEMT/zpwiki",
|
||||||
|
}
|
||||||
|
}
|
||||||
).encode("utf-8")
|
).encode("utf-8")
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
|
|||||||
@ -10,9 +10,16 @@ import scripts.build_sqlite_index as indexer
|
|||||||
|
|
||||||
|
|
||||||
def write_json(path: Path, value) -> None:
|
def write_json(path: Path, value) -> None:
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
path.parent.mkdir(
|
||||||
|
parents=True,
|
||||||
|
exist_ok=True,
|
||||||
|
)
|
||||||
|
|
||||||
path.write_text(
|
path.write_text(
|
||||||
json.dumps(value, ensure_ascii=False),
|
json.dumps(
|
||||||
|
value,
|
||||||
|
ensure_ascii=False,
|
||||||
|
),
|
||||||
encoding="utf-8",
|
encoding="utf-8",
|
||||||
)
|
)
|
||||||
|
|
||||||
@ -25,7 +32,9 @@ def sample_documents() -> list[dict]:
|
|||||||
"author": "Autor",
|
"author": "Autor",
|
||||||
"published": True,
|
"published": True,
|
||||||
"content_length": 100,
|
"content_length": 100,
|
||||||
"metadata": {"published": True},
|
"metadata": {
|
||||||
|
"published": True,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
@ -33,19 +42,33 @@ def sample_documents() -> list[dict]:
|
|||||||
def sample_chunks() -> list[dict]:
|
def sample_chunks() -> list[dict]:
|
||||||
return [
|
return [
|
||||||
{
|
{
|
||||||
"chunk_id": "pages/test/README.md::chunk-0",
|
"chunk_id": (
|
||||||
"document_path": "pages/test/README.md",
|
"pages/test/README.md::chunk-0"
|
||||||
|
),
|
||||||
|
"document_path": (
|
||||||
|
"pages/test/README.md"
|
||||||
|
),
|
||||||
"title": "Strojový preklad",
|
"title": "Strojový preklad",
|
||||||
"author": "Autor",
|
"author": "Autor",
|
||||||
"published": True,
|
"published": True,
|
||||||
"chunk_index": 0,
|
"chunk_index": 0,
|
||||||
"heading_paths": [["Úvod"]],
|
"heading_paths": [
|
||||||
"text": "Dokument: Strojový preklad. Neurónový preklad textu.",
|
["Úvod"]
|
||||||
|
],
|
||||||
|
"text": (
|
||||||
|
"Dokument: Strojový preklad. "
|
||||||
|
"Neurónový preklad textu."
|
||||||
|
),
|
||||||
"text_length": 58,
|
"text_length": 58,
|
||||||
"token_count": 16,
|
"token_count": 16,
|
||||||
"content_hash": "abc",
|
"content_hash": "abc",
|
||||||
"tags": ["translation", "nlp"],
|
"tags": [
|
||||||
"categories": ["project"],
|
"translation",
|
||||||
|
"nlp",
|
||||||
|
],
|
||||||
|
"categories": [
|
||||||
|
"project",
|
||||||
|
],
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
@ -53,26 +76,66 @@ def sample_chunks() -> list[dict]:
|
|||||||
def configure_indexer(
|
def configure_indexer(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
) -> tuple[Path, Path, Path]:
|
) -> tuple[
|
||||||
documents_file = tmp_path / "documents.json"
|
Path,
|
||||||
chunks_file = tmp_path / "chunks.json"
|
Path,
|
||||||
db_file = tmp_path / "zp_index.sqlite"
|
Path,
|
||||||
|
]:
|
||||||
|
documents_file = (
|
||||||
|
tmp_path / "documents.json"
|
||||||
|
)
|
||||||
|
|
||||||
write_json(documents_file, sample_documents())
|
chunks_file = (
|
||||||
write_json(chunks_file, sample_chunks())
|
tmp_path / "chunks.json"
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setattr(indexer, "DOCUMENTS_FILE", documents_file)
|
db_file = (
|
||||||
monkeypatch.setattr(indexer, "CHUNKS_FILE", chunks_file)
|
tmp_path / "zp_index.sqlite"
|
||||||
monkeypatch.setattr(indexer, "DB_FILE", db_file)
|
)
|
||||||
|
|
||||||
return documents_file, chunks_file, db_file
|
write_json(
|
||||||
|
documents_file,
|
||||||
|
sample_documents(),
|
||||||
|
)
|
||||||
|
|
||||||
|
write_json(
|
||||||
|
chunks_file,
|
||||||
|
sample_chunks(),
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"DOCUMENTS_FILE",
|
||||||
|
documents_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"CHUNKS_FILE",
|
||||||
|
chunks_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"DB_FILE",
|
||||||
|
db_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
return (
|
||||||
|
documents_file,
|
||||||
|
chunks_file,
|
||||||
|
db_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_database_contains_documents_chunks_metadata_and_fts(
|
def test_database_contains_documents_chunks_metadata_and_fts(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
_, _, db_file = configure_indexer(monkeypatch, tmp_path)
|
_, _, db_file = configure_indexer(
|
||||||
|
monkeypatch,
|
||||||
|
tmp_path,
|
||||||
|
)
|
||||||
|
|
||||||
counts = indexer.build_database()
|
counts = indexer.build_database()
|
||||||
|
|
||||||
@ -80,55 +143,205 @@ def test_database_contains_documents_chunks_metadata_and_fts(
|
|||||||
"documents": 1,
|
"documents": 1,
|
||||||
"chunks": 1,
|
"chunks": 1,
|
||||||
"fts_chunks": 1,
|
"fts_chunks": 1,
|
||||||
|
"embedding_chunks": 1,
|
||||||
"tags": 2,
|
"tags": 2,
|
||||||
"categories": 1,
|
"categories": 1,
|
||||||
}
|
}
|
||||||
|
|
||||||
with sqlite3.connect(db_file) as conn:
|
with sqlite3.connect(
|
||||||
assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok"
|
db_file
|
||||||
assert conn.execute("PRAGMA foreign_key_check").fetchall() == []
|
) as conn:
|
||||||
assert conn.execute("SELECT published FROM chunks").fetchone()[0] == 1
|
assert (
|
||||||
assert conn.execute("SELECT COUNT(*) FROM chunk_tags").fetchone()[0] == 2
|
conn.execute(
|
||||||
assert conn.execute("SELECT COUNT(*) FROM chunk_categories").fetchone()[0] == 1
|
"PRAGMA integrity_check"
|
||||||
assert conn.execute(
|
).fetchone()[0]
|
||||||
"SELECT COUNT(*) FROM chunks_fts WHERE chunks_fts MATCH 'strojovy'"
|
== "ok"
|
||||||
).fetchone()[0] == 1
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"PRAGMA foreign_key_check"
|
||||||
|
).fetchall()
|
||||||
|
== []
|
||||||
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT published
|
||||||
|
FROM chunks
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_tags
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
== 2
|
||||||
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_categories
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunks_fts
|
||||||
|
WHERE chunks_fts
|
||||||
|
MATCH 'strojovy'
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM chunk_embeddings
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
|
||||||
|
embedding_row = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT
|
||||||
|
model,
|
||||||
|
dimensions,
|
||||||
|
LENGTH(embedding)
|
||||||
|
FROM chunk_embeddings
|
||||||
|
"""
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
|
assert embedding_row is not None
|
||||||
|
|
||||||
|
model, dimensions, blob_size = (
|
||||||
|
embedding_row
|
||||||
|
)
|
||||||
|
|
||||||
|
assert model
|
||||||
|
assert dimensions > 0
|
||||||
|
|
||||||
|
# float32 = 4 bajty.
|
||||||
|
assert (
|
||||||
|
blob_size
|
||||||
|
== dimensions * 4
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_database_rebuild_replaces_old_database_only_after_success(
|
def test_database_rebuild_replaces_old_database_only_after_success(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
_, _, db_file = configure_indexer(monkeypatch, tmp_path)
|
_, _, db_file = configure_indexer(
|
||||||
|
monkeypatch,
|
||||||
|
tmp_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
with sqlite3.connect(
|
||||||
|
db_file
|
||||||
|
) as conn:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE marker(
|
||||||
|
value TEXT
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO marker
|
||||||
|
VALUES ('old database')
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
|
||||||
with sqlite3.connect(db_file) as conn:
|
|
||||||
conn.execute("CREATE TABLE marker(value TEXT)")
|
|
||||||
conn.execute("INSERT INTO marker VALUES ('old database')")
|
|
||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
def fail_validation(conn: sqlite3.Connection) -> None:
|
def fail_validation(
|
||||||
raise RuntimeError("úmyselná chyba validácie")
|
conn: sqlite3.Connection,
|
||||||
|
) -> None:
|
||||||
|
raise RuntimeError(
|
||||||
|
"úmyselná chyba validácie"
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setattr(indexer, "validate_database", fail_validation)
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"validate_database",
|
||||||
|
fail_validation,
|
||||||
|
)
|
||||||
|
|
||||||
with pytest.raises(RuntimeError, match="úmyselná chyba"):
|
with pytest.raises(
|
||||||
|
RuntimeError,
|
||||||
|
match="úmyselná chyba",
|
||||||
|
):
|
||||||
indexer.build_database()
|
indexer.build_database()
|
||||||
|
|
||||||
with sqlite3.connect(db_file) as conn:
|
with sqlite3.connect(
|
||||||
assert conn.execute("SELECT value FROM marker").fetchone()[0] == "old database"
|
db_file
|
||||||
|
) as conn:
|
||||||
|
assert (
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT value
|
||||||
|
FROM marker
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
== "old database"
|
||||||
|
)
|
||||||
|
|
||||||
assert not indexer.temporary_database_path(db_file).exists()
|
assert not (
|
||||||
|
indexer.temporary_database_path(
|
||||||
|
db_file
|
||||||
|
).exists()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_database_rejects_chunk_without_chunk_id(
|
def test_database_rejects_chunk_without_chunk_id(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
documents_file, chunks_file, _ = configure_indexer(monkeypatch, tmp_path)
|
(
|
||||||
broken = sample_chunks()
|
documents_file,
|
||||||
broken[0]["chunk_id"] = ""
|
chunks_file,
|
||||||
write_json(documents_file, sample_documents())
|
_,
|
||||||
write_json(chunks_file, broken)
|
) = configure_indexer(
|
||||||
|
monkeypatch,
|
||||||
|
tmp_path,
|
||||||
|
)
|
||||||
|
|
||||||
with pytest.raises(ValueError, match="chunk_id"):
|
broken = sample_chunks()
|
||||||
|
|
||||||
|
broken[0][
|
||||||
|
"chunk_id"
|
||||||
|
] = ""
|
||||||
|
|
||||||
|
write_json(
|
||||||
|
documents_file,
|
||||||
|
sample_documents(),
|
||||||
|
)
|
||||||
|
|
||||||
|
write_json(
|
||||||
|
chunks_file,
|
||||||
|
broken,
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
ValueError,
|
||||||
|
match="chunk_id",
|
||||||
|
):
|
||||||
indexer.build_database()
|
indexer.build_database()
|
||||||
|
|||||||
@ -11,7 +11,13 @@ from scripts.search_utils import search_database
|
|||||||
|
|
||||||
|
|
||||||
def write_json(path: Path, value) -> None:
|
def write_json(path: Path, value) -> None:
|
||||||
path.write_text(json.dumps(value, ensure_ascii=False), encoding="utf-8")
|
path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
value,
|
||||||
|
ensure_ascii=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_search_database(
|
def build_search_database(
|
||||||
@ -141,18 +147,43 @@ def build_search_database(
|
|||||||
"deep::0",
|
"deep::0",
|
||||||
"pages/topics/deep/README.md",
|
"pages/topics/deep/README.md",
|
||||||
"Neurónové siete",
|
"Neurónové siete",
|
||||||
"Modely hlbokého učenia a trénovanie hlbokých neurónových sietí.",
|
(
|
||||||
|
"Modely hlbokého učenia a trénovanie "
|
||||||
|
"hlbokých neurónových sietí."
|
||||||
|
),
|
||||||
False,
|
False,
|
||||||
tags=["nn"],
|
tags=["nn"],
|
||||||
),
|
),
|
||||||
]
|
]
|
||||||
|
|
||||||
write_json(documents_file, documents)
|
write_json(
|
||||||
write_json(chunks_file, chunks)
|
documents_file,
|
||||||
|
documents,
|
||||||
|
)
|
||||||
|
|
||||||
|
write_json(
|
||||||
|
chunks_file,
|
||||||
|
chunks,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"DOCUMENTS_FILE",
|
||||||
|
documents_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"CHUNKS_FILE",
|
||||||
|
chunks_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
indexer,
|
||||||
|
"DB_FILE",
|
||||||
|
db_file,
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setattr(indexer, "DOCUMENTS_FILE", documents_file)
|
|
||||||
monkeypatch.setattr(indexer, "CHUNKS_FILE", chunks_file)
|
|
||||||
monkeypatch.setattr(indexer, "DB_FILE", db_file)
|
|
||||||
indexer.build_database()
|
indexer.build_database()
|
||||||
|
|
||||||
return db_file
|
return db_file
|
||||||
@ -162,58 +193,124 @@ def test_person_query_does_not_add_any_term_noise(
|
|||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
response = search_database(db_file, "jan ptak", limit=5)
|
response = search_database(
|
||||||
|
db_file,
|
||||||
|
"jan ptak",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
|
||||||
assert response["strategies"] == ["all_terms"]
|
assert response["strategies"] == [
|
||||||
assert [item["title"] for item in response["results"]] == ["Ján Pták"]
|
"all_terms"
|
||||||
|
]
|
||||||
|
|
||||||
|
assert [
|
||||||
|
item["title"]
|
||||||
|
for item in response["results"]
|
||||||
|
] == [
|
||||||
|
"Ján Pták"
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_topic_query_returns_best_topic_first(
|
def test_topic_query_returns_best_topic_first(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
response = search_database(db_file, "strojovy preklad", limit=5)
|
response = search_database(
|
||||||
|
db_file,
|
||||||
|
"strojovy preklad",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
|
||||||
assert response["strategies"] == ["all_terms"]
|
assert response["strategies"] == [
|
||||||
assert response["results"][0]["title"] == "Strojový preklad"
|
"all_terms"
|
||||||
|
]
|
||||||
|
|
||||||
|
assert (
|
||||||
|
response["results"][0]["title"]
|
||||||
|
== "Strojový preklad"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_prefix_fallback_handles_slovak_inflection(
|
def test_prefix_fallback_handles_slovak_inflection(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
response = search_database(db_file, "hlboke ucenie", limit=5)
|
response = search_database(
|
||||||
|
db_file,
|
||||||
|
"hlboke ucenie",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
|
||||||
assert response["strategies"] == ["prefix_terms"]
|
assert response["strategies"] == [
|
||||||
assert response["results"][0]["title"] == "Neurónové siete"
|
"prefix_terms"
|
||||||
|
]
|
||||||
|
|
||||||
|
assert (
|
||||||
|
response["results"][0]["title"]
|
||||||
|
== "Neurónové siete"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_any_term_is_used_only_when_stricter_queries_find_nothing(
|
def test_any_term_is_used_only_when_stricter_queries_find_nothing(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
response = search_database(db_file, "nezmysel preklad", limit=5)
|
response = search_database(
|
||||||
|
db_file,
|
||||||
|
"nezmysel preklad",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response["strategies"] == [
|
||||||
|
"any_term"
|
||||||
|
]
|
||||||
|
|
||||||
assert response["strategies"] == ["any_term"]
|
|
||||||
assert response["results"]
|
assert response["results"]
|
||||||
assert all("preklad" in item["text"].casefold() for item in response["results"])
|
|
||||||
|
assert any(
|
||||||
|
item.get("match_strategy")
|
||||||
|
== "any_term"
|
||||||
|
and "preklad"
|
||||||
|
in item["text"].casefold()
|
||||||
|
for item in response["results"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_published_only_excludes_unpublished_chunks(
|
def test_published_only_excludes_unpublished_chunks(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
|
all_results = search_database(
|
||||||
|
db_file,
|
||||||
|
"hlboke ucenie",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
|
||||||
all_results = search_database(db_file, "hlboke ucenie", limit=5)
|
|
||||||
public_results = search_database(
|
public_results = search_database(
|
||||||
db_file,
|
db_file,
|
||||||
"hlboke ucenie",
|
"hlboke ucenie",
|
||||||
@ -222,14 +319,26 @@ def test_published_only_excludes_unpublished_chunks(
|
|||||||
)
|
)
|
||||||
|
|
||||||
assert all_results["results"]
|
assert all_results["results"]
|
||||||
assert public_results["results"] == []
|
|
||||||
|
assert any(
|
||||||
|
item["published"] is False
|
||||||
|
for item in all_results["results"]
|
||||||
|
)
|
||||||
|
|
||||||
|
assert all(
|
||||||
|
item["published"] is True
|
||||||
|
for item in public_results["results"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_results_are_diversified_by_document(
|
def test_results_are_diversified_by_document(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
response = search_database(
|
response = search_database(
|
||||||
db_file,
|
db_file,
|
||||||
@ -238,48 +347,97 @@ def test_results_are_diversified_by_document(
|
|||||||
max_per_document=1,
|
max_per_document=1,
|
||||||
)
|
)
|
||||||
|
|
||||||
paths = [item["document_path"] for item in response["results"]]
|
paths = [
|
||||||
assert len(paths) == len(set(paths))
|
item["document_path"]
|
||||||
|
for item in response["results"]
|
||||||
|
]
|
||||||
|
|
||||||
|
assert len(paths) == len(
|
||||||
|
set(paths)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_public_result_format_has_no_internal_id_and_uses_boolean(
|
def test_public_result_format_has_no_internal_id_and_uses_boolean(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
result = search_database(db_file, "jan ptak", limit=1)["results"][0]
|
result = search_database(
|
||||||
|
db_file,
|
||||||
|
"jan ptak",
|
||||||
|
limit=1,
|
||||||
|
)["results"][0]
|
||||||
|
|
||||||
assert "id" not in result
|
assert "id" not in result
|
||||||
assert result["published"] is True
|
assert result["published"] is True
|
||||||
assert result["chunk_id"] == "jan-ptak::0"
|
assert result["chunk_id"] == "jan-ptak::0"
|
||||||
assert result["source_url"].endswith("students/jan_ptak")
|
|
||||||
|
assert result["source_url"].endswith(
|
||||||
|
"students/jan_ptak"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_empty_query_returns_empty_result(
|
def test_empty_query_returns_empty_result(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
db_file = build_search_database(tmp_path, monkeypatch)
|
db_file = build_search_database(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
)
|
||||||
|
|
||||||
response = search_database(db_file, " ", limit=5)
|
response = search_database(
|
||||||
|
db_file,
|
||||||
|
" ",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
|
||||||
assert response == {
|
assert response == {
|
||||||
"engine": "sqlite_fts5",
|
"engine": "hybrid_fts5_embeddings",
|
||||||
"strategies": [],
|
"strategies": [],
|
||||||
"results": [],
|
"results": [],
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def test_missing_database_is_reported(tmp_path: Path) -> None:
|
def test_missing_database_is_reported(
|
||||||
with pytest.raises(FileNotFoundError):
|
tmp_path: Path,
|
||||||
search_database(tmp_path / "missing.sqlite", "rag")
|
) -> None:
|
||||||
|
with pytest.raises(
|
||||||
|
FileNotFoundError
|
||||||
|
):
|
||||||
|
search_database(
|
||||||
|
tmp_path / "missing.sqlite",
|
||||||
|
"rag",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_missing_fts_schema_is_reported(tmp_path: Path) -> None:
|
def test_missing_fts_schema_is_reported(
|
||||||
db_file = tmp_path / "broken.sqlite"
|
tmp_path: Path,
|
||||||
with sqlite3.connect(db_file) as conn:
|
) -> None:
|
||||||
conn.execute("CREATE TABLE chunks(id INTEGER PRIMARY KEY)")
|
db_file = (
|
||||||
|
tmp_path / "broken.sqlite"
|
||||||
|
)
|
||||||
|
|
||||||
with pytest.raises(RuntimeError, match="FTS5 index"):
|
with sqlite3.connect(
|
||||||
search_database(db_file, "rag")
|
db_file
|
||||||
|
) as conn:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE chunks(
|
||||||
|
id INTEGER PRIMARY KEY
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
RuntimeError,
|
||||||
|
match="FTS5 index",
|
||||||
|
):
|
||||||
|
search_database(
|
||||||
|
db_file,
|
||||||
|
"rag",
|
||||||
|
)
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user