Presun nasadenia
This commit is contained in:
parent
31ecefed40
commit
aaa0a65f93
33
README.md
33
README.md
@ -274,6 +274,7 @@ zp-agent/
|
|||||||
│
|
│
|
||||||
├── scripts/
|
├── scripts/
|
||||||
│ ├── start_project.sh
|
│ ├── start_project.sh
|
||||||
|
│ ├── compose.sh
|
||||||
│ └── verify_project.py
|
│ └── verify_project.py
|
||||||
│
|
│
|
||||||
├── data/
|
├── data/
|
||||||
@ -281,10 +282,11 @@ zp-agent/
|
|||||||
│ ├── chunks.json
|
│ ├── chunks.json
|
||||||
│ └── zp_index.sqlite
|
│ └── zp_index.sqlite
|
||||||
│
|
│
|
||||||
├── Dockerfile
|
├── deployment/
|
||||||
├── docker-compose.yml
|
│ ├── Dockerfile
|
||||||
├── requirements.txt
|
│ ├── docker-compose.yml
|
||||||
├── requirements-dev.txt
|
│ ├── requirements.txt
|
||||||
|
│ └── requirements-dev.txt
|
||||||
└── README.md
|
└── README.md
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -679,6 +681,13 @@ Po vytvorení `.env`, klonu `../zpwiki` a SSH konfigurácie spusti:
|
|||||||
./scripts/start_project.sh
|
./scripts/start_project.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Dockerfile, Compose a oba zoznamy závislostí sú v `deployment/`.
|
||||||
|
`.env`, `.gitignore` a `.dockerignore` zostávajú v koreni projektu.
|
||||||
|
Príkazy Compose spúšťaj cez `./scripts/compose.sh`, napríklad
|
||||||
|
`./scripts/compose.sh ps`. Obal nastavuje absolútne cesty, koreň build
|
||||||
|
kontextu a pôvodný názov projektu `zp-agent`, takže sa nemenia dátové volume.
|
||||||
|
Funguje aj pri spustení cez absolútnu cestu z iného pracovného priečinka.
|
||||||
|
|
||||||
Skript štandardne používa kontext `default`, nemení globálny Docker kontext
|
Skript štandardne používa kontext `default`, nemení globálny Docker kontext
|
||||||
a spravuje iba projekt `zp-agent`. Iný cieľ nastav cez `ZP_DOCKER_CONTEXT`.
|
a spravuje iba projekt `zp-agent`. Iný cieľ nastav cez `ZP_DOCKER_CONTEXT`.
|
||||||
Zostaví image, počká na zdravý Neo4j, overí existujúce indexy a podľa
|
Zostaví image, počká na zdravý Neo4j, overí existujúce indexy a podľa
|
||||||
@ -721,13 +730,13 @@ http://127.0.0.1:8000/openapi.json
|
|||||||
Logy:
|
Logy:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose logs -f zp-agent-api
|
./scripts/compose.sh logs -f zp-agent-api
|
||||||
```
|
```
|
||||||
|
|
||||||
Zastavenie:
|
Zastavenie:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose down
|
./scripts/compose.sh down
|
||||||
```
|
```
|
||||||
|
|
||||||
Hugging Face cache volume zostáva persistentné; predvolený embeddingový
|
Hugging Face cache volume zostáva persistentné; predvolený embeddingový
|
||||||
@ -737,7 +746,7 @@ Neo4j je spustené ako samostatná služba v rovnakom Docker Compose
|
|||||||
prostredí. Stav služieb je možné skontrolovať:
|
prostredí. Stav služieb je možné skontrolovať:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose ps
|
./scripts/compose.sh ps
|
||||||
```
|
```
|
||||||
|
|
||||||
Neo4j Browser je lokálne dostupný na:
|
Neo4j Browser je lokálne dostupný na:
|
||||||
@ -749,7 +758,7 @@ http://127.0.0.1:7474/browser/
|
|||||||
Kontrola spojenia z API kontajnera:
|
Kontrola spojenia z API kontajnera:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose exec zp-agent-api \
|
./scripts/compose.sh exec zp-agent-api \
|
||||||
python -m tools.graphrag.neo4j_smoke
|
python -m tools.graphrag.neo4j_smoke
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -760,7 +769,7 @@ a Neo4j graf z jedného snapshotu wiki. Nová generácia sa aktivuje až po
|
|||||||
úspešnej príprave oboch indexov:
|
úspešnej príprave oboch indexov:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose run --rm zp-agent-api python -m tools.rag.rebuild_index
|
./scripts/compose.sh run --rm zp-agent-api python -m tools.rag.rebuild_index
|
||||||
```
|
```
|
||||||
|
|
||||||
Vzniknú súbory:
|
Vzniknú súbory:
|
||||||
@ -831,7 +840,7 @@ význam môže dostať embeddingové vyhľadávanie.
|
|||||||
Test z terminálu:
|
Test z terminálu:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose run --rm zp-agent-api \
|
./scripts/compose.sh run --rm zp-agent-api \
|
||||||
python -m tools.rag.search_db "rag agent" --limit 5
|
python -m tools.rag.search_db "rag agent" --limit 5
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -943,7 +952,7 @@ Document -[:DESCRIBES]-> Topic
|
|||||||
Knowledge graph sa vytvorí príkazom:
|
Knowledge graph sa vytvorí príkazom:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose exec zp-agent-api \
|
./scripts/compose.sh exec zp-agent-api \
|
||||||
python -m tools.graphrag.build_graphrag
|
python -m tools.graphrag.build_graphrag
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -1233,7 +1242,7 @@ hodnôt.
|
|||||||
Inštalácia testovacích závislostí:
|
Inštalácia testovacích závislostí:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -r requirements-dev.txt
|
pip install -r deployment/requirements-dev.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
Bežné automatizované testy:
|
Bežné automatizované testy:
|
||||||
|
|||||||
@ -19,10 +19,10 @@ RUN python -m pip install --no-cache-dir \
|
|||||||
torch==2.11.0 \
|
torch==2.11.0 \
|
||||||
--index-url https://download.pytorch.org/whl/cpu
|
--index-url https://download.pytorch.org/whl/cpu
|
||||||
|
|
||||||
COPY requirements.txt .
|
COPY deployment/requirements.txt ./deployment/requirements.txt
|
||||||
|
|
||||||
RUN python -m pip install --no-cache-dir \
|
RUN python -m pip install --no-cache-dir \
|
||||||
-r requirements.txt
|
-r deployment/requirements.txt
|
||||||
|
|
||||||
COPY app ./app
|
COPY app ./app
|
||||||
COPY tools ./tools
|
COPY tools ./tools
|
||||||
@ -1,7 +1,10 @@
|
|||||||
|
name: zp-agent
|
||||||
|
|
||||||
services:
|
services:
|
||||||
zp-agent-api:
|
zp-agent-api:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: ..
|
||||||
|
dockerfile: deployment/Dockerfile
|
||||||
args:
|
args:
|
||||||
EMBEDDING_MODEL: ${EMBEDDING_MODEL:-intfloat/multilingual-e5-small}
|
EMBEDDING_MODEL: ${EMBEDDING_MODEL:-intfloat/multilingual-e5-small}
|
||||||
EMBEDDING_REVISION: ${EMBEDDING_REVISION:-}
|
EMBEDDING_REVISION: ${EMBEDDING_REVISION:-}
|
||||||
@ -12,7 +15,7 @@ services:
|
|||||||
- "8000:8000"
|
- "8000:8000"
|
||||||
|
|
||||||
env_file:
|
env_file:
|
||||||
- .env
|
- ../.env
|
||||||
|
|
||||||
environment:
|
environment:
|
||||||
ZPWIKI_ROOT: /zpwiki
|
ZPWIKI_ROOT: /zpwiki
|
||||||
@ -37,8 +40,8 @@ services:
|
|||||||
NEO4J_DATABASE: ${NEO4J_DATABASE:-neo4j}
|
NEO4J_DATABASE: ${NEO4J_DATABASE:-neo4j}
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
- ./data:/app/data
|
- ../data:/app/data
|
||||||
- ../zpwiki:/zpwiki
|
- ../../zpwiki:/zpwiki
|
||||||
- hf-cache:/cache/huggingface
|
- hf-cache:/cache/huggingface
|
||||||
- type: bind
|
- type: bind
|
||||||
source: ${ZPWIKI_SSH_KEY:-${HOME}/.ssh/id_ed25519}
|
source: ${ZPWIKI_SSH_KEY:-${HOME}/.ssh/id_ed25519}
|
||||||
10
scripts/compose.sh
Executable file
10
scripts/compose.sh
Executable file
@ -0,0 +1,10 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
project_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
docker_context="${ZP_DOCKER_CONTEXT:-default}"
|
||||||
|
|
||||||
|
exec docker --context "$docker_context" compose --project-name zp-agent \
|
||||||
|
--project-directory "$project_root/deployment" \
|
||||||
|
--env-file "$project_root/.env" \
|
||||||
|
-f "$project_root/deployment/docker-compose.yml" "$@"
|
||||||
@ -3,8 +3,7 @@ set -euo pipefail
|
|||||||
|
|
||||||
project_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
project_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
docker_context="${ZP_DOCKER_CONTEXT:-default}"
|
docker_context="${ZP_DOCKER_CONTEXT:-default}"
|
||||||
compose=(docker --context "$docker_context" compose --project-name zp-agent
|
compose=("$project_root/scripts/compose.sh")
|
||||||
--project-directory "$project_root" -f "$project_root/docker-compose.yml")
|
|
||||||
|
|
||||||
docker --context "$docker_context" info --format '{{.ServerVersion}}' >/dev/null
|
docker --context "$docker_context" info --format '{{.ServerVersion}}' >/dev/null
|
||||||
"${compose[@]}" version >/dev/null
|
"${compose[@]}" version >/dev/null
|
||||||
|
|||||||
@ -124,7 +124,10 @@ def main() -> int:
|
|||||||
return result.stdout.strip()
|
return result.stdout.strip()
|
||||||
|
|
||||||
docker = ['docker', '--context', args.docker_context]
|
docker = ['docker', '--context', args.docker_context]
|
||||||
compose = [*docker, 'compose', '--project-directory', str(ROOT)]
|
compose = [*docker, 'compose', '--project-name', 'zp-agent',
|
||||||
|
'--project-directory', str(ROOT / 'deployment'),
|
||||||
|
'--env-file', str(ROOT / '.env'),
|
||||||
|
'-f', str(ROOT / 'deployment/docker-compose.yml')]
|
||||||
check('python_and_shell', verify_code)
|
check('python_and_shell', verify_code)
|
||||||
check('git_diff', lambda: command('git', 'diff', '--check'))
|
check('git_diff', lambda: command('git', 'diff', '--check'))
|
||||||
check('git_integrity', lambda: command('git', 'fsck', '--connectivity-only'))
|
check('git_integrity', lambda: command('git', 'fsck', '--connectivity-only'))
|
||||||
|
|||||||
@ -28,7 +28,7 @@ def requirement_names(text: str) -> set[str]:
|
|||||||
|
|
||||||
def test_no_development_secret_is_committed_in_runtime_config() -> None:
|
def test_no_development_secret_is_committed_in_runtime_config() -> None:
|
||||||
files = [
|
files = [
|
||||||
read_project_file("docker-compose.yml"),
|
read_project_file("deployment/docker-compose.yml"),
|
||||||
read_project_file("app/main.py"),
|
read_project_file("app/main.py"),
|
||||||
]
|
]
|
||||||
|
|
||||||
@ -36,7 +36,7 @@ def test_no_development_secret_is_committed_in_runtime_config() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def test_compose_uses_env_file_and_keeps_chunk_configuration() -> None:
|
def test_compose_uses_env_file_and_keeps_chunk_configuration() -> None:
|
||||||
compose = read_project_file("docker-compose.yml")
|
compose = read_project_file("deployment/docker-compose.yml")
|
||||||
|
|
||||||
assert "env_file:" in compose
|
assert "env_file:" in compose
|
||||||
assert ".env" in compose
|
assert ".env" in compose
|
||||||
@ -44,8 +44,8 @@ def test_compose_uses_env_file_and_keeps_chunk_configuration() -> None:
|
|||||||
assert "CHUNK_OVERLAP_TOKENS" in compose
|
assert "CHUNK_OVERLAP_TOKENS" in compose
|
||||||
assert "CHUNK_MIN_TOKENS" in compose
|
assert "CHUNK_MIN_TOKENS" in compose
|
||||||
assert "CHUNK_TOKEN_ENCODING" in compose
|
assert "CHUNK_TOKEN_ENCODING" in compose
|
||||||
assert "./data:/app/data" in compose
|
assert "../data:/app/data" in compose
|
||||||
assert "../zpwiki:/zpwiki" in compose
|
assert "../../zpwiki:/zpwiki" in compose
|
||||||
|
|
||||||
|
|
||||||
def test_secrets_are_ignored_by_git_and_docker() -> None:
|
def test_secrets_are_ignored_by_git_and_docker() -> None:
|
||||||
@ -57,7 +57,7 @@ def test_secrets_are_ignored_by_git_and_docker() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def test_runtime_requirements_contain_only_direct_dependencies() -> None:
|
def test_runtime_requirements_contain_only_direct_dependencies() -> None:
|
||||||
names = requirement_names(read_project_file("requirements.txt"))
|
names = requirement_names(read_project_file("deployment/requirements.txt"))
|
||||||
|
|
||||||
required = {
|
required = {
|
||||||
"fastapi",
|
"fastapi",
|
||||||
|
|||||||
@ -37,9 +37,9 @@ def content_fingerprints(questions: Path, root: Path = ROOT) -> dict[str, Any]:
|
|||||||
# too, so a freeze cannot silently overlook new runtime/delivery helpers.
|
# too, so a freeze cannot silently overlook new runtime/delivery helpers.
|
||||||
for package in ('answers', 'retrieval', 'runtime', 'sync'):
|
for package in ('answers', 'retrieval', 'runtime', 'sync'):
|
||||||
files.extend(sorted((root / 'tools' / package).rglob('*.py')))
|
files.extend(sorted((root / 'tools' / package).rglob('*.py')))
|
||||||
# requirements.txt and tools/ are present both in the checkout and image.
|
# Deployment requirements and tools/ exist in both checkout and image.
|
||||||
files.extend(root / name for name in [
|
files.extend(root / name for name in [
|
||||||
'requirements.txt', 'tools/retrieval/publication.py', 'tools/retrieval/retrieval_contract.py',
|
'deployment/requirements.txt', 'tools/retrieval/publication.py', 'tools/retrieval/retrieval_contract.py',
|
||||||
'tools/sync/index_state.py', 'tools/rag/common.py', 'tools/sync/sync_indexes.py',
|
'tools/sync/index_state.py', 'tools/rag/common.py', 'tools/sync/sync_indexes.py',
|
||||||
'tools/rag/rag_utils.py', 'tools/answer_evaluation/context_policy.py',
|
'tools/rag/rag_utils.py', 'tools/answer_evaluation/context_policy.py',
|
||||||
'tools/answer_evaluation/checked_generation.py', 'tools/answer_evaluation/answer_format.py',
|
'tools/answer_evaluation/checked_generation.py', 'tools/answer_evaluation/answer_format.py',
|
||||||
|
|||||||
@ -101,6 +101,7 @@ def test_freeze_includes_shared_modules_and_effective_publication_policy():
|
|||||||
questions=Path('tools/graphrag/evaluation/graphrag_questions_v4.json')
|
questions=Path('tools/graphrag/evaluation/graphrag_questions_v4.json')
|
||||||
fp=content_fingerprints(questions)
|
fp=content_fingerprints(questions)
|
||||||
assert {'tools/retrieval/publication.py','tools/retrieval/retrieval_contract.py','tools/sync/index_state.py'} <= fp['implementation'].keys()
|
assert {'tools/retrieval/publication.py','tools/retrieval/retrieval_contract.py','tools/sync/index_state.py'} <= fp['implementation'].keys()
|
||||||
|
assert 'deployment/requirements.txt' in fp['implementation']
|
||||||
assert fp['retrieval_policy']['published_only'] is True
|
assert fp['retrieval_policy']['published_only'] is True
|
||||||
assert fp['retrieval_policy']['academic_preparation_is_work'] is False
|
assert fp['retrieval_policy']['academic_preparation_is_work'] is False
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user