perun + fix
This commit is contained in:
parent
020768b261
commit
59b0b6ef39
|
Before Width: | Height: | Size: 197 KiB After Width: | Height: | Size: 197 KiB |
340
perun/README.md
Normal file
340
perun/README.md
Normal file
@ -0,0 +1,340 @@
|
||||
# Slovenská Aya: Mistral-SK a OLMo
|
||||
|
||||
## Scenár
|
||||
|
||||
Cieľom je inštrukčne dotrénovať Mistral-SK-7B a základný OLMo 2 7B na spoločnej slovenskej časti Aya Collection a porovnať ich adaptáciu. Použije sa **SFT s LoRA v LLaMA-Factory, zrýchlený Unslothom**. Jeden tréning využije **štyri H200 na jednom uzle Peruna cez DDP**.
|
||||
|
||||
Možnosti LoRA do návrhu scenára:
|
||||
|
||||
- **LoRA (Low-Rank Adaptation)**
|
||||
- **QLoRA (Quantized Low-Rank Adaptation)**
|
||||
- **rsLoRA (Rank-Stabilized LoRA)**
|
||||
- **DoRA (Weight-Decomposed Low-Rank Adaptation)**
|
||||
|
||||
|
||||
## Prehľad súborov
|
||||
|
||||
| Súbor | Úloha |
|
||||
| --- | --- |
|
||||
| `configs/mistral.yaml` | Základný tréningový config pre `slovak-nlp/mistral-sk-7b`. |
|
||||
| `configs/olmo.yaml` | Základný tréningový config pre `allenai/OLMo-2-1124-7B`. |
|
||||
| `scripts/prepare_data.py` | Načítanie, čistenie, deduplikácia a rozdelenie dát. |
|
||||
| `scripts/train.py` | Spustenie LLaMA-Factory podľa zvoleného configu. |
|
||||
| `scripts/evaluate.py` | Loss, perplexita, generovanie a slovný F1 / presná zhoda na eval alebo test množine. |
|
||||
|
||||
Balík má spolu šesť súborov. `evaluate.py` odvodí nastavenia z rovnakého modelového YAML a konkrétnu použitú konfiguráciu uloží až k výsledkom.
|
||||
|
||||
Poradie používania je **príprava dát → tréning → vyhodnotenie**. Príprava dát sa vykoná raz pre oba modely. Tréning sa spustí samostatne pre každý model alebo experiment a následne sa vyhodnotí vytvorený adaptér. Model a hyperparametre sa nastavujú v YAML; Python skripty vykonávajú jednotlivé kroky.
|
||||
|
||||
## Spoločný dataset do 4 096 tokenov a nastavenie H200
|
||||
|
||||
Oba modely používajú **`cutoff_len: 4096`**. Limit zahŕňa celú SFT sekvenciu: systémový text Alpaca, inštrukciu a prípadný vstup, celú referenčnú odpoveď, EOS a koncové oddeľovače. Spúšťací skript kontroluje pôvodnú kontextovú kapacitu modelu podľa `cutoff_len`.
|
||||
|
||||
`prepare_data.py` si načíta oba YAML configy a ich presné tokenizéry. **Celý pár vyradí, ak prekročí limit čo i len pri jednom modeli.** Všetky ponechané príklady zapíše bez orezania. Filtrovanie prebehne pred deduplikáciou a spoločným rozdelením na train/eval/test, takže oba modely dostanú rovnaké príklady. Inštrukcia a pole `input` sa spájajú rovnako ako v Alpaca konvertore LLaMA-Factory.
|
||||
|
||||
|
||||
Nastavenie jedného tréningu:
|
||||
|
||||
- štyri H200 cez DDP, `bf16: true`, `use_unsloth: true`, `use_unsloth_gc: true`, bez kvantizácie;
|
||||
- batch 8 na GPU, akumulácia 1: **4 × 8 × 1 = efektívny batch 32**;
|
||||
- eval loss s batchom 8 na GPU, záverečné generovanie po jednom príklade;
|
||||
- LoRA rank 16, alpha 32, dropout 0, projekcie attention aj MLP;
|
||||
- jedna epocha, AdamW, cosine scheduler, warmup 8 %, clipping gradientov 0,5;
|
||||
- loss iba z odpovedí, bez packingu, dynamický padding na dĺžku aktuálnej dávky;
|
||||
- validácia/checkpoint každých 1 000 optimizer krokov, výber podľa najnižšieho loss, limit checkpointov 2.
|
||||
|
||||
|
||||
Spúšťač overuje, že dáta majú záznam o dĺžkovom filtrovaní pre daný model, revíziu, šablónu a limit. Kontrola pri tokenizácii tiež zastaví beh, ak by predspracovanie chcelo orezať vstup alebo odpoveď. Konfigurácia načítaného základného modelu sa uloží do `adapter/base_model_config.json`.
|
||||
|
||||
LR `0.0001` pri Mistrale nadväzuje na 300k experiment; pri OLMo je východiskový kandidát. Zmenené nastavenia stále treba overiť podľa eval dát. OLMo je základný model pre vlastné SFT a jeho karta uvádza angličtinu.
|
||||
|
||||
## Prostredie na Perunovi
|
||||
|
||||
Na spustenie sa používa kompatibilné GPU prostredie s LLaMA-Factory, Unsloth, Unsloth Zoo, PyTorch a CUDA pre H200. LLaMA-Factory používa aj `datasets`, `huggingface_hub` a `PyYAML`; generatívna evaluácia potrebuje `rouge-chinese`, `jieba` a `nltk`. Python má byť aspoň 3.11 podľa aktuálnych požiadaviek LLaMA-Factory. Dĺžkový filter pri príprave dát používa Transformers a PyYAML a načíta oba tokenizéry; váhy modelov na túto CPU úlohu nepotrebuje.
|
||||
|
||||
### Požiadavky a inštalácia prostredí
|
||||
|
||||
Priečinok `requirements` obsahuje verzie odvodené z exportu Hrušky.
|
||||
|
||||
| Súbor | Prostredie |
|
||||
| --- | --- |
|
||||
| `requirements/training.txt` | Python 3.11.16; LLaMA-Factory, Unsloth, PyTorch 2.6.0 s CUDA 12.4 a generatívna evaluácia. |
|
||||
| `requirements/lm-eval.txt` | Python 3.11.16; lm-evaluation-harness, Transformers 4.48.2 a PyTorch 2.6.0 s CUDA 12.4. |
|
||||
| `requirements/skbench.txt` | Python 3.13.17; export 308 balíkov SK-bench vrátane vLLM 0.28.0, Transformers 5.12.1 a PyTorch 2.13.0. |
|
||||
| `requirements/skbench/pyproject.toml` a `uv.lock` | Pôvodné závislosti a uzamknuté verzie SK-bench pre obnovenie cez uv. |
|
||||
|
||||
Git závislosti LLaMA-Factory, lm-eval a lighteval sú pripnuté na pôvodné commity. Neplatná lokálna cesta k pip bola v tréningovom a lm-eval exporte nahradená verziou `pip==26.2.1` z Condy. Výpočtové knižnice zachovávajú pôvodné verzie.
|
||||
|
||||
Do tréningových požiadaviek boli doplnené `jieba==0.42.1`, `rouge-chinese==1.0.3`, `nltk==3.10.3` a jeho závislosti `joblib==1.6.0` a `defusedxml==0.7.1` pre generatívnu evaluáciu. Doplnený je aj `torchao==0.13.0`, pretože ho vyžaduje zachytená verzia Unsloth Zoo a export ho neobsahoval. Táto verzia TorchAO podporuje Python API s PyTorch 2.6; jej natívne C++ rozšírenia sú určené pre PyTorch 2.8. Scenár používa LoRA bez kvantizácie TorchAO.
|
||||
|
||||
Pre tréning je `setuptools` upravený na `80.10.2`, rovnako ako vo funkčnom SK-bench exporte. Knižnica jieba používa `pkg_resources`, ktorý bol zo setuptools 82 odstránený. Tréningové a benchmarkové verzie ostatných zachytených balíkov sa nemenia.
|
||||
|
||||
Ide o požiadavky pripravené z funkčných prostredí Hrušky a uvedených doplnení. Čistá inštalácia a výpočet na H200 sa overia na Perunovi.
|
||||
|
||||
#### Tréning a evaluácia na pripravených Aya dátach
|
||||
|
||||
Nasledujúce príkazy sa zadávajú z koreňa balíka v prostredí s dostupnou Condou a Gitom. PyTorch a xformers sa najprv nainštalujú z rovnakého oficiálneho CUDA 12.4 indexu:
|
||||
|
||||
```bash
|
||||
conda create -n aya_train_perun python=3.11.16 pip=26.2.1 -y
|
||||
conda activate aya_train_perun
|
||||
|
||||
python -m pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 xformers==0.0.29.post3 \
|
||||
--index-url https://download.pytorch.org/whl/cu124
|
||||
|
||||
python -m pip install -r requirements/training.txt
|
||||
python -m pip check
|
||||
```
|
||||
|
||||
Na pridelenom GPU uzle sa overia importy, CUDA a BF16:
|
||||
|
||||
```bash
|
||||
python -c 'import unsloth, torch, jieba, nltk; from rouge_chinese import Rouge; assert torch.cuda.is_available(); assert torch.cuda.is_bf16_supported(); print("GPU:", torch.cuda.get_device_name(0)); print("Viditelne GPU:", torch.cuda.device_count()); print("CUDA:", torch.version.cuda)'
|
||||
```
|
||||
|
||||
Príprava dát, tréning aj `scripts/evaluate.py` sa potom spúšťajú v `aya_train_perun` podľa ďalších častí dokumentácie. Overené opravy generovania z Titanu sú súčasťou `scripts/train.py`.
|
||||
|
||||
#### lm-evaluation-harness
|
||||
|
||||
Príkazy sa zadávajú z koreňa balíka:
|
||||
|
||||
```bash
|
||||
conda create -n aya_lmeval_perun python=3.11.16 pip=26.2.1 -y
|
||||
conda activate aya_lmeval_perun
|
||||
|
||||
python -m pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
|
||||
python -m pip install -r requirements/lm-eval.txt
|
||||
python -m pip check
|
||||
lm_eval --help
|
||||
```
|
||||
|
||||
Súbor inštaluje framework pri revízii `6d2abda4fd171e68a8789330c4149e37c1ca0bda`. Vlastné slovenské úlohy, zoznam benchmarkov a evaluačné dáta sa prenášajú samostatne; požiadavky obsahujú Python knižnice.
|
||||
|
||||
#### SK-bench
|
||||
|
||||
Na obnovenie SK-bench sa používa uv 0.13.5 a zdrojová kópia projektu z Hrušky pri revízii `15823bbcc2d995e69441fcb7d34abb61bd8180c5`. Súbory `pyproject.toml` a `uv.lock` sa skopírujú z tohto balíka do koreňa zdrojovej kópie SK-bench. Ak je uv potrebné doplniť, nainštaluje sa cez `python -m pip install uv==0.13.5` v existujúcom Python prostredí; vlastné závislosti SK-bench sa vytvoria v jeho samostatnej `.venv`.
|
||||
|
||||
Z koreňa balíka, pri zdrojovej kópii SK-bench uloženej v `../sk-bench`:
|
||||
|
||||
```bash
|
||||
cp requirements/skbench/pyproject.toml requirements/skbench/uv.lock ../sk-bench/
|
||||
cd ../sk-bench
|
||||
|
||||
uv sync --locked --no-default-groups --python 3.13.17
|
||||
uv pip check --python .venv/bin/python
|
||||
uv run --locked python -c 'import torch, vllm, transformers; print(torch.__version__, vllm.__version__, transformers.__version__)'
|
||||
```
|
||||
|
||||
Základná uzamknutá zostava zodpovedá všetkým 308 balíkom exportovaným z Hrušky. Skupiny `dev` a `hpc` sa pri tomto obnovení nepridávajú. `requirements/skbench.txt` uchováva rovnaké konkrétne verzie v pip formáte; pri používaní projektu cez uv sa vychádza z jeho `uv.lock`.
|
||||
|
||||
SK-bench používa CUDA 13; pri jeho prenose sa na pridelenom GPU uzle overí ovládač cez `nvidia-smi` a dostupnosť CUDA v PyTorchi. Pôvodný export Hrušky uvádza ovládač 580.178.04.
|
||||
|
||||
Zdroje: [PyTorch CUDA zostavy](https://pytorch.org/get-started/previous-versions/#v260), [kompatibilita TorchAO](https://github.com/pytorch/ao/issues/2919), [závislosti Unsloth Zoo 2026.8.16](https://pypi.org/pypi/unsloth-zoo/2026.8.16/json), [zmeny setuptools](https://setuptools.pypa.io/en/latest/history.html#v82-0-0), [uzamknuté prostredia uv](https://docs.astral.sh/uv/concepts/projects/sync/).
|
||||
|
||||
### Odosielanie úloh cez Slurm
|
||||
|
||||
Priame Python príkazy sú určené pre už pridelený výpočtový uzol. Z prihlasovacieho uzla sa úlohy odosielajú cez `sbatch --wrap`. Príkazy sa zadávajú z koreňa balíka s aktivovaným Python prostredím; tento priečinok sa použije aj pri výpočte.
|
||||
|
||||
Pred odoslaním sa v rovnakom termináli nastavia projektový účet a QoS. Hodnoty `NAZOV_PROJEKTU` a `PRIDELENE_QOS` sa nahradia údajmi prideleného projektu:
|
||||
|
||||
```bash
|
||||
AYA_ACCOUNT="NAZOV_PROJEKTU"
|
||||
AYA_QOS="PRIDELENE_QOS"
|
||||
```
|
||||
|
||||
Na Perunovi sa pre výpočet používa pracovný priečinok na SCRATCH. Dáta a výsledky určené na uchovanie sa po výpočte skopírujú do projektového úložiska. Parameter `--time` určuje maximálny čas úlohy.
|
||||
|
||||
## 1. Príprava dát
|
||||
|
||||
Príkazy sa spúšťajú z koreňa balíka s aktivovaným Python prostredím. Príprava dát používa CPU; na Perunovi sa vykonáva na výpočtovom uzle s pridelenými CPU.
|
||||
|
||||
```bash
|
||||
python scripts/prepare_data.py --output-dir data/aya_slovak_4096
|
||||
```
|
||||
|
||||
Odoslanie prípravy dát ako CPU úlohy:
|
||||
|
||||
```bash
|
||||
sbatch --job-name=aya_prepare \
|
||||
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
|
||||
--partition=cpu_short --nodes=1 --ntasks=1 \
|
||||
--cpus-per-task=8 --mem=64G --time=12:00:00 \
|
||||
--output=%x_%j.log \
|
||||
--wrap='python -u scripts/prepare_data.py --output-dir data/aya_slovak_4096'
|
||||
```
|
||||
|
||||
Tréningové úlohy sa odosielajú po úspešnom dokončení prípravy dát.
|
||||
|
||||
Výstupný priečinok prípravy dát musí byť nový alebo prázdny.
|
||||
|
||||
### `prepare_data.py`
|
||||
|
||||
Jeho vstupom sú slovenské zdrojové dáta Aya Collection a obe modelové konfigurácie. Používa CPU a tokenizéry Mistralu a OLMo. Výsledkom je jeden spoločný dátový balík pre oba modely.
|
||||
|
||||
1. Načíta slovenský split `train` datasetu `CohereLabs/aya_collection_language_split` pri pevnej revízii. Zdroj číta postupne.
|
||||
2. Vyradí neplatné riadky, prázdne zadania a prázdne odpovede. Polia `inputs` a `targets` prevedie na `instruction/input/output` a odstráni okrajové medzery.
|
||||
3. Pomocou oboch tokenizérov zmeria celú SFT sekvenciu vrátane formátu Alpaca a odpovede. Ak príklad prekročí `cutoff_len` pri ktoromkoľvek modeli, vyradí celý pár.
|
||||
4. Odstráni duplicitné normalizované páry zadania a odpovede. Rovnaké normalizované zadanie s rôznymi odpoveďami ponechá ako jednu skupinu.
|
||||
5. Podľa seedu, predvolene `42`, vytvorí train/eval/test. Celá skupina zadania zostane v jednej množine. Do eval a test vyberie najviac 1 000 príkladov na každú množinu; ostatné ponechané príklady patria do train.
|
||||
6. Uloží dáta vo formáte, ktorý následne načíta LLaMA-Factory.
|
||||
|
||||
Príklad uloženého riadka:
|
||||
|
||||
```json
|
||||
{"instruction":"Aké je hlavné mesto Slovenska?","input":"","output":"Bratislava."}
|
||||
```
|
||||
|
||||
Výstupy v `data/aya_slovak_4096`:
|
||||
|
||||
| Súbor | Obsah a použitie |
|
||||
| --- | --- |
|
||||
| `train.jsonl` | Príklady na učenie LoRA parametrov. |
|
||||
| `eval.jsonl` | Validačné príklady na sledovanie loss a ladenie. |
|
||||
| `test.jsonl` | Odložené príklady na záverečné vyhodnotenie. |
|
||||
| `dataset_info.json` | Registrácia súborov pod názvami `aya_train`, `aya_eval`, `aya_test` a mapovanie ich stĺpcov pre LLaMA-Factory. |
|
||||
| `manifest.json` | Zdrojová revízia, seed, skutočné počty, nastavenia dĺžkového filtra a kontrolné súčty súborov. |
|
||||
| `*.metadata.jsonl` | Pôvod a identifikácia príkladov v rovnakom poradí ako v príslušnom dátovom súbore. |
|
||||
| `excluded_long.jsonl` | Identifikácia a namerané dĺžky vyradených dlhých príkladov. |
|
||||
|
||||
Na deduplikáciu a rozdelenie dát používa dočasnú SQLite databázu. Po úspešnom dokončení ju odstráni; prepínač `--keep-db` ju ponechá. Počty eval a test príkladov možno nastaviť cez `--eval-size` a `--test-size`, seed cez `--seed`. Vlastný vstupný JSONL možno načítať cez `--local-jsonl`.
|
||||
|
||||
Oba modely potom používajú tie isté výsledné súbory. Každý model si ich pri tréningu tokenizuje svojím tokenizérom. LLaMA-Factory používa pripravené `aya_train` a `aya_eval` bez ďalšieho automatického delenia cez `val_size`.
|
||||
|
||||
## 2. Tréning
|
||||
|
||||
### `train.py`
|
||||
|
||||
Je to spúšťač tréningu v LLaMA-Factory. Ako vstup dostane modelový YAML a už pripravený dátový balík.
|
||||
|
||||
1. Načíta model, revíziu, cesty k dátam, LoRA parametre, learning rate, batch a ostatné nastavenia z YAML.
|
||||
2. Skontroluje existenciu dát, ich kontrolné súčty a zhodu dĺžkového filtra s použitým modelom, revíziou, šablónou a limitom.
|
||||
3. Uloží presnú použitú konfiguráciu a záznam behu.
|
||||
4. Cez `torchrun` spustí štyri procesy DDP a v každom zavolá tréning LLaMA-Factory.
|
||||
|
||||
Pri aktuálnych konfiguráciách prebieha SFT s LoRA, Unslothom a BF16. Základné váhy modelu sú zmrazené a učia sa LoRA parametre. Loss sa počíta z odpovedí. LLaMA-Factory každých 1 000 optimizer krokov vykoná validáciu a uloží checkpoint; na konci načíta checkpoint s najnižším eval loss.
|
||||
|
||||
Kontrola pomocou `--dry-run` sa vykonáva po príprave dát. Overí základné nastavenia a dátové súbory bez načítania váh a bez spustenia tréningu.
|
||||
|
||||
Kontrola konfigurácie a pripravených dát bez spustenia tréningu:
|
||||
|
||||
```bash
|
||||
python scripts/train.py configs/mistral.yaml --dry-run
|
||||
python scripts/train.py configs/olmo.yaml --dry-run
|
||||
```
|
||||
|
||||
Každý tréning sa spúšťa samostatne v prostredí so štyrmi dostupnými GPU:
|
||||
|
||||
```bash
|
||||
python scripts/train.py configs/mistral.yaml
|
||||
python scripts/train.py configs/olmo.yaml
|
||||
```
|
||||
|
||||
Odoslanie tréningov cez Slurm na štyri H200. Nasledujúci príklad spustí modely postupne; `afterok` umožní spustenie OLMo po úspešnom dokončení Mistralu:
|
||||
|
||||
```bash
|
||||
AYA_MISTRAL_JOB=$(sbatch --parsable --job-name=aya_mistral \
|
||||
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
|
||||
--partition=gpu_long --nodes=1 --ntasks=1 --gres=gpu:4 \
|
||||
--cpus-per-task=32 --mem=256G --time=2-00:00:00 \
|
||||
--output=%x_%j.log \
|
||||
--wrap='python -u scripts/train.py configs/mistral.yaml')
|
||||
|
||||
sbatch --job-name=aya_olmo \
|
||||
--dependency="afterok:$AYA_MISTRAL_JOB" \
|
||||
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
|
||||
--partition=gpu_long --nodes=1 --ntasks=1 --gres=gpu:4 \
|
||||
--cpus-per-task=32 --mem=256G --time=2-00:00:00 \
|
||||
--output=%x_%j.log \
|
||||
--wrap='python -u scripts/train.py configs/olmo.yaml'
|
||||
```
|
||||
|
||||
Jedna úloha Slurmu spustí `train.py`, ktorý vytvorí štyri procesy DDP. Parametre `--run-dir` a `--resume` sa pri odosielaní uvádzajú v Python príkaze vo vnútri `--wrap`.
|
||||
|
||||
Každý proces DDP má na pridelenej GPU kópiu modelu a spracúva inú časť dávky. Gradienty sa synchronizujú medzi GPU, takže štyri GPU vykonávajú jeden spoločný tréning. Pri batchi 8 na GPU a akumulácii 1 je efektívny batch **4 × 8 × 1 = 32**. Každý proces importuje Unsloth pred Transformers a LLaMA-Factory.
|
||||
|
||||
Výsledkom tréningu je **LoRA adaptér**, ktorý sa používa spolu so základným modelom. Pre Mistral sa uloží do `runs/mistral/adapter`, pre OLMo do `runs/olmo/adapter`. Checkpointy sú v podpriečinkoch adaptéra.
|
||||
|
||||
V nadradenom priečinku behu zostanú:
|
||||
|
||||
- `train.log` – priebeh a hlásenia tréningu;
|
||||
- `used_config.yaml` – presné nastavenia odovzdané LLaMA-Factory;
|
||||
- `run.json` – záznam behu, efektívny batch a identifikácia použitých dát.
|
||||
|
||||
|
||||
## 3. Vyhodnotenie na eval dátach
|
||||
|
||||
### `evaluate.py`
|
||||
|
||||
Ako vstup dostane rovnaký modelový YAML, pripravené dáta a dotrénovaný adaptér. Načíta základný model spolu s adaptérom a vykoná dve samostatné fázy na jednej GPU:
|
||||
|
||||
1. **Eval loss a perplexita:** model spracuje zadania s pôvodnými referenčnými odpoveďami. Vypočíta sa strata pri predpovedaní odpovedí a z nej perplexita.
|
||||
2. **Generovanie a porovnanie odpovedí:** model dostane zadania a vytvorí vlastné odpovede. Skript ich porovná s úplnými referenciami pomocou slovného F1 a normalizovanej presnej zhody.
|
||||
|
||||
Predvolene sa vyhodnotí pripravená množina `eval.jsonl`:
|
||||
|
||||
```bash
|
||||
python scripts/evaluate.py configs/mistral.yaml --output-dir results/mistral_eval
|
||||
|
||||
python scripts/evaluate.py configs/olmo.yaml --output-dir results/olmo_eval
|
||||
```
|
||||
|
||||
Po úspešnom dokončení príslušného tréningu sa vyhodnotenie odošle ako úloha s jednou GPU:
|
||||
|
||||
```bash
|
||||
sbatch --job-name=aya_mistral_eval \
|
||||
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
|
||||
--partition=gpu_short --nodes=1 --ntasks=1 --gres=gpu:1 \
|
||||
--cpus-per-task=8 --mem=64G --time=12:00:00 \
|
||||
--output=%x_%j.log \
|
||||
--wrap='python -u scripts/evaluate.py configs/mistral.yaml --output-dir results/mistral_eval'
|
||||
|
||||
sbatch --job-name=aya_olmo_eval \
|
||||
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
|
||||
--partition=gpu_short --nodes=1 --ntasks=1 --gres=gpu:1 \
|
||||
--cpus-per-task=8 --mem=64G --time=12:00:00 \
|
||||
--output=%x_%j.log \
|
||||
--wrap='python -u scripts/evaluate.py configs/olmo.yaml --output-dir results/olmo_eval'
|
||||
```
|
||||
|
||||
Pre vyhodnotenie test množiny sa do Python príkazu vo vnútri `--wrap` pridá `--split test` a zvolí sa nový výstupný priečinok, napríklad `results/mistral_test`.
|
||||
|
||||
V priečinku behu sa uchovávajú pôvodné súbory `run.json` a `used_config.yaml`. Vyhodnotenie porovná manifest pripravených dát so záznamom z tréningu; iné rozdelenie alebo iný dataset odmietne. Rovnaká kontrola platí pre eval aj test.
|
||||
|
||||
Adaptér z iného behu sa určí parametrom `--adapter runs/mistral_dalsi/adapter`. Predvolene sa použije adaptér z `output_dir` v modelovom YAML. Prepínač `--dry-run` overí pripravené vstupy a zobrazí odvodené evaluačné nastavenia.
|
||||
|
||||
Generovanie je greedy, s jedným lúčom a limitom najviac 256 nových tokenov. Limit možno zmeniť cez `--max-new-tokens`. Pri každom vstupe sa počet nových tokenov obmedzí aj zostávajúcim miestom do 4 096 tokenov. Celý vstup a úplná referencia zostávajú zachované; prípadné zníženie limitu sa zapíše do logu.
|
||||
|
||||
Skript zapisuje výsledky do priečinka z `--output-dir`:
|
||||
|
||||
| Súbor | Obsah súboru |
|
||||
| --- | --- |
|
||||
| `summary.json` | Hlavný súhrn: eval loss, perplexita, F1, presná zhoda a počet vyhodnotených príkladov. |
|
||||
| `loss/eval_results.json` | Výsledky samostatného výpočtu eval loss v LLaMA-Factory. |
|
||||
| `generation/generated_predictions.jsonl` | Vygenerované odpovede modelu. |
|
||||
| `scores/per_sample.jsonl` | Predikcia, úplná referencia a skóre pre každý príklad. |
|
||||
| `scores/metrics.json` | Podrobné súhrny metrík aj rozdelenie podľa typu úlohy. |
|
||||
| `evaluation.json` | Zvolený split, modelový config, adaptér a stav vyhodnotenia. |
|
||||
|
||||
Pri výsledkoch zostanú aj logy a konkrétne použité konfigurácie oboch evaluačných fáz. Pre vyhodnotenie možno ako vstupný config použiť aj `runs/mistral/used_config.yaml` alebo OLMo ekvivalent.
|
||||
|
||||
|
||||
Eval slúži na ladenie a výber checkpointu. Po ukončení výberu nastavení sa vyhodnotí odložená testovacia množina a výsledky sa uložia do nového výstupného priečinka:
|
||||
|
||||
```bash
|
||||
python scripts/evaluate.py configs/mistral.yaml --split test --output-dir results/mistral_test
|
||||
python scripts/evaluate.py configs/olmo.yaml --split test --output-dir results/olmo_test
|
||||
```
|
||||
|
||||
Externé lm-eval a SK-bench majú vlastné spúšťanie. Tento balík obsahuje prípravu dát, tréning a vyhodnotenie na pripravených eval/test dátach.
|
||||
|
||||
### Sledovanie odoslaných úloh
|
||||
|
||||
```bash
|
||||
squeue -u "$USER"
|
||||
```
|
||||
|
||||
Výstup úlohy sa zapisuje do logu v pracovnom priečinku. V názve `%x_%j.log` sa `%x` nahradí názvom úlohy a `%j` jej číslom. Pre prípravu dát s číslom úlohy 12345:
|
||||
|
||||
```bash
|
||||
tail -f aya_prepare_12345.log
|
||||
```
|
||||
59
perun/configs/mistral.yaml
Normal file
59
perun/configs/mistral.yaml
Normal file
@ -0,0 +1,59 @@
|
||||
# Jeden trening tohto modelu: 4 H200, DDP, Unsloth a bf16.
|
||||
model_name_or_path: "slovak-nlp/mistral-sk-7b"
|
||||
model_revision: "089497ae0d72018e9591895c54f774bf0b4a83a4"
|
||||
stage: sft
|
||||
do_train: true
|
||||
do_eval: true
|
||||
finetuning_type: lora
|
||||
template: alpaca
|
||||
use_unsloth: true
|
||||
use_unsloth_gc: true
|
||||
bf16: true
|
||||
infer_dtype: bfloat16
|
||||
flash_attn: sdpa
|
||||
|
||||
dataset_dir: data/aya_slovak_4096
|
||||
dataset: aya_train
|
||||
eval_dataset: aya_eval
|
||||
cutoff_len: 4096
|
||||
packing: false
|
||||
train_on_prompt: false
|
||||
preprocessing_num_workers: 8
|
||||
dataloader_num_workers: 2
|
||||
overwrite_cache: false
|
||||
|
||||
lora_rank: 16
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
|
||||
|
||||
per_device_train_batch_size: 8
|
||||
per_device_eval_batch_size: 8
|
||||
gradient_accumulation_steps: 1
|
||||
learning_rate: 0.0001
|
||||
num_train_epochs: 1.0
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.08
|
||||
optim: adamw_torch
|
||||
weight_decay: 0.0
|
||||
max_grad_norm: 0.5
|
||||
gradient_checkpointing: true
|
||||
ddp_find_unused_parameters: false
|
||||
|
||||
output_dir: runs/mistral/adapter
|
||||
overwrite_output_dir: false
|
||||
logging_steps: 10
|
||||
eval_strategy: steps
|
||||
eval_steps: 1000
|
||||
save_strategy: steps
|
||||
save_steps: 1000
|
||||
save_total_limit: 2
|
||||
save_only_model: false
|
||||
load_best_model_at_end: true
|
||||
metric_for_best_model: loss
|
||||
greater_is_better: false
|
||||
predict_with_generate: false
|
||||
plot_loss: true
|
||||
report_to: none
|
||||
seed: 42
|
||||
data_seed: 42
|
||||
59
perun/configs/olmo.yaml
Normal file
59
perun/configs/olmo.yaml
Normal file
@ -0,0 +1,59 @@
|
||||
# Jeden trening tohto modelu: 4 H200, DDP, Unsloth a bf16.
|
||||
model_name_or_path: "allenai/OLMo-2-1124-7B"
|
||||
model_revision: "7df9a82518afdecae4e8c026b27adccc8c1f0032"
|
||||
stage: sft
|
||||
do_train: true
|
||||
do_eval: true
|
||||
finetuning_type: lora
|
||||
template: alpaca
|
||||
use_unsloth: true
|
||||
use_unsloth_gc: true
|
||||
bf16: true
|
||||
infer_dtype: bfloat16
|
||||
flash_attn: sdpa
|
||||
|
||||
dataset_dir: data/aya_slovak_4096
|
||||
dataset: aya_train
|
||||
eval_dataset: aya_eval
|
||||
cutoff_len: 4096
|
||||
packing: false
|
||||
train_on_prompt: false
|
||||
preprocessing_num_workers: 8
|
||||
dataloader_num_workers: 2
|
||||
overwrite_cache: false
|
||||
|
||||
lora_rank: 16
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
|
||||
|
||||
per_device_train_batch_size: 8
|
||||
per_device_eval_batch_size: 8
|
||||
gradient_accumulation_steps: 1
|
||||
learning_rate: 0.0001
|
||||
num_train_epochs: 1.0
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.08
|
||||
optim: adamw_torch
|
||||
weight_decay: 0.0
|
||||
max_grad_norm: 0.5
|
||||
gradient_checkpointing: true
|
||||
ddp_find_unused_parameters: false
|
||||
|
||||
output_dir: runs/olmo/adapter
|
||||
overwrite_output_dir: false
|
||||
logging_steps: 10
|
||||
eval_strategy: steps
|
||||
eval_steps: 1000
|
||||
save_strategy: steps
|
||||
save_steps: 1000
|
||||
save_total_limit: 2
|
||||
save_only_model: false
|
||||
load_best_model_at_end: true
|
||||
metric_for_best_model: loss
|
||||
greater_is_better: false
|
||||
predict_with_generate: false
|
||||
plot_loss: true
|
||||
report_to: none
|
||||
seed: 42
|
||||
data_seed: 42
|
||||
131
perun/requirements/lm-eval.txt
Normal file
131
perun/requirements/lm-eval.txt
Normal file
@ -0,0 +1,131 @@
|
||||
# Python 3.11.16; CUDA 12.4.
|
||||
absl-py==2.5.0
|
||||
accelerate==1.15.0
|
||||
aiohappyeyeballs==2.7.1
|
||||
aiohttp==3.14.3
|
||||
aiosignal==1.4.0
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
attrs==26.1.0
|
||||
certifi==2026.7.22
|
||||
chardet==6.0.0.post1
|
||||
charset-normalizer==3.5.2
|
||||
click==8.5.0
|
||||
cloudpickle==3.1.2
|
||||
colorama==0.4.6
|
||||
contourpy==1.3.3
|
||||
cycler==0.12.1
|
||||
DataProperty==1.1.1
|
||||
datasets==5.0.1
|
||||
defusedxml==0.7.1
|
||||
dill==0.4.1
|
||||
evaluate==0.4.6
|
||||
filelock==3.32.3
|
||||
fonttools==4.66.1
|
||||
frozenlist==1.8.0
|
||||
fsspec==2026.6.0
|
||||
googleapis-common-protos==1.75.5
|
||||
h11==0.16.0
|
||||
hf-xet==1.6.0
|
||||
httpcore==1.0.9
|
||||
httpx==0.28.1
|
||||
huggingface_hub==0.36.2
|
||||
idna==3.20
|
||||
Jinja2==3.1.6
|
||||
joblib==1.6.0
|
||||
jsonlines==4.0.0
|
||||
kiwisolver==1.5.1
|
||||
lm_eval @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@6d2abda4fd171e68a8789330c4149e37c1ca0bda
|
||||
lxml==6.1.3
|
||||
markdown-it-py==4.2.0
|
||||
MarkupSafe==3.0.3
|
||||
matplotlib==3.11.2
|
||||
mbstrdecoder==1.1.5
|
||||
mdurl==0.1.2
|
||||
more-itertools==11.1.0
|
||||
mpmath==1.3.0
|
||||
multidict==6.9.1
|
||||
multiprocess==0.70.19
|
||||
narwhals==2.26.0
|
||||
networkx==3.6.1
|
||||
nltk==3.10.3
|
||||
numexpr==2.14.2
|
||||
numpy==2.4.6
|
||||
nvidia-cublas-cu12==12.4.5.8
|
||||
nvidia-cuda-cupti-cu12==12.4.127
|
||||
nvidia-cuda-nvrtc-cu12==12.4.127
|
||||
nvidia-cuda-runtime-cu12==12.4.127
|
||||
nvidia-cudnn-cu12==9.1.0.70
|
||||
nvidia-cufft-cu12==11.2.1.3
|
||||
nvidia-curand-cu12==10.3.5.147
|
||||
nvidia-cusolver-cu12==11.6.1.9
|
||||
nvidia-cusparse-cu12==12.3.1.170
|
||||
nvidia-cusparselt-cu12==0.6.2
|
||||
nvidia-nccl-cu12==2.21.5
|
||||
nvidia-nvjitlink-cu12==12.4.127
|
||||
nvidia-nvtx-cu12==12.4.127
|
||||
opentelemetry-api==1.45.0
|
||||
opentelemetry-exporter-http-transport==0.66b0
|
||||
opentelemetry-exporter-otlp-common==0.66b0
|
||||
opentelemetry-exporter-otlp-proto-common==1.45.0
|
||||
opentelemetry-exporter-otlp-proto-http==1.45.0
|
||||
opentelemetry-proto==1.45.0
|
||||
opentelemetry-sdk==1.45.0
|
||||
opentelemetry-semantic-conventions==0.66b0
|
||||
packaging==26.3
|
||||
pandas==3.0.6
|
||||
pathvalidate==3.3.1
|
||||
peft==0.14.0
|
||||
pillow==12.3.0
|
||||
pip==26.2.1
|
||||
platformdirs==4.12.2
|
||||
portalocker==4.4.0
|
||||
propcache==0.5.4
|
||||
protobuf==7.36.2
|
||||
psutil==7.2.2
|
||||
pyarrow==25.0.1
|
||||
pybind11==3.1.0
|
||||
pydantic==2.13.5
|
||||
pydantic_core==2.46.5
|
||||
Pygments==2.21.0
|
||||
pyparsing==3.3.3
|
||||
pytablewriter==1.2.1
|
||||
python-dateutil==2.9.0.post0
|
||||
pytz==2026.4
|
||||
PyYAML==6.0.3
|
||||
regex==2026.9.29
|
||||
requests==2.34.2
|
||||
rich==15.0.0
|
||||
rouge_score==0.1.2
|
||||
sacrebleu==2.6.0
|
||||
safetensors==0.8.0
|
||||
scikit-learn==1.9.1
|
||||
scipy==1.17.1
|
||||
sentencepiece==0.2.2
|
||||
setuptools==84.0.0
|
||||
shellingham==1.5.4
|
||||
six==1.17.0
|
||||
sqlitedict==2.1.0
|
||||
sympy==1.13.1
|
||||
tabledata==1.3.5
|
||||
tabulate==0.10.0
|
||||
tcolorpy==0.1.7
|
||||
threadpoolctl==3.7.0
|
||||
tokenizers==0.21.4
|
||||
torch==2.6.0+cu124
|
||||
tqdm==4.70.1
|
||||
tqdm-multiprocess==0.0.11
|
||||
transformers==4.48.2
|
||||
triton==3.2.0
|
||||
typepy==1.3.5
|
||||
typer==0.27.2
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
urllib3==2.8.0
|
||||
wandb==0.30.0
|
||||
wheel==0.48.0
|
||||
word2number==1.1
|
||||
xxhash==4.0.1
|
||||
yarl==1.25.1
|
||||
zstandard==0.25.0
|
||||
309
perun/requirements/skbench.txt
Normal file
309
perun/requirements/skbench.txt
Normal file
@ -0,0 +1,309 @@
|
||||
# Python 3.13.17;
|
||||
absl-py==2.4.0
|
||||
accelerate==1.14.0
|
||||
aenum==3.1.15
|
||||
aioboto3==15.5.0
|
||||
aiobotocore==2.25.1
|
||||
aiofiles==25.1.0
|
||||
aiohappyeyeballs==2.7.1
|
||||
aiohttp==3.14.1
|
||||
aioitertools==0.13.0
|
||||
aiosignal==1.4.0
|
||||
annotated-doc==0.0.4
|
||||
annotated-types==0.7.0
|
||||
anthropic==0.116.0
|
||||
antlr4-python3-runtime==4.13.2
|
||||
anyio==4.14.1
|
||||
apache-tvm-ffi==0.1.11
|
||||
astor==0.8.1
|
||||
attrs==26.1.0
|
||||
beautifulsoup4==4.15.0
|
||||
blake3==1.0.9
|
||||
blis==1.3.3
|
||||
boto3==1.40.61
|
||||
botocore==1.40.61
|
||||
cachetools==7.1.4
|
||||
catalogue==2.0.10
|
||||
cbor2==6.1.2
|
||||
certifi==2026.6.17
|
||||
cffi==2.0.0
|
||||
cfgv==3.5.0
|
||||
chardet==6.0.0.post1
|
||||
charset-normalizer==3.4.7
|
||||
click==8.4.2
|
||||
cloudpathlib==0.24.0
|
||||
cloudpickle==3.1.2
|
||||
colorama==0.4.6
|
||||
colorlog==6.10.1
|
||||
compressed-tensors==0.17.0
|
||||
confection==1.3.3
|
||||
cryptography==49.0.0
|
||||
cuda-bindings==13.3.1
|
||||
cuda-core==1.0.1
|
||||
cuda-pathfinder==1.5.6
|
||||
cuda-python==13.3.1
|
||||
cuda-tile==1.5.0
|
||||
cuda-toolkit==13.0.3.0
|
||||
cymem==2.0.13
|
||||
dataproperty==1.1.1
|
||||
datasets==5.0.0
|
||||
debugpy==1.8.21
|
||||
depyf==0.20.0
|
||||
detect-installer==0.1.0
|
||||
dill==0.4.1
|
||||
diskcache==5.6.3
|
||||
distlib==0.4.3
|
||||
distro==1.9.0
|
||||
dnspython==2.8.0
|
||||
docstring-parser==0.18.0
|
||||
einops==0.8.2
|
||||
email-validator==2.3.0
|
||||
emoji==2.15.0
|
||||
fastapi==0.136.3
|
||||
fastapi-cli==0.0.27
|
||||
fastapi-cloud-cli==0.22.1
|
||||
fastar==0.11.0
|
||||
fastsafetensors==0.3.3
|
||||
fastuuid==0.14.0
|
||||
filelock==3.29.5
|
||||
flashinfer-python==0.6.16.post3
|
||||
frozendict==2.4.7
|
||||
frozenlist==1.8.0
|
||||
fsspec==2025.9.0
|
||||
gitdb==4.0.12
|
||||
gitpython==3.1.50
|
||||
googleapis-common-protos==1.75.0
|
||||
grpcio==1.81.1
|
||||
h11==0.16.0
|
||||
hf==1.29.0
|
||||
hf-xet==1.6.0
|
||||
httpcore==1.0.9
|
||||
httptools==0.8.0
|
||||
httpx==0.28.1
|
||||
httpx-sse==0.4.3
|
||||
huggingface-hub==1.29.0
|
||||
humming-kernels==0.1.12
|
||||
identify==2.6.19
|
||||
idna==3.18
|
||||
ijson==3.5.0
|
||||
immutabledict==4.3.1
|
||||
importlib-metadata==8.9.0
|
||||
inspect-ai==0.3.145
|
||||
interegular==0.3.3
|
||||
jieba==0.42.1
|
||||
jinja2==3.1.6
|
||||
jiter==0.16.0
|
||||
jmespath==1.1.0
|
||||
joblib==1.5.3
|
||||
jsonlines==4.0.0
|
||||
jsonpatch==1.33
|
||||
jsonpath-ng==1.8.0
|
||||
jsonpointer==3.1.1
|
||||
jsonref==1.1.0
|
||||
jsonschema==4.26.0
|
||||
jsonschema-specifications==2025.9.1
|
||||
langcodes==3.5.1
|
||||
langdetect==1.0.9
|
||||
lark==1.2.2
|
||||
latex2sympy2-extended==1.0.6
|
||||
lighteval @ git+https://github.com/huggingface/lighteval.git@10b9104eb16778b174e9e240140974a039018981
|
||||
linkify-it-py==2.1.0
|
||||
litellm==1.90.2
|
||||
llguidance==1.7.6
|
||||
llvmlite==0.47.0
|
||||
lm-format-enforcer==0.11.3
|
||||
loguru==0.7.3
|
||||
lxml==6.1.1
|
||||
markdown-it-py==4.2.0
|
||||
markupsafe==3.0.3
|
||||
mbstrdecoder==1.1.5
|
||||
mcp==1.28.1
|
||||
mdit-py-plugins==0.6.1
|
||||
mdurl==0.1.2
|
||||
mistral-common==1.11.7
|
||||
ml-dtypes==0.5.4
|
||||
mmh3==5.2.1
|
||||
model-hosting-container-standards==0.1.16
|
||||
more-itertools==11.1.0
|
||||
mpmath==1.3.0
|
||||
msgpack==1.2.1
|
||||
msgspec==0.21.1
|
||||
multidict==6.7.1
|
||||
multiprocess==0.70.19
|
||||
murmurhash==1.0.15
|
||||
narwhals==2.23.0
|
||||
natto-py==1.0.1
|
||||
nccl4py==0.4.1
|
||||
nest-asyncio==1.6.0
|
||||
networkx==3.6.1
|
||||
ninja==1.13.0
|
||||
nltk==3.9.1
|
||||
nodeenv==1.10.0
|
||||
numba==0.65.0
|
||||
numpy==2.4.6
|
||||
nvidia-cublas==13.1.1.3
|
||||
nvidia-cuda-cccl==13.3.3.4.1
|
||||
nvidia-cuda-crt==13.3.73
|
||||
nvidia-cuda-cupti==13.0.85
|
||||
nvidia-cuda-nvcc==13.2.78
|
||||
nvidia-cuda-nvdisasm==13.3.73
|
||||
nvidia-cuda-nvrtc==13.0.88
|
||||
nvidia-cuda-runtime==13.0.96
|
||||
nvidia-cudnn-cu13==9.20.0.48
|
||||
nvidia-cudnn-frontend==1.25.0
|
||||
nvidia-cufft==12.0.0.61
|
||||
nvidia-cufile==1.15.1.6
|
||||
nvidia-curand==10.4.0.35
|
||||
nvidia-cusolver==12.0.4.66
|
||||
nvidia-cusparse==12.6.3.3
|
||||
nvidia-cusparselt-cu13==0.8.1
|
||||
nvidia-cutlass-dsl==4.6.2
|
||||
nvidia-cutlass-dsl-libs-base==4.6.2
|
||||
nvidia-cutlass-dsl-libs-core==4.6.2
|
||||
nvidia-cutlass-dsl-libs-cu12==4.6.2
|
||||
nvidia-cutlass-dsl-libs-cu13==4.6.2
|
||||
nvidia-ml-py==13.610.43
|
||||
nvidia-nccl-cu13==2.29.7
|
||||
nvidia-nvjitlink==13.3.33
|
||||
nvidia-nvshmem-cu13==3.4.5
|
||||
nvidia-nvtx==13.0.85
|
||||
nvidia-nvvm==13.2.78
|
||||
nvtx==0.2.15
|
||||
openai==2.44.0
|
||||
openai-harmony==0.0.8
|
||||
opencv-python-headless==5.0.0.93
|
||||
opentelemetry-api==1.43.0
|
||||
opentelemetry-exporter-otlp==1.43.0
|
||||
opentelemetry-exporter-otlp-proto-common==1.43.0
|
||||
opentelemetry-exporter-otlp-proto-grpc==1.43.0
|
||||
opentelemetry-exporter-otlp-proto-http==1.43.0
|
||||
opentelemetry-proto==1.43.0
|
||||
opentelemetry-sdk==1.43.0
|
||||
opentelemetry-semantic-conventions==0.64b0
|
||||
opentelemetry-semantic-conventions-ai==0.5.1
|
||||
outlines-core==0.2.14
|
||||
packaging==26.2
|
||||
pandas==3.0.3
|
||||
partial-json-parser==0.2.1.1.post7
|
||||
pathlib-abc==0.5.2
|
||||
pathvalidate==3.3.1
|
||||
pillow==12.3.0
|
||||
pip==26.1.2
|
||||
platformdirs==4.10.0
|
||||
portalocker==3.2.0
|
||||
pre-commit==4.6.0
|
||||
preshed==3.0.13
|
||||
prometheus-client==0.25.0
|
||||
prometheus-fastapi-instrumentator==8.0.2
|
||||
propcache==0.5.2
|
||||
protobuf==6.33.6
|
||||
psutil==7.2.2
|
||||
py-cpuinfo==9.0.0
|
||||
pyarrow==24.0.0
|
||||
pybase64==1.4.3
|
||||
pycountry==26.2.16
|
||||
pycparser==3.0
|
||||
pydantic==2.13.4
|
||||
pydantic-core==2.46.4
|
||||
pydantic-extra-types==2.11.1
|
||||
pydantic-settings==2.14.2
|
||||
pygments==2.20.0
|
||||
pyjwt==2.13.0
|
||||
pynvvideocodec==2.0.4
|
||||
pytablewriter==1.2.1
|
||||
pythainlp==5.3.4
|
||||
python-crfsuite==0.9.12
|
||||
python-dateutil==2.9.0.post0
|
||||
python-discovery==1.4.2
|
||||
python-dotenv==1.2.2
|
||||
python-json-logger==4.1.0
|
||||
python-multipart==0.0.32
|
||||
pytz==2026.2
|
||||
pyvi==0.1.1
|
||||
pyyaml==6.0.3
|
||||
pyzmq==27.1.0
|
||||
quack-kernels==0.6.4
|
||||
ray==2.56.0
|
||||
referencing==0.37.0
|
||||
regex==2026.6.28
|
||||
requests==2.34.2
|
||||
rich==15.0.0
|
||||
rich-toolkit==0.20.1
|
||||
rignore==0.7.6
|
||||
rouge-score==0.1.2
|
||||
rpds-py==2026.6.3
|
||||
s3fs==2025.9.0
|
||||
s3transfer==0.14.0
|
||||
sacrebleu==2.6.0
|
||||
safetensors==0.8.0
|
||||
scikit-learn==1.9.0
|
||||
scipy==1.18.0
|
||||
semver==3.0.4
|
||||
sentencepiece==0.2.1
|
||||
sentry-sdk==2.64.0
|
||||
setproctitle==1.3.7
|
||||
setuptools==80.10.2
|
||||
shellingham==1.5.4
|
||||
shortuuid==1.0.13
|
||||
six==1.17.0
|
||||
sklearn-crfsuite==0.5.0
|
||||
smart-open==8.0.0
|
||||
smmap==5.0.3
|
||||
sniffio==1.3.1
|
||||
soupsieve==2.8.4
|
||||
spacy==3.8.14
|
||||
spacy-legacy==3.0.12
|
||||
spacy-loggers==1.0.5
|
||||
srsly==2.5.3
|
||||
sse-starlette==3.4.5
|
||||
stanza==1.13.0
|
||||
starlette==1.3.1
|
||||
sudachidict-core==20260428
|
||||
sudachipy==0.6.11
|
||||
supervisor==4.3.0
|
||||
sympy==1.14.0
|
||||
tabledata==1.3.5
|
||||
tabulate==0.10.0
|
||||
tcolorpy==0.1.7
|
||||
tenacity==9.1.4
|
||||
termcolor==2.3.0
|
||||
textual==8.2.8
|
||||
thinc==8.3.13
|
||||
threadpoolctl==3.6.0
|
||||
tiktoken==0.13.0
|
||||
tilelang==0.1.12
|
||||
tokenizers==0.22.2
|
||||
tokenspeed-mla==0.1.8
|
||||
tokenspeed-triton==3.7.10.post20260531
|
||||
torch==2.13.0
|
||||
torch-c-dlpack-ext==0.1.5
|
||||
torchaudio==2.11.0
|
||||
torchcodec==0.16.0
|
||||
torchvision==0.28.0
|
||||
tqdm==4.68.3
|
||||
transformers==5.12.1
|
||||
triton==3.7.1
|
||||
typepy==1.3.5
|
||||
typer==0.26.8
|
||||
typing-extensions==4.16.0
|
||||
typing-inspection==0.4.2
|
||||
uc-micro-py==2.0.0
|
||||
udapi==0.5.2
|
||||
udtools==0.2.8
|
||||
universal-pathlib==0.3.10
|
||||
urllib3==2.7.0
|
||||
uvicorn==0.49.0
|
||||
uvloop==0.22.1
|
||||
virtualenv==21.5.1
|
||||
vllm==0.28.0
|
||||
wandb==0.28.0
|
||||
wasabi==1.1.3
|
||||
watchfiles==1.2.0
|
||||
weasel==1.0.0
|
||||
websockets==16.0
|
||||
wrapt==1.17.3
|
||||
xgrammar==0.2.3
|
||||
xxhash==3.8.0
|
||||
yarl==1.24.2
|
||||
z3-solver==4.15.4.0
|
||||
zipp==4.1.0
|
||||
103
perun/requirements/skbench/pyproject.toml
Normal file
103
perun/requirements/skbench/pyproject.toml
Normal file
@ -0,0 +1,103 @@
|
||||
[project]
|
||||
name = "sk-bench"
|
||||
version = "0.1.0"
|
||||
description = "Elevating Slovak LLM evaluation through the power of lighteval."
|
||||
license = "Apache-2.0"
|
||||
license-files = ["LICENSE"]
|
||||
requires-python = ">=3.13,<3.14"
|
||||
dependencies = [
|
||||
"lighteval[multilingual,vllm]>=0.13.0",
|
||||
"pip>=25.3",
|
||||
"transformers==5.12.1",
|
||||
"pyyaml>=6.0.3",
|
||||
"more-itertools>=10.8.0",
|
||||
"immutabledict>=4.3.1",
|
||||
"langdetect>=1.0.9",
|
||||
"litellm[caching]>=1.82.0",
|
||||
"wandb>=0.26.1",
|
||||
"hf>=1.1.0",
|
||||
"pre-commit>=4.5.1",
|
||||
"vllm==0.28.0",
|
||||
"torch==2.13.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
vllm = ["vllm==0.28.0", "torch==2.13.0"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
{include-group = "lint"},
|
||||
{include-group = "typing"},
|
||||
{include-group = "test"},
|
||||
{include-group = "notebooks"},
|
||||
]
|
||||
lint = [
|
||||
"ruff>=0.9.0",
|
||||
"pre-commit>=4.1.0",
|
||||
"typos>=1.29.0",
|
||||
]
|
||||
typing = [
|
||||
"mypy>=1.14.0",
|
||||
"pandas-stubs>=2.3.3.260113",
|
||||
]
|
||||
test = [
|
||||
"pytest>=8.0.0",
|
||||
]
|
||||
notebooks = [
|
||||
"pandas",
|
||||
"pyarrow",
|
||||
"jupyter",
|
||||
"matplotlib",
|
||||
]
|
||||
hpc = ["deep-gemm"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 119
|
||||
|
||||
[tool.ruff.lint]
|
||||
# Enable pycodestyle (`E`) and Pyflakes (`F`) codes by default.
|
||||
# Never enforce `E501` (line length violations).
|
||||
ignore = ["E501", "D100", "D101", "D102", "D103", "D104", "D415", "D105", "DOC501", "DOC201", "CPY001"]
|
||||
select = ["C", "E", "F", "I", "W", "CPY", "D417", "DOC"]
|
||||
preview = true
|
||||
|
||||
[tool.ruff.lint.pydocstyle]
|
||||
convention = "google"
|
||||
|
||||
[tool.ruff.lint.isort]
|
||||
lines-after-imports = 2
|
||||
known-first-party = ["lighteval"]
|
||||
|
||||
[tool.ruff.format]
|
||||
quote-style = "double" # Like Black, use double quotes for strings.
|
||||
indent-style = "space" # Like Black, indent with spaces, rather than tabs.
|
||||
skip-magic-trailing-comma = false # Like Black, respect magic trailing commas.
|
||||
line-ending = "auto" # Like Black, automatically detect the appropriate line ending.
|
||||
|
||||
[tool.uv]
|
||||
default-groups = []
|
||||
index-strategy = "unsafe-best-match"
|
||||
extra-build-dependencies = { deep-gemm = ["torch"] }
|
||||
|
||||
[tool.uv.sources]
|
||||
lighteval = { git = "https://github.com/huggingface/lighteval.git", rev = "10b9104eb16778b174e9e240140974a039018981" }
|
||||
deep-gemm = { git = "https://github.com/deepseek-ai/DeepGEMM.git", rev = "v2.1.1.post3" }
|
||||
# No default-groups: uv sync will only install core dependencies.
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = ["src.evaluation_tracker_patch", "src.pipeline_patch"]
|
||||
ignore_errors = true
|
||||
|
||||
[tool.typos.default.extend-words]
|
||||
Ony = "Ony"
|
||||
tak = "tak"
|
||||
biologia = "biologia"
|
||||
|
||||
[[tool.uv.dependency-metadata]]
|
||||
name = "deep-gemm"
|
||||
version = "2.1.1.post3"
|
||||
requires-dist = []
|
||||
|
||||
# vllm >= 0.28.0 ships CUDA 13 wheels on PyPI directly (no cu128 build
|
||||
# exists anymore); the GPU nodes' driver (580.159.04) supports CUDA 13.0,
|
||||
# so the default PyPI index is used for both vllm and torch.
|
||||
6274
perun/requirements/skbench/uv.lock
Normal file
6274
perun/requirements/skbench/uv.lock
Normal file
File diff suppressed because it is too large
Load Diff
148
perun/requirements/training.txt
Normal file
148
perun/requirements/training.txt
Normal file
@ -0,0 +1,148 @@
|
||||
# Python 3.11.16; CUDA 12.4.
|
||||
accelerate==1.11.0
|
||||
aiofiles==24.1.0
|
||||
aiohappyeyeballs==2.7.1
|
||||
aiohttp==3.14.3
|
||||
aiosignal==1.4.0
|
||||
alembic==1.20.0
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
antlr4-python3-runtime==4.9.3
|
||||
anyio==4.14.2
|
||||
attrs==26.1.0
|
||||
av==16.0.0
|
||||
bitsandbytes==0.50.2
|
||||
brotli==1.2.0
|
||||
certifi==2026.7.22
|
||||
charset-normalizer==3.5.1
|
||||
click==8.5.0
|
||||
colorlog==6.12.0
|
||||
contourpy==1.3.3
|
||||
cut-cross-entropy==25.1.1
|
||||
cycler==0.12.1
|
||||
datasets==3.6.0
|
||||
diffusers==0.40.0
|
||||
dill==0.3.8
|
||||
docstring_parser==0.18.0
|
||||
einops==0.8.2
|
||||
fastapi==0.141.1
|
||||
ffmpy==1.0.0
|
||||
filelock==3.32.3
|
||||
fire==0.7.1
|
||||
fonttools==4.63.0
|
||||
frozenlist==1.8.0
|
||||
fsspec==2025.3.0
|
||||
gradio==5.50.0
|
||||
gradio_client==1.14.0
|
||||
groovy==0.1.2
|
||||
h11==0.16.0
|
||||
hf-xet==1.6.0
|
||||
hf_transfer==0.1.9
|
||||
httpcore==1.0.9
|
||||
httpx==0.28.1
|
||||
huggingface_hub==1.29.0
|
||||
idna==3.19
|
||||
importlib_metadata==9.0.1
|
||||
Jinja2==3.1.6
|
||||
kiwisolver==1.5.1
|
||||
llamafactory @ git+https://github.com/hiyouga/LLaMA-Factory.git@7fcf5b3b130e5713b52415bb7404c476fada9c8c
|
||||
Mako==1.4.3
|
||||
markdown-it-py==4.2.0
|
||||
MarkupSafe==3.0.3
|
||||
matplotlib==3.11.1
|
||||
mdurl==0.1.2
|
||||
modelscope==1.39.1
|
||||
modelscope-hub==0.3.0
|
||||
mpmath==1.3.0
|
||||
msgspec==0.21.1
|
||||
multidict==6.7.1
|
||||
multiprocess==0.70.16
|
||||
nest-asyncio==1.6.0
|
||||
networkx==3.6.1
|
||||
numpy==2.4.6
|
||||
nvidia-cublas-cu12==12.4.5.8
|
||||
nvidia-cuda-cupti-cu12==12.4.127
|
||||
nvidia-cuda-nvrtc-cu12==12.4.127
|
||||
nvidia-cuda-runtime-cu12==12.4.127
|
||||
nvidia-cudnn-cu12==9.1.0.70
|
||||
nvidia-cufft-cu12==11.2.1.3
|
||||
nvidia-curand-cu12==10.3.5.147
|
||||
nvidia-cusolver-cu12==11.6.1.9
|
||||
nvidia-cusparse-cu12==12.3.1.170
|
||||
nvidia-cusparselt-cu12==0.6.2
|
||||
nvidia-nccl-cu12==2.21.5
|
||||
nvidia-nvjitlink-cu12==12.4.127
|
||||
nvidia-nvtx-cu12==12.4.127
|
||||
omegaconf==2.3.1
|
||||
optuna==5.0.0
|
||||
orjson==3.12.0
|
||||
packaging==26.3
|
||||
pandas==2.3.3
|
||||
peft==0.18.1
|
||||
pillow==11.3.0
|
||||
pip==26.2.1
|
||||
propcache==0.5.2
|
||||
protobuf==7.36.0
|
||||
psutil==7.2.2
|
||||
pyarrow==25.0.1
|
||||
pydantic==2.12.3
|
||||
pydantic_core==2.41.4
|
||||
pydub==0.25.1
|
||||
Pygments==2.21.0
|
||||
pyparsing==3.3.2
|
||||
python-dateutil==2.9.0.post0
|
||||
python-multipart==0.0.32
|
||||
pytz==2026.3.post1
|
||||
PyYAML==6.0.3
|
||||
regex==2026.7.19
|
||||
requests==2.34.2
|
||||
rich==15.0.0
|
||||
ruff==0.16.5
|
||||
safehttpx==0.1.7
|
||||
safetensors==0.8.0
|
||||
scipy==1.17.1
|
||||
semantic-version==2.10.0
|
||||
sentencepiece==0.2.2
|
||||
setuptools==80.10.2
|
||||
shellingham==1.5.4
|
||||
shtab==1.12.0
|
||||
six==1.17.0
|
||||
SQLAlchemy==2.1.4
|
||||
sse-starlette==3.4.8
|
||||
starlette==0.52.1
|
||||
sympy==1.13.1
|
||||
termcolor==3.3.0
|
||||
tiktoken==0.14.0
|
||||
tokenizers==0.22.2
|
||||
tomlkit==0.13.3
|
||||
torch==2.6.0+cu124
|
||||
torchaudio==2.6.0+cu124
|
||||
torchdata==0.11.0
|
||||
torchvision==0.21.0+cu124
|
||||
tqdm==4.70.0
|
||||
transformers==5.5.0
|
||||
triton==3.2.0
|
||||
trl==0.24.0
|
||||
typer==0.27.2
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
tyro==0.8.14
|
||||
tzdata==2026.3
|
||||
unsloth==2026.6.1
|
||||
unsloth_zoo==2026.8.16
|
||||
urllib3==2.7.0
|
||||
uvicorn==0.52.4
|
||||
websockets==15.0.1
|
||||
wheel==0.47.0
|
||||
xformers==0.0.29.post3
|
||||
xxhash==4.0.1
|
||||
yarl==1.24.5
|
||||
zipp==4.1.0
|
||||
|
||||
# Doplnenia pre generativnu evaluaciu a deklarovane zavislosti Zoo:
|
||||
jieba==0.42.1
|
||||
rouge-chinese==1.0.3
|
||||
nltk==3.10.3
|
||||
joblib==1.6.0
|
||||
defusedxml==0.7.1
|
||||
torchao==0.13.0
|
||||
302
perun/scripts/evaluate.py
Normal file
302
perun/scripts/evaluate.py
Normal file
@ -0,0 +1,302 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate the trained adapter on the prepared eval or test split; report loss, F1 and exact match."""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
from collections import Counter, defaultdict
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import unicodedata
|
||||
from train import ROOT, read_config, write_config, verify_data, verify_length_filter, run_logged
|
||||
|
||||
def normalized_tokens(text: str) -> list[str]:
|
||||
text = unicodedata.normalize("NFKC", text).casefold()
|
||||
return "".join(
|
||||
" " if unicodedata.category(char).startswith("P") else char
|
||||
for char in text
|
||||
).split()
|
||||
|
||||
|
||||
def score_pair(prediction: str, reference: str) -> dict:
|
||||
predicted = normalized_tokens(prediction)
|
||||
expected = normalized_tokens(reference)
|
||||
if not expected:
|
||||
return {
|
||||
"token_f1": None,
|
||||
"normalized_exact_match": None,
|
||||
"scored": False,
|
||||
"skip_reason": "reference_has_zero_normalized_tokens",
|
||||
"prediction_tokens": len(predicted),
|
||||
"reference_tokens": 0,
|
||||
}
|
||||
shared = sum((Counter(predicted) & Counter(expected)).values())
|
||||
f1 = 200.0 * shared / (len(predicted) + len(expected))
|
||||
return {
|
||||
"token_f1": f1,
|
||||
"normalized_exact_match": 100.0 if predicted == expected else 0.0,
|
||||
"scored": True,
|
||||
"skip_reason": None,
|
||||
"prediction_tokens": len(predicted),
|
||||
"reference_tokens": len(expected),
|
||||
}
|
||||
|
||||
|
||||
def read_jsonl(path: Path) -> list[dict]:
|
||||
result = []
|
||||
with path.open("r", encoding="utf-8-sig") as handle:
|
||||
for line_number, line in enumerate(handle, 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"{path}:{line_number}: invalid JSON: {exc.msg}") from exc
|
||||
if not isinstance(row, dict):
|
||||
raise ValueError(f"{path}:{line_number}: expected a JSON object")
|
||||
result.append(row)
|
||||
return result
|
||||
|
||||
|
||||
def required_text(row: dict, field: str, context: str) -> str:
|
||||
value = row.get(field)
|
||||
if not isinstance(value, str):
|
||||
raise ValueError(f"{context}: {field!r} must contain a string")
|
||||
return value
|
||||
|
||||
|
||||
def reference_text(row: dict, index: int) -> str:
|
||||
for field in ("output", "response", "label"):
|
||||
if field in row:
|
||||
return required_text(row, field, f"reference row {index}")
|
||||
raise ValueError(f"reference row {index}: missing output/response/label")
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def verify_training_dataset(adapter: Path, data_dir: Path) -> Path:
|
||||
run_dir = adapter.parent
|
||||
if adapter.name.startswith("checkpoint-") and not (run_dir / "run.json").is_file():
|
||||
run_dir = run_dir.parent
|
||||
record_path = run_dir / "run.json"
|
||||
if not record_path.is_file():
|
||||
raise ValueError(f"Missing training record: {record_path}. Keep the adapter with its original run files.")
|
||||
record = json.loads(record_path.read_text(encoding="utf-8"))
|
||||
if record.get("data_manifest_sha256") != sha256(data_dir / "manifest.json"):
|
||||
raise ValueError("The prepared dataset differs from this adapter's training data. "
|
||||
"Use the original dataset and its eval/test split recorded in run.json.")
|
||||
return run_dir
|
||||
|
||||
|
||||
def summarize(rows: list[dict]) -> dict:
|
||||
scored = [row for row in rows if row["scored"]]
|
||||
count = len(scored)
|
||||
return {
|
||||
"count_total": len(rows),
|
||||
"count_scored": count,
|
||||
"count_skipped_zero_token_references": len(rows) - count,
|
||||
"count_empty_predictions_raw": sum(not row["prediction"].strip() for row in rows),
|
||||
"count_zero_token_predictions": sum(row["prediction_tokens"] == 0 for row in rows),
|
||||
"count_nonempty_predictions_raw": sum(bool(row["prediction"].strip()) for row in rows),
|
||||
"count_truncated_or_modified_labels": sum(
|
||||
not row["label_matches_full_reference"] for row in rows
|
||||
),
|
||||
"count_labels_different_before_normalization": sum(
|
||||
not row["label_matches_full_reference_raw"] for row in rows
|
||||
),
|
||||
"token_f1": sum(row["token_f1"] for row in scored) / count if count else None,
|
||||
"normalized_exact_match": (
|
||||
sum(row["normalized_exact_match"] for row in scored) / count if count else None
|
||||
),
|
||||
"prediction_tokens_total": sum(row["prediction_tokens"] for row in rows),
|
||||
"reference_tokens_total": sum(row["reference_tokens"] for row in rows),
|
||||
}
|
||||
|
||||
|
||||
def score_files(
|
||||
predictions_path: Path,
|
||||
references_path: Path,
|
||||
output_dir: Path,
|
||||
metadata_path: Path | None = None,
|
||||
) -> dict:
|
||||
predictions = read_jsonl(predictions_path)
|
||||
references = read_jsonl(references_path)
|
||||
if not references:
|
||||
raise ValueError("The reference file contains no examples")
|
||||
if len(predictions) != len(references):
|
||||
raise ValueError(
|
||||
f"Row count mismatch: {len(predictions)} predictions != {len(references)} references. "
|
||||
"No scores written; evaluate the complete split in the original order."
|
||||
)
|
||||
metadata = read_jsonl(metadata_path) if metadata_path is not None else [{} for _ in references]
|
||||
if len(metadata) != len(references):
|
||||
raise ValueError(
|
||||
f"Row count mismatch: {len(metadata)} metadata rows != {len(references)} references"
|
||||
)
|
||||
|
||||
rows = []
|
||||
groups = defaultdict(list)
|
||||
for index, (prediction_row, reference_row, meta) in enumerate(
|
||||
zip(predictions, references, metadata)
|
||||
):
|
||||
prediction = required_text(prediction_row, "predict", f"prediction row {index}")
|
||||
label = required_text(prediction_row, "label", f"prediction row {index}")
|
||||
reference = reference_text(reference_row, index)
|
||||
if "row_index" in meta and meta["row_index"] != index:
|
||||
raise ValueError(f"metadata row {index}: row_index does not match the reference order")
|
||||
source_metadata = meta.get("source_metadata", {})
|
||||
if not isinstance(source_metadata, dict):
|
||||
raise ValueError(f"metadata row {index}: source_metadata must be an object")
|
||||
task_type = meta.get(
|
||||
"task_type", source_metadata.get("task_type", reference_row.get("task_type", "unknown"))
|
||||
)
|
||||
if not isinstance(task_type, str):
|
||||
raise ValueError(f"metadata row {index}: task_type must be a string")
|
||||
row = {
|
||||
"id": meta.get("stable_id", meta.get("id", reference_row.get("id", index))),
|
||||
"index": index,
|
||||
"task_type": task_type,
|
||||
"prediction": prediction,
|
||||
"reference": reference,
|
||||
"exported_label": label,
|
||||
"label_matches_full_reference": normalized_tokens(label) == normalized_tokens(reference),
|
||||
"label_matches_full_reference_raw": label == reference,
|
||||
**score_pair(prediction, reference),
|
||||
}
|
||||
rows.append(row)
|
||||
groups[task_type].append(row)
|
||||
|
||||
metrics = {
|
||||
"schema_version": 1,
|
||||
"metric_scale": "0..100",
|
||||
"metric_version": "aya_overlap_v1",
|
||||
"overall": summarize(rows),
|
||||
"by_task_type": {name: summarize(group) for name, group in sorted(groups.items())},
|
||||
"inputs": {
|
||||
"predictions": {"path": str(predictions_path.resolve()), "sha256": sha256(predictions_path)},
|
||||
"references": {"path": str(references_path.resolve()), "sha256": sha256(references_path)},
|
||||
},
|
||||
}
|
||||
if metadata_path is not None:
|
||||
metrics["inputs"]["metadata"] = {"path": str(metadata_path.resolve()), "sha256": sha256(metadata_path)}
|
||||
|
||||
if output_dir.exists() and (not output_dir.is_dir() or any(output_dir.iterdir())):
|
||||
raise ValueError(f"Output directory must be new or empty: {output_dir}")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
with (output_dir / "per_sample.jsonl").open("w", encoding="utf-8", newline="\n") as handle:
|
||||
for row in rows:
|
||||
handle.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n")
|
||||
(output_dir / "metrics.json").write_text(
|
||||
json.dumps(metrics, ensure_ascii=False, indent=2, allow_nan=False) + "\n", encoding="utf-8"
|
||||
)
|
||||
return metrics
|
||||
|
||||
|
||||
|
||||
def evaluation_configs(training: dict, adapter: Path, output_dir: Path, split: str, max_new_tokens: int):
|
||||
keys = ("model_name_or_path", "model_revision", "stage", "finetuning_type", "template",
|
||||
"dataset_dir", "cutoff_len", "packing", "train_on_prompt",
|
||||
"bf16", "infer_dtype", "flash_attn", "use_unsloth", "rope_scaling", "seed", "data_seed",
|
||||
"preprocessing_num_workers", "dataloader_num_workers", "report_to")
|
||||
base = {key: training[key] for key in keys if key in training}
|
||||
base.update({"adapter_name_or_path": str(adapter), "eval_dataset": f"aya_{split}",
|
||||
"use_unsloth_gc": False, "overwrite_output_dir": False})
|
||||
loss = dict(base, output_dir=str(output_dir / "loss"), do_eval=True,
|
||||
predict_with_generate=False, per_device_eval_batch_size=8)
|
||||
generation = dict(base, output_dir=str(output_dir / "generation"), do_predict=True,
|
||||
predict_with_generate=True, per_device_eval_batch_size=1,
|
||||
do_sample=False, num_beams=1, max_new_tokens=max_new_tokens)
|
||||
return loss, generation
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("config", type=Path, help="The same model YAML used for training.")
|
||||
parser.add_argument("--adapter", type=Path, help="Defaults to output_dir in the model YAML.")
|
||||
parser.add_argument("--output-dir", type=Path, required=True)
|
||||
parser.add_argument("--split", choices=("eval", "test"), default="eval")
|
||||
parser.add_argument("--max-new-tokens", type=int, default=256)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.max_new_tokens < 1:
|
||||
parser.error("--max-new-tokens must be positive.")
|
||||
config = read_config(args.config.resolve())
|
||||
data_dir = Path(config["dataset_dir"])
|
||||
manifest = verify_data(data_dir, (args.split,))
|
||||
verify_length_filter(manifest, config)
|
||||
adapter = args.adapter.resolve() if args.adapter else Path(config["output_dir"])
|
||||
if not (adapter / "adapter_config.json").is_file():
|
||||
raise ValueError(f"Missing trained adapter: {adapter}")
|
||||
run_dir = verify_training_dataset(adapter, data_dir)
|
||||
used_config = run_dir / "used_config.yaml"
|
||||
if not used_config.is_file():
|
||||
raise ValueError(f"Missing recorded training configuration: {used_config}")
|
||||
if used_config.is_file():
|
||||
trained = read_config(used_config)
|
||||
for key in ("model_name_or_path", "model_revision", "template", "cutoff_len", "rope_scaling"):
|
||||
if trained.get(key) != config.get(key):
|
||||
raise ValueError(f"Evaluation setting {key} differs from this adapter's training; use {used_config}.")
|
||||
output_dir = args.output_dir.resolve()
|
||||
if output_dir.exists() and any(output_dir.iterdir()):
|
||||
raise ValueError(f"Evaluation output directory must be new or empty: {output_dir}")
|
||||
loss, generation = evaluation_configs(config, adapter, output_dir, args.split, args.max_new_tokens)
|
||||
if args.dry_run:
|
||||
print(json.dumps({"split": args.split, "adapter": str(adapter), "evaluation_gpus": 1,
|
||||
"loss_config": loss, "generation_config": generation}, ensure_ascii=False, indent=2))
|
||||
print("Evaluation configuration/data checks passed. No inference started.")
|
||||
return 0
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
record_path = output_dir / "evaluation.json"
|
||||
record = {"status": "running", "split": args.split, "adapter": str(adapter),
|
||||
"config": str(args.config.resolve()), "gpus": 1, "backend": "Unsloth", "precision": "bf16"}
|
||||
record_path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
|
||||
env = os.environ.copy()
|
||||
env.update({"PYTHONUNBUFFERED": "1", "TOKENIZERS_PARALLELISM": "false"})
|
||||
try:
|
||||
for name, settings in (("eval_loss", loss), ("eval_generate", generation)):
|
||||
path = output_dir / f"{name}.yaml"
|
||||
write_config(path, settings)
|
||||
command = [sys.executable, "-m", "torch.distributed.run", "--standalone", "--nproc_per_node=1",
|
||||
str(ROOT / "scripts/train.py"), str(path), "--worker"]
|
||||
run_logged(command, output_dir / f"{name}.log", env)
|
||||
metrics = score_files(output_dir / "generation/generated_predictions.jsonl",
|
||||
data_dir / f"{args.split}.jsonl", output_dir / "scores",
|
||||
data_dir / f"{args.split}.metadata.jsonl")
|
||||
reported = json.loads((output_dir / "loss/eval_results.json").read_text(encoding="utf-8"))
|
||||
value = reported.get("eval_loss")
|
||||
if value is None or not math.isfinite(value):
|
||||
raise ValueError("LLaMA-Factory did not report a finite eval_loss.")
|
||||
summary = {"model": config["model_name_or_path"], "split": args.split,
|
||||
"eval_loss": value, "perplexity": math.exp(value) if value < 700 else None,
|
||||
"token_f1": metrics["overall"]["token_f1"],
|
||||
"normalized_exact_match": metrics["overall"]["normalized_exact_match"],
|
||||
"examples": metrics["overall"]["count_total"], "metric_scale": "0..100",
|
||||
"max_new_tokens": args.max_new_tokens, "cutoff_len": config["cutoff_len"],
|
||||
"generation_context_limit": config["cutoff_len"]}
|
||||
|
||||
(output_dir / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
record["status"] = "complete"
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
except BaseException:
|
||||
record["status"] = "failed_or_interrupted"
|
||||
raise
|
||||
finally:
|
||||
record_path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (OSError, ValueError, RuntimeError, ImportError, subprocess.CalledProcessError) as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
428
perun/scripts/prepare_data.py
Normal file
428
perun/scripts/prepare_data.py
Normal file
@ -0,0 +1,428 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare Slovak Aya train/eval/test data for LLaMA-Factory."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
from itertools import islice
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import sys
|
||||
import unicodedata
|
||||
from typing import Any, Iterator
|
||||
|
||||
|
||||
HF_DATASET = "CohereLabs/aya_collection_language_split"
|
||||
HF_REVISION = "a3af2fde4b4cb5b2775830b11244a1a20b5f004f"
|
||||
HF_CONFIG = "slovak"
|
||||
HF_SPLIT = "train"
|
||||
SPLITS = ("train", "eval", "test")
|
||||
|
||||
|
||||
def normalized(text: str, *, fold_case: bool = False) -> str:
|
||||
text = " ".join(unicodedata.normalize("NFKC", text).split())
|
||||
return text.casefold() if fold_case else text
|
||||
|
||||
|
||||
def digest_json(value: Any) -> str:
|
||||
payload = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
result = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1024 * 1024), b""):
|
||||
result.update(chunk)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, content: Any) -> None:
|
||||
with path.open("x", encoding="utf-8", newline="\n") as target:
|
||||
json.dump(content, target, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
target.write("\n")
|
||||
|
||||
|
||||
def local_rows(path: Path) -> Iterator[Any]:
|
||||
with path.open("r", encoding="utf-8-sig") as source:
|
||||
for line_number, line in enumerate(source, 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
yield json.loads(line)
|
||||
except json.JSONDecodeError as error:
|
||||
raise ValueError(f"Invalid JSON at {path}:{line_number}: {error}") from error
|
||||
|
||||
|
||||
def open_source(args: argparse.Namespace) -> tuple[Iterator[Any], dict[str, Any]]:
|
||||
if args.local_jsonl:
|
||||
path = args.local_jsonl.expanduser().resolve(strict=True)
|
||||
if not path.is_file():
|
||||
raise ValueError(f"Local source is not a file: {path}")
|
||||
return local_rows(path), {
|
||||
"kind": "local_jsonl",
|
||||
"path": str(path),
|
||||
"sha256": file_sha256(path),
|
||||
"bytes": path.stat().st_size,
|
||||
"revision": None,
|
||||
}
|
||||
|
||||
try:
|
||||
from datasets import load_dataset
|
||||
from huggingface_hub import HfApi
|
||||
except ImportError as error:
|
||||
raise RuntimeError("Install datasets and huggingface_hub for Hugging Face input.") from error
|
||||
|
||||
info = HfApi().dataset_info(HF_DATASET, revision=args.revision or HF_REVISION)
|
||||
revision = info.sha
|
||||
if not revision:
|
||||
raise RuntimeError("Hugging Face did not return an immutable dataset revision.")
|
||||
data = load_dataset(
|
||||
HF_DATASET, HF_CONFIG, split=HF_SPLIT, revision=revision, streaming=True
|
||||
)
|
||||
return iter(data), {
|
||||
"kind": "huggingface",
|
||||
"dataset": HF_DATASET,
|
||||
"config": HF_CONFIG,
|
||||
"split": HF_SPLIT,
|
||||
"requested_revision": args.revision,
|
||||
"revision": revision,
|
||||
"streaming": True,
|
||||
}
|
||||
|
||||
|
||||
def convert_row(row: Any) -> tuple[dict[str, str], dict[str, Any]] | None:
|
||||
if not isinstance(row, dict):
|
||||
return None
|
||||
if "inputs" in row or "targets" in row:
|
||||
instruction, query, response = row.get("inputs"), "", row.get("targets")
|
||||
excluded = {"inputs", "targets"}
|
||||
else:
|
||||
instruction, query, response = row.get("instruction"), row.get("input", ""), row.get("output")
|
||||
query = "" if query is None else query
|
||||
excluded = {"instruction", "input", "output"}
|
||||
if not all(isinstance(value, str) for value in (instruction, query, response)):
|
||||
return None
|
||||
if not instruction.strip() or not response.strip():
|
||||
return None
|
||||
metadata = {key: value for key, value in row.items() if key not in excluded}
|
||||
return {
|
||||
"instruction": instruction.strip(),
|
||||
"input": query.strip(),
|
||||
"output": response.strip(),
|
||||
}, metadata
|
||||
|
||||
|
||||
def create_database(path: Path) -> sqlite3.Connection:
|
||||
connection = sqlite3.connect(path)
|
||||
connection.executescript(
|
||||
"""
|
||||
PRAGMA journal_mode = DELETE;
|
||||
PRAGMA synchronous = NORMAL;
|
||||
PRAGMA temp_store = FILE;
|
||||
PRAGMA cache_size = -32768;
|
||||
CREATE TABLE prompt_groups (
|
||||
group_id TEXT PRIMARY KEY,
|
||||
order_key TEXT NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
split TEXT NOT NULL DEFAULT 'train'
|
||||
);
|
||||
CREATE TABLE records (
|
||||
position INTEGER PRIMARY KEY,
|
||||
stable_id TEXT NOT NULL UNIQUE,
|
||||
group_id TEXT NOT NULL,
|
||||
instruction TEXT NOT NULL,
|
||||
input TEXT NOT NULL,
|
||||
output TEXT NOT NULL,
|
||||
metadata TEXT NOT NULL,
|
||||
source_row_index INTEGER NOT NULL,
|
||||
source_dataset TEXT NOT NULL,
|
||||
source_task TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
return connection
|
||||
|
||||
|
||||
SYSTEM_TEXT = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"
|
||||
USER_TEXT = "### Instruction:\n{}\n\n### Response:\n"
|
||||
|
||||
|
||||
class SFTLengthFilter:
|
||||
"""Keep a whole pair only when its exact Alpaca length fits both model configs."""
|
||||
def __init__(self, config_dir: Path):
|
||||
import yaml
|
||||
from transformers import AutoTokenizer
|
||||
self.profiles = {}
|
||||
self.excluded_by_model = Counter()
|
||||
self.max_accepted_tokens = Counter()
|
||||
for name in ("mistral", "olmo"):
|
||||
settings = yaml.safe_load((config_dir / f"{name}.yaml").read_text(encoding="utf-8-sig"))
|
||||
if settings.get("template") != "alpaca":
|
||||
raise ValueError("Length filtering currently requires the Alpaca template for both models.")
|
||||
limit = int(settings["cutoff_len"])
|
||||
if limit < 1:
|
||||
raise ValueError("cutoff_len must be positive.")
|
||||
tokenizer = AutoTokenizer.from_pretrained(settings["model_name_or_path"],
|
||||
revision=settings["model_revision"], use_fast=True, fix_mistral_regex=True,
|
||||
model_max_length=10**9)
|
||||
backend = tokenizer.backend_tokenizer
|
||||
backend.no_truncation()
|
||||
backend.no_padding()
|
||||
if tokenizer.eos_token_id is None:
|
||||
raise ValueError(f"Missing EOS token: {name}")
|
||||
self.profiles[name] = {
|
||||
"model_name_or_path": settings["model_name_or_path"],
|
||||
"model_revision": settings["model_revision"], "template": "alpaca",
|
||||
"cutoff_len": limit, "tokenizer_class": type(tokenizer).__name__,
|
||||
"system_tokens": len(tokenizer.encode(SYSTEM_TEXT, add_special_tokens=False)),
|
||||
"ending_tokens": 1 + len(tokenizer.encode("\n\n", add_special_tokens=False)),
|
||||
"backend": backend,
|
||||
}
|
||||
|
||||
def lengths(self, examples: list[dict[str, str]]) -> list[dict[str, int]]:
|
||||
prompts = [USER_TEXT.format("\n".join(part for part in (row["instruction"], row["input"]) if part))
|
||||
for row in examples]
|
||||
responses = [row["output"] for row in examples]
|
||||
result = [{} for _ in examples]
|
||||
for name, profile in self.profiles.items():
|
||||
backend = profile["backend"]
|
||||
encode = getattr(backend, "encode_batch_fast", backend.encode_batch)
|
||||
inputs = encode(prompts, add_special_tokens=False)
|
||||
targets = encode(responses, add_special_tokens=False)
|
||||
for item, source, target in zip(result, inputs, targets):
|
||||
item[name] = profile["system_tokens"] + len(source) + len(target) + profile["ending_tokens"]
|
||||
return result
|
||||
|
||||
def over_limit(self, lengths: dict[str, int]) -> list[str]:
|
||||
return [name for name, length in lengths.items() if length > self.profiles[name]["cutoff_len"]]
|
||||
|
||||
def report(self) -> dict:
|
||||
return {"enabled": True,
|
||||
"models": {name: {key: value for key, value in profile.items() if key != "backend"}
|
||||
for name, profile in self.profiles.items()},
|
||||
"excluded_by_model": {name: self.excluded_by_model[name] for name in self.profiles},
|
||||
"max_accepted_tokens_before_pair_dedup": {name: self.max_accepted_tokens[name] for name in self.profiles},
|
||||
"excluded_log": "excluded_long.jsonl"}
|
||||
|
||||
|
||||
def ingest(connection: sqlite3.Connection, rows: Iterator[Any], seed: int,
|
||||
length_filter: SFTLengthFilter, excluded) -> Counter:
|
||||
counts: Counter = Counter()
|
||||
indexed = enumerate(rows)
|
||||
next_progress = 10000
|
||||
while batch := list(islice(indexed, 512)):
|
||||
candidates = []
|
||||
for source_index, row in batch:
|
||||
counts["source_rows"] += 1
|
||||
converted = convert_row(row)
|
||||
if converted is None:
|
||||
counts["invalid_or_empty_rows"] += 1
|
||||
else:
|
||||
candidates.append((source_index, *converted))
|
||||
measured = length_filter.lengths([data for _, data, _ in candidates])
|
||||
for (source_index, data, metadata), lengths in zip(candidates, measured):
|
||||
over = length_filter.over_limit(lengths)
|
||||
if over:
|
||||
counts["filtered_long_rows"] += 1
|
||||
length_filter.excluded_by_model.update(over)
|
||||
excluded.write(json.dumps({"source_row_index": source_index, "source_id": metadata.get("id"),
|
||||
"dataset_name": metadata.get("dataset_name"), "task_type": metadata.get("task_type"),
|
||||
"token_lengths": lengths, "over_limit_models": over}, ensure_ascii=False) + "\n")
|
||||
continue
|
||||
for name, length in lengths.items():
|
||||
length_filter.max_accepted_tokens[name] = max(length_filter.max_accepted_tokens[name], length)
|
||||
prompt_key = [normalized(data["instruction"], fold_case=True), normalized(data["input"], fold_case=True)]
|
||||
group_id = digest_json(prompt_key)
|
||||
stable_id = digest_json([prompt_key, normalized(data["output"])])
|
||||
cursor = connection.execute(
|
||||
"""INSERT OR IGNORE INTO records
|
||||
(stable_id, group_id, instruction, input, output, metadata,
|
||||
source_row_index, source_dataset, source_task)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(stable_id, group_id, data["instruction"], data["input"], data["output"],
|
||||
json.dumps(metadata, ensure_ascii=False, separators=(",", ":")), source_index,
|
||||
str(metadata.get("dataset_name") or "unknown"), str(metadata.get("task_type") or "unknown")))
|
||||
if cursor.rowcount == 0:
|
||||
counts["duplicate_pairs_removed"] += 1
|
||||
else:
|
||||
counts["retained_rows"] += 1
|
||||
connection.execute(
|
||||
"""INSERT INTO prompt_groups (group_id, order_key, size) VALUES (?, ?, 1)
|
||||
ON CONFLICT(group_id) DO UPDATE SET size = size + 1""",
|
||||
(group_id, digest_json([seed, group_id])))
|
||||
if counts["source_rows"] >= next_progress:
|
||||
connection.commit()
|
||||
print(f"Read {counts['source_rows']:,}; filtered long {counts['filtered_long_rows']:,}; retained {counts['retained_rows']:,}", flush=True)
|
||||
next_progress += 10000
|
||||
connection.commit()
|
||||
for key in ("source_rows", "invalid_or_empty_rows", "filtered_long_rows", "duplicate_pairs_removed", "retained_rows"):
|
||||
counts.setdefault(key, 0)
|
||||
if not counts["retained_rows"]:
|
||||
raise ValueError("The source contains no usable examples after validation and length filtering.")
|
||||
connection.execute("CREATE INDEX prompt_group_order ON prompt_groups(order_key, group_id)")
|
||||
connection.execute("CREATE INDEX records_group ON records(group_id)")
|
||||
connection.commit()
|
||||
counts["prompt_groups"] = connection.execute("SELECT COUNT(*) FROM prompt_groups").fetchone()[0]
|
||||
counts["groups_with_multiple_responses"] = connection.execute("SELECT COUNT(*) FROM prompt_groups WHERE size > 1").fetchone()[0]
|
||||
return counts
|
||||
|
||||
|
||||
def assign_splits(connection: sqlite3.Connection, eval_size: int, test_size: int) -> dict[str, dict[str, int]]:
|
||||
remaining = {"eval": eval_size, "test": test_size}
|
||||
assignments: list[tuple[str, str]] = []
|
||||
# Selection depends only on seed, normalized prompt, and the number of its
|
||||
# distinct responses. An entire prompt group always stays in one split.
|
||||
for group_id, size in connection.execute(
|
||||
"SELECT group_id, size FROM prompt_groups ORDER BY order_key, group_id"
|
||||
):
|
||||
split = next((name for name in ("eval", "test") if size <= remaining[name]), None)
|
||||
if split is not None:
|
||||
remaining[split] -= size
|
||||
assignments.append((split, group_id))
|
||||
if not any(remaining.values()):
|
||||
break
|
||||
connection.executemany("UPDATE prompt_groups SET split = ? WHERE group_id = ?", assignments)
|
||||
connection.commit()
|
||||
result = {name: {"rows": 0, "prompt_groups": 0} for name in SPLITS}
|
||||
for split, group_count, row_count in connection.execute(
|
||||
"SELECT split, COUNT(*), SUM(size) FROM prompt_groups GROUP BY split"
|
||||
):
|
||||
result[split] = {"rows": row_count, "prompt_groups": group_count}
|
||||
if not result["train"]["rows"]:
|
||||
raise ValueError("No rows remain for train. Reduce --eval-size and/or --test-size.")
|
||||
# A record belongs to one prompt group; each group has exactly one split.
|
||||
conflicts = connection.execute(
|
||||
"""SELECT COUNT(*) FROM (
|
||||
SELECT r.group_id FROM records r JOIN prompt_groups g USING(group_id)
|
||||
GROUP BY r.group_id HAVING COUNT(DISTINCT g.split) != 1
|
||||
)"""
|
||||
).fetchone()[0]
|
||||
if conflicts:
|
||||
raise RuntimeError("Prompt-disjoint split verification failed.")
|
||||
if result["eval"]["rows"] > eval_size or result["test"]["rows"] > test_size:
|
||||
raise RuntimeError("A holdout exceeds its requested row limit.")
|
||||
return result
|
||||
|
||||
|
||||
def emit_splits(connection: sqlite3.Connection, output: Path, expected: dict[str, dict[str, int]]) -> dict[str, Any]:
|
||||
files: dict[str, Any] = {}
|
||||
for split in SPLITS:
|
||||
data_path = output / f"{split}.jsonl"
|
||||
metadata_path = output / f"{split}.metadata.jsonl"
|
||||
row_count = 0
|
||||
with data_path.open("x", encoding="utf-8", newline="\n") as data_file, metadata_path.open(
|
||||
"x", encoding="utf-8", newline="\n"
|
||||
) as metadata_file:
|
||||
cursor = connection.execute(
|
||||
"""SELECT r.stable_id, r.group_id, r.instruction, r.input, r.output,
|
||||
r.metadata, r.source_row_index
|
||||
FROM records r JOIN prompt_groups g USING(group_id)
|
||||
WHERE g.split = ? ORDER BY g.order_key, r.stable_id""", (split,)
|
||||
)
|
||||
for stable_id, group_id, instruction, query, response, metadata, source_index in cursor:
|
||||
item = {"instruction": instruction, "input": query, "output": response}
|
||||
provenance = {
|
||||
"row_index": row_count,
|
||||
"stable_id": stable_id,
|
||||
"prompt_group_id": group_id,
|
||||
"source_row_index": source_index,
|
||||
"source_metadata": json.loads(metadata),
|
||||
}
|
||||
provenance["source_id"] = provenance["source_metadata"].get("id")
|
||||
data_file.write(json.dumps(item, ensure_ascii=False, separators=(",", ":")) + "\n")
|
||||
metadata_file.write(json.dumps(provenance, ensure_ascii=False, separators=(",", ":")) + "\n")
|
||||
row_count += 1
|
||||
if row_count != expected[split]["rows"]:
|
||||
raise RuntimeError(f"Unexpected {split} output count: {row_count}")
|
||||
for path in (data_path, metadata_path):
|
||||
files[path.name] = {"sha256": file_sha256(path), "bytes": path.stat().st_size, "rows": row_count}
|
||||
return files
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output-dir", type=Path, required=True)
|
||||
parser.add_argument("--revision", help="Hugging Face revision; resolved to an immutable commit before loading.")
|
||||
parser.add_argument("--local-jsonl", type=Path, help="Offline JSONL input in Aya or Alpaca format.")
|
||||
parser.add_argument("--eval-size", type=int, default=1000, help="Maximum number of eval records (default: 1000).")
|
||||
parser.add_argument("--test-size", type=int, default=1000, help="Maximum number of test records (default: 1000).")
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
parser.add_argument("--keep-db", action="store_true", help="Keep the temporary SQLite database.")
|
||||
args = parser.parse_args()
|
||||
if min(args.eval_size, args.test_size) < 0:
|
||||
parser.error("Holdout sizes must be nonnegative.")
|
||||
if args.local_jsonl and args.revision:
|
||||
parser.error("--revision applies only to Hugging Face input.")
|
||||
output = args.output_dir.expanduser().resolve()
|
||||
if output.exists() and (not output.is_dir() or any(output.iterdir())):
|
||||
parser.error(f"Output must be a new or empty directory: {output}")
|
||||
# Resolve source before creating output, so authentication or dependency
|
||||
# errors do not produce a misleading empty data directory.
|
||||
rows, source = open_source(args)
|
||||
length_filter = SFTLengthFilter(Path(__file__).resolve().parents[1] / "configs")
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
database_path = output / "preparation.sqlite3"
|
||||
connection = create_database(database_path)
|
||||
try:
|
||||
excluded_path = output / "excluded_long.jsonl"
|
||||
with excluded_path.open("x", encoding="utf-8", newline="\n") as excluded:
|
||||
counts = ingest(connection, rows, args.seed, length_filter, excluded)
|
||||
splits = assign_splits(connection, args.eval_size, args.test_size)
|
||||
if sum(item["rows"] for item in splits.values()) != counts["retained_rows"]:
|
||||
raise RuntimeError("Split counts do not sum to the retained record count.")
|
||||
files = emit_splits(connection, output, splits)
|
||||
files[excluded_path.name] = {"sha256": file_sha256(excluded_path),
|
||||
"bytes": excluded_path.stat().st_size,
|
||||
"rows": counts["filtered_long_rows"]}
|
||||
registration = {
|
||||
f"aya_{split}": {
|
||||
"file_name": f"{split}.jsonl", "formatting": "alpaca",
|
||||
"columns": {"prompt": "instruction", "query": "input", "response": "output"},
|
||||
} for split in SPLITS
|
||||
}
|
||||
info_path = output / "dataset_info.json"
|
||||
write_json(info_path, registration)
|
||||
files[info_path.name] = {"sha256": file_sha256(info_path), "bytes": info_path.stat().st_size}
|
||||
breakdown = [
|
||||
{"split": split, "dataset_name": dataset, "task_type": task, "rows": rows_count, "prompt_groups": groups_count}
|
||||
for split, dataset, task, rows_count, groups_count in connection.execute(
|
||||
"""SELECT g.split, r.source_dataset, r.source_task, COUNT(*), COUNT(DISTINCT r.group_id)
|
||||
FROM records r JOIN prompt_groups g USING(group_id)
|
||||
GROUP BY g.split, r.source_dataset, r.source_task
|
||||
ORDER BY g.split, r.source_dataset, r.source_task"""
|
||||
)
|
||||
]
|
||||
manifest = {
|
||||
"format_version": 1,
|
||||
"prepared_at_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"preparation_script_sha256": file_sha256(Path(__file__).resolve()),
|
||||
"source": source,
|
||||
"seed": args.seed,
|
||||
"counts": dict(counts),
|
||||
"length_filter": length_filter.report(),
|
||||
"requested_holdout_max_rows": {"eval": args.eval_size, "test": args.test_size},
|
||||
"splits": splits,
|
||||
"source_breakdown": breakdown,
|
||||
"files": files,
|
||||
}
|
||||
write_json(output / "manifest.json", manifest)
|
||||
finally:
|
||||
connection.close()
|
||||
if not args.keep_db:
|
||||
# Only our explicitly named staging file inside this new output is removed.
|
||||
database_path.unlink()
|
||||
print(json.dumps({"output_dir": str(output), "source": source, "counts": dict(counts), "splits": splits}, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (ValueError, RuntimeError) as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
325
perun/scripts/train.py
Normal file
325
perun/scripts/train.py
Normal file
@ -0,0 +1,325 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Train from one LLaMA-Factory YAML on four GPUs; import Unsloth in each worker first."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import functools
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
from types import MethodType
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
GPUS = 4
|
||||
|
||||
|
||||
def read_config(path: Path) -> dict:
|
||||
import yaml
|
||||
with path.open(encoding="utf-8-sig") as handle:
|
||||
config = yaml.safe_load(handle)
|
||||
if not isinstance(config, dict):
|
||||
raise ValueError("The configuration must be a YAML mapping.")
|
||||
if config.get("use_unsloth") is not True:
|
||||
raise ValueError("This package requires use_unsloth: true.")
|
||||
if config.get("bf16") is not True or config.get("fp16", False):
|
||||
raise ValueError("The H200 configuration must use bf16, not fp16.")
|
||||
if "model_max_length" in config:
|
||||
raise ValueError("Remove model_max_length from YAML; LLaMA-Factory derives it from cutoff_len.")
|
||||
if int(config.get("cutoff_len", 0)) < 1:
|
||||
raise ValueError("cutoff_len must be positive.")
|
||||
for key in ("dataset_dir", "output_dir"):
|
||||
if key not in config:
|
||||
raise ValueError(f"Missing configuration key: {key}")
|
||||
value = Path(config[key]).expanduser()
|
||||
config[key] = str((ROOT / value).resolve() if not value.is_absolute() else value.resolve())
|
||||
return config
|
||||
|
||||
|
||||
def write_config(path: Path, config: dict) -> None:
|
||||
import yaml
|
||||
path.write_text(yaml.safe_dump(config, allow_unicode=True, sort_keys=False), encoding="utf-8")
|
||||
|
||||
|
||||
def digest(path: Path) -> str:
|
||||
result = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
result.update(block)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def verify_data(data_dir: Path, splits: tuple[str, ...]) -> dict:
|
||||
manifest_path = data_dir / "manifest.json"
|
||||
if not manifest_path.is_file():
|
||||
raise ValueError(f"Run prepare_data.py first; missing {manifest_path}")
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
for split in splits:
|
||||
if manifest.get("splits", {}).get(split, {}).get("rows", 0) < 1:
|
||||
raise ValueError(f"The prepared {split} split is empty.")
|
||||
names = ["dataset_info.json"]
|
||||
for split in splits:
|
||||
names.extend((f"{split}.jsonl", f"{split}.metadata.jsonl"))
|
||||
for name in names:
|
||||
item = manifest.get("files", {}).get(name, {})
|
||||
if not item.get("sha256") or digest(data_dir / name) != item["sha256"]:
|
||||
raise ValueError(f"Prepared data changed or is missing: {name}")
|
||||
return manifest
|
||||
|
||||
|
||||
def verify_length_filter(manifest: dict, config: dict) -> None:
|
||||
report = manifest.get("length_filter", {})
|
||||
if report.get("enabled") is not True:
|
||||
raise ValueError("Prepared data has no shared length filter. Rerun prepare_data.py into data/aya_slovak_4096.")
|
||||
candidates = [profile for profile in report.get("models", {}).values()
|
||||
if profile.get("model_name_or_path") == config["model_name_or_path"]
|
||||
and profile.get("model_revision") == config["model_revision"]
|
||||
and profile.get("template") == config["template"]]
|
||||
if not candidates or not any(0 < int(profile["cutoff_len"]) <= int(config["cutoff_len"]) for profile in candidates):
|
||||
raise ValueError("The data was not length-filtered for this model, revision, template and context limit.")
|
||||
|
||||
|
||||
def run_logged(command: list[str], log: Path, env: dict | None = None) -> None:
|
||||
print("Running:", " ".join(command), flush=True)
|
||||
with log.open("a", encoding="utf-8") as handle:
|
||||
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||
text=True, encoding="utf-8", errors="replace", env=env)
|
||||
try:
|
||||
for line in process.stdout:
|
||||
print(line, end="", flush=True)
|
||||
handle.write(line)
|
||||
handle.flush()
|
||||
code = process.wait()
|
||||
except BaseException:
|
||||
process.terminate()
|
||||
process.wait()
|
||||
raise
|
||||
if code:
|
||||
raise subprocess.CalledProcessError(code, command)
|
||||
|
||||
|
||||
def context_settings(settings: dict, original_config) -> dict:
|
||||
requested = int(settings["cutoff_len"])
|
||||
native = int(original_config.max_position_embeddings)
|
||||
if requested > native or settings.get("rope_scaling"):
|
||||
raise ValueError("The shared 4096-token scenario uses the original model contexts without RoPE extension.")
|
||||
return {"requested_length": requested, "base_positions": native}
|
||||
|
||||
|
||||
def reject_truncation(source_len: int, target_len: int, cutoff_len: int) -> None:
|
||||
if source_len + target_len > cutoff_len:
|
||||
raise ValueError(f"A complete SFT example requires {source_len + target_len} tokens, "
|
||||
f"but only {cutoff_len} are available. Stopping instead of truncating input or answer.")
|
||||
|
||||
|
||||
def require_complete_examples() -> None:
|
||||
from llamafactory.data.processor import supervised, unsupervised
|
||||
for module in (supervised, unsupervised):
|
||||
original = module.infer_seqlen
|
||||
|
||||
def checked(source_len, target_len, cutoff_len, _original=original):
|
||||
reject_truncation(source_len, target_len, cutoff_len)
|
||||
return _original(source_len, target_len, cutoff_len)
|
||||
module.infer_seqlen = checked
|
||||
|
||||
|
||||
def bound_generation_context(model, context_length: int, default_max_new_tokens: int) -> None:
|
||||
original_generate = model.generate
|
||||
|
||||
@functools.wraps(original_generate)
|
||||
def bounded_generate(self, *args, **kwargs):
|
||||
inputs = kwargs.get("input_ids")
|
||||
if inputs is None:
|
||||
inputs = kwargs.get("inputs")
|
||||
if inputs is None and args:
|
||||
inputs = args[0]
|
||||
if inputs is None:
|
||||
raise ValueError("Cannot verify the text input length before generation.")
|
||||
prompt_length = int(inputs.shape[-1])
|
||||
available = context_length - prompt_length
|
||||
if available < 1:
|
||||
raise ValueError("The complete input leaves no room for an answer inside the context limit.")
|
||||
requested = kwargs.get("max_new_tokens")
|
||||
requested = int(default_max_new_tokens if requested is None else requested)
|
||||
if requested < 1:
|
||||
raise ValueError("max_new_tokens must be positive.")
|
||||
kwargs["max_new_tokens"] = min(requested, available)
|
||||
if requested > available:
|
||||
print(f"Generation limit: {available} new tokens after {prompt_length} input tokens "
|
||||
f"(context {context_length}); the complete input is preserved.", flush=True)
|
||||
return original_generate(*args, **kwargs)
|
||||
|
||||
model.generate = MethodType(bounded_generate, model)
|
||||
|
||||
|
||||
def require_unsloth_backend(settings: dict, context: dict) -> dict:
|
||||
# LLaMA-Factory may otherwise silently fall back when Unsloth rejects a model.
|
||||
# The wrappers affect this process only; no installed package is edited.
|
||||
from llamafactory.model import loader
|
||||
from llamafactory.model.model_utils import unsloth as integration
|
||||
loaded = {}
|
||||
|
||||
if hasattr(integration, "_get_unsloth_kwargs"):
|
||||
original_kwargs = integration._get_unsloth_kwargs
|
||||
|
||||
@functools.wraps(original_kwargs)
|
||||
def pinned_kwargs(config, model_name_or_path, model_args, finetuning_args):
|
||||
values = original_kwargs(config, model_name_or_path, model_args, finetuning_args)
|
||||
values["use_exact_model_name"] = True
|
||||
if model_args.model_revision:
|
||||
values["revision"] = model_args.model_revision
|
||||
return values
|
||||
|
||||
integration._get_unsloth_kwargs = pinned_kwargs
|
||||
|
||||
def strict(original):
|
||||
@functools.wraps(original)
|
||||
def wrapped(*args, **kwargs):
|
||||
model = original(*args, **kwargs)
|
||||
if model is None:
|
||||
raise RuntimeError("Unsloth did not load this model. Training/evaluation stopped; "
|
||||
"use a compatible Unsloth/LLaMA-Factory environment.")
|
||||
if model.config.max_position_embeddings < context["requested_length"]:
|
||||
raise RuntimeError("The loaded model's configured context is shorter than the requested SFT context.")
|
||||
if settings.get("do_train"):
|
||||
loaded["base_config_json"] = model.config.to_json_string()
|
||||
return model
|
||||
return wrapped
|
||||
|
||||
for name in ("load_unsloth_pretrained_model", "load_unsloth_peft_model"):
|
||||
original = getattr(loader, name, None)
|
||||
if original is None:
|
||||
raise RuntimeError("The installed LLaMA-Factory has an unsupported Unsloth integration.")
|
||||
setattr(loader, name, strict(original))
|
||||
if settings.get("do_predict") and settings.get("predict_with_generate"):
|
||||
# Keep PEFT/Unsloth delegation: GenerationMixin.generate bound directly
|
||||
# to a PeftModel fails in Transformers 5 because its class has no _sample.
|
||||
original_patch_model = loader.patch_model
|
||||
|
||||
@functools.wraps(original_patch_model)
|
||||
def patch_model_preserving_generate(model, *args, **kwargs):
|
||||
original_generate = model.generate
|
||||
result = original_patch_model(model, *args, **kwargs)
|
||||
model.generate = original_generate
|
||||
return result
|
||||
|
||||
loader.patch_model = patch_model_preserving_generate
|
||||
# Attach the guard to the final model, after LLaMA-Factory patches generate
|
||||
# and finishes adapter initialization. Earlier attachment can be replaced.
|
||||
from llamafactory.train.sft import workflow
|
||||
original_load_model = workflow.load_model
|
||||
|
||||
@functools.wraps(original_load_model)
|
||||
def load_bounded_model(*args, **kwargs):
|
||||
model = original_load_model(*args, **kwargs)
|
||||
bound_generation_context(model, context["requested_length"], int(settings["max_new_tokens"]))
|
||||
return model
|
||||
|
||||
workflow.load_model = load_bounded_model
|
||||
return loaded
|
||||
|
||||
|
||||
def worker(config_path: Path) -> None:
|
||||
# This order is important for Unsloth's patches.
|
||||
import unsloth # noqa: F401
|
||||
import torch
|
||||
config = read_config(config_path)
|
||||
world = int(os.environ.get("WORLD_SIZE", "1"))
|
||||
if config.get("do_train") and world != GPUS:
|
||||
raise RuntimeError("This training scenario requires four processes / four GPUs.")
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() < world:
|
||||
raise RuntimeError(f"This job requires {world} visible CUDA GPU(s).")
|
||||
local_rank = int(os.environ.get("LOCAL_RANK", "0"))
|
||||
torch.cuda.set_device(local_rank)
|
||||
if not torch.cuda.is_bf16_supported():
|
||||
raise RuntimeError("The allocated GPU does not support the requested bf16 precision.")
|
||||
print(f"rank={local_rank}, world_size={world}, GPU={torch.cuda.get_device_name(local_rank)}, "
|
||||
"precision=bf16, backend=Unsloth", flush=True)
|
||||
from transformers import AutoConfig
|
||||
original_config = AutoConfig.from_pretrained(config["model_name_or_path"], revision=config["model_revision"], trust_remote_code=False)
|
||||
context = context_settings(config, original_config)
|
||||
print("Model context:", json.dumps(context), flush=True)
|
||||
loaded = require_unsloth_backend(config, context)
|
||||
require_complete_examples()
|
||||
from llamafactory.train.tuner import run_exp
|
||||
run_exp(config)
|
||||
if config.get("do_train") and local_rank == 0:
|
||||
(Path(config["output_dir"]) / "base_model_config.json").write_text(loaded["base_config_json"], encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("config", type=Path, help="configs/mistral.yaml or configs/olmo.yaml")
|
||||
parser.add_argument("--run-dir", type=Path, help="New run directory; adapter is saved inside adapter/.")
|
||||
parser.add_argument("--resume", type=Path, help="Checkpoint inside the original run's adapter/ directory.")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Validate data/config; do not import GPU libraries.")
|
||||
parser.add_argument("--worker", action="store_true", help=argparse.SUPPRESS)
|
||||
args = parser.parse_args()
|
||||
if args.worker:
|
||||
worker(args.config.resolve())
|
||||
return 0
|
||||
config = read_config(args.config.resolve())
|
||||
if config.get("stage") != "sft" or config.get("finetuning_type") != "lora":
|
||||
raise ValueError("Expected an SFT / LoRA training configuration.")
|
||||
if not config.get("do_train") or config.get("eval_dataset") != "aya_eval":
|
||||
raise ValueError("Training must use the prepared train and eval splits.")
|
||||
if config.get("dataset") != "aya_train" or "val_size" in config:
|
||||
raise ValueError("Use aya_train and aya_eval without another automatic split.")
|
||||
if any(key in config for key in ("max_steps", "max_samples")):
|
||||
raise ValueError("The full-data scenario must not impose a smoke-test sample/step limit.")
|
||||
batch = GPUS * config["per_device_train_batch_size"] * config["gradient_accumulation_steps"]
|
||||
data_dir = Path(config["dataset_dir"])
|
||||
manifest = verify_data(data_dir, ("train", "eval"))
|
||||
verify_length_filter(manifest, config)
|
||||
if args.run_dir:
|
||||
config["output_dir"] = str(args.run_dir.resolve() / "adapter")
|
||||
run_dir = Path(config["output_dir"]).parent
|
||||
if args.resume:
|
||||
checkpoint = args.resume.resolve()
|
||||
checkpoint.relative_to(Path(config["output_dir"]))
|
||||
for name in ("trainer_state.json", "adapter_config.json", "optimizer.pt", "scheduler.pt"):
|
||||
if not (checkpoint / name).is_file():
|
||||
raise ValueError(f"Incomplete checkpoint: {checkpoint / name}")
|
||||
previous = json.loads((run_dir / "run.json").read_text(encoding="utf-8"))
|
||||
if previous["config"] != config or previous["data_manifest_sha256"] != digest(data_dir / "manifest.json"):
|
||||
raise ValueError("Resume requires the original configuration and prepared data.")
|
||||
elif run_dir.exists() and any(run_dir.iterdir()):
|
||||
raise ValueError(f"Run directory is not empty; choose --run-dir for a new run: {run_dir}")
|
||||
print(json.dumps({"gpus_per_training_run": GPUS, "effective_batch": batch,
|
||||
"cutoff_len": config["cutoff_len"], "bf16": True, "use_unsloth": True,
|
||||
"train_rows": manifest["splits"]["train"]["rows"],
|
||||
"eval_rows": manifest["splits"]["eval"]["rows"],
|
||||
"adapter_output": config["output_dir"]}, ensure_ascii=False, indent=2))
|
||||
if args.dry_run:
|
||||
print("Configuration/data checks passed. No training started.")
|
||||
return 0
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
if not args.resume:
|
||||
record = {"config": config, "gpus": GPUS, "effective_batch": batch,
|
||||
"data_manifest_sha256": digest(data_dir / "manifest.json"),
|
||||
"created_at_utc": datetime.now(timezone.utc).isoformat()}
|
||||
(run_dir / "run.json").write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
execution = dict(config)
|
||||
if args.resume:
|
||||
execution["resume_from_checkpoint"] = str(args.resume.resolve())
|
||||
execution_path = run_dir / "used_config.yaml"
|
||||
write_config(execution_path, execution)
|
||||
env = os.environ.copy()
|
||||
env.update({"PYTHONUNBUFFERED": "1", "TOKENIZERS_PARALLELISM": "false"})
|
||||
env.setdefault("OMP_NUM_THREADS", "4")
|
||||
command = [sys.executable, "-m", "torch.distributed.run", "--standalone",
|
||||
f"--nproc_per_node={GPUS}", str(Path(__file__).resolve()),
|
||||
str(execution_path), "--worker"]
|
||||
run_logged(command, run_dir / "train.log", env)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (OSError, ValueError, RuntimeError, ImportError, subprocess.CalledProcessError) as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
Loading…
Reference in New Issue
Block a user