js886io/Diplomovka
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Perun

Browse Source
This commit is contained in:
Jakub Schwarc 2026-11-12 16:51:42 +01:00
parent 59b0b6ef39
commit aabd0b5874
31 changed files with 9173 additions and 395 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@ -975,3 +975,10 @@ Z pôvodných 412 slovenských položiek sa vynechalo 187 položiek pre 12 nedos
Pri príprave sa sampler FlashInfer pokúšal kompilovať cez starší systémový CUDA toolkit. Úspešný beh používal `VLLM_USE_FLASHINFER_SAMPLER=0` a `--enforce-eager`. Pri príprave sa sampler FlashInfer pokúšal kompilovať cez starší systémový CUDA toolkit. Úspešný beh používal `VLLM_USE_FLASHINFER_SAMPLER=0` a `--enforce-eager`.
# Perun – dotrénovanie Ministralu 3B na slovenskej Aya
Model `ministral/Ministral-3b-instruct` bol dotrénovaný metódou SFT s LoRA na 3 490 041 tréningových príkladoch na štyroch NVIDIA H200. Beh dokončil jednu epochu a 109 064 krokov pri LR `0.0002`. Najnižší validačný loss bol **1,13089**.
Na odloženej Aya množine dosiahol posledný checkpoint tokenové F1 **35,06 zo 100**. V čiastočnom slovenskom zero-shot SK-bench sa porovnali oba modely na rovnakých **385 variantoch a 29 z 33 úloh**. Overall podľa metodiky leaderboardu stúpol z **7,18 na 14,07 bodu** pri pokrytí piatich zo šiestich skupín.
[Dokumentácia experimentu, konfigurácie a výsledky](perun/ministral_3b_aya/README.md)

304
perun/README.md
View File

@ -1,56 +1,50 @@
# Slovenská Aya: Mistral-SK a OLMo # Slovenská Aya na Perunovi
## Scenár Prvým modelom pre ďalšiu sériu je `ministral/Ministral-3b-instruct`. Po SFT s LoRA v LLaMA-Factory sa pôvodný a dotrénovaný model porovnajú na rovnakých úlohách SK-bench. `configs/ministral.yaml` pripína rovnakú východiskovú revíziu modelu, aká sa stiahla pre baseline. Formát chatu sa odvodí z tokenizéra modelu. Konfigurácie Mistral-SK a OLMo sú v balíku ako ďalšie možnosti.
Cieľom je inštrukčne dotrénovať Mistral-SK-7B a základný OLMo 2 7B na spoločnej slovenskej časti Aya Collection a porovnať ich adaptáciu. Použije sa **SFT s LoRA v LLaMA-Factory, zrýchlený Unslothom**. Jeden tréning využije **štyri H200 na jednom uzle Peruna cez DDP**.
Možnosti LoRA do návrhu scenára:
- **LoRA (Low-Rank Adaptation)**
- **QLoRA (Quantized Low-Rank Adaptation)**
- **rsLoRA (Rank-Stabilized LoRA)**
- **DoRA (Weight-Decomposed Low-Rank Adaptation)**
## Prehľad súborov
| Súbor | Úloha | | Súbor | Úloha |
| --- | --- | | --- | --- |
| `configs/mistral.yaml` | Základný tréningový config pre `slovak-nlp/mistral-sk-7b`. | | `scripts/prepare_data.py` | Jednorazová príprava spoločného datasetu. |
| `configs/olmo.yaml` | Základný tréningový config pre `allenai/OLMo-2-1124-7B`. | | `configs/ministral.yaml` | Model, revízia, LoRA a tréningové nastavenia. |
| `scripts/prepare_data.py` | Načítanie, čistenie, deduplikácia a rozdelenie dát. | | `scripts/train.py` | Kontrola dát a spustenie tréningu s Unslothom. |
| `scripts/train.py` | Spustenie LLaMA-Factory podľa zvoleného configu. | | `scripts/evaluate.py` | Loss, perplexita, slovný F1 a presná zhoda na spoločnom splite eval alebo test. |
| `scripts/evaluate.py` | Loss, perplexita, generovanie a slovný F1 / presná zhoda na eval alebo test množine. |
Balík má spolu šesť súborov. `evaluate.py` odvodí nastavenia z rovnakého modelového YAML a konkrétnu použitú konfiguráciu uloží až k výsledkom. ## Jeden pevný dataset
Poradie používania je **príprava dát → tréning → vyhodnotenie**. Príprava dát sa vykoná raz pre oba modely. Tréning sa spustí samostatne pre každý model alebo experiment a následne sa vyhodnotí vytvorený adaptér. Model a hyperparametre sa nastavujú v YAML; Python skripty vykonávajú jednotlivé kroky. Príprava dát sa vykoná **raz**. Skript načíta slovenskú Aya, vynechá neplatné a prázdne riadky, odstráni normalizované duplicity a rozdelí dáta so seedom 42. Rovnaké zadania s rôznymi odpoveďami zostávajú v jednej množine. Eval aj test dostanú najviac 1 000 príkladov. Zvyšok patrí do train.
## Spoločný dataset do 4 096 tokenov a nastavenie H200 V `data/aya_slovak_fixed` sa uložia **úplné zadania a referenčné odpovede**. Pri zmene modelu sa použijú tie isté súbory a ten istý manifest. Tokenizácia prebehne až v LLaMA-Factory podľa modelovej konfigurácie. Pri `cutoff_len: 4096` sa prípadné orezanie týka spracovaných tokenov pre daný beh. Dĺžky a počet orezaných príkladov sa môžu medzi modelmi líšiť.
Oba modely používajú **`cutoff_len: 4096`**. Limit zahŕňa celú SFT sekvenciu: systémový text Alpaca, inštrukciu a prípadný vstup, celú referenčnú odpoveď, EOS a koncové oddeľovače. Spúšťací skript kontroluje pôvodnú kontextovú kapacitu modelu podľa `cutoff_len`. Manifest zaznamená zdroj, seed, počty a kontrolné súčty. `dataset_info.json` registruje `aya_train`, `aya_eval` a `aya_test`. Súbory `*.metadata.jsonl` uchovávajú pôvod príkladov v rovnakom poradí. Dočasná SQLite databáza zabezpečuje deduplikáciu a rozdelenie bez držania celého datasetu v RAM.
`prepare_data.py` si načíta oba YAML configy a ich presné tokenizéry. **Celý pár vyradí, ak prekročí limit čo i len pri jednom modeli.** Všetky ponechané príklady zapíše bez orezania. Filtrovanie prebehne pred deduplikáciou a spoločným rozdelením na train/eval/test, takže oba modely dostanú rovnaké príklady. Inštrukcia a pole `input` sa spájajú rovnako ako v Alpaca konvertore LLaMA-Factory. ## 1. Príprava dát na CPU
Balík sa prenesie do `/mnt/project/perun26012578/perun`. Zdrojový Parquet je už v `/mnt/project/perun26012578/raw/slovak_train.parquet`. Pri príprave stačí Python s PyArrow, ktorý je dostupný aj v pripravenom kontajneri SK-bench.
Nastavenie jedného tréningu: Z prihlasovacieho uzla sa príprava odošle do `cpu_short`. Príkaz použije existujúci obraz a Python prostredie v `apphome`.
- štyri H200 cez DDP, `bf16: true`, `use_unsloth: true`, `use_unsloth_gc: true`, bez kvantizácie; ```bash
- batch 8 na GPU, akumulácia 1: **4 × 8 × 1 = efektívny batch 32**; cd /mnt/project/perun26012578/perun
- eval loss s batchom 8 na GPU, záverečné generovanie po jednom príklade; mkdir -p logs
- LoRA rank 16, alpha 32, dropout 0, projekcie attention aj MLP; export APPHOME=/mnt/home/jascke919/apphome
- jedna epocha, AdamW, cosine scheduler, warmup 8 %, clipping gradientov 0,5;
- loss iba z odpovedí, bez packingu, dynamický padding na dĺžku aktuálnej dávky;
- validácia/checkpoint každých 1 000 optimizer krokov, výber podľa najnižšieho loss, limit checkpointov 2.
sbatch --job-name=aya_prepare \
--account=perun26012578 --partition=cpu_short \
--nodes=1 --ntasks=1 --cpus-per-task=4 --mem=24G \
--time=08:00:00 --output=logs/aya_prepare_%j.out \
--wrap='apptainer exec --home "$APPHOME" \
--bind /mnt/project/perun26012578:/mnt/project/perun26012578 \
--pwd /mnt/project/perun26012578/perun "$APPHOME/cuda.sif" \
"$APPHOME/venv/bin/python" -u scripts/prepare_data.py \
--local-parquet /mnt/project/perun26012578/raw/slovak_train.parquet \
--output-dir data/aya_slovak_fixed'
```
Spúšťač overuje, že dáta majú záznam o dĺžkovom filtrovaní pre daný model, revíziu, šablónu a limit. Kontrola pri tokenizácii tiež zastaví beh, ak by predspracovanie chcelo orezať vstup alebo odpoveď. Konfigurácia načítaného základného modelu sa uloží do `adapter/base_model_config.json`. Po dokončení sa skutočné počty a kontrolné súčty nachádzajú v `data/aya_slovak_fixed/manifest.json`. Výstupný priečinok prípravy musí byť nový alebo prázdny. Pri vlastnom lokálnom JSONL slúži `--local-jsonl`. Bez lokálneho vstupu sa použije pripnutá revízia slovenskej Aya na Hugging Face.
LR `0.0001` pri Mistrale nadväzuje na 300k experiment; pri OLMo je východiskový kandidát. Zmenené nastavenia stále treba overiť podľa eval dát. OLMo je základný model pre vlastné SFT a jeho karta uvádza angličtinu. ## Prostredie pre tréning
## Prostredie na Perunovi Tréning používa samostatné prostredie s LLaMA-Factory, Unslothom a CUDA pre H200. Balík obsahuje zachytené verzie knižníc. Funkčnosť konkrétnej zostavy sa overí pri prvom pilote na Perunovi.
Na spustenie sa používa kompatibilné GPU prostredie s LLaMA-Factory, Unsloth, Unsloth Zoo, PyTorch a CUDA pre H200. LLaMA-Factory používa aj `datasets`, `huggingface_hub` a `PyYAML`; generatívna evaluácia potrebuje `rouge-chinese`, `jieba` a `nltk`. Python má byť aspoň 3.11 podľa aktuálnych požiadaviek LLaMA-Factory. Dĺžkový filter pri príprave dát používa Transformers a PyYAML a načíta oba tokenizéry; váhy modelov na túto CPU úlohu nepotrebuje.
### Požiadavky a inštalácia prostredí ### Požiadavky a inštalácia prostredí
@ -92,7 +86,7 @@ Na pridelenom GPU uzle sa overia importy, CUDA a BF16:
python -c 'import unsloth, torch, jieba, nltk; from rouge_chinese import Rouge; assert torch.cuda.is_available(); assert torch.cuda.is_bf16_supported(); print("GPU:", torch.cuda.get_device_name(0)); print("Viditelne GPU:", torch.cuda.device_count()); print("CUDA:", torch.version.cuda)' python -c 'import unsloth, torch, jieba, nltk; from rouge_chinese import Rouge; assert torch.cuda.is_available(); assert torch.cuda.is_bf16_supported(); print("GPU:", torch.cuda.get_device_name(0)); print("Viditelne GPU:", torch.cuda.device_count()); print("CUDA:", torch.version.cuda)'
``` ```
Príprava dát, tréning aj `scripts/evaluate.py` sa potom spúšťajú v `aya_train_perun` podľa ďalších častí dokumentácie. Overené opravy generovania z Titanu sú súčasťou `scripts/train.py`. Tréning aj `scripts/evaluate.py` sa potom spúšťajú v `aya_train_perun` podľa ďalších častí dokumentácie. Overené opravy generovania z Titanu sú súčasťou `scripts/train.py`.
#### lm-evaluation-harness #### lm-evaluation-harness
@ -110,231 +104,47 @@ lm_eval --help
Súbor inštaluje framework pri revízii `6d2abda4fd171e68a8789330c4149e37c1ca0bda`. Vlastné slovenské úlohy, zoznam benchmarkov a evaluačné dáta sa prenášajú samostatne; požiadavky obsahujú Python knižnice. Súbor inštaluje framework pri revízii `6d2abda4fd171e68a8789330c4149e37c1ca0bda`. Vlastné slovenské úlohy, zoznam benchmarkov a evaluačné dáta sa prenášajú samostatne; požiadavky obsahujú Python knižnice.
#### SK-bench ## 2. Pilot a tréning
Na obnovenie SK-bench sa používa uv 0.13.5 a zdrojová kópia projektu z Hrušky pri revízii `15823bbcc2d995e69441fcb7d34abb61bd8180c5`. Súbory `pyproject.toml` a `uv.lock` sa skopírujú z tohto balíka do koreňa zdrojovej kópie SK-bench. Ak je uv potrebné doplniť, nainštaluje sa cez `python -m pip install uv==0.13.5` v existujúcom Python prostredí; vlastné závislosti SK-bench sa vytvoria v jeho samostatnej `.venv`. Z koreňa balíka s aktivovaným tréningovým prostredím sa overí konfigurácia a integrita spoločných dát.
Z koreňa balíka, pri zdrojovej kópii SK-bench uloženej v `../sk-bench`:
```bash ```bash
cp requirements/skbench/pyproject.toml requirements/skbench/uv.lock ../sk-bench/ python scripts/train.py configs/ministral.yaml --dry-run
cd ../sk-bench
uv sync --locked --no-default-groups --python 3.13.17
uv pip check --python .venv/bin/python
uv run --locked python -c 'import torch, vllm, transformers; print(torch.__version__, vllm.__version__, transformers.__version__)'
``` ```
Základná uzamknutá zostava zodpovedá všetkým 308 balíkom exportovaným z Hrušky. Skupiny `dev` a `hpc` sa pri tomto obnovení nepridávajú. `requirements/skbench.txt` uchováva rovnaké konkrétne verzie v pip formáte; pri používaní projektu cez uv sa vychádza z jeho `uv.lock`. Plný tréning využíva štyri H200 na jednom uzle cez DDP. `train.py` spustí procesy sám. Slurm job preto požaduje jeden task a štyri GPU. Základná konfigurácia používa BF16, LoRA rank 16, alpha 32, batch 8 na GPU a efektívny batch 32. LR 0,0001 je východisková hodnota pre Ministral.
SK-bench používa CUDA 13; pri jeho prenose sa na pridelenom GPU uzle overí ovládač cez `nvidia-smi` a dostupnosť CUDA v PyTorchi. Pôvodný export Hrušky uvádza ovládač 580.178.04. Prvý pilot overí načítanie modelu, jeho natívny formát chatu, priebeh lossu a uloženie adaptéra. `--pilot-steps 30` obmedzí počet krokov a pri predvolenom splite použije najviac 2 048 tréningových riadkov z pevného poradia. Celý validačný split zostane dostupný. Pilot použije vlastný výstupný priečinok. LLaMA-Factory načíta zdrojový JSONL a obmedzí rozsah pred tokenizáciou.
Zdroje: [PyTorch CUDA zostavy](https://pytorch.org/get-started/previous-versions/#v260), [kompatibilita TorchAO](https://github.com/pytorch/ao/issues/2919), [závislosti Unsloth Zoo 2026.8.16](https://pypi.org/pypi/unsloth-zoo/2026.8.16/json), [zmeny setuptools](https://setuptools.pypa.io/en/latest/history.html#v82-0-0), [uzamknuté prostredia uv](https://docs.astral.sh/uv/concepts/projects/sync/).
### Odosielanie úloh cez Slurm
Priame Python príkazy sú určené pre už pridelený výpočtový uzol. Z prihlasovacieho uzla sa úlohy odosielajú cez `sbatch --wrap`. Príkazy sa zadávajú z koreňa balíka s aktivovaným Python prostredím; tento priečinok sa použije aj pri výpočte.
Pred odoslaním sa v rovnakom termináli nastavia projektový účet a QoS. Hodnoty `NAZOV_PROJEKTU` a `PRIDELENE_QOS` sa nahradia údajmi prideleného projektu:
```bash ```bash
AYA_ACCOUNT="NAZOV_PROJEKTU" sbatch --job-name=ministral_pilot \
AYA_QOS="PRIDELENE_QOS" --account=perun26012578 --partition=gpu_short \
--nodes=1 --ntasks=1 --gres=gpu:4 --cpus-per-task=32 --mem=128G \
--time=02:00:00 --output=logs/ministral_pilot_%j.out \
--wrap='python -u scripts/train.py configs/ministral.yaml --run-dir runs/ministral_pilot --pilot-steps 30'
``` ```
Na Perunovi sa pre výpočet používa pracovný priečinok na SCRATCH. Dáta a výsledky určené na uchovanie sa po výpočte skopírujú do projektového úložiska. Parameter `--time` určuje maximálny čas úlohy. Plný tréning sa spustí po kontrole pilotu a spresnení času. Pri dlhom behu sa checkpointy ukladajú každých 1 000 optimizer krokov a vyberá sa podľa najnižšieho validačného lossu.
## 1. Príprava dát
Príkazy sa spúšťajú z koreňa balíka s aktivovaným Python prostredím. Príprava dát používa CPU; na Perunovi sa vykonáva na výpočtovom uzle s pridelenými CPU.
```bash ```bash
python scripts/prepare_data.py --output-dir data/aya_slovak_4096 sbatch --job-name=ministral_train \
--account=perun26012578 --partition=gpu_long \
--nodes=1 --ntasks=1 --gres=gpu:4 --cpus-per-task=32 --mem=256G \
--time=2-00:00:00 --output=logs/ministral_train_%j.out \
--wrap='python -u scripts/train.py configs/ministral.yaml'
``` ```
Odoslanie prípravy dát ako CPU úlohy: Adaptér sa uloží do `runs/ministral/adapter`. V nadradenom priečinku zostanú `train.log`, `used_config.yaml` a `run.json` s identifikáciou datasetu. Pre iný beh sa zvolí nový `--run-dir`. Pri obnovení z checkpointu sa použije `--resume` s pôvodnými dátami a nastaveniami.
## 3. Vyhodnotenie
Po tréningu môže `evaluate.py` na jednej GPU vypočítať validačný loss a perplexitu a vygenerovať odpovede pre F1 a presnú zhodu. Pri generovaní sa ponechá priestor aspoň pre jeden nový token. Pri skórovaní generovaných odpovedí sa používajú úplné referencie zo spoločného JSONL. Výsledky obsahujú aj počty referencií, ktoré sa líšia od exportovaných labelov. Po výbere modelu sa môže použiť `--split test` s novým výstupným priečinkom.
```bash ```bash
sbatch --job-name=aya_prepare \ python scripts/evaluate.py configs/ministral.yaml --output-dir results/ministral_eval
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \ python scripts/evaluate.py configs/ministral.yaml --split test --output-dir results/ministral_test
--partition=cpu_short --nodes=1 --ntasks=1 \
--cpus-per-task=8 --mem=64G --time=12:00:00 \
--output=%x_%j.log \
--wrap='python -u scripts/prepare_data.py --output-dir data/aya_slovak_4096'
``` ```
Tréningové úlohy sa odosielajú po úspešnom dokončení prípravy dát. Externý SK-bench sa spúšťa zo samostatného klonu v `/mnt/project/perun26012578/sk-bench`. Na porovnanie s baseline sa použije rovnaká revízia SK-bench, rovnaké nastavenia generovania a rovnaký súbor `tasks_ministral_sk_baseline.txt` s 340 variantmi. Lokálne exportovaný model musí byť viditeľný z kontajnera. [SK-bench na Slurme](https://github.com/slovak-nlp/sk-bench/blob/main/container/slurm/README.md)
Výstupný priečinok prípravy dát musí byť nový alebo prázdny. Stav jobu sa sleduje cez `squeue -u "$USER"`. Pri chybách sa kontroluje log v `logs/`. Revízia Gitu sa v prípade potreby zistí pred odoslaním na prihlasovacom uzle.
### `prepare_data.py`
Jeho vstupom sú slovenské zdrojové dáta Aya Collection a obe modelové konfigurácie. Používa CPU a tokenizéry Mistralu a OLMo. Výsledkom je jeden spoločný dátový balík pre oba modely.
1. Načíta slovenský split `train` datasetu `CohereLabs/aya_collection_language_split` pri pevnej revízii. Zdroj číta postupne.
2. Vyradí neplatné riadky, prázdne zadania a prázdne odpovede. Polia `inputs` a `targets` prevedie na `instruction/input/output` a odstráni okrajové medzery.
3. Pomocou oboch tokenizérov zmeria celú SFT sekvenciu vrátane formátu Alpaca a odpovede. Ak príklad prekročí `cutoff_len` pri ktoromkoľvek modeli, vyradí celý pár.
4. Odstráni duplicitné normalizované páry zadania a odpovede. Rovnaké normalizované zadanie s rôznymi odpoveďami ponechá ako jednu skupinu.
5. Podľa seedu, predvolene `42`, vytvorí train/eval/test. Celá skupina zadania zostane v jednej množine. Do eval a test vyberie najviac 1 000 príkladov na každú množinu; ostatné ponechané príklady patria do train.
6. Uloží dáta vo formáte, ktorý následne načíta LLaMA-Factory.
Príklad uloženého riadka:
```json
{"instruction":"Aké je hlavné mesto Slovenska?","input":"","output":"Bratislava."}
```
Výstupy v `data/aya_slovak_4096`:
| Súbor | Obsah a použitie |
| --- | --- |
| `train.jsonl` | Príklady na učenie LoRA parametrov. |
| `eval.jsonl` | Validačné príklady na sledovanie loss a ladenie. |
| `test.jsonl` | Odložené príklady na záverečné vyhodnotenie. |
| `dataset_info.json` | Registrácia súborov pod názvami `aya_train`, `aya_eval`, `aya_test` a mapovanie ich stĺpcov pre LLaMA-Factory. |
| `manifest.json` | Zdrojová revízia, seed, skutočné počty, nastavenia dĺžkového filtra a kontrolné súčty súborov. |
| `*.metadata.jsonl` | Pôvod a identifikácia príkladov v rovnakom poradí ako v príslušnom dátovom súbore. |
| `excluded_long.jsonl` | Identifikácia a namerané dĺžky vyradených dlhých príkladov. |
Na deduplikáciu a rozdelenie dát používa dočasnú SQLite databázu. Po úspešnom dokončení ju odstráni; prepínač `--keep-db` ju ponechá. Počty eval a test príkladov možno nastaviť cez `--eval-size` a `--test-size`, seed cez `--seed`. Vlastný vstupný JSONL možno načítať cez `--local-jsonl`.
Oba modely potom používajú tie isté výsledné súbory. Každý model si ich pri tréningu tokenizuje svojím tokenizérom. LLaMA-Factory používa pripravené `aya_train` a `aya_eval` bez ďalšieho automatického delenia cez `val_size`.
## 2. Tréning
### `train.py`
Je to spúšťač tréningu v LLaMA-Factory. Ako vstup dostane modelový YAML a už pripravený dátový balík.
1. Načíta model, revíziu, cesty k dátam, LoRA parametre, learning rate, batch a ostatné nastavenia z YAML.
2. Skontroluje existenciu dát, ich kontrolné súčty a zhodu dĺžkového filtra s použitým modelom, revíziou, šablónou a limitom.
3. Uloží presnú použitú konfiguráciu a záznam behu.
4. Cez `torchrun` spustí štyri procesy DDP a v každom zavolá tréning LLaMA-Factory.
Pri aktuálnych konfiguráciách prebieha SFT s LoRA, Unslothom a BF16. Základné váhy modelu sú zmrazené a učia sa LoRA parametre. Loss sa počíta z odpovedí. LLaMA-Factory každých 1 000 optimizer krokov vykoná validáciu a uloží checkpoint; na konci načíta checkpoint s najnižším eval loss.
Kontrola pomocou `--dry-run` sa vykonáva po príprave dát. Overí základné nastavenia a dátové súbory bez načítania váh a bez spustenia tréningu.
Kontrola konfigurácie a pripravených dát bez spustenia tréningu:
```bash
python scripts/train.py configs/mistral.yaml --dry-run
python scripts/train.py configs/olmo.yaml --dry-run
```
Každý tréning sa spúšťa samostatne v prostredí so štyrmi dostupnými GPU:
```bash
python scripts/train.py configs/mistral.yaml
python scripts/train.py configs/olmo.yaml
```
Odoslanie tréningov cez Slurm na štyri H200. Nasledujúci príklad spustí modely postupne; `afterok` umožní spustenie OLMo po úspešnom dokončení Mistralu:
```bash
AYA_MISTRAL_JOB=$(sbatch --parsable --job-name=aya_mistral \
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
--partition=gpu_long --nodes=1 --ntasks=1 --gres=gpu:4 \
--cpus-per-task=32 --mem=256G --time=2-00:00:00 \
--output=%x_%j.log \
--wrap='python -u scripts/train.py configs/mistral.yaml')
sbatch --job-name=aya_olmo \
--dependency="afterok:$AYA_MISTRAL_JOB" \
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
--partition=gpu_long --nodes=1 --ntasks=1 --gres=gpu:4 \
--cpus-per-task=32 --mem=256G --time=2-00:00:00 \
--output=%x_%j.log \
--wrap='python -u scripts/train.py configs/olmo.yaml'
```
Jedna úloha Slurmu spustí `train.py`, ktorý vytvorí štyri procesy DDP. Parametre `--run-dir` a `--resume` sa pri odosielaní uvádzajú v Python príkaze vo vnútri `--wrap`.
Každý proces DDP má na pridelenej GPU kópiu modelu a spracúva inú časť dávky. Gradienty sa synchronizujú medzi GPU, takže štyri GPU vykonávajú jeden spoločný tréning. Pri batchi 8 na GPU a akumulácii 1 je efektívny batch **4 × 8 × 1 = 32**. Každý proces importuje Unsloth pred Transformers a LLaMA-Factory.
Výsledkom tréningu je **LoRA adaptér**, ktorý sa používa spolu so základným modelom. Pre Mistral sa uloží do `runs/mistral/adapter`, pre OLMo do `runs/olmo/adapter`. Checkpointy sú v podpriečinkoch adaptéra.
V nadradenom priečinku behu zostanú:
- `train.log` – priebeh a hlásenia tréningu;
- `used_config.yaml` – presné nastavenia odovzdané LLaMA-Factory;
- `run.json` – záznam behu, efektívny batch a identifikácia použitých dát.
## 3. Vyhodnotenie na eval dátach
### `evaluate.py`
Ako vstup dostane rovnaký modelový YAML, pripravené dáta a dotrénovaný adaptér. Načíta základný model spolu s adaptérom a vykoná dve samostatné fázy na jednej GPU:
1. **Eval loss a perplexita:** model spracuje zadania s pôvodnými referenčnými odpoveďami. Vypočíta sa strata pri predpovedaní odpovedí a z nej perplexita.
2. **Generovanie a porovnanie odpovedí:** model dostane zadania a vytvorí vlastné odpovede. Skript ich porovná s úplnými referenciami pomocou slovného F1 a normalizovanej presnej zhody.
Predvolene sa vyhodnotí pripravená množina `eval.jsonl`:
```bash
python scripts/evaluate.py configs/mistral.yaml --output-dir results/mistral_eval
python scripts/evaluate.py configs/olmo.yaml --output-dir results/olmo_eval
```
Po úspešnom dokončení príslušného tréningu sa vyhodnotenie odošle ako úloha s jednou GPU:
```bash
sbatch --job-name=aya_mistral_eval \
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
--partition=gpu_short --nodes=1 --ntasks=1 --gres=gpu:1 \
--cpus-per-task=8 --mem=64G --time=12:00:00 \
--output=%x_%j.log \
--wrap='python -u scripts/evaluate.py configs/mistral.yaml --output-dir results/mistral_eval'
sbatch --job-name=aya_olmo_eval \
--account="$AYA_ACCOUNT" --qos="$AYA_QOS" \
--partition=gpu_short --nodes=1 --ntasks=1 --gres=gpu:1 \
--cpus-per-task=8 --mem=64G --time=12:00:00 \
--output=%x_%j.log \
--wrap='python -u scripts/evaluate.py configs/olmo.yaml --output-dir results/olmo_eval'
```
Pre vyhodnotenie test množiny sa do Python príkazu vo vnútri `--wrap` pridá `--split test` a zvolí sa nový výstupný priečinok, napríklad `results/mistral_test`.
V priečinku behu sa uchovávajú pôvodné súbory `run.json` a `used_config.yaml`. Vyhodnotenie porovná manifest pripravených dát so záznamom z tréningu; iné rozdelenie alebo iný dataset odmietne. Rovnaká kontrola platí pre eval aj test.
Adaptér z iného behu sa určí parametrom `--adapter runs/mistral_dalsi/adapter`. Predvolene sa použije adaptér z `output_dir` v modelovom YAML. Prepínač `--dry-run` overí pripravené vstupy a zobrazí odvodené evaluačné nastavenia.
Generovanie je greedy, s jedným lúčom a limitom najviac 256 nových tokenov. Limit možno zmeniť cez `--max-new-tokens`. Pri každom vstupe sa počet nových tokenov obmedzí aj zostávajúcim miestom do 4 096 tokenov. Celý vstup a úplná referencia zostávajú zachované; prípadné zníženie limitu sa zapíše do logu.
Skript zapisuje výsledky do priečinka z `--output-dir`:
| Súbor | Obsah súboru |
| --- | --- |
| `summary.json` | Hlavný súhrn: eval loss, perplexita, F1, presná zhoda a počet vyhodnotených príkladov. |
| `loss/eval_results.json` | Výsledky samostatného výpočtu eval loss v LLaMA-Factory. |
| `generation/generated_predictions.jsonl` | Vygenerované odpovede modelu. |
| `scores/per_sample.jsonl` | Predikcia, úplná referencia a skóre pre každý príklad. |
| `scores/metrics.json` | Podrobné súhrny metrík aj rozdelenie podľa typu úlohy. |
| `evaluation.json` | Zvolený split, modelový config, adaptér a stav vyhodnotenia. |
Pri výsledkoch zostanú aj logy a konkrétne použité konfigurácie oboch evaluačných fáz. Pre vyhodnotenie možno ako vstupný config použiť aj `runs/mistral/used_config.yaml` alebo OLMo ekvivalent.
Eval slúži na ladenie a výber checkpointu. Po ukončení výberu nastavení sa vyhodnotí odložená testovacia množina a výsledky sa uložia do nového výstupného priečinka:
```bash
python scripts/evaluate.py configs/mistral.yaml --split test --output-dir results/mistral_test
python scripts/evaluate.py configs/olmo.yaml --split test --output-dir results/olmo_test
```
Externé lm-eval a SK-bench majú vlastné spúšťanie. Tento balík obsahuje prípravu dát, tréning a vyhodnotenie na pripravených eval/test dátach.
### Sledovanie odoslaných úloh
```bash
squeue -u "$USER"
```
Výstup úlohy sa zapisuje do logu v pracovnom priečinku. V názve `%x_%j.log` sa `%x` nahradí názvom úlohy a `%j` jej číslom. Pre prípravu dát s číslom úlohy 12345:
```bash
tail -f aya_prepare_12345.log
```

58
perun/configs/ministral.yaml Normal file
View File

@ -0,0 +1,58 @@
# SFT pôvodného Ministral-3b-instruct. Šablóna sa odvodí z jeho tokenizéra.
model_name_or_path: "ministral/Ministral-3b-instruct"
model_revision: "2c95908929198d6e69af8638f0dbbd9bc6b93f9e"
stage: sft
do_train: true
do_eval: true
finetuning_type: lora
use_unsloth: true
use_unsloth_gc: true
bf16: true
infer_dtype: bfloat16
flash_attn: sdpa
dataset_dir: data/aya_slovak_fixed
dataset: aya_train
eval_dataset: aya_eval
cutoff_len: 4096
packing: false
train_on_prompt: false
preprocessing_num_workers: 8
dataloader_num_workers: 2
overwrite_cache: false
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
gradient_accumulation_steps: 1
learning_rate: 0.0001
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.08
optim: adamw_torch
weight_decay: 0.0
max_grad_norm: 0.5
gradient_checkpointing: true
ddp_find_unused_parameters: false
output_dir: runs/ministral/adapter
overwrite_output_dir: false
logging_steps: 10
eval_strategy: steps
eval_steps: 1000
save_strategy: steps
save_steps: 1000
save_total_limit: 2
save_only_model: false
load_best_model_at_end: true
metric_for_best_model: loss
greater_is_better: false
predict_with_generate: false
plot_loss: true
report_to: none
seed: 42
data_seed: 42

2
perun/configs/mistral.yaml
View File

@ -12,7 +12,7 @@ bf16: true
infer_dtype: bfloat16 infer_dtype: bfloat16
flash_attn: sdpa flash_attn: sdpa
dataset_dir: data/aya_slovak_4096 dataset_dir: data/aya_slovak_fixed
dataset: aya_train dataset: aya_train
eval_dataset: aya_eval eval_dataset: aya_eval
cutoff_len: 4096 cutoff_len: 4096

2
perun/configs/olmo.yaml
View File

@ -12,7 +12,7 @@ bf16: true
infer_dtype: bfloat16 infer_dtype: bfloat16
flash_attn: sdpa flash_attn: sdpa
dataset_dir: data/aya_slovak_4096 dataset_dir: data/aya_slovak_fixed
dataset: aya_train dataset: aya_train
eval_dataset: aya_eval eval_dataset: aya_eval
cutoff_len: 4096 cutoff_len: 4096

3
perun/ministral_3b_aya/.gitignore vendored Normal file
View File

@ -0,0 +1,3 @@
/data/
/runs/
__pycache__/

78
perun/ministral_3b_aya/README.md Normal file
View File

@ -0,0 +1,78 @@
# Perun – dotrénovanie Ministralu 3B na slovenskej Aya
Model [`ministral/Ministral-3b-instruct`](https://huggingface.co/ministral/Ministral-3b-instruct) bol dotrénovaný metódou SFT s LoRA cez LLaMA-Factory a Unsloth. Použila sa základná revízia `2c95908929198d6e69af8638f0dbbd9bc6b93f9e`. Behy prebehli na Perune.
## Dáta a tréning
Slovenská časť Aya Collection bola načítaná z Parquetu. Po vyradení prázdnych alebo neplatných záznamov a odstránení duplicitných párov zostalo 3 492 041 príkladov. Rovnaké normalizované prompty sa držali v jednej skupine pri delení dát.
| Množina | Príklady |
|---|---:|
| Tréningová | 3 490 041 |
| Validačná | 1 000 |
| Testovacia | 1 000 |
[Manifest dát](training/data_manifest.json) obsahuje počty, seed a kontrolné súčty vstupu aj pripravených súborov. Jeho SHA-256 sa zhoduje s hodnotou zaznamenanou v [metadátach tréningu](training/run.json).
| Nastavenie jobu 148065 | Hodnota |
|---|---|
| Hardvér a presnosť | 4× NVIDIA H200 na jednom uzle, BF16 |
| Tréning | 1 epocha, 109 064 krokov |
| Efektívna dávka | 32 príkladov, 8 na GPU a akumulácia 1 |
| LoRA | Rank 16, alpha 32, dropout 0 |
| Learning rate | `0.0002`, kosínusový plán a warmup 8 % |
| Kontext | 4 096 tokenov |
| Seed | 42 pre model aj dáta |
Tréningová slučka trvala **24 h 06 min**. Priemerný tréningový loss bol **1,33344**. Najnižší validačný loss bol **1,13089** v kroku **109 000**. Posledný checkpoint z kroku **109 064** mal validačný loss **1,13097**. Na testovanie Aya aj porovnanie v SK-bench sa použil posledný checkpoint.
[Tréningové metriky](training/train_results.json) · [Validačný priebeh](training/validation_loss.csv) · [Graf tréningového lossu](training/training_loss.png) · [Graf validačného lossu](training/training_eval_loss.png)
## Test na odloženej Aya množine
Job 152588 vyhodnotil checkpoint 109 064 na 1 000 testovacích príkladoch. Pri generovaní bol limit 256 nových tokenov. Pri jednom príklade zostala po normalizácii prázdna referencia, takže tokenové F1 a presná zhoda používajú 999 hodnotiteľných odpovedí.
| Metrika | Hodnota |
|---|---:|
| Testovací loss | 1,08998 |
| Perplexita | 2,97423 |
| Tokenové F1 (0–100) | 35,06 |
| Normalizovaná presná zhoda | 3,60 % (36 z 999) |
[Súhrn testu](aya_test/summary.json) · [Konfigurácia výpočtu lossu](aya_test/eval_loss.yaml) · [Konfigurácia generovania](aya_test/eval_generate.yaml)
## Čiastočné výsledky SK-bench
Pôvodný Ministral a model so zlúčeným adaptérom z checkpointu 109 064 boli vyhodnotené na **rovnakých 385 z 412 slovenských zero-shot variantov**, ktoré pokrývajú **29 z 33 úloh**. Výsledok spája pôvodných 340 variantov (joby 145668 a 152980) a 45 doplnených variantov (joby 157393 a 157394).
| Skupina | Pokrytie | Pôvodný | Dotrénovaný | Rozdiel |
|---|---:|---:|---:|---:|
| Knowledge & Reasoning | 10/10 | 20,57 | 23,37 | +2,80 |
| Reading & QA | 8/9 | 7,42 | 12,53 | +5,11 |
| Classification | 8/9 | 0,00 | 23,34 | +23,34 |
| Instruction | 1/1 | 7,82 | 10,24 | +2,42 |
| Grammar & Morphology | 2/2 | 0,12 | 0,88 | +0,76 |
| **Overall** | **5/6 skupín** | **7,18** | **14,07** | **+6,89** |
Chýbajú OneRuler-SK (18 variantov), NLI-SK-annotated (3), SMESum (3) a SlovakSum (3). Skupina Generation preto nemá skóre. Overall je priemer piatich dostupných skupín s rovnakou váhou, vypočítaný postupom verejného leaderboardu. Dotrénovaný model sa zlepšil v 26 z 29 úloh. Najväčší pokles bol pri SKJ1, z 59,67 na 0,00.
[Skóre po úlohách](sk_bench/scores.csv) · [Zoznam variantov](sk_bench/tasks_sk0_385.txt) · [Agregát pôvodného modelu](sk_bench/aggregated/models__ministral-base.json) · [Agregát dotrénovaného modelu](sk_bench/aggregated/models__ministral-last-109064.json)
Agregované JSON-y vznikli oficiálnym skriptom z revízie SK-bench `15823bbcc2d995e69441fcb7d34abb61bd8180c5`. CSV používa hlavné metriky úloh z revízie leaderboardu `833bd28e96fd392b7064de053084dc0a6ca9a3e3`. Hodnoty sú na škále 0–100 a rozdiel je v bodoch. [Agregačný skript](https://github.com/slovak-nlp/sk-bench/blob/15823bbcc2d995e69441fcb7d34abb61bd8180c5/scripts/generate_final_results.py) · [Výpočet leaderboardu](https://huggingface.co/spaces/slovak-nlp/sk-bench-leaderboard/blob/833bd28e96fd392b7064de053084dc0a6ca9a3e3/backend/app/leaderboard.py)
## Spustenie tréningu a testu
Príkazy sa spúšťajú z tohto priečinka v pripravenom prostredí s [tréningovými závislosťami](requirements/training.txt). Príprava používa pôvodný slovenský Parquet. Tréningový skript spúšťa štyri procesy na uzle so štyrmi GPU. Evaluácia testovacej množiny sa spúšťa na jednej GPU.
```bash
python scripts/prepare_data.py --local-parquet /cesta/k/slovak_train.parquet \
--output-dir data/aya_slovak_fixed --seed 42 --eval-size 1000 --test-size 1000
python scripts/train.py configs/input_config.yaml --run-dir runs/ministral_aya
python scripts/evaluate.py runs/ministral_aya/used_config.yaml \
--adapter runs/ministral_aya/adapter/checkpoint-109064 \
--split test --output-dir runs/ministral_aya/evaluations/aya_test --max-new-tokens 256
```
[`input_config.yaml`](configs/input_config.yaml) je pôvodný vstup a `--run-dir` určuje nový výstupný priečinok. [`used_config.yaml`](configs/used_config.yaml) zaznamenáva konfiguráciu skutočne použitú pri jobe 148065 vrátane ciest na Perune.

28
perun/ministral_3b_aya/aya_test/eval_generate.yaml Normal file
View File

@ -0,0 +1,28 @@
model_name_or_path: ministral/Ministral-3b-instruct
model_revision: 2c95908929198d6e69af8638f0dbbd9bc6b93f9e
stage: sft
finetuning_type: lora
dataset_dir: /mnt/project/perun26012578/perun/data/aya_slovak_fixed
cutoff_len: 4096
packing: false
train_on_prompt: false
bf16: true
infer_dtype: bfloat16
flash_attn: sdpa
use_unsloth: true
seed: 42
data_seed: 42
preprocessing_num_workers: 8
dataloader_num_workers: 2
report_to: none
adapter_name_or_path: /mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/adapter/checkpoint-109064
eval_dataset: aya_test
use_unsloth_gc: false
overwrite_output_dir: false
output_dir: /mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/evaluations/aya_test_last_109064/generation
do_predict: true
predict_with_generate: true
per_device_eval_batch_size: 1
do_sample: false
num_beams: 1
max_new_tokens: 256

25
perun/ministral_3b_aya/aya_test/eval_loss.yaml Normal file
View File

@ -0,0 +1,25 @@
model_name_or_path: ministral/Ministral-3b-instruct
model_revision: 2c95908929198d6e69af8638f0dbbd9bc6b93f9e
stage: sft
finetuning_type: lora
dataset_dir: /mnt/project/perun26012578/perun/data/aya_slovak_fixed
cutoff_len: 4096
packing: false
train_on_prompt: false
bf16: true
infer_dtype: bfloat16
flash_attn: sdpa
use_unsloth: true
seed: 42
data_seed: 42
preprocessing_num_workers: 8
dataloader_num_workers: 2
report_to: none
adapter_name_or_path: /mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/adapter/checkpoint-109064
eval_dataset: aya_test
use_unsloth_gc: false
overwrite_output_dir: false
output_dir: /mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/evaluations/aya_test_last_109064/loss
do_eval: true
predict_with_generate: false
per_device_eval_batch_size: 8

9
perun/ministral_3b_aya/aya_test/evaluation.json Normal file
View File

@ -0,0 +1,9 @@
{
"status": "complete",
"split": "test",
"adapter": "/mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/adapter/checkpoint-109064",
"config": "/mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/used_config.yaml",
"gpus": 1,
"backend": "Unsloth",
"precision": "bf16"
}

13
perun/ministral_3b_aya/aya_test/summary.json Normal file
View File

@ -0,0 +1,13 @@
{
"model": "ministral/Ministral-3b-instruct",
"split": "test",
"eval_loss": 1.089984655380249,
"perplexity": 2.974228433808542,
"token_f1": 35.06239252655205,
"normalized_exact_match": 3.6036036036036037,
"examples": 1000,
"metric_scale": "0..100",
"max_new_tokens": 256,
"cutoff_len": 4096,
"generation_context_limit": 4096
}

53
perun/ministral_3b_aya/configs/input_config.yaml Normal file
View File

@ -0,0 +1,53 @@
model_name_or_path: ministral/Ministral-3b-instruct
model_revision: 2c95908929198d6e69af8638f0dbbd9bc6b93f9e
stage: sft
do_train: true
do_eval: true
finetuning_type: lora
use_unsloth: true
use_unsloth_gc: true
bf16: true
infer_dtype: bfloat16
flash_attn: sdpa
dataset_dir: data/aya_slovak_fixed
dataset: aya_train
eval_dataset: aya_eval
cutoff_len: 4096
packing: false
train_on_prompt: false
preprocessing_num_workers: 8
dataloader_num_workers: 2
overwrite_cache: false
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
gradient_accumulation_steps: 1
learning_rate: 0.0002
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.08
optim: adamw_torch
weight_decay: 0.0
max_grad_norm: 0.5
gradient_checkpointing: true
ddp_find_unused_parameters: false
output_dir: /mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/adapter
overwrite_output_dir: false
logging_steps: 10
eval_strategy: steps
eval_steps: 1000
save_strategy: steps
save_steps: 1000
save_total_limit: 2
save_only_model: false
load_best_model_at_end: true
metric_for_best_model: loss
greater_is_better: false
predict_with_generate: false
plot_loss: true
report_to: none
seed: 42
data_seed: 42

53
perun/ministral_3b_aya/configs/used_config.yaml Normal file
View File

@ -0,0 +1,53 @@
model_name_or_path: ministral/Ministral-3b-instruct
model_revision: 2c95908929198d6e69af8638f0dbbd9bc6b93f9e
stage: sft
do_train: true
do_eval: true
finetuning_type: lora
use_unsloth: true
use_unsloth_gc: true
bf16: true
infer_dtype: bfloat16
flash_attn: sdpa
dataset_dir: /mnt/project/perun26012578/perun/data/aya_slovak_fixed
dataset: aya_train
eval_dataset: aya_eval
cutoff_len: 4096
packing: false
train_on_prompt: false
preprocessing_num_workers: 8
dataloader_num_workers: 2
overwrite_cache: false
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
gradient_accumulation_steps: 1
learning_rate: 0.0002
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.08
optim: adamw_torch
weight_decay: 0.0
max_grad_norm: 0.5
gradient_checkpointing: true
ddp_find_unused_parameters: false
output_dir: /mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/adapter
overwrite_output_dir: false
logging_steps: 10
eval_strategy: steps
eval_steps: 1000
save_strategy: steps
save_steps: 1000
save_total_limit: 2
save_only_model: false
load_best_model_at_end: true
metric_for_best_model: loss
greater_is_better: false
predict_with_generate: false
plot_loss: true
report_to: none
seed: 42
data_seed: 42

148
perun/ministral_3b_aya/requirements/training.txt Normal file
View File

@ -0,0 +1,148 @@
# Python 3.11.16; CUDA 12.4.
accelerate==1.11.0
aiofiles==24.1.0
aiohappyeyeballs==2.7.1
aiohttp==3.14.3
aiosignal==1.4.0
alembic==1.20.0
annotated-doc==0.0.5
annotated-types==0.8.0
antlr4-python3-runtime==4.9.3
anyio==4.14.2
attrs==26.1.0
av==16.0.0
bitsandbytes==0.50.2
brotli==1.2.0
certifi==2026.7.22
charset-normalizer==3.5.1
click==8.5.0
colorlog==6.12.0
contourpy==1.3.3
cut-cross-entropy==25.1.1
cycler==0.12.1
datasets==3.6.0
diffusers==0.40.0
dill==0.3.8
docstring_parser==0.18.0
einops==0.8.2
fastapi==0.141.1
ffmpy==1.0.0
filelock==3.32.3
fire==0.7.1
fonttools==4.63.0
frozenlist==1.8.0
fsspec==2025.3.0
gradio==5.50.0
gradio_client==1.14.0
groovy==0.1.2
h11==0.16.0
hf-xet==1.6.0
hf_transfer==0.1.9
httpcore==1.0.9
httpx==0.28.1
huggingface_hub==1.29.0
idna==3.19
importlib_metadata==9.0.1
Jinja2==3.1.6
kiwisolver==1.5.1
llamafactory @ git+https://github.com/hiyouga/LLaMA-Factory.git@7fcf5b3b130e5713b52415bb7404c476fada9c8c
Mako==1.4.3
markdown-it-py==4.2.0
MarkupSafe==3.0.3
matplotlib==3.11.1
mdurl==0.1.2
modelscope==1.39.1
modelscope-hub==0.3.0
mpmath==1.3.0
msgspec==0.21.1
multidict==6.7.1
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.6.1
numpy==2.4.6
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-cusparselt-cu12==0.6.2
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
omegaconf==2.3.1
optuna==5.0.0
orjson==3.12.0
packaging==26.3
pandas==2.3.3
peft==0.18.1
pillow==11.3.0
pip==26.2.1
propcache==0.5.2
protobuf==7.36.0
psutil==7.2.2
pyarrow==25.0.1
pydantic==2.12.3
pydantic_core==2.41.4
pydub==0.25.1
Pygments==2.21.0
pyparsing==3.3.2
python-dateutil==2.9.0.post0
python-multipart==0.0.32
pytz==2026.3.post1
PyYAML==6.0.3
regex==2026.7.19
requests==2.34.2
rich==15.0.0
ruff==0.16.5
safehttpx==0.1.7
safetensors==0.8.0
scipy==1.17.1
semantic-version==2.10.0
sentencepiece==0.2.2
setuptools==80.10.2
shellingham==1.5.4
shtab==1.12.0
six==1.17.0
SQLAlchemy==2.1.4
sse-starlette==3.4.8
starlette==0.52.1
sympy==1.13.1
termcolor==3.3.0
tiktoken==0.14.0
tokenizers==0.22.2
tomlkit==0.13.3
torch==2.6.0+cu124
torchaudio==2.6.0+cu124
torchdata==0.11.0
torchvision==0.21.0+cu124
tqdm==4.70.0
transformers==5.5.0
triton==3.2.0
trl==0.24.0
typer==0.27.2
typing-inspection==0.4.4
typing_extensions==4.16.0
tyro==0.8.14
tzdata==2026.3
unsloth==2026.6.1
unsloth_zoo==2026.8.16
urllib3==2.7.0
uvicorn==0.52.4
websockets==15.0.1
wheel==0.47.0
xformers==0.0.29.post3
xxhash==4.0.1
yarl==1.24.5
zipp==4.1.0
# Doplnenia pre generativnu evaluaciu a deklarovane zavislosti Zoo:
jieba==0.42.1
rouge-chinese==1.0.3
nltk==3.10.3
joblib==1.6.0
defusedxml==0.7.1
torchao==0.13.0

301
perun/ministral_3b_aya/scripts/evaluate.py Normal file
View File

@ -0,0 +1,301 @@
#!/usr/bin/env python3
"""Evaluate the trained adapter on the prepared eval or test split; report loss, F1 and exact match."""
from __future__ import annotations
import argparse
from collections import Counter, defaultdict
import hashlib
import json
import math
import os
from pathlib import Path
import subprocess
import sys
import unicodedata
from train import ROOT, read_config, write_config, verify_data, run_logged
def normalized_tokens(text: str) -> list[str]:
text = unicodedata.normalize("NFKC", text).casefold()
return "".join(
" " if unicodedata.category(char).startswith("P") else char
for char in text
).split()
def score_pair(prediction: str, reference: str) -> dict:
predicted = normalized_tokens(prediction)
expected = normalized_tokens(reference)
if not expected:
return {
"token_f1": None,
"normalized_exact_match": None,
"scored": False,
"skip_reason": "reference_has_zero_normalized_tokens",
"prediction_tokens": len(predicted),
"reference_tokens": 0,
}
shared = sum((Counter(predicted) & Counter(expected)).values())
f1 = 200.0 * shared / (len(predicted) + len(expected))
return {
"token_f1": f1,
"normalized_exact_match": 100.0 if predicted == expected else 0.0,
"scored": True,
"skip_reason": None,
"prediction_tokens": len(predicted),
"reference_tokens": len(expected),
}
def read_jsonl(path: Path) -> list[dict]:
result = []
with path.open("r", encoding="utf-8-sig") as handle:
for line_number, line in enumerate(handle, 1):
if not line.strip():
continue
try:
row = json.loads(line)
except json.JSONDecodeError as exc:
raise ValueError(f"{path}:{line_number}: invalid JSON: {exc.msg}") from exc
if not isinstance(row, dict):
raise ValueError(f"{path}:{line_number}: expected a JSON object")
result.append(row)
return result
def required_text(row: dict, field: str, context: str) -> str:
value = row.get(field)
if not isinstance(value, str):
raise ValueError(f"{context}: {field!r} must contain a string")
return value
def reference_text(row: dict, index: int) -> str:
for field in ("output", "response", "label"):
if field in row:
return required_text(row, field, f"reference row {index}")
raise ValueError(f"reference row {index}: missing output/response/label")
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()
def verify_training_dataset(adapter: Path, data_dir: Path) -> Path:
run_dir = adapter.parent
if adapter.name.startswith("checkpoint-") and not (run_dir / "run.json").is_file():
run_dir = run_dir.parent
record_path = run_dir / "run.json"
if not record_path.is_file():
raise ValueError(f"Missing training record: {record_path}. Keep the adapter with its original run files.")
record = json.loads(record_path.read_text(encoding="utf-8"))
if record.get("data_manifest_sha256") != sha256(data_dir / "manifest.json"):
raise ValueError("The prepared dataset differs from this adapter's training data. "
"Use the original dataset and its eval/test split recorded in run.json.")
return run_dir
def summarize(rows: list[dict]) -> dict:
scored = [row for row in rows if row["scored"]]
count = len(scored)
return {
"count_total": len(rows),
"count_scored": count,
"count_skipped_zero_token_references": len(rows) - count,
"count_empty_predictions_raw": sum(not row["prediction"].strip() for row in rows),
"count_zero_token_predictions": sum(row["prediction_tokens"] == 0 for row in rows),
"count_nonempty_predictions_raw": sum(bool(row["prediction"].strip()) for row in rows),
"count_truncated_or_modified_labels": sum(
not row["label_matches_full_reference"] for row in rows
),
"count_labels_different_before_normalization": sum(
not row["label_matches_full_reference_raw"] for row in rows
),
"token_f1": sum(row["token_f1"] for row in scored) / count if count else None,
"normalized_exact_match": (
sum(row["normalized_exact_match"] for row in scored) / count if count else None
),
"prediction_tokens_total": sum(row["prediction_tokens"] for row in rows),
"reference_tokens_total": sum(row["reference_tokens"] for row in rows),
}
def score_files(
predictions_path: Path,
references_path: Path,
output_dir: Path,
metadata_path: Path | None = None,
) -> dict:
predictions = read_jsonl(predictions_path)
references = read_jsonl(references_path)
if not references:
raise ValueError("The reference file contains no examples")
if len(predictions) != len(references):
raise ValueError(
f"Row count mismatch: {len(predictions)} predictions != {len(references)} references. "
"No scores written; evaluate the complete split in the original order."
)
metadata = read_jsonl(metadata_path) if metadata_path is not None else [{} for _ in references]
if len(metadata) != len(references):
raise ValueError(
f"Row count mismatch: {len(metadata)} metadata rows != {len(references)} references"
)
rows = []
groups = defaultdict(list)
for index, (prediction_row, reference_row, meta) in enumerate(
zip(predictions, references, metadata)
):
prediction = required_text(prediction_row, "predict", f"prediction row {index}")
label = required_text(prediction_row, "label", f"prediction row {index}")
reference = reference_text(reference_row, index)
if "row_index" in meta and meta["row_index"] != index:
raise ValueError(f"metadata row {index}: row_index does not match the reference order")
source_metadata = meta.get("source_metadata", {})
if not isinstance(source_metadata, dict):
raise ValueError(f"metadata row {index}: source_metadata must be an object")
task_type = meta.get(
"task_type", source_metadata.get("task_type", reference_row.get("task_type", "unknown"))
)
if not isinstance(task_type, str):
raise ValueError(f"metadata row {index}: task_type must be a string")
row = {
"id": meta.get("stable_id", meta.get("id", reference_row.get("id", index))),
"index": index,
"task_type": task_type,
"prediction": prediction,
"reference": reference,
"exported_label": label,
"label_matches_full_reference": normalized_tokens(label) == normalized_tokens(reference),
"label_matches_full_reference_raw": label == reference,
**score_pair(prediction, reference),
}
rows.append(row)
groups[task_type].append(row)
metrics = {
"schema_version": 1,
"metric_scale": "0..100",
"metric_version": "aya_overlap_v1",
"overall": summarize(rows),
"by_task_type": {name: summarize(group) for name, group in sorted(groups.items())},
"inputs": {
"predictions": {"path": str(predictions_path.resolve()), "sha256": sha256(predictions_path)},
"references": {"path": str(references_path.resolve()), "sha256": sha256(references_path)},
},
}
if metadata_path is not None:
metrics["inputs"]["metadata"] = {"path": str(metadata_path.resolve()), "sha256": sha256(metadata_path)}
if output_dir.exists() and (not output_dir.is_dir() or any(output_dir.iterdir())):
raise ValueError(f"Output directory must be new or empty: {output_dir}")
output_dir.mkdir(parents=True, exist_ok=True)
with (output_dir / "per_sample.jsonl").open("w", encoding="utf-8", newline="\n") as handle:
for row in rows:
handle.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n")
(output_dir / "metrics.json").write_text(
json.dumps(metrics, ensure_ascii=False, indent=2, allow_nan=False) + "\n", encoding="utf-8"
)
return metrics
def evaluation_configs(training: dict, adapter: Path, output_dir: Path, split: str, max_new_tokens: int):
keys = ("model_name_or_path", "model_revision", "stage", "finetuning_type", "template",
"dataset_dir", "cutoff_len", "packing", "train_on_prompt",
"bf16", "infer_dtype", "flash_attn", "use_unsloth", "rope_scaling", "seed", "data_seed",
"preprocessing_num_workers", "dataloader_num_workers", "report_to")
base = {key: training[key] for key in keys if key in training}
base.update({"adapter_name_or_path": str(adapter), "eval_dataset": f"aya_{split}",
"use_unsloth_gc": False, "overwrite_output_dir": False})
loss = dict(base, output_dir=str(output_dir / "loss"), do_eval=True,
predict_with_generate=False, per_device_eval_batch_size=8)
generation = dict(base, output_dir=str(output_dir / "generation"), do_predict=True,
predict_with_generate=True, per_device_eval_batch_size=1,
do_sample=False, num_beams=1, max_new_tokens=max_new_tokens)
return loss, generation
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("config", type=Path, help="The same model YAML used for training.")
parser.add_argument("--adapter", type=Path, help="Defaults to output_dir in the model YAML.")
parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--split", choices=("eval", "test"), default="eval")
parser.add_argument("--max-new-tokens", type=int, default=256)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
if args.max_new_tokens < 1:
parser.error("--max-new-tokens must be positive.")
config = read_config(args.config.resolve())
data_dir = Path(config["dataset_dir"])
verify_data(data_dir, (args.split,))
adapter = args.adapter.resolve() if args.adapter else Path(config["output_dir"])
if not (adapter / "adapter_config.json").is_file():
raise ValueError(f"Missing trained adapter: {adapter}")
run_dir = verify_training_dataset(adapter, data_dir)
used_config = run_dir / "used_config.yaml"
if not used_config.is_file():
raise ValueError(f"Missing recorded training configuration: {used_config}")
if used_config.is_file():
trained = read_config(used_config)
for key in ("model_name_or_path", "model_revision", "template", "cutoff_len", "rope_scaling"):
if trained.get(key) != config.get(key):
raise ValueError(f"Evaluation setting {key} differs from this adapter's training; use {used_config}.")
output_dir = args.output_dir.resolve()
if output_dir.exists() and any(output_dir.iterdir()):
raise ValueError(f"Evaluation output directory must be new or empty: {output_dir}")
loss, generation = evaluation_configs(config, adapter, output_dir, args.split, args.max_new_tokens)
if args.dry_run:
print(json.dumps({"split": args.split, "adapter": str(adapter), "evaluation_gpus": 1,
"loss_config": loss, "generation_config": generation}, ensure_ascii=False, indent=2))
print("Evaluation configuration/data checks passed. No inference started.")
return 0
output_dir.mkdir(parents=True, exist_ok=True)
record_path = output_dir / "evaluation.json"
record = {"status": "running", "split": args.split, "adapter": str(adapter),
"config": str(args.config.resolve()), "gpus": 1, "backend": "Unsloth", "precision": "bf16"}
record_path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
env = os.environ.copy()
env.update({"PYTHONUNBUFFERED": "1", "TOKENIZERS_PARALLELISM": "false"})
try:
for name, settings in (("eval_loss", loss), ("eval_generate", generation)):
path = output_dir / f"{name}.yaml"
write_config(path, settings)
command = [sys.executable, "-m", "torch.distributed.run", "--standalone", "--nproc_per_node=1",
str(ROOT / "scripts/train.py"), str(path), "--worker"]
run_logged(command, output_dir / f"{name}.log", env)
metrics = score_files(output_dir / "generation/generated_predictions.jsonl",
data_dir / f"{args.split}.jsonl", output_dir / "scores",
data_dir / f"{args.split}.metadata.jsonl")
reported = json.loads((output_dir / "loss/eval_results.json").read_text(encoding="utf-8"))
value = reported.get("eval_loss")
if value is None or not math.isfinite(value):
raise ValueError("LLaMA-Factory did not report a finite eval_loss.")
summary = {"model": config["model_name_or_path"], "split": args.split,
"eval_loss": value, "perplexity": math.exp(value) if value < 700 else None,
"token_f1": metrics["overall"]["token_f1"],
"normalized_exact_match": metrics["overall"]["normalized_exact_match"],
"examples": metrics["overall"]["count_total"], "metric_scale": "0..100",
"max_new_tokens": args.max_new_tokens, "cutoff_len": config["cutoff_len"],
"generation_context_limit": config["cutoff_len"]}
(output_dir / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
record["status"] = "complete"
print(json.dumps(summary, ensure_ascii=False, indent=2))
except BaseException:
record["status"] = "failed_or_interrupted"
raise
finally:
record_path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except (OSError, ValueError, RuntimeError, ImportError, subprocess.CalledProcessError) as error:
print(f"ERROR: {error}", file=sys.stderr)
raise SystemExit(1)

356
perun/ministral_3b_aya/scripts/prepare_data.py Normal file
View File

@ -0,0 +1,356 @@
#!/usr/bin/env python3
"""Prepare one shared Slovak Aya train/eval/test dataset for LLaMA-Factory."""
from __future__ import annotations
import argparse
from collections import Counter
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import sqlite3
import sys
import unicodedata
from typing import Any, Iterator
HF_DATASET = "CohereLabs/aya_collection_language_split"
HF_REVISION = "a3af2fde4b4cb5b2775830b11244a1a20b5f004f"
HF_CONFIG = "slovak"
HF_SPLIT = "train"
SPLITS = ("train", "eval", "test")
def normalized(text: str, *, fold_case: bool = False) -> str:
text = " ".join(unicodedata.normalize("NFKC", text).split())
return text.casefold() if fold_case else text
def digest_json(value: Any) -> str:
payload = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def file_sha256(path: Path) -> str:
result = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1024 * 1024), b""):
result.update(chunk)
return result.hexdigest()
def write_json(path: Path, content: Any) -> None:
with path.open("x", encoding="utf-8", newline="\n") as target:
json.dump(content, target, ensure_ascii=False, indent=2, sort_keys=True)
target.write("\n")
def local_rows(path: Path) -> Iterator[Any]:
with path.open("r", encoding="utf-8-sig") as source:
for line_number, line in enumerate(source, 1):
if not line.strip():
continue
try:
yield json.loads(line)
except json.JSONDecodeError as error:
raise ValueError(f"Invalid JSON at {path}:{line_number}: {error}") from error
def parquet_rows(parquet: Any) -> Iterator[dict[str, Any]]:
with parquet:
for batch in parquet.iter_batches(batch_size=2048):
yield from batch.to_pylist()
def open_source(args: argparse.Namespace) -> tuple[Iterator[Any], dict[str, Any]]:
local_path = args.local_parquet or args.local_jsonl
if local_path:
path = local_path.expanduser().resolve(strict=True)
if not path.is_file():
raise ValueError(f"Local source is not a file: {path}")
source = {
"kind": "local_parquet" if args.local_parquet else "local_jsonl",
"path": str(path),
"sha256": file_sha256(path),
"bytes": path.stat().st_size,
"revision": None,
}
if args.local_parquet:
try:
from pyarrow.parquet import ParquetFile
except ImportError as error:
raise RuntimeError("Install pyarrow to read --local-parquet input.") from error
return parquet_rows(ParquetFile(path)), source
return local_rows(path), source
try:
from datasets import load_dataset
from huggingface_hub import HfApi
except ImportError as error:
raise RuntimeError("Install datasets and huggingface_hub for Hugging Face input.") from error
info = HfApi().dataset_info(HF_DATASET, revision=args.revision or HF_REVISION)
revision = info.sha
if not revision:
raise RuntimeError("Hugging Face did not return an immutable dataset revision.")
data = load_dataset(
HF_DATASET, HF_CONFIG, split=HF_SPLIT, revision=revision, streaming=True
)
return iter(data), {
"kind": "huggingface",
"dataset": HF_DATASET,
"config": HF_CONFIG,
"split": HF_SPLIT,
"requested_revision": args.revision,
"revision": revision,
"streaming": True,
}
def convert_row(row: Any) -> tuple[dict[str, str], dict[str, Any]] | None:
if not isinstance(row, dict):
return None
if "inputs" in row or "targets" in row:
instruction, query, response = row.get("inputs"), "", row.get("targets")
excluded = {"inputs", "targets"}
else:
instruction, query, response = row.get("instruction"), row.get("input", ""), row.get("output")
query = "" if query is None else query
excluded = {"instruction", "input", "output"}
if not all(isinstance(value, str) for value in (instruction, query, response)):
return None
if not instruction.strip() or not response.strip():
return None
metadata = {key: value for key, value in row.items() if key not in excluded}
return {
"instruction": instruction.strip(),
"input": query.strip(),
"output": response.strip(),
}, metadata
def create_database(path: Path) -> sqlite3.Connection:
connection = sqlite3.connect(path)
connection.executescript(
"""
PRAGMA journal_mode = DELETE;
PRAGMA synchronous = NORMAL;
PRAGMA temp_store = FILE;
PRAGMA cache_size = -32768;
CREATE TABLE prompt_groups (
group_id TEXT PRIMARY KEY,
order_key TEXT NOT NULL,
size INTEGER NOT NULL,
split TEXT NOT NULL DEFAULT 'train'
);
CREATE TABLE records (
position INTEGER PRIMARY KEY,
stable_id TEXT NOT NULL UNIQUE,
group_id TEXT NOT NULL,
instruction TEXT NOT NULL,
input TEXT NOT NULL,
output TEXT NOT NULL,
metadata TEXT NOT NULL,
source_row_index INTEGER NOT NULL,
source_dataset TEXT NOT NULL,
source_task TEXT NOT NULL
);
"""
)
return connection
def ingest(connection: sqlite3.Connection, rows: Iterator[Any], seed: int) -> Counter:
counts: Counter = Counter()
for source_index, row in enumerate(rows):
counts["source_rows"] += 1
converted = convert_row(row)
if converted is None:
counts["invalid_or_empty_rows"] += 1
else:
data, metadata = converted
prompt_key = [normalized(data["instruction"], fold_case=True), normalized(data["input"], fold_case=True)]
group_id = digest_json(prompt_key)
stable_id = digest_json([prompt_key, normalized(data["output"])])
cursor = connection.execute(
"""INSERT OR IGNORE INTO records
(stable_id, group_id, instruction, input, output, metadata,
source_row_index, source_dataset, source_task)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(stable_id, group_id, data["instruction"], data["input"], data["output"],
json.dumps(metadata, ensure_ascii=False, separators=(",", ":")), source_index,
str(metadata.get("dataset_name") or "unknown"), str(metadata.get("task_type") or "unknown")))
if cursor.rowcount == 0:
counts["duplicate_pairs_removed"] += 1
else:
counts["retained_rows"] += 1
connection.execute(
"""INSERT INTO prompt_groups (group_id, order_key, size) VALUES (?, ?, 1)
ON CONFLICT(group_id) DO UPDATE SET size = size + 1""",
(group_id, digest_json([seed, group_id])))
if counts["source_rows"] % 10000 == 0:
connection.commit()
print(f"Read {counts['source_rows']:,}; retained {counts['retained_rows']:,}", flush=True)
connection.commit()
for key in ("source_rows", "invalid_or_empty_rows", "duplicate_pairs_removed", "retained_rows"):
counts.setdefault(key, 0)
if not counts["retained_rows"]:
raise ValueError("The source contains no usable examples after validation.")
connection.execute("CREATE INDEX prompt_group_order ON prompt_groups(order_key, group_id)")
connection.execute("CREATE INDEX records_group ON records(group_id)")
connection.commit()
counts["prompt_groups"] = connection.execute("SELECT COUNT(*) FROM prompt_groups").fetchone()[0]
counts["groups_with_multiple_responses"] = connection.execute("SELECT COUNT(*) FROM prompt_groups WHERE size > 1").fetchone()[0]
return counts
def assign_splits(connection: sqlite3.Connection, eval_size: int, test_size: int) -> dict[str, dict[str, int]]:
remaining = {"eval": eval_size, "test": test_size}
assignments: list[tuple[str, str]] = []
# Selection depends only on seed, normalized prompt, and the number of its
# distinct responses. An entire prompt group always stays in one split.
for group_id, size in connection.execute(
"SELECT group_id, size FROM prompt_groups ORDER BY order_key, group_id"
):
split = next((name for name in ("eval", "test") if size <= remaining[name]), None)
if split is not None:
remaining[split] -= size
assignments.append((split, group_id))
if not any(remaining.values()):
break
connection.executemany("UPDATE prompt_groups SET split = ? WHERE group_id = ?", assignments)
connection.commit()
result = {name: {"rows": 0, "prompt_groups": 0} for name in SPLITS}
for split, group_count, row_count in connection.execute(
"SELECT split, COUNT(*), SUM(size) FROM prompt_groups GROUP BY split"
):
result[split] = {"rows": row_count, "prompt_groups": group_count}
if not result["train"]["rows"]:
raise ValueError("No rows remain for train. Reduce --eval-size and/or --test-size.")
# A record belongs to one prompt group; each group has exactly one split.
conflicts = connection.execute(
"""SELECT COUNT(*) FROM (
SELECT r.group_id FROM records r JOIN prompt_groups g USING(group_id)
GROUP BY r.group_id HAVING COUNT(DISTINCT g.split) != 1
)"""
).fetchone()[0]
if conflicts:
raise RuntimeError("Prompt-disjoint split verification failed.")
if result["eval"]["rows"] > eval_size or result["test"]["rows"] > test_size:
raise RuntimeError("A holdout exceeds its requested row limit.")
return result
def emit_splits(connection: sqlite3.Connection, output: Path, expected: dict[str, dict[str, int]]) -> dict[str, Any]:
files: dict[str, Any] = {}
for split in SPLITS:
data_path = output / f"{split}.jsonl"
metadata_path = output / f"{split}.metadata.jsonl"
row_count = 0
with data_path.open("x", encoding="utf-8", newline="\n") as data_file, metadata_path.open(
"x", encoding="utf-8", newline="\n"
) as metadata_file:
cursor = connection.execute(
"""SELECT r.stable_id, r.group_id, r.instruction, r.input, r.output,
r.metadata, r.source_row_index
FROM records r JOIN prompt_groups g USING(group_id)
WHERE g.split = ? ORDER BY g.order_key, r.stable_id""", (split,)
)
for stable_id, group_id, instruction, query, response, metadata, source_index in cursor:
item = {"instruction": instruction, "input": query, "output": response}
provenance = {
"row_index": row_count,
"stable_id": stable_id,
"prompt_group_id": group_id,
"source_row_index": source_index,
"source_metadata": json.loads(metadata),
}
provenance["source_id"] = provenance["source_metadata"].get("id")
data_file.write(json.dumps(item, ensure_ascii=False, separators=(",", ":")) + "\n")
metadata_file.write(json.dumps(provenance, ensure_ascii=False, separators=(",", ":")) + "\n")
row_count += 1
if row_count != expected[split]["rows"]:
raise RuntimeError(f"Unexpected {split} output count: {row_count}")
for path in (data_path, metadata_path):
files[path.name] = {"sha256": file_sha256(path), "bytes": path.stat().st_size, "rows": row_count}
return files
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--revision", help="Hugging Face revision; resolved to an immutable commit before loading.")
local_input = parser.add_mutually_exclusive_group()
local_input.add_argument("--local-jsonl", type=Path, help="Offline JSONL input in Aya or Alpaca format.")
local_input.add_argument("--local-parquet", type=Path, help="Offline Parquet input in Aya or Alpaca format.")
parser.add_argument("--eval-size", type=int, default=1000, help="Maximum number of eval records (default: 1000).")
parser.add_argument("--test-size", type=int, default=1000, help="Maximum number of test records (default: 1000).")
parser.add_argument("--seed", type=int, default=42)
parser.add_argument("--keep-db", action="store_true", help="Keep the temporary SQLite database.")
args = parser.parse_args()
if min(args.eval_size, args.test_size) < 0:
parser.error("Holdout sizes must be nonnegative.")
if (args.local_jsonl or args.local_parquet) and args.revision:
parser.error("--revision applies only to Hugging Face input.")
output = args.output_dir.expanduser().resolve()
if output.exists() and (not output.is_dir() or any(output.iterdir())):
parser.error(f"Output must be a new or empty directory: {output}")
# Resolve source before creating output, so authentication or dependency
# errors do not produce a misleading empty data directory.
rows, source = open_source(args)
output.mkdir(parents=True, exist_ok=True)
database_path = output / "preparation.sqlite3"
connection = create_database(database_path)
try:
counts = ingest(connection, rows, args.seed)
splits = assign_splits(connection, args.eval_size, args.test_size)
if sum(item["rows"] for item in splits.values()) != counts["retained_rows"]:
raise RuntimeError("Split counts do not sum to the retained record count.")
files = emit_splits(connection, output, splits)
registration = {
f"aya_{split}": {
"file_name": f"{split}.jsonl", "formatting": "alpaca",
"columns": {"prompt": "instruction", "query": "input", "response": "output"},
} for split in SPLITS
}
info_path = output / "dataset_info.json"
write_json(info_path, registration)
files[info_path.name] = {"sha256": file_sha256(info_path), "bytes": info_path.stat().st_size}
breakdown = [
{"split": split, "dataset_name": dataset, "task_type": task, "rows": rows_count, "prompt_groups": groups_count}
for split, dataset, task, rows_count, groups_count in connection.execute(
"""SELECT g.split, r.source_dataset, r.source_task, COUNT(*), COUNT(DISTINCT r.group_id)
FROM records r JOIN prompt_groups g USING(group_id)
GROUP BY g.split, r.source_dataset, r.source_task
ORDER BY g.split, r.source_dataset, r.source_task"""
)
]
manifest = {
"format_version": 1,
"prepared_at_utc": datetime.now(timezone.utc).isoformat(),
"preparation_script_sha256": file_sha256(Path(__file__).resolve()),
"source": source,
"seed": args.seed,
"counts": dict(counts),
"requested_holdout_max_rows": {"eval": args.eval_size, "test": args.test_size},
"splits": splits,
"source_breakdown": breakdown,
"files": files,
}
write_json(output / "manifest.json", manifest)
finally:
connection.close()
if not args.keep_db:
# Only our explicitly named staging file inside this new output is removed.
database_path.unlink()
print(json.dumps({"output_dir": str(output), "source": source, "counts": dict(counts), "splits": splits}, ensure_ascii=False, indent=2))
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except (ValueError, RuntimeError) as error:
print(f"ERROR: {error}", file=sys.stderr)
raise SystemExit(1)

345
perun/ministral_3b_aya/scripts/train.py Normal file
View File

@ -0,0 +1,345 @@
#!/usr/bin/env python3
"""Train from one LLaMA-Factory YAML on four GPUs; import Unsloth in each worker first."""
from __future__ import annotations
import argparse
from datetime import datetime, timezone
import functools
import hashlib
import json
import os
from pathlib import Path
import subprocess
import sys
from types import MethodType
ROOT = Path(__file__).resolve().parents[1]
GPUS = 4
def read_config(path: Path) -> dict:
import yaml
with path.open(encoding="utf-8-sig") as handle:
config = yaml.safe_load(handle)
if not isinstance(config, dict):
raise ValueError("The configuration must be a YAML mapping.")
if config.get("use_unsloth") is not True:
raise ValueError("This package requires use_unsloth: true.")
if config.get("bf16") is not True or config.get("fp16", False):
raise ValueError("The H200 configuration must use bf16, not fp16.")
if "model_max_length" in config:
raise ValueError("Remove model_max_length from YAML; LLaMA-Factory derives it from cutoff_len.")
if int(config.get("cutoff_len", 0)) < 1:
raise ValueError("cutoff_len must be positive.")
for key in ("dataset_dir", "output_dir"):
if key not in config:
raise ValueError(f"Missing configuration key: {key}")
value = Path(config[key]).expanduser()
config[key] = str((ROOT / value).resolve() if not value.is_absolute() else value.resolve())
return config
def write_config(path: Path, config: dict) -> None:
import yaml
path.write_text(yaml.safe_dump(config, allow_unicode=True, sort_keys=False), encoding="utf-8")
def digest(path: Path) -> str:
result = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1024 * 1024), b""):
result.update(block)
return result.hexdigest()
def verify_data(data_dir: Path, splits: tuple[str, ...]) -> dict:
manifest_path = data_dir / "manifest.json"
if not manifest_path.is_file():
raise ValueError(f"Run prepare_data.py first; missing {manifest_path}")
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
for split in splits:
if manifest.get("splits", {}).get(split, {}).get("rows", 0) < 1:
raise ValueError(f"The prepared {split} split is empty.")
names = ["dataset_info.json"]
for split in splits:
names.extend((f"{split}.jsonl", f"{split}.metadata.jsonl"))
for name in names:
item = manifest.get("files", {}).get(name, {})
if not item.get("sha256") or digest(data_dir / name) != item["sha256"]:
raise ValueError(f"Prepared data changed or is missing: {name}")
return manifest
def run_logged(command: list[str], log: Path, env: dict | None = None) -> None:
print("Running:", " ".join(command), flush=True)
with log.open("a", encoding="utf-8") as handle:
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True, encoding="utf-8", errors="replace", env=env)
try:
for line in process.stdout:
print(line, end="", flush=True)
handle.write(line)
handle.flush()
code = process.wait()
except BaseException:
process.terminate()
process.wait()
raise
if code:
raise subprocess.CalledProcessError(code, command)
def context_settings(settings: dict, original_config) -> dict:
requested = int(settings["cutoff_len"])
native = int(original_config.max_position_embeddings)
if requested > native or settings.get("rope_scaling"):
raise ValueError(f"Use cutoff_len <= {native} within the model's native context and omit rope_scaling.")
return {"requested_length": requested, "base_positions": native}
def reserve_generation_space() -> None:
# Reserve space before batching so the trainer sees the same prompt length
# as model.generate when it masks the prompt in the generated sequence.
from llamafactory.data.processor import unsupervised
original = unsupervised.infer_seqlen
@functools.wraps(original)
def with_generation_space(source_len: int, target_len: int, cutoff_len: int):
source, target = original(source_len, target_len, cutoff_len)
source = min(source, cutoff_len - 1)
if source < source_len or target < target_len:
print(f"Generation preprocessing truncated tokens: input {source_len} -> {source}, "
f"label {target_len} -> {target}, context {cutoff_len}.", flush=True)
return source, target
unsupervised.infer_seqlen = with_generation_space
def bound_generation_context(model, context_length: int, default_max_new_tokens: int) -> None:
original_generate = model.generate
@functools.wraps(original_generate)
def bounded_generate(self, *args, **kwargs):
inputs = kwargs.get("input_ids")
if inputs is None:
inputs = kwargs.get("inputs")
if inputs is None and args:
inputs = args[0]
if inputs is None:
raise ValueError("Cannot verify the text input length before generation.")
prompt_length = int(inputs.shape[-1])
available = context_length - prompt_length
if available < 1:
raise ValueError("The preprocessed input leaves no room for an answer inside the context limit.")
requested = kwargs.get("max_new_tokens")
requested = int(default_max_new_tokens if requested is None else requested)
if requested < 1:
raise ValueError("max_new_tokens must be positive.")
kwargs["max_new_tokens"] = min(requested, available)
if requested > available:
print(f"Generation limit: {available} new tokens after {prompt_length} input tokens "
f"(context {context_length}).", flush=True)
return original_generate(*args, **kwargs)
model.generate = MethodType(bounded_generate, model)
def require_unsloth_backend(settings: dict, context: dict) -> dict:
# LLaMA-Factory may otherwise silently fall back when Unsloth rejects a model.
# The wrappers affect this process only; no installed package is edited.
from llamafactory.model import loader
from llamafactory.model.model_utils import unsloth as integration
loaded = {}
if hasattr(integration, "_get_unsloth_kwargs"):
original_kwargs = integration._get_unsloth_kwargs
@functools.wraps(original_kwargs)
def pinned_kwargs(config, model_name_or_path, model_args, finetuning_args):
values = original_kwargs(config, model_name_or_path, model_args, finetuning_args)
values["use_exact_model_name"] = True
if model_args.model_revision:
values["revision"] = model_args.model_revision
return values
integration._get_unsloth_kwargs = pinned_kwargs
def strict(original):
@functools.wraps(original)
def wrapped(*args, **kwargs):
model = original(*args, **kwargs)
if model is None:
raise RuntimeError("Unsloth did not load this model. Training/evaluation stopped; "
"use a compatible Unsloth/LLaMA-Factory environment.")
if model.config.max_position_embeddings < context["requested_length"]:
raise RuntimeError("The loaded model's configured context is shorter than the requested SFT context.")
if settings.get("do_train"):
loaded["base_config_json"] = model.config.to_json_string()
return model
return wrapped
for name in ("load_unsloth_pretrained_model", "load_unsloth_peft_model"):
original = getattr(loader, name, None)
if original is None:
raise RuntimeError("The installed LLaMA-Factory has an unsupported Unsloth integration.")
setattr(loader, name, strict(original))
if settings.get("do_predict") and settings.get("predict_with_generate"):
# Keep PEFT/Unsloth delegation: GenerationMixin.generate bound directly
# to a PeftModel fails in Transformers 5 because its class has no _sample.
original_patch_model = loader.patch_model
@functools.wraps(original_patch_model)
def patch_model_preserving_generate(model, *args, **kwargs):
original_generate = model.generate
result = original_patch_model(model, *args, **kwargs)
model.generate = original_generate
return result
loader.patch_model = patch_model_preserving_generate
# Attach the guard to the final model, after LLaMA-Factory patches generate
# and finishes adapter initialization. Earlier attachment can be replaced.
from llamafactory.train.sft import workflow
original_load_model = workflow.load_model
@functools.wraps(original_load_model)
def load_bounded_model(*args, **kwargs):
model = original_load_model(*args, **kwargs)
bound_generation_context(model, context["requested_length"], int(settings["max_new_tokens"]))
return model
workflow.load_model = load_bounded_model
return loaded
def worker(config_path: Path) -> None:
# This order is important for Unsloth's patches.
import unsloth # noqa: F401
import torch
config = read_config(config_path)
world = int(os.environ.get("WORLD_SIZE", "1"))
if config.get("do_train") and world != GPUS:
raise RuntimeError("This training scenario requires four processes / four GPUs.")
if not torch.cuda.is_available() or torch.cuda.device_count() < world:
raise RuntimeError(f"This job requires {world} visible CUDA GPU(s).")
local_rank = int(os.environ.get("LOCAL_RANK", "0"))
torch.cuda.set_device(local_rank)
if not torch.cuda.is_bf16_supported():
raise RuntimeError("The allocated GPU does not support the requested bf16 precision.")
print(f"rank={local_rank}, world_size={world}, GPU={torch.cuda.get_device_name(local_rank)}, "
"precision=bf16, backend=Unsloth", flush=True)
from transformers import AutoConfig
original_config = AutoConfig.from_pretrained(config["model_name_or_path"], revision=config["model_revision"], trust_remote_code=False)
context = context_settings(config, original_config)
print("Model context:", json.dumps(context), flush=True)
loaded = require_unsloth_backend(config, context)
if config.get("do_predict") and config.get("predict_with_generate"):
if context["requested_length"] < 2:
raise ValueError("Generation requires cutoff_len >= 2.")
reserve_generation_space()
from llamafactory.train.tuner import run_exp
run_exp(config)
if config.get("do_train") and local_rank == 0:
(Path(config["output_dir"]) / "base_model_config.json").write_text(loaded["base_config_json"], encoding="utf-8")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("config", type=Path, help="LLaMA-Factory model YAML.")
parser.add_argument("--run-dir", type=Path, help="New run directory; adapter is saved inside adapter/.")
parser.add_argument("--resume", type=Path, help="Checkpoint inside the original run's adapter/ directory.")
parser.add_argument("--pilot-steps", type=int, help="Limit a pilot to this many optimizer steps; requires --run-dir.")
parser.add_argument("--pilot-samples", type=int,
help="Maximum rows per dataset in a pilot (default: 2048 or all eval rows, whichever is larger).")
parser.add_argument("--pilot-eval-steps", type=int,
help="Evaluate and save at this interval in a pilot (default: at the last step).")
parser.add_argument("--dry-run", action="store_true", help="Validate data/config; do not import GPU libraries.")
parser.add_argument("--worker", action="store_true", help=argparse.SUPPRESS)
args = parser.parse_args()
if args.worker:
worker(args.config.resolve())
return 0
if args.pilot_steps is None and args.pilot_samples is not None:
parser.error("--pilot-samples requires --pilot-steps.")
if args.pilot_steps is not None and (args.pilot_steps < 1 or args.run_dir is None):
parser.error("--pilot-steps must be positive and requires --run-dir.")
if args.pilot_samples is not None and args.pilot_samples < 1:
parser.error("--pilot-samples must be positive.")
if args.pilot_eval_steps is not None and not 1 <= args.pilot_eval_steps <= (args.pilot_steps or 0):
parser.error("--pilot-eval-steps requires --pilot-steps and must be between 1 and that limit.")
config = read_config(args.config.resolve())
if config.get("stage") != "sft" or config.get("finetuning_type") != "lora":
raise ValueError("Expected an SFT / LoRA training configuration.")
if not config.get("do_train") or config.get("eval_dataset") != "aya_eval":
raise ValueError("Training must use the prepared train and eval splits.")
if config.get("dataset") != "aya_train" or "val_size" in config:
raise ValueError("Use aya_train and aya_eval without another automatic split.")
if any(key in config for key in ("max_steps", "max_samples")):
raise ValueError("The full-data scenario must not impose a smoke-test sample/step limit.")
batch = GPUS * config["per_device_train_batch_size"] * config["gradient_accumulation_steps"]
data_dir = Path(config["dataset_dir"])
manifest = verify_data(data_dir, ("train", "eval"))
train_rows = manifest["splits"]["train"]["rows"]
eval_rows = manifest["splits"]["eval"]["rows"]
if args.pilot_steps is not None:
# LLaMA-Factory limits both splits before tokenization. Keep the complete
# validation split, even if the caller prepared more than 2048 eval rows.
sample_limit = args.pilot_samples or max(2048, eval_rows)
if sample_limit < eval_rows:
parser.error(f"--pilot-samples must be at least {eval_rows} to retain the full eval split.")
eval_interval = args.pilot_eval_steps or args.pilot_steps
config.update(max_steps=args.pilot_steps, max_samples=sample_limit,
eval_strategy="steps", save_strategy="steps",
eval_steps=eval_interval, save_steps=eval_interval)
if args.run_dir:
config["output_dir"] = str(args.run_dir.resolve() / "adapter")
run_dir = Path(config["output_dir"]).parent
if args.resume:
checkpoint = args.resume.resolve()
checkpoint.relative_to(Path(config["output_dir"]))
for name in ("trainer_state.json", "adapter_config.json", "optimizer.pt", "scheduler.pt"):
if not (checkpoint / name).is_file():
raise ValueError(f"Incomplete checkpoint: {checkpoint / name}")
previous = json.loads((run_dir / "run.json").read_text(encoding="utf-8"))
if previous["config"] != config or previous["data_manifest_sha256"] != digest(data_dir / "manifest.json"):
raise ValueError("Resume requires the original configuration and prepared data.")
elif run_dir.exists() and any(run_dir.iterdir()):
raise ValueError(f"Run directory is not empty; choose --run-dir for a new run: {run_dir}")
summary = {"gpus_per_training_run": GPUS, "effective_batch": batch,
"cutoff_len": config["cutoff_len"], "bf16": True, "use_unsloth": True,
"train_rows": train_rows, "eval_rows": eval_rows,
"adapter_output": config["output_dir"]}
if args.pilot_steps is not None:
summary.update(pilot_steps=args.pilot_steps, pilot_train_rows=min(train_rows, config["max_samples"]),
pilot_eval_steps=config["eval_steps"])
print(json.dumps(summary, ensure_ascii=False, indent=2))
if args.dry_run:
print("Configuration/data checks passed. No training started.")
return 0
run_dir.mkdir(parents=True, exist_ok=True)
if not args.resume:
record = {"config": config, "gpus": GPUS, "effective_batch": batch,
"data_manifest_sha256": digest(data_dir / "manifest.json"),
"created_at_utc": datetime.now(timezone.utc).isoformat()}
(run_dir / "run.json").write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
execution = dict(config)
if args.resume:
execution["resume_from_checkpoint"] = str(args.resume.resolve())
execution_path = run_dir / "used_config.yaml"
write_config(execution_path, execution)
env = os.environ.copy()
env.update({"PYTHONUNBUFFERED": "1", "TOKENIZERS_PARALLELISM": "false"})
env.setdefault("OMP_NUM_THREADS", "4")
command = [sys.executable, "-m", "torch.distributed.run", "--standalone",
f"--nproc_per_node={GPUS}", str(Path(__file__).resolve()),
str(execution_path), "--worker"]
run_logged(command, run_dir / "train.log", env)
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except (OSError, ValueError, RuntimeError, ImportError, subprocess.CalledProcessError) as error:
print(f"ERROR: {error}", file=sys.stderr)
raise SystemExit(1)

3250
perun/ministral_3b_aya/sk_bench/aggregated/models__ministral-base.json Normal file
View File

File diff suppressed because it is too large Load Diff

3250
perun/ministral_3b_aya/sk_bench/aggregated/models__ministral-last-109064.json Normal file
View File

File diff suppressed because it is too large Load Diff

30
perun/ministral_3b_aya/sk_bench/scores.csv Normal file
View File

@ -0,0 +1,30 @@
group,task,metric,base_score,finetuned_score,difference_points
Knowledge & Reasoning,ARC,acc,24.4939,24.4654,-0.0285
Knowledge & Reasoning,MMLU,acc,22.2452,24.1489,1.9037
Knowledge & Reasoning,G-MMLU,acc,21.0833,27.5833,6.5000
Knowledge & Reasoning,TruthQA,acc,6.4696,31.8766,25.4070
Knowledge & Reasoning,SCIO,acc,23.8469,25.0047,1.1578
Knowledge & Reasoning,FinExam,acc,8.8706,18.6157,9.7451
Knowledge & Reasoning,Driving,acc,13.7423,27.1041,13.3618
Knowledge & Reasoning,COPA,acc,5.3333,27.0000,21.6667
Knowledge & Reasoning,SKJ1,acc,59.6667,0.0000,-59.6667
Knowledge & Reasoning,Trivia,acc,19.9245,27.8565,7.9320
Reading & QA,Belebele,acc,25.3704,26.6667,1.2963
Reading & QA,Chiby,acc,0.0000,14.1123,14.1123
Reading & QA,WikiQA,qem,0.0534,7.8847,7.8313
Reading & QA,ETest-M,acc,15.0087,18.1501,3.1414
Reading & QA,ETest-Q,qem,0.2257,1.2039,0.9782
Reading & QA,SKTest-M,acc,17.3900,26.0080,8.6180
Reading & QA,SKTest-Q,qem,1.2605,0.9804,-0.2801
Reading & QA,sKLEP-QA,qem,0.0278,5.2037,5.1759
Classification,sKLEP-NLI,acc,0.0000,1.8718,1.8718
Classification,sKLEP-RTE,acc,0.0000,31.4056,31.4056
Classification,Senti-3,acc,0.0000,38.1370,38.1370
Classification,Senti-2,acc,0.0000,48.7393,48.7393
Classification,sKLEP-Sent,acc,0.0000,10.7486,10.7486
Classification,MovieRev,acc,0.0000,14.5750,14.5750
Classification,Toxic,acc,0.0000,19.9016,19.9016
Classification,sKLEP-Hate,acc,0.0000,21.3798,21.3798
Instruction,IFEval,prompt_level_strict_acc,7.8212,10.2421,2.4209
Grammar & Morphology,Analogies,acc,0.0000,0.0629,0.0629
Grammar & Morphology,Slohry,acc,0.2331,1.6983,1.4652
1 group task metric base_score finetuned_score difference_points
2 Knowledge & Reasoning ARC acc 24.4939 24.4654 -0.0285
3 Knowledge & Reasoning MMLU acc 22.2452 24.1489 1.9037
4 Knowledge & Reasoning G-MMLU acc 21.0833 27.5833 6.5000
5 Knowledge & Reasoning TruthQA acc 6.4696 31.8766 25.4070
6 Knowledge & Reasoning SCIO acc 23.8469 25.0047 1.1578
7 Knowledge & Reasoning FinExam acc 8.8706 18.6157 9.7451
8 Knowledge & Reasoning Driving acc 13.7423 27.1041 13.3618
9 Knowledge & Reasoning COPA acc 5.3333 27.0000 21.6667
10 Knowledge & Reasoning SKJ1 acc 59.6667 0.0000 -59.6667
11 Knowledge & Reasoning Trivia acc 19.9245 27.8565 7.9320
12 Reading & QA Belebele acc 25.3704 26.6667 1.2963
13 Reading & QA Chiby acc 0.0000 14.1123 14.1123
14 Reading & QA WikiQA qem 0.0534 7.8847 7.8313
15 Reading & QA ETest-M acc 15.0087 18.1501 3.1414
16 Reading & QA ETest-Q qem 0.2257 1.2039 0.9782
17 Reading & QA SKTest-M acc 17.3900 26.0080 8.6180
18 Reading & QA SKTest-Q qem 1.2605 0.9804 -0.2801
19 Reading & QA sKLEP-QA qem 0.0278 5.2037 5.1759
20 Classification sKLEP-NLI acc 0.0000 1.8718 1.8718
21 Classification sKLEP-RTE acc 0.0000 31.4056 31.4056
22 Classification Senti-3 acc 0.0000 38.1370 38.1370
23 Classification Senti-2 acc 0.0000 48.7393 48.7393
24 Classification sKLEP-Sent acc 0.0000 10.7486 10.7486
25 Classification MovieRev acc 0.0000 14.5750 14.5750
26 Classification Toxic acc 0.0000 19.9016 19.9016
27 Classification sKLEP-Hate acc 0.0000 21.3798 21.3798
28 Instruction IFEval prompt_level_strict_acc 7.8212 10.2421 2.4209
29 Grammar & Morphology Analogies acc 0.0000 0.0629 0.0629
30 Grammar & Morphology Slohry acc 0.2331 1.6983 1.4652

385
perun/ministral_3b_aya/sk_bench/tasks_sk0_385.txt Normal file
View File

@ -0,0 +1,385 @@
arc.prompt3_mqa_single_correct_sk|0
arc.prompt1_mqa_single_correct_sk|0
arc.prompt2_mqa_single_correct_sk|0
belebele.prompt3_mqa_single_correct_sk|0
belebele.prompt1_mqa_single_correct_sk|0
belebele.prompt2_mqa_single_correct_sk|0
chiby.prompt2_chiby_sk|0
chiby.prompt1_chiby_sk|0
chiby.prompt3_chiby_sk|0
financial_exams.prompt1_completion_sk|0
financial_exams.prompt3_completion_sk|0
financial_exams.prompt2_completion_sk|0
ifeval_sk.ifeval|0
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt3_mqa_single_correct_sk:anatomy|0
mmlu.prompt1_mqa_single_correct_sk:anatomy|0
mmlu.prompt2_mqa_single_correct_sk:anatomy|0
mmlu.prompt3_mqa_single_correct_sk:astronomy|0
mmlu.prompt1_mqa_single_correct_sk:astronomy|0
mmlu.prompt2_mqa_single_correct_sk:astronomy|0
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt3_mqa_single_correct_sk:college_biology|0
mmlu.prompt1_mqa_single_correct_sk:college_biology|0
mmlu.prompt2_mqa_single_correct_sk:college_biology|0
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
mmlu.prompt3_mqa_single_correct_sk:college_physics|0
mmlu.prompt1_mqa_single_correct_sk:college_physics|0
mmlu.prompt2_mqa_single_correct_sk:college_physics|0
mmlu.prompt3_mqa_single_correct_sk:computer_security|0
mmlu.prompt1_mqa_single_correct_sk:computer_security|0
mmlu.prompt2_mqa_single_correct_sk:computer_security|0
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt3_mqa_single_correct_sk:econometrics|0
mmlu.prompt1_mqa_single_correct_sk:econometrics|0
mmlu.prompt2_mqa_single_correct_sk:econometrics|0
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
mmlu.prompt3_mqa_single_correct_sk:global_facts|0
mmlu.prompt1_mqa_single_correct_sk:global_facts|0
mmlu.prompt2_mqa_single_correct_sk:global_facts|0
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt3_mqa_single_correct_sk:human_aging|0
mmlu.prompt1_mqa_single_correct_sk:human_aging|0
mmlu.prompt2_mqa_single_correct_sk:human_aging|0
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt3_mqa_single_correct_sk:international_law|0
mmlu.prompt1_mqa_single_correct_sk:international_law|0
mmlu.prompt2_mqa_single_correct_sk:international_law|0
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
mmlu.prompt3_mqa_single_correct_sk:management|0
mmlu.prompt1_mqa_single_correct_sk:management|0
mmlu.prompt2_mqa_single_correct_sk:management|0
mmlu.prompt3_mqa_single_correct_sk:marketing|0
mmlu.prompt1_mqa_single_correct_sk:marketing|0
mmlu.prompt2_mqa_single_correct_sk:marketing|0
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt3_mqa_single_correct_sk:nutrition|0
mmlu.prompt1_mqa_single_correct_sk:nutrition|0
mmlu.prompt2_mqa_single_correct_sk:nutrition|0
mmlu.prompt3_mqa_single_correct_sk:philosophy|0
mmlu.prompt1_mqa_single_correct_sk:philosophy|0
mmlu.prompt2_mqa_single_correct_sk:philosophy|0
mmlu.prompt3_mqa_single_correct_sk:prehistory|0
mmlu.prompt1_mqa_single_correct_sk:prehistory|0
mmlu.prompt2_mqa_single_correct_sk:prehistory|0
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt3_mqa_single_correct_sk:professional_law|0
mmlu.prompt1_mqa_single_correct_sk:professional_law|0
mmlu.prompt2_mqa_single_correct_sk:professional_law|0
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt3_mqa_single_correct_sk:public_relations|0
mmlu.prompt1_mqa_single_correct_sk:public_relations|0
mmlu.prompt2_mqa_single_correct_sk:public_relations|0
mmlu.prompt3_mqa_single_correct_sk:security_studies|0
mmlu.prompt1_mqa_single_correct_sk:security_studies|0
mmlu.prompt2_mqa_single_correct_sk:security_studies|0
mmlu.prompt3_mqa_single_correct_sk:sociology|0
mmlu.prompt1_mqa_single_correct_sk:sociology|0
mmlu.prompt2_mqa_single_correct_sk:sociology|0
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt3_mqa_single_correct_sk:virology|0
mmlu.prompt1_mqa_single_correct_sk:virology|0
mmlu.prompt2_mqa_single_correct_sk:virology|0
mmlu.prompt3_mqa_single_correct_sk:world_religions|0
mmlu.prompt1_mqa_single_correct_sk:world_religions|0
mmlu.prompt2_mqa_single_correct_sk:world_religions|0
multi_wiki_qa.prompt3_qa_sk|0
multi_wiki_qa.prompt2_qa_sk|0
multi_wiki_qa.prompt1_qa_sk|0
senti_sk.prompt2_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_binary_sk|0
senti_sk.prompt1_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_3_class_sk|0
senti_sk.prompt1_sentiment_3_class_sk|0
senti_sk.prompt2_sentiment_3_class_sk|0
skj1.prompt3_skj1_sk|0
skj1.prompt1_skj1_sk|0
skj1.prompt2_skj1_sk|0
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_hate_speech_sk:hate-speech|0
sklep.prompt3_hate_speech_sk:hate-speech|0
sklep.prompt2_hate_speech_sk:hate-speech|0
sklep.prompt1_nli_3_classes_sk:nli|0
sklep.prompt2_nli_3_classes_sk:nli|0
sklep.prompt3_nli_3_classes_sk:nli|0
sklep.prompt1_nli_binary_sk:rte|0
sklep.prompt3_nli_binary_sk:rte|0
sklep.prompt2_nli_binary_sk:rte|0
sklep.prompt3_qa_sk:question-answering|0
sklep.prompt2_qa_sk:question-answering|0
sklep.prompt1_qa_sk:question-answering|0
toxic_sk.prompt2_toxicity_sk|0
toxic_sk.prompt3_toxicity_sk|0
toxic_sk.prompt1_toxicity_sk|0
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
slovakcopa.prompt3_copa_sk|0
slovakcopa.prompt1_copa_sk|0
slovakcopa.prompt2_copa_sk|0
movie_review.prompt3_sentiment_binary_sk|0
movie_review.prompt2_sentiment_binary_sk|0
movie_review.prompt1_sentiment_binary_sk|0
slovak_trivia.prompt2_mqa_single_correct_sk|0
slovak_trivia.prompt3_mqa_single_correct_sk|0
slovak_trivia.prompt1_mqa_single_correct_sk|0
analogies_sk.prompt1_analogy_sk|0
analogies_sk.prompt2_analogy_sk|0
analogies_sk.prompt3_analogy_sk|0
driving_test.prompt3_mqa_single_correct_sk|0
driving_test.prompt1_mqa_single_correct_sk|0
driving_test.prompt2_mqa_single_correct_sk|0
e_tests.prompt3_mqa_single_correct_sk:biologia|0
e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_single_correct_sk:informatika|0
e_tests.prompt3_mqa_single_correct_sk:geografia|0
e_tests.prompt3_mqa_single_correct_sk:matematika|0
e_tests.prompt3_mqa_single_correct_sk:chemia|0
e_tests.prompt3_mqa_single_correct_sk:dejepis|0
e_tests.prompt3_mqa_single_correct_sk:fyzika|0
e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_single_correct_sk:biologia|0
e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_single_correct_sk:informatika|0
e_tests.prompt1_mqa_single_correct_sk:geografia|0
e_tests.prompt1_mqa_single_correct_sk:matematika|0
e_tests.prompt1_mqa_single_correct_sk:chemia|0
e_tests.prompt1_mqa_single_correct_sk:dejepis|0
e_tests.prompt1_mqa_single_correct_sk:fyzika|0
e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_single_correct_sk:biologia|0
e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_single_correct_sk:informatika|0
e_tests.prompt2_mqa_single_correct_sk:geografia|0
e_tests.prompt2_mqa_single_correct_sk:matematika|0
e_tests.prompt2_mqa_single_correct_sk:chemia|0
e_tests.prompt2_mqa_single_correct_sk:dejepis|0
e_tests.prompt2_mqa_single_correct_sk:fyzika|0
e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
e_tests.prompt1_qa_sk:biologia|0
e_tests.prompt1_qa_sk:obcianska-nauka|0
e_tests.prompt1_qa_sk:informatika|0
e_tests.prompt1_qa_sk:geografia|0
e_tests.prompt1_qa_sk:chemia|0
e_tests.prompt1_qa_sk:dejepis|0
e_tests.prompt1_qa_sk:fyzika|0
e_tests.prompt1_qa_sk:slovensky-jazyk|0
e_tests.prompt3_qa_sk:biologia|0
e_tests.prompt3_qa_sk:obcianska-nauka|0
e_tests.prompt3_qa_sk:informatika|0
e_tests.prompt3_qa_sk:geografia|0
e_tests.prompt3_qa_sk:chemia|0
e_tests.prompt3_qa_sk:dejepis|0
e_tests.prompt3_qa_sk:fyzika|0
e_tests.prompt3_qa_sk:slovensky-jazyk|0
e_tests.prompt2_qa_sk:biologia|0
e_tests.prompt2_qa_sk:obcianska-nauka|0
e_tests.prompt2_qa_sk:informatika|0
e_tests.prompt2_qa_sk:geografia|0
e_tests.prompt2_qa_sk:chemia|0
e_tests.prompt2_qa_sk:dejepis|0
e_tests.prompt2_qa_sk:fyzika|0
e_tests.prompt2_qa_sk:slovensky-jazyk|0
slovak_tests.prompt1_qa_sk:monitor9|0
slovak_tests.prompt1_qa_sk:maturita|0
slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
slovak_tests.prompt3_qa_sk:monitor9|0
slovak_tests.prompt3_qa_sk:maturita|0
slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
slovak_tests.prompt2_qa_sk:monitor9|0
slovak_tests.prompt2_qa_sk:maturita|0
slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
slohry.prompt2_declension_sk:declension_high|0
slohry.prompt2_declension_sk:declension_mid|0
slohry.prompt2_declension_sk:declension_low|0
slohry.prompt1_declension_sk:declension_high|0
slohry.prompt1_declension_sk:declension_mid|0
slohry.prompt1_declension_sk:declension_low|0
slohry.prompt3_declension_sk:declension_high|0
slohry.prompt3_declension_sk:declension_mid|0
slohry.prompt3_declension_sk:declension_low|0
slohry.prompt3_gradation_sk:gradation_high|0
slohry.prompt3_gradation_sk:gradation_mid|0
slohry.prompt3_gradation_sk:gradation_low|0
slohry.prompt1_gradation_sk:gradation_high|0
slohry.prompt1_gradation_sk:gradation_mid|0
slohry.prompt1_gradation_sk:gradation_low|0
slohry.prompt2_gradation_sk:gradation_high|0
slohry.prompt2_gradation_sk:gradation_mid|0
slohry.prompt2_gradation_sk:gradation_low|0
slohry.prompt1_identity_form_sk:identity_form_high|0
slohry.prompt1_identity_form_sk:identity_form_mid|0
slohry.prompt1_identity_form_sk:identity_form_low|0
slohry.prompt2_identity_form_sk:identity_form_high|0
slohry.prompt2_identity_form_sk:identity_form_mid|0
slohry.prompt2_identity_form_sk:identity_form_low|0
slohry.prompt3_identity_form_sk:identity_form_high|0
slohry.prompt3_identity_form_sk:identity_form_mid|0
slohry.prompt3_identity_form_sk:identity_form_low|0
slohry.prompt2_inflection_sk:inflection_high|0
slohry.prompt2_inflection_sk:inflection_mid|0
slohry.prompt2_inflection_sk:inflection_low|0
slohry.prompt1_inflection_sk:inflection_high|0
slohry.prompt1_inflection_sk:inflection_mid|0
slohry.prompt1_inflection_sk:inflection_low|0
slohry.prompt3_inflection_sk:inflection_high|0
slohry.prompt3_inflection_sk:inflection_mid|0
slohry.prompt3_inflection_sk:inflection_low|0
scio.prompt2_mqa_single_correct_sk:aj|0
scio.prompt2_mqa_single_correct_sk:mat|0
scio.prompt2_mqa_single_correct_sk:bio|0
scio.prompt2_mqa_single_correct_sk:che|0
scio.prompt2_mqa_single_correct_sk:vsp|0
scio.prompt3_mqa_single_correct_sk:aj|0
scio.prompt3_mqa_single_correct_sk:mat|0
scio.prompt3_mqa_single_correct_sk:bio|0
scio.prompt3_mqa_single_correct_sk:che|0
scio.prompt3_mqa_single_correct_sk:vsp|0
scio.prompt1_mqa_single_correct_sk:aj|0
scio.prompt1_mqa_single_correct_sk:mat|0
scio.prompt1_mqa_single_correct_sk:bio|0
scio.prompt1_mqa_single_correct_sk:che|0
scio.prompt1_mqa_single_correct_sk:vsp|0
globalmmlu-lite.prompt3_mqa_single_correct_sk|0
globalmmlu-lite.prompt1_mqa_single_correct_sk|0
globalmmlu-lite.prompt2_mqa_single_correct_sk|0

448
perun/ministral_3b_aya/training/data_manifest.json Normal file
View File

@ -0,0 +1,448 @@
{
"counts": {
"duplicate_pairs_removed": 101879,
"groups_with_multiple_responses": 133562,
"invalid_or_empty_rows": 283,
"prompt_groups": 2339573,
"retained_rows": 3492041,
"source_rows": 3594203
},
"files": {
"dataset_info.json": {
"bytes": 560,
"sha256": "049ff7f6702cdf4e49f0afb491bb2aad4552e96c96565f191564665fe8c74d1d"
},
"eval.jsonl": {
"bytes": 475745,
"rows": 1000,
"sha256": "154ebcf5f73f1711865d389b3a7d483ca1006098ff5d25881bbabb5cfcb8a1e0"
},
"eval.metadata.jsonl": {
"bytes": 416790,
"rows": 1000,
"sha256": "7864cb71daf35d56ea3caebb496108839d7d4e36653effd0741507e21a8e1e47"
},
"test.jsonl": {
"bytes": 449384,
"rows": 1000,
"sha256": "9e6df865c825979b5b3d458a8a832406ce5a859e8f9aaa9e567b7377e41c1324"
},
"test.metadata.jsonl": {
"bytes": 414881,
"rows": 1000,
"sha256": "a5a46723acc22f20c99a29e3b338d6986201c943c6b937f7402e2f04f671c8f3"
},
"train.jsonl": {
"bytes": 1636857404,
"rows": 3490041,
"sha256": "39acf85df1401f9748f665d29754bd57234f45bca37195eb891f3ce696e5a1ab"
},
"train.metadata.jsonl": {
"bytes": 1464274344,
"rows": 3490041,
"sha256": "2d8ce0a5394487463ea8292b6e9bf376a11dd4f79700292cf8a65fe9ca3a66a3"
}
},
"format_version": 1,
"preparation_script_sha256": "5b126cdcfcaf0e38a38ff45f98fde9409b69eb10a925d6444990a90f99830b9b",
"prepared_at_utc": "2026-11-08T03:52:39.903856+00:00",
"requested_holdout_max_rows": {
"eval": 1000,
"test": 1000
},
"seed": 42,
"source": {
"bytes": 931479534,
"kind": "local_parquet",
"path": "/mnt/project/perun26012578/raw/slovak_train.parquet",
"revision": null,
"sha256": "6a21c52b80549cbb8f9d13de981a2d8f65c273a17f651fd09c1efcc952821e82"
},
"source_breakdown": [
{
"dataset_name": "Adversarial QA (T)",
"prompt_groups": 4,
"rows": 4,
"split": "eval",
"task_type": "question-answering"
},
{
"dataset_name": "CNN-Daily-Mail (T)",
"prompt_groups": 29,
"rows": 29,
"split": "eval",
"task_type": "summarization"
},
{
"dataset_name": "Dolly-v2 (T)",
"prompt_groups": 6,
"rows": 6,
"split": "eval",
"task_type": "generation"
},
{
"dataset_name": "Flan-CoT-submix (T)",
"prompt_groups": 30,
"rows": 30,
"split": "eval",
"task_type": "generation"
},
{
"dataset_name": "Flan-Coqa (T)",
"prompt_groups": 1,
"rows": 1,
"split": "eval",
"task_type": "question-answering"
},
{
"dataset_name": "Flan-GEM-wiki-lingua (T)",
"prompt_groups": 9,
"rows": 9,
"split": "eval",
"task_type": "summarization"
},
{
"dataset_name": "Flan-lambada (T)",
"prompt_groups": 2,
"rows": 2,
"split": "eval",
"task_type": "generation"
},
{
"dataset_name": "Flan-unified-QA (T)",
"prompt_groups": 2,
"rows": 2,
"split": "eval",
"task_type": "question-answering"
},
{
"dataset_name": "HotpotQA (T)",
"prompt_groups": 80,
"rows": 80,
"split": "eval",
"task_type": "question-answering"
},
{
"dataset_name": "NQ-Open (T)",
"prompt_groups": 60,
"rows": 60,
"split": "eval",
"task_type": "question-answering"
},
{
"dataset_name": "PAWS-Wiki (T)",
"prompt_groups": 7,
"rows": 7,
"split": "eval",
"task_type": "paraphrase-identification"
},
{
"dataset_name": "PIQA (T)",
"prompt_groups": 7,
"rows": 7,
"split": "eval",
"task_type": "question-answering"
},
{
"dataset_name": "SODA-inst (T)",
"prompt_groups": 34,
"rows": 267,
"split": "eval",
"task_type": "dialogue"
},
{
"dataset_name": "UNER-LLM-inst",
"prompt_groups": 2,
"rows": 2,
"split": "eval",
"task_type": "named-entity-recognition"
},
{
"dataset_name": "Wiki-split-inst (T)",
"prompt_groups": 329,
"rows": 334,
"split": "eval",
"task_type": "text-simplification"
},
{
"dataset_name": "Xlel_wd-inst",
"prompt_groups": 4,
"rows": 4,
"split": "eval",
"task_type": "event-linking"
},
{
"dataset_name": "Xlel_wd-inst (T)",
"prompt_groups": 156,
"rows": 156,
"split": "eval",
"task_type": "event-linking"
},
{
"dataset_name": "Adversarial QA (T)",
"prompt_groups": 3,
"rows": 3,
"split": "test",
"task_type": "question-answering"
},
{
"dataset_name": "CNN-Daily-Mail (T)",
"prompt_groups": 19,
"rows": 19,
"split": "test",
"task_type": "summarization"
},
{
"dataset_name": "Dolly-v2 (T)",
"prompt_groups": 3,
"rows": 3,
"split": "test",
"task_type": "generation"
},
{
"dataset_name": "Flan-CoT-submix (T)",
"prompt_groups": 28,
"rows": 28,
"split": "test",
"task_type": "generation"
},
{
"dataset_name": "Flan-Coqa (T)",
"prompt_groups": 3,
"rows": 3,
"split": "test",
"task_type": "question-answering"
},
{
"dataset_name": "Flan-GEM-wiki-lingua (T)",
"prompt_groups": 8,
"rows": 8,
"split": "test",
"task_type": "summarization"
},
{
"dataset_name": "HotpotQA (T)",
"prompt_groups": 67,
"rows": 67,
"split": "test",
"task_type": "question-answering"
},
{
"dataset_name": "Joke-explaination-inst (T)",
"prompt_groups": 1,
"rows": 1,
"split": "test",
"task_type": "generation"
},
{
"dataset_name": "NQ-Open (T)",
"prompt_groups": 49,
"rows": 51,
"split": "test",
"task_type": "question-answering"
},
{
"dataset_name": "PAWS-Wiki (T)",
"prompt_groups": 15,
"rows": 15,
"split": "test",
"task_type": "paraphrase-identification"
},
{
"dataset_name": "PIQA (T)",
"prompt_groups": 7,
"rows": 7,
"split": "test",
"task_type": "question-answering"
},
{
"dataset_name": "SODA-inst (T)",
"prompt_groups": 37,
"rows": 379,
"split": "test",
"task_type": "dialogue"
},
{
"dataset_name": "UNER-LLM-inst",
"prompt_groups": 4,
"rows": 4,
"split": "test",
"task_type": "named-entity-recognition"
},
{
"dataset_name": "Wiki-split-inst (T)",
"prompt_groups": 273,
"rows": 277,
"split": "test",
"task_type": "text-simplification"
},
{
"dataset_name": "Xlel_wd-inst",
"prompt_groups": 4,
"rows": 4,
"split": "test",
"task_type": "event-linking"
},
{
"dataset_name": "Xlel_wd-inst (T)",
"prompt_groups": 131,
"rows": 131,
"split": "test",
"task_type": "event-linking"
},
{
"dataset_name": "Adversarial QA (T)",
"prompt_groups": 9686,
"rows": 9984,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "CNN-Daily-Mail (T)",
"prompt_groups": 98366,
"rows": 98381,
"split": "train",
"task_type": "summarization"
},
{
"dataset_name": "Dolly-v2 (T)",
"prompt_groups": 14559,
"rows": 14783,
"split": "train",
"task_type": "generation"
},
{
"dataset_name": "Flan-CoT-submix (T)",
"prompt_groups": 88084,
"rows": 91852,
"split": "train",
"task_type": "generation"
},
{
"dataset_name": "Flan-Coqa (T)",
"prompt_groups": 6405,
"rows": 6405,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "Flan-GEM-wiki-lingua (T)",
"prompt_groups": 27128,
"rows": 27130,
"split": "train",
"task_type": "summarization"
},
{
"dataset_name": "Flan-lambada (T)",
"prompt_groups": 4277,
"rows": 4277,
"split": "train",
"task_type": "generation"
},
{
"dataset_name": "Flan-unified-QA (T)",
"prompt_groups": 538,
"rows": 538,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "HotpotQA (T)",
"prompt_groups": 264664,
"rows": 266105,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "Joke-explaination-inst (T)",
"prompt_groups": 724,
"rows": 733,
"split": "train",
"task_type": "generation"
},
{
"dataset_name": "Mintaka-inst (T)",
"prompt_groups": 8,
"rows": 13834,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "NQ-Open (T)",
"prompt_groups": 170153,
"rows": 172361,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "PAWS-Wiki (T)",
"prompt_groups": 46682,
"rows": 46746,
"split": "train",
"task_type": "paraphrase-identification"
},
{
"dataset_name": "PIQA (T)",
"prompt_groups": 16092,
"rows": 16093,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "SODA-inst (T)",
"prompt_groups": 116604,
"rows": 1190936,
"split": "train",
"task_type": "dialogue"
},
{
"dataset_name": "UNER-LLM-inst",
"prompt_groups": 8195,
"rows": 8197,
"split": "train",
"task_type": "named-entity-recognition"
},
{
"dataset_name": "WIKI QA (T)",
"prompt_groups": 1030,
"rows": 1039,
"split": "train",
"task_type": "question-answering"
},
{
"dataset_name": "Wiki-split-inst (T)",
"prompt_groups": 970135,
"rows": 988524,
"split": "train",
"task_type": "text-simplification"
},
{
"dataset_name": "Xlel_wd-inst",
"prompt_groups": 12433,
"rows": 13268,
"split": "train",
"task_type": "event-linking"
},
{
"dataset_name": "Xlel_wd-inst (T)",
"prompt_groups": 482400,
"rows": 518855,
"split": "train",
"task_type": "event-linking"
}
],
"splits": {
"eval": {
"prompt_groups": 762,
"rows": 1000
},
"test": {
"prompt_groups": 652,
"rows": 1000
},
"train": {
"prompt_groups": 2338159,
"rows": 3490041
}
}
}

7
perun/ministral_3b_aya/training/eval_results.json Normal file
View File

@ -0,0 +1,7 @@
{
"epoch": 1.0,
"eval_loss": 1.1308903694152832,
"eval_runtime": 3.3257,
"eval_samples_per_second": 300.687,
"eval_steps_per_second": 9.622
}

61
perun/ministral_3b_aya/training/run.json Normal file
View File

@ -0,0 +1,61 @@
{
"config": {
"model_name_or_path": "ministral/Ministral-3b-instruct",
"model_revision": "2c95908929198d6e69af8638f0dbbd9bc6b93f9e",
"stage": "sft",
"do_train": true,
"do_eval": true,
"finetuning_type": "lora",
"use_unsloth": true,
"use_unsloth_gc": true,
"bf16": true,
"infer_dtype": "bfloat16",
"flash_attn": "sdpa",
"dataset_dir": "/mnt/project/perun26012578/perun/data/aya_slovak_fixed",
"dataset": "aya_train",
"eval_dataset": "aya_eval",
"cutoff_len": 4096,
"packing": false,
"train_on_prompt": false,
"preprocessing_num_workers": 8,
"dataloader_num_workers": 2,
"overwrite_cache": false,
"lora_rank": 16,
"lora_alpha": 32,
"lora_dropout": 0.0,
"lora_target": "q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj",
"per_device_train_batch_size": 8,
"per_device_eval_batch_size": 8,
"gradient_accumulation_steps": 1,
"learning_rate": 0.0002,
"num_train_epochs": 1.0,
"lr_scheduler_type": "cosine",
"warmup_ratio": 0.08,
"optim": "adamw_torch",
"weight_decay": 0.0,
"max_grad_norm": 0.5,
"gradient_checkpointing": true,
"ddp_find_unused_parameters": false,
"output_dir": "/mnt/project/perun26012578/perun/models/ministral-3b-instruct/runs/full_aya_lr_0p000200_epoch1/adapter",
"overwrite_output_dir": false,
"logging_steps": 10,
"eval_strategy": "steps",
"eval_steps": 1000,
"save_strategy": "steps",
"save_steps": 1000,
"save_total_limit": 2,
"save_only_model": false,
"load_best_model_at_end": true,
"metric_for_best_model": "loss",
"greater_is_better": false,
"predict_with_generate": false,
"plot_loss": true,
"report_to": "none",
"seed": 42,
"data_seed": 42
},
"gpus": 4,
"effective_batch": 32,
"data_manifest_sha256": "ca76b5f54a7e899c535443db58d4438eb8267b974a99e928c9d83603b05afbc6",
"created_at_utc": "2026-11-09T16:46:52.760203+00:00"
}

8
perun/ministral_3b_aya/training/train_results.json Normal file
View File

@ -0,0 +1,8 @@
{
"epoch": 1.0,
"total_flos": 3.332743094900831e+19,
"train_loss": 1.3334437627037912,
"train_runtime": 86760.7126,
"train_samples_per_second": 40.226,
"train_steps_per_second": 1.257
}

BIN
perun/ministral_3b_aya/training/training_eval_loss.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

BIN
perun/ministral_3b_aya/training/training_loss.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

111
perun/ministral_3b_aya/training/validation_loss.csv Normal file
View File

@ -0,0 +1,111 @@
step,epoch,eval_loss
1000,0.009168928335656128,2.360072612762451
2000,0.018337856671312255,2.0450692176818848
3000,0.027506785006968387,1.8085086345672607
4000,0.03667571334262451,1.6807985305786133
5000,0.045844641678280645,1.6042695045471191
6000,0.05501357001393677,1.5622832775115967
7000,0.0641824983495929,1.5304133892059326
8000,0.07335142668524902,1.5210760831832886
9000,0.08252035502090516,1.5131747722625732
10000,0.09168928335656129,1.5016822814941406
11000,0.10085821169221741,1.4817888736724854
12000,0.11002714002787355,1.4736380577087402
13000,0.11919606836352967,1.4612903594970703
14000,0.1283649966991858,1.4510605335235596
15000,0.13753392503484194,1.4477511644363403
16000,0.14670285337049804,1.440198540687561
17000,0.15587178170615418,1.4364843368530273
18000,0.1650407100418103,1.430030345916748
19000,0.17420963837746645,1.4217251539230347
20000,0.18337856671312258,1.4251708984375
21000,0.1925474950487787,1.4137701988220215
22000,0.20171642338443482,1.414902687072754
23000,0.21088535172009096,1.4114439487457275
24000,0.2200542800557471,1.4006237983703613
25000,0.2292232083914032,1.396726369857788
26000,0.23839213672705933,1.3982290029525757
27000,0.24756106506271547,1.3934869766235352
28000,0.2567299933983716,1.3892569541931152
29000,0.26589892173402774,1.3871415853500366
30000,0.2750678500696839,1.3804394006729126
31000,0.28423677840534,1.3778815269470215
32000,0.2934057067409961,1.369511365890503
33000,0.3025746350766522,1.3662126064300537
34000,0.31174356341230836,1.362453579902649
35000,0.3209124917479645,1.3640830516815186
36000,0.3300814200836206,1.3563292026519775
37000,0.33925034841927676,1.350153923034668
38000,0.3484192767549329,1.3465571403503418
39000,0.35758820509058903,1.3438115119934082
40000,0.36675713342624516,1.3429770469665527
41000,0.37592606176190124,1.3332430124282837
42000,0.3850949900975574,1.334341049194336
43000,0.3942639184332135,1.3271080255508423
44000,0.40343284676886965,1.322904348373413
45000,0.4126017751045258,1.3139293193817139
46000,0.4217707034401819,1.3120628595352173
47000,0.43093963177583805,1.3108934164047241
48000,0.4401085601114942,1.3062814474105835
49000,0.4492774884471503,1.3023930788040161
50000,0.4584464167828064,1.2967934608459473
51000,0.46761534511846253,1.2952461242675781
52000,0.47678427345411867,1.288960576057434
53000,0.4859532017897748,1.2882647514343262
54000,0.49512213012543094,1.2822253704071045
55000,0.504291058461087,1.275566577911377
56000,0.5134599867967432,1.2699377536773682
57000,0.5226289151323993,1.268863320350647
58000,0.5317978434680555,1.2632348537445068
59000,0.5409667718037116,1.2605316638946533
60000,0.5501357001393677,1.258490800857544
61000,0.5593046284750238,1.2527873516082764
62000,0.56847355681068,1.249812364578247
63000,0.5776424851463361,1.2420779466629028
64000,0.5868114134819922,1.2373592853546143
65000,0.5959803418176484,1.2303643226623535
66000,0.6051492701533044,1.230921983718872
67000,0.6143181984889606,1.2246050834655762
68000,0.6234871268246167,1.224005103111267
69000,0.6326560551602729,1.2206971645355225
70000,0.641824983495929,1.2165570259094238
71000,0.6509939118315852,1.2117177248001099
72000,0.6601628401672412,1.2068102359771729
73000,0.6693317685028973,1.2016609907150269
74000,0.6785006968385535,1.1967077255249023
75000,0.6876696251742096,1.1926217079162598
76000,0.6968385535098658,1.1919411420822144
77000,0.7060074818455219,1.1873736381530762
78000,0.7151764101811781,1.1825459003448486
79000,0.7243453385168341,1.1790175437927246
80000,0.7335142668524903,1.1748923063278198
81000,0.7426831951881464,1.1718367338180542
82000,0.7518521235238025,1.1701050996780396
83000,0.7610210518594587,1.1669533252716064
84000,0.7701899801951148,1.163935899734497
85000,0.7793589085307709,1.1622910499572754
86000,0.788527836866427,1.159493088722229
87000,0.7976967652020832,1.1554956436157227
88000,0.8068656935377393,1.1514334678649902
89000,0.8160346218733955,1.149964690208435
90000,0.8252035502090516,1.1484603881835938
91000,0.8343724785447076,1.1455662250518799
92000,0.8435414068803638,1.145362377166748
93000,0.8527103352160199,1.1426365375518799
94000,0.8618792635516761,1.1414161920547485
95000,0.8710481918873322,1.139862060546875
96000,0.8802171202229884,1.1378090381622314
97000,0.8893860485586444,1.13651442527771
98000,0.8985549768943006,1.1353960037231445
99000,0.9077239052299567,1.1344630718231201
100000,0.9168928335656128,1.1332764625549316
101000,0.926061761901269,1.1325769424438477
102000,0.9352306902369251,1.132089614868164
103000,0.9443996185725813,1.1317222118377686
104000,0.9535685469082373,1.1315722465515137
105000,0.9627374752438935,1.1312942504882812
106000,0.9719064035795496,1.1310818195343018
107000,0.9810753319152058,1.131012201309204
108000,0.9902442602508619,1.130906343460083
109000,0.999413188586518,1.1308903694152832
109064,1.0,1.1309707164764404
1 step epoch eval_loss
2 1000 0.009168928335656128 2.360072612762451
3 2000 0.018337856671312255 2.0450692176818848
4 3000 0.027506785006968387 1.8085086345672607
5 4000 0.03667571334262451 1.6807985305786133
6 5000 0.045844641678280645 1.6042695045471191
7 6000 0.05501357001393677 1.5622832775115967
8 7000 0.0641824983495929 1.5304133892059326
9 8000 0.07335142668524902 1.5210760831832886
10 9000 0.08252035502090516 1.5131747722625732
11 10000 0.09168928335656129 1.5016822814941406
12 11000 0.10085821169221741 1.4817888736724854
13 12000 0.11002714002787355 1.4736380577087402
14 13000 0.11919606836352967 1.4612903594970703
15 14000 0.1283649966991858 1.4510605335235596
16 15000 0.13753392503484194 1.4477511644363403
17 16000 0.14670285337049804 1.440198540687561
18 17000 0.15587178170615418 1.4364843368530273
19 18000 0.1650407100418103 1.430030345916748
20 19000 0.17420963837746645 1.4217251539230347
21 20000 0.18337856671312258 1.4251708984375
22 21000 0.1925474950487787 1.4137701988220215
23 22000 0.20171642338443482 1.414902687072754
24 23000 0.21088535172009096 1.4114439487457275
25 24000 0.2200542800557471 1.4006237983703613
26 25000 0.2292232083914032 1.396726369857788
27 26000 0.23839213672705933 1.3982290029525757
28 27000 0.24756106506271547 1.3934869766235352
29 28000 0.2567299933983716 1.3892569541931152
30 29000 0.26589892173402774 1.3871415853500366
31 30000 0.2750678500696839 1.3804394006729126
32 31000 0.28423677840534 1.3778815269470215
33 32000 0.2934057067409961 1.369511365890503
34 33000 0.3025746350766522 1.3662126064300537
35 34000 0.31174356341230836 1.362453579902649
36 35000 0.3209124917479645 1.3640830516815186
37 36000 0.3300814200836206 1.3563292026519775
38 37000 0.33925034841927676 1.350153923034668
39 38000 0.3484192767549329 1.3465571403503418
40 39000 0.35758820509058903 1.3438115119934082
41 40000 0.36675713342624516 1.3429770469665527
42 41000 0.37592606176190124 1.3332430124282837
43 42000 0.3850949900975574 1.334341049194336
44 43000 0.3942639184332135 1.3271080255508423
45 44000 0.40343284676886965 1.322904348373413
46 45000 0.4126017751045258 1.3139293193817139
47 46000 0.4217707034401819 1.3120628595352173
48 47000 0.43093963177583805 1.3108934164047241
49 48000 0.4401085601114942 1.3062814474105835
50 49000 0.4492774884471503 1.3023930788040161
51 50000 0.4584464167828064 1.2967934608459473
52 51000 0.46761534511846253 1.2952461242675781
53 52000 0.47678427345411867 1.288960576057434
54 53000 0.4859532017897748 1.2882647514343262
55 54000 0.49512213012543094 1.2822253704071045
56 55000 0.504291058461087 1.275566577911377
57 56000 0.5134599867967432 1.2699377536773682
58 57000 0.5226289151323993 1.268863320350647
59 58000 0.5317978434680555 1.2632348537445068
60 59000 0.5409667718037116 1.2605316638946533
61 60000 0.5501357001393677 1.258490800857544
62 61000 0.5593046284750238 1.2527873516082764
63 62000 0.56847355681068 1.249812364578247
64 63000 0.5776424851463361 1.2420779466629028
65 64000 0.5868114134819922 1.2373592853546143
66 65000 0.5959803418176484 1.2303643226623535
67 66000 0.6051492701533044 1.230921983718872
68 67000 0.6143181984889606 1.2246050834655762
69 68000 0.6234871268246167 1.224005103111267
70 69000 0.6326560551602729 1.2206971645355225
71 70000 0.641824983495929 1.2165570259094238
72 71000 0.6509939118315852 1.2117177248001099
73 72000 0.6601628401672412 1.2068102359771729
74 73000 0.6693317685028973 1.2016609907150269
75 74000 0.6785006968385535 1.1967077255249023
76 75000 0.6876696251742096 1.1926217079162598
77 76000 0.6968385535098658 1.1919411420822144
78 77000 0.7060074818455219 1.1873736381530762
79 78000 0.7151764101811781 1.1825459003448486
80 79000 0.7243453385168341 1.1790175437927246
81 80000 0.7335142668524903 1.1748923063278198
82 81000 0.7426831951881464 1.1718367338180542
83 82000 0.7518521235238025 1.1701050996780396
84 83000 0.7610210518594587 1.1669533252716064
85 84000 0.7701899801951148 1.163935899734497
86 85000 0.7793589085307709 1.1622910499572754
87 86000 0.788527836866427 1.159493088722229
88 87000 0.7976967652020832 1.1554956436157227
89 88000 0.8068656935377393 1.1514334678649902
90 89000 0.8160346218733955 1.149964690208435
91 90000 0.8252035502090516 1.1484603881835938
92 91000 0.8343724785447076 1.1455662250518799
93 92000 0.8435414068803638 1.145362377166748
94 93000 0.8527103352160199 1.1426365375518799
95 94000 0.8618792635516761 1.1414161920547485
96 95000 0.8710481918873322 1.139862060546875
97 96000 0.8802171202229884 1.1378090381622314
98 97000 0.8893860485586444 1.13651442527771
99 98000 0.8985549768943006 1.1353960037231445
100 99000 0.9077239052299567 1.1344630718231201
101 100000 0.9168928335656128 1.1332764625549316
102 101000 0.926061761901269 1.1325769424438477
103 102000 0.9352306902369251 1.132089614868164
104 103000 0.9443996185725813 1.1317222118377686
105 104000 0.9535685469082373 1.1315722465515137
106 105000 0.9627374752438935 1.1312942504882812
107 106000 0.9719064035795496 1.1310818195343018
108 107000 0.9810753319152058 1.131012201309204
109 108000 0.9902442602508619 1.130906343460083
110 109000 0.999413188586518 1.1308903694152832
111 109064 1.0 1.1309707164764404

5
perun/scripts/evaluate.py
View File

@ -11,7 +11,7 @@ from pathlib import Path
import subprocess import subprocess
import sys import sys
import unicodedata import unicodedata
from train import ROOT, read_config, write_config, verify_data, verify_length_filter, run_logged from train import ROOT, read_config, write_config, verify_data, run_logged
def normalized_tokens(text: str) -> list[str]: def normalized_tokens(text: str) -> list[str]:
text = unicodedata.normalize("NFKC", text).casefold() text = unicodedata.normalize("NFKC", text).casefold()
@ -231,8 +231,7 @@ def main() -> int:
parser.error("--max-new-tokens must be positive.") parser.error("--max-new-tokens must be positive.")
config = read_config(args.config.resolve()) config = read_config(args.config.resolve())
data_dir = Path(config["dataset_dir"]) data_dir = Path(config["dataset_dir"])
manifest = verify_data(data_dir, (args.split,)) verify_data(data_dir, (args.split,))
verify_length_filter(manifest, config)
adapter = args.adapter.resolve() if args.adapter else Path(config["output_dir"]) adapter = args.adapter.resolve() if args.adapter else Path(config["output_dir"])
if not (adapter / "adapter_config.json").is_file(): if not (adapter / "adapter_config.json").is_file():
raise ValueError(f"Missing trained adapter: {adapter}") raise ValueError(f"Missing trained adapter: {adapter}")

144
perun/scripts/prepare_data.py
View File

@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Prepare Slovak Aya train/eval/test data for LLaMA-Factory.""" """Prepare one shared Slovak Aya train/eval/test dataset for LLaMA-Factory."""
from __future__ import annotations from __future__ import annotations
@ -7,7 +7,6 @@ import argparse
from collections import Counter from collections import Counter
from datetime import datetime, timezone from datetime import datetime, timezone
import hashlib import hashlib
from itertools import islice
import json import json
from pathlib import Path from pathlib import Path
import sqlite3 import sqlite3
@ -58,18 +57,32 @@ def local_rows(path: Path) -> Iterator[Any]:
raise ValueError(f"Invalid JSON at {path}:{line_number}: {error}") from error raise ValueError(f"Invalid JSON at {path}:{line_number}: {error}") from error
def parquet_rows(parquet: Any) -> Iterator[dict[str, Any]]:
with parquet:
for batch in parquet.iter_batches(batch_size=2048):
yield from batch.to_pylist()
def open_source(args: argparse.Namespace) -> tuple[Iterator[Any], dict[str, Any]]: def open_source(args: argparse.Namespace) -> tuple[Iterator[Any], dict[str, Any]]:
if args.local_jsonl: local_path = args.local_parquet or args.local_jsonl
path = args.local_jsonl.expanduser().resolve(strict=True) if local_path:
path = local_path.expanduser().resolve(strict=True)
if not path.is_file(): if not path.is_file():
raise ValueError(f"Local source is not a file: {path}") raise ValueError(f"Local source is not a file: {path}")
return local_rows(path), { source = {
"kind": "local_jsonl", "kind": "local_parquet" if args.local_parquet else "local_jsonl",
"path": str(path), "path": str(path),
"sha256": file_sha256(path), "sha256": file_sha256(path),
"bytes": path.stat().st_size, "bytes": path.stat().st_size,
"revision": None, "revision": None,
} }
if args.local_parquet:
try:
from pyarrow.parquet import ParquetFile
except ImportError as error:
raise RuntimeError("Install pyarrow to read --local-parquet input.") from error
return parquet_rows(ParquetFile(path)), source
return local_rows(path), source
try: try:
from datasets import load_dataset from datasets import load_dataset
@ -148,94 +161,15 @@ def create_database(path: Path) -> sqlite3.Connection:
return connection return connection
SYSTEM_TEXT = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n" def ingest(connection: sqlite3.Connection, rows: Iterator[Any], seed: int) -> Counter:
USER_TEXT = "### Instruction:\n{}\n\n### Response:\n"
class SFTLengthFilter:
"""Keep a whole pair only when its exact Alpaca length fits both model configs."""
def __init__(self, config_dir: Path):
import yaml
from transformers import AutoTokenizer
self.profiles = {}
self.excluded_by_model = Counter()
self.max_accepted_tokens = Counter()
for name in ("mistral", "olmo"):
settings = yaml.safe_load((config_dir / f"{name}.yaml").read_text(encoding="utf-8-sig"))
if settings.get("template") != "alpaca":
raise ValueError("Length filtering currently requires the Alpaca template for both models.")
limit = int(settings["cutoff_len"])
if limit < 1:
raise ValueError("cutoff_len must be positive.")
tokenizer = AutoTokenizer.from_pretrained(settings["model_name_or_path"],
revision=settings["model_revision"], use_fast=True, fix_mistral_regex=True,
model_max_length=10**9)
backend = tokenizer.backend_tokenizer
backend.no_truncation()
backend.no_padding()
if tokenizer.eos_token_id is None:
raise ValueError(f"Missing EOS token: {name}")
self.profiles[name] = {
"model_name_or_path": settings["model_name_or_path"],
"model_revision": settings["model_revision"], "template": "alpaca",
"cutoff_len": limit, "tokenizer_class": type(tokenizer).__name__,
"system_tokens": len(tokenizer.encode(SYSTEM_TEXT, add_special_tokens=False)),
"ending_tokens": 1 + len(tokenizer.encode("\n\n", add_special_tokens=False)),
"backend": backend,
}
def lengths(self, examples: list[dict[str, str]]) -> list[dict[str, int]]:
prompts = [USER_TEXT.format("\n".join(part for part in (row["instruction"], row["input"]) if part))
for row in examples]
responses = [row["output"] for row in examples]
result = [{} for _ in examples]
for name, profile in self.profiles.items():
backend = profile["backend"]
encode = getattr(backend, "encode_batch_fast", backend.encode_batch)
inputs = encode(prompts, add_special_tokens=False)
targets = encode(responses, add_special_tokens=False)
for item, source, target in zip(result, inputs, targets):
item[name] = profile["system_tokens"] + len(source) + len(target) + profile["ending_tokens"]
return result
def over_limit(self, lengths: dict[str, int]) -> list[str]:
return [name for name, length in lengths.items() if length > self.profiles[name]["cutoff_len"]]
def report(self) -> dict:
return {"enabled": True,
"models": {name: {key: value for key, value in profile.items() if key != "backend"}
for name, profile in self.profiles.items()},
"excluded_by_model": {name: self.excluded_by_model[name] for name in self.profiles},
"max_accepted_tokens_before_pair_dedup": {name: self.max_accepted_tokens[name] for name in self.profiles},
"excluded_log": "excluded_long.jsonl"}
def ingest(connection: sqlite3.Connection, rows: Iterator[Any], seed: int,
length_filter: SFTLengthFilter, excluded) -> Counter:
counts: Counter = Counter() counts: Counter = Counter()
indexed = enumerate(rows) for source_index, row in enumerate(rows):
next_progress = 10000 counts["source_rows"] += 1
while batch := list(islice(indexed, 512)): converted = convert_row(row)
candidates = [] if converted is None:
for source_index, row in batch: counts["invalid_or_empty_rows"] += 1
counts["source_rows"] += 1 else:
converted = convert_row(row) data, metadata = converted
if converted is None:
counts["invalid_or_empty_rows"] += 1
else:
candidates.append((source_index, *converted))
measured = length_filter.lengths([data for _, data, _ in candidates])
for (source_index, data, metadata), lengths in zip(candidates, measured):
over = length_filter.over_limit(lengths)
if over:
counts["filtered_long_rows"] += 1
length_filter.excluded_by_model.update(over)
excluded.write(json.dumps({"source_row_index": source_index, "source_id": metadata.get("id"),
"dataset_name": metadata.get("dataset_name"), "task_type": metadata.get("task_type"),
"token_lengths": lengths, "over_limit_models": over}, ensure_ascii=False) + "\n")
continue
for name, length in lengths.items():
length_filter.max_accepted_tokens[name] = max(length_filter.max_accepted_tokens[name], length)
prompt_key = [normalized(data["instruction"], fold_case=True), normalized(data["input"], fold_case=True)] prompt_key = [normalized(data["instruction"], fold_case=True), normalized(data["input"], fold_case=True)]
group_id = digest_json(prompt_key) group_id = digest_json(prompt_key)
stable_id = digest_json([prompt_key, normalized(data["output"])]) stable_id = digest_json([prompt_key, normalized(data["output"])])
@ -255,15 +189,14 @@ def ingest(connection: sqlite3.Connection, rows: Iterator[Any], seed: int,
"""INSERT INTO prompt_groups (group_id, order_key, size) VALUES (?, ?, 1) """INSERT INTO prompt_groups (group_id, order_key, size) VALUES (?, ?, 1)
ON CONFLICT(group_id) DO UPDATE SET size = size + 1""", ON CONFLICT(group_id) DO UPDATE SET size = size + 1""",
(group_id, digest_json([seed, group_id]))) (group_id, digest_json([seed, group_id])))
if counts["source_rows"] >= next_progress: if counts["source_rows"] % 10000 == 0:
connection.commit() connection.commit()
print(f"Read {counts['source_rows']:,}; filtered long {counts['filtered_long_rows']:,}; retained {counts['retained_rows']:,}", flush=True) print(f"Read {counts['source_rows']:,}; retained {counts['retained_rows']:,}", flush=True)
next_progress += 10000
connection.commit() connection.commit()
for key in ("source_rows", "invalid_or_empty_rows", "filtered_long_rows", "duplicate_pairs_removed", "retained_rows"): for key in ("source_rows", "invalid_or_empty_rows", "duplicate_pairs_removed", "retained_rows"):
counts.setdefault(key, 0) counts.setdefault(key, 0)
if not counts["retained_rows"]: if not counts["retained_rows"]:
raise ValueError("The source contains no usable examples after validation and length filtering.") raise ValueError("The source contains no usable examples after validation.")
connection.execute("CREATE INDEX prompt_group_order ON prompt_groups(order_key, group_id)") connection.execute("CREATE INDEX prompt_group_order ON prompt_groups(order_key, group_id)")
connection.execute("CREATE INDEX records_group ON records(group_id)") connection.execute("CREATE INDEX records_group ON records(group_id)")
connection.commit() connection.commit()
@ -348,7 +281,9 @@ def main() -> int:
parser = argparse.ArgumentParser(description=__doc__) parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output-dir", type=Path, required=True) parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--revision", help="Hugging Face revision; resolved to an immutable commit before loading.") parser.add_argument("--revision", help="Hugging Face revision; resolved to an immutable commit before loading.")
parser.add_argument("--local-jsonl", type=Path, help="Offline JSONL input in Aya or Alpaca format.") local_input = parser.add_mutually_exclusive_group()
local_input.add_argument("--local-jsonl", type=Path, help="Offline JSONL input in Aya or Alpaca format.")
local_input.add_argument("--local-parquet", type=Path, help="Offline Parquet input in Aya or Alpaca format.")
parser.add_argument("--eval-size", type=int, default=1000, help="Maximum number of eval records (default: 1000).") parser.add_argument("--eval-size", type=int, default=1000, help="Maximum number of eval records (default: 1000).")
parser.add_argument("--test-size", type=int, default=1000, help="Maximum number of test records (default: 1000).") parser.add_argument("--test-size", type=int, default=1000, help="Maximum number of test records (default: 1000).")
parser.add_argument("--seed", type=int, default=42) parser.add_argument("--seed", type=int, default=42)
@ -356,7 +291,7 @@ def main() -> int:
args = parser.parse_args() args = parser.parse_args()
if min(args.eval_size, args.test_size) < 0: if min(args.eval_size, args.test_size) < 0:
parser.error("Holdout sizes must be nonnegative.") parser.error("Holdout sizes must be nonnegative.")
if args.local_jsonl and args.revision: if (args.local_jsonl or args.local_parquet) and args.revision:
parser.error("--revision applies only to Hugging Face input.") parser.error("--revision applies only to Hugging Face input.")
output = args.output_dir.expanduser().resolve() output = args.output_dir.expanduser().resolve()
if output.exists() and (not output.is_dir() or any(output.iterdir())): if output.exists() and (not output.is_dir() or any(output.iterdir())):
@ -364,21 +299,15 @@ def main() -> int:
# Resolve source before creating output, so authentication or dependency # Resolve source before creating output, so authentication or dependency
# errors do not produce a misleading empty data directory. # errors do not produce a misleading empty data directory.
rows, source = open_source(args) rows, source = open_source(args)
length_filter = SFTLengthFilter(Path(__file__).resolve().parents[1] / "configs")
output.mkdir(parents=True, exist_ok=True) output.mkdir(parents=True, exist_ok=True)
database_path = output / "preparation.sqlite3" database_path = output / "preparation.sqlite3"
connection = create_database(database_path) connection = create_database(database_path)
try: try:
excluded_path = output / "excluded_long.jsonl" counts = ingest(connection, rows, args.seed)
with excluded_path.open("x", encoding="utf-8", newline="\n") as excluded:
counts = ingest(connection, rows, args.seed, length_filter, excluded)
splits = assign_splits(connection, args.eval_size, args.test_size) splits = assign_splits(connection, args.eval_size, args.test_size)
if sum(item["rows"] for item in splits.values()) != counts["retained_rows"]: if sum(item["rows"] for item in splits.values()) != counts["retained_rows"]:
raise RuntimeError("Split counts do not sum to the retained record count.") raise RuntimeError("Split counts do not sum to the retained record count.")
files = emit_splits(connection, output, splits) files = emit_splits(connection, output, splits)
files[excluded_path.name] = {"sha256": file_sha256(excluded_path),
"bytes": excluded_path.stat().st_size,
"rows": counts["filtered_long_rows"]}
registration = { registration = {
f"aya_{split}": { f"aya_{split}": {
"file_name": f"{split}.jsonl", "formatting": "alpaca", "file_name": f"{split}.jsonl", "formatting": "alpaca",
@ -404,7 +333,6 @@ def main() -> int:
"source": source, "source": source,
"seed": args.seed, "seed": args.seed,
"counts": dict(counts), "counts": dict(counts),
"length_filter": length_filter.report(),
"requested_holdout_max_rows": {"eval": args.eval_size, "test": args.test_size}, "requested_holdout_max_rows": {"eval": args.eval_size, "test": args.test_size},
"splits": splits, "splits": splits,
"source_breakdown": breakdown, "source_breakdown": breakdown,

84
perun/scripts/train.py
View File

@ -70,18 +70,6 @@ def verify_data(data_dir: Path, splits: tuple[str, ...]) -> dict:
return manifest return manifest
def verify_length_filter(manifest: dict, config: dict) -> None:
report = manifest.get("length_filter", {})
if report.get("enabled") is not True:
raise ValueError("Prepared data has no shared length filter. Rerun prepare_data.py into data/aya_slovak_4096.")
candidates = [profile for profile in report.get("models", {}).values()
if profile.get("model_name_or_path") == config["model_name_or_path"]
and profile.get("model_revision") == config["model_revision"]
and profile.get("template") == config["template"]]
if not candidates or not any(0 < int(profile["cutoff_len"]) <= int(config["cutoff_len"]) for profile in candidates):
raise ValueError("The data was not length-filtered for this model, revision, template and context limit.")
def run_logged(command: list[str], log: Path, env: dict | None = None) -> None: def run_logged(command: list[str], log: Path, env: dict | None = None) -> None:
print("Running:", " ".join(command), flush=True) print("Running:", " ".join(command), flush=True)
with log.open("a", encoding="utf-8") as handle: with log.open("a", encoding="utf-8") as handle:
@ -105,25 +93,27 @@ def context_settings(settings: dict, original_config) -> dict:
requested = int(settings["cutoff_len"]) requested = int(settings["cutoff_len"])
native = int(original_config.max_position_embeddings) native = int(original_config.max_position_embeddings)
if requested > native or settings.get("rope_scaling"): if requested > native or settings.get("rope_scaling"):
raise ValueError("The shared 4096-token scenario uses the original model contexts without RoPE extension.") raise ValueError(f"Use cutoff_len <= {native} within the model's native context and omit rope_scaling.")
return {"requested_length": requested, "base_positions": native} return {"requested_length": requested, "base_positions": native}
def reject_truncation(source_len: int, target_len: int, cutoff_len: int) -> None: def reserve_generation_space() -> None:
if source_len + target_len > cutoff_len: # Reserve space before batching so the trainer sees the same prompt length
raise ValueError(f"A complete SFT example requires {source_len + target_len} tokens, " # as model.generate when it masks the prompt in the generated sequence.
f"but only {cutoff_len} are available. Stopping instead of truncating input or answer.") from llamafactory.data.processor import unsupervised
original = unsupervised.infer_seqlen
def require_complete_examples() -> None: @functools.wraps(original)
from llamafactory.data.processor import supervised, unsupervised def with_generation_space(source_len: int, target_len: int, cutoff_len: int):
for module in (supervised, unsupervised): source, target = original(source_len, target_len, cutoff_len)
original = module.infer_seqlen source = min(source, cutoff_len - 1)
if source < source_len or target < target_len:
print(f"Generation preprocessing truncated tokens: input {source_len} -> {source}, "
f"label {target_len} -> {target}, context {cutoff_len}.", flush=True)
return source, target
def checked(source_len, target_len, cutoff_len, _original=original): unsupervised.infer_seqlen = with_generation_space
reject_truncation(source_len, target_len, cutoff_len)
return _original(source_len, target_len, cutoff_len)
module.infer_seqlen = checked
def bound_generation_context(model, context_length: int, default_max_new_tokens: int) -> None: def bound_generation_context(model, context_length: int, default_max_new_tokens: int) -> None:
@ -141,7 +131,7 @@ def bound_generation_context(model, context_length: int, default_max_new_tokens:
prompt_length = int(inputs.shape[-1]) prompt_length = int(inputs.shape[-1])
available = context_length - prompt_length available = context_length - prompt_length
if available < 1: if available < 1:
raise ValueError("The complete input leaves no room for an answer inside the context limit.") raise ValueError("The preprocessed input leaves no room for an answer inside the context limit.")
requested = kwargs.get("max_new_tokens") requested = kwargs.get("max_new_tokens")
requested = int(default_max_new_tokens if requested is None else requested) requested = int(default_max_new_tokens if requested is None else requested)
if requested < 1: if requested < 1:
@ -149,7 +139,7 @@ def bound_generation_context(model, context_length: int, default_max_new_tokens:
kwargs["max_new_tokens"] = min(requested, available) kwargs["max_new_tokens"] = min(requested, available)
if requested > available: if requested > available:
print(f"Generation limit: {available} new tokens after {prompt_length} input tokens " print(f"Generation limit: {available} new tokens after {prompt_length} input tokens "
f"(context {context_length}); the complete input is preserved.", flush=True) f"(context {context_length}).", flush=True)
return original_generate(*args, **kwargs) return original_generate(*args, **kwargs)
model.generate = MethodType(bounded_generate, model) model.generate = MethodType(bounded_generate, model)
@ -243,7 +233,10 @@ def worker(config_path: Path) -> None:
context = context_settings(config, original_config) context = context_settings(config, original_config)
print("Model context:", json.dumps(context), flush=True) print("Model context:", json.dumps(context), flush=True)
loaded = require_unsloth_backend(config, context) loaded = require_unsloth_backend(config, context)
require_complete_examples() if config.get("do_predict") and config.get("predict_with_generate"):
if context["requested_length"] < 2:
raise ValueError("Generation requires cutoff_len >= 2.")
reserve_generation_space()
from llamafactory.train.tuner import run_exp from llamafactory.train.tuner import run_exp
run_exp(config) run_exp(config)
if config.get("do_train") and local_rank == 0: if config.get("do_train") and local_rank == 0:
@ -252,15 +245,24 @@ def worker(config_path: Path) -> None:
def main() -> int: def main() -> int:
parser = argparse.ArgumentParser(description=__doc__) parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("config", type=Path, help="configs/mistral.yaml or configs/olmo.yaml") parser.add_argument("config", type=Path, help="LLaMA-Factory model YAML.")
parser.add_argument("--run-dir", type=Path, help="New run directory; adapter is saved inside adapter/.") parser.add_argument("--run-dir", type=Path, help="New run directory; adapter is saved inside adapter/.")
parser.add_argument("--resume", type=Path, help="Checkpoint inside the original run's adapter/ directory.") parser.add_argument("--resume", type=Path, help="Checkpoint inside the original run's adapter/ directory.")
parser.add_argument("--pilot-steps", type=int, help="Limit a pilot to this many optimizer steps; requires --run-dir.")
parser.add_argument("--pilot-samples", type=int,
help="Maximum rows per dataset in a pilot (default: 2048 or all eval rows, whichever is larger).")
parser.add_argument("--dry-run", action="store_true", help="Validate data/config; do not import GPU libraries.") parser.add_argument("--dry-run", action="store_true", help="Validate data/config; do not import GPU libraries.")
parser.add_argument("--worker", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--worker", action="store_true", help=argparse.SUPPRESS)
args = parser.parse_args() args = parser.parse_args()
if args.worker: if args.worker:
worker(args.config.resolve()) worker(args.config.resolve())
return 0 return 0
if args.pilot_steps is None and args.pilot_samples is not None:
parser.error("--pilot-samples requires --pilot-steps.")
if args.pilot_steps is not None and (args.pilot_steps < 1 or args.run_dir is None):
parser.error("--pilot-steps must be positive and requires --run-dir.")
if args.pilot_samples is not None and args.pilot_samples < 1:
parser.error("--pilot-samples must be positive.")
config = read_config(args.config.resolve()) config = read_config(args.config.resolve())
if config.get("stage") != "sft" or config.get("finetuning_type") != "lora": if config.get("stage") != "sft" or config.get("finetuning_type") != "lora":
raise ValueError("Expected an SFT / LoRA training configuration.") raise ValueError("Expected an SFT / LoRA training configuration.")
@ -273,7 +275,17 @@ def main() -> int:
batch = GPUS * config["per_device_train_batch_size"] * config["gradient_accumulation_steps"] batch = GPUS * config["per_device_train_batch_size"] * config["gradient_accumulation_steps"]
data_dir = Path(config["dataset_dir"]) data_dir = Path(config["dataset_dir"])
manifest = verify_data(data_dir, ("train", "eval")) manifest = verify_data(data_dir, ("train", "eval"))
verify_length_filter(manifest, config) train_rows = manifest["splits"]["train"]["rows"]
eval_rows = manifest["splits"]["eval"]["rows"]
if args.pilot_steps is not None:
# LLaMA-Factory limits both splits before tokenization. Keep the complete
# validation split, even if the caller prepared more than 2048 eval rows.
sample_limit = args.pilot_samples or max(2048, eval_rows)
if sample_limit < eval_rows:
parser.error(f"--pilot-samples must be at least {eval_rows} to retain the full eval split.")
config.update(max_steps=args.pilot_steps, max_samples=sample_limit,
eval_strategy="steps", save_strategy="steps",
eval_steps=args.pilot_steps, save_steps=args.pilot_steps)
if args.run_dir: if args.run_dir:
config["output_dir"] = str(args.run_dir.resolve() / "adapter") config["output_dir"] = str(args.run_dir.resolve() / "adapter")
run_dir = Path(config["output_dir"]).parent run_dir = Path(config["output_dir"]).parent
@ -288,11 +300,13 @@ def main() -> int:
raise ValueError("Resume requires the original configuration and prepared data.") raise ValueError("Resume requires the original configuration and prepared data.")
elif run_dir.exists() and any(run_dir.iterdir()): elif run_dir.exists() and any(run_dir.iterdir()):
raise ValueError(f"Run directory is not empty; choose --run-dir for a new run: {run_dir}") raise ValueError(f"Run directory is not empty; choose --run-dir for a new run: {run_dir}")
print(json.dumps({"gpus_per_training_run": GPUS, "effective_batch": batch, summary = {"gpus_per_training_run": GPUS, "effective_batch": batch,
"cutoff_len": config["cutoff_len"], "bf16": True, "use_unsloth": True, "cutoff_len": config["cutoff_len"], "bf16": True, "use_unsloth": True,
"train_rows": manifest["splits"]["train"]["rows"], "train_rows": train_rows, "eval_rows": eval_rows,
"eval_rows": manifest["splits"]["eval"]["rows"], "adapter_output": config["output_dir"]}
"adapter_output": config["output_dir"]}, ensure_ascii=False, indent=2)) if args.pilot_steps is not None:
summary.update(pilot_steps=args.pilot_steps, pilot_train_rows=min(train_rows, config["max_samples"]))
print(json.dumps(summary, ensure_ascii=False, indent=2))
if args.dry_run: if args.dry_run:
print("Configuration/data checks passed. No training started.") print("Configuration/data checks passed. No training started.")
return 0 return 0