oprava

README.md

@@ -871,7 +871,7 @@ Presné hodnoty LR, finálne aj minimálne eval loss, kroky, training loss a run
 
 # Hruška – hľadanie learning rate na 300k Aya príkladoch
 
-Po skúšobnom hľadaní LR pomocou Optuny na 30k príkladoch nasledovalo **overenie šiestich ručne zvolených hodnôt LR na väčšej, fixnej 300k vzorke**. Cieľom bolo nájsť vhodné východiskové nastavenie pre plánovaný tréning na celej slovenskej Aya množine na **Perune**. Tento 300k experiment ešte nebol tréningom na celej množine.
+Po skúšobnom hľadaní LR pomocou Optuny na 30k príkladoch nasledovalo **overenie šiestich ručne zvolených hodnôt LR na väčšej, fixnej 300k vzorke**. Cieľom bolo nájsť vhodné východiskové nastavenie pre plánovaný tréning na celej slovenskej Aya množine na **Perune**.
 
 Podklady k tréningu a následným benchmarkom vybraného modelu sú v priečinku [aya_300k_lr_experiments/](aya_300k_lr_experiments/). Postup opakovania a technické podmienky sú v [reproduction.md](aya_300k_lr_experiments/reproduction.md).
 
@@ -898,7 +898,7 @@ Zo slovenského `train` splitu `CohereLabs/aya_collection_language_split` sa vyf
 | Evaluácia / logovanie | každých 1000 / 10 krokov + finálna evaluácia |
 | Seed | 42 |
 
-Pôvodné [konfigurácie](aya_300k_lr_experiments/configs/) a [tréningové logy](aya_300k_lr_experiments/logs/) sú zachované. Konfigurácie sa líšia hodnotou LR, výstupnou cestou a prevádzkovým príznakom `overwrite_output_dir`. Archív neobsahuje checksum vstupného datasetu ani identifikátory validačných vzoriek; úplnú identitu dát preto nemožno nezávisle overiť iba z uložených konfigurácií.
+Pôvodné [konfigurácie](aya_300k_lr_experiments/configs/) a [tréningové logy](aya_300k_lr_experiments/logs/) sú zachované.
 
 ## Výsledky a výber LR pre ďalší tréning
 
@@ -913,34 +913,26 @@ Prvá séria obsahovala LR `0.00018`, `0.00020` a `0.00023`. Po jej vyhodnotení
 | 0.00020 | 0.897669 | 0.897507 | 18 000 |
 | 0.00023 | 0.903236 | 0.903193 | 18 000 |
 
-**LR `0.00010` bol vybraný preto, že dosiahol najnižší finálny validačný loss zo všetkých šiestich 300k behov: `0.87261962890625`.** Rozdiel oproti LR `0.00008` je približne 0.002597. Ide o najlepší bodový výsledok v tejto sérii; neprebehli opakovania s viacerými seedmi ani overenie optimálneho LR na celej Aya množine. Pre Perun slúži ako východisko.
+**LR `0.00010` bol vybraný preto, že dosiahol najnižší finálny validačný loss zo všetkých šiestich 300k behov: `0.87261962890625`.** Rozdiel oproti LR `0.00008` je približne 0.002597. Ide o najlepší bodový výsledok v tejto sérii.
 
-Na 30k skúške bol najlepší skúšaný LR `0.0002`. Výsledky 300k série preto ukazujú význam overenia LR na väčšej vzorke. Absolútne hodnoty eval loss medzi 30k, 100k a 300k sériami nie sú priamym porovnaním kvality modelov, keďže nemožno predpokladať zhodné validačné vzorky.
+Na 30k skúške bol najlepší skúšaný LR `0.0002`. Výsledky 300k série preto ukazujú význam overenia LR na väčšej vzorke.
 
 Presné údaje sú v [pôvodnom LR súhrne](aya_300k_lr_experiments/results/aya_300k_lr_summary.csv), [rozšírenom súhrne behov](aya_300k_lr_experiments/results/training_run_summary.csv) a [histórii validačného loss](aya_300k_lr_experiments/results/eval_loss_history.csv).
 
-<p align="center">
-  <img src="aya_300k_lr_experiments/results/lr_vs_final_eval_loss.png" alt="Finálny eval loss podľa learning rate na 300k Aya príkladoch" width="50%">
-</p>
 
 <p align="center">
   <img src="aya_300k_lr_experiments/results/eval_loss_over_steps.png" alt="Priebeh validačného loss pre šesť learning rate" width="50%">
 </p>
 
-<p align="center">
-  <img src="aya_300k_lr_experiments/results/grad_norm_over_steps.png" alt="Priebeh zaznamenaného gradient norm pre šesť learning rate" width="50%">
-</p>
 
 ## Export vybraného modelu a benchmarky
 
-Na následné benchmarky sa vybral model s LR `0.00010`. Jeho LoRA adaptér sa zlúčil s pôvodným základným modelom do samostatného modelu v `bfloat16`. Export nepredstavoval ďalší tréning. [Exportná konfigurácia](aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml) používa rovnaký snapshot základného modelu a template `alpaca`.
+Na následné benchmarky sa vybral model s LR `0.00010`. Jeho LoRA adaptér sa zlúčil s pôvodným základným modelom do samostatného modelu v `bfloat16`.
 
-Benchmarky skúmajú schopnosti tohto vybraného modelu. **Ostatných päť LR zatiaľ týmito benchmarkmi nebolo porovnaných**, takže z nich nemožno vyhlásiť LR `0.00010` za najlepší LR podľa downstream metrík.
+Benchmarky skúmajú schopnosti tohto vybraného modelu. **Ostatných päť LR zatiaľ týmito benchmarkmi nebolo porovnaných**.
 
 ### lm-evaluation-harness
 
-Použilo sa pôvodné prostredie `lm_eval_hruska`, HF backend, `bf16`, batch size 1 a zero-shot nastavenie bez aplikovania chat template. Zlúčený model sa načítal samostatne; tokenizer pochádzal z pôvodného modelového snapshotu. Tento postup obchádza nekompatibilitu exportovaného označenia `TokenizersBackend` so starším Transformers 4.48.2.
-
 Vyhodnotenie dokončilo všetkých sedem pôvodných úloh a skončilo s návratovým kódom 0. Zachované sú [výsledky](aya_300k_lr_experiments/benchmarks/lmeval/results/), [log](aya_300k_lr_experiments/benchmarks/lmeval/logs/), custom úlohy a metadáta prostredia. V nasledujúcej tabuľke sú accuracy metriky prevedené na percentá; SKQuad F1 je už v zdroji v škále 0–100.
 
 | Model | ARC acc_norm | HellaSwag acc_norm | M-MMLU acc | TQA MC1 | TQA MC2 | SKLegal acc_norm | SKQuad F1 |
@@ -950,19 +942,16 @@ Vyhodnotenie dokončilo všetkých sedem pôvodných úloh a skončilo s návrat
 | 100k – experiment 10, r16/a32/LR 0.0001 | 32.93 | 51.57 | 34.48 | 20.57 | 34.80 | 46.18 | 37.76 |
 | **300k – vybraný LR 0.00010** | 34.73 | 51.64 | 31.03 | 19.67 | 35.07 | 46.10 | 38.73 |
 
-Oproti 100k experimentu 10 dosiahol vybraný 300k model bodovo vyššie ARC acc_norm (+1.80 percentuálneho bodu) a SKQuad F1 (+0.97 bodu), ale nižšie M-MMLU acc (−3.45 percentuálneho bodu). Ostatné zmeny sú menšie. V žiadnej zo siedmich hlavných metrík neprekonal maximum z celej desiatky 100k behov; tieto maximá navyše pochádzajú z rôznych konfigurácií. Výber najnižšieho validačného loss teda neznamená automaticky najlepší výsledok v každej downstream úlohe. Nejde o vyhodnotenie všetkých šiestich 300k LR ani o test štatistickej významnosti rozdielov.
+Oproti 100k experimentu 10 dosiahol vybraný 300k model bodovo vyššie ARC acc_norm (+1.80 percentuálneho bodu) a SKQuad F1 (+0.97 bodu), ale nižšie M-MMLU acc (−3.45 percentuálneho bodu). Ostatné zmeny sú menšie. V žiadnej zo siedmich hlavných metrík neprekonal maximum z celej desiatky 100k behov; tieto maximá navyše pochádzajú z rôznych konfigurácií. Výber najnižšieho validačného loss teda neznamená automaticky najlepší výsledok v každej downstream úlohe.
-
-Úplné porovnanie so všetkými desiatimi staršími Hruška experimentmi je v [comparison_with_100k.csv](aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv). Vo všetkých porovnávaných Hruška behoch sa zhodujú konfigurácie a verzie úloh, počty vzoriek, seeds, evaluator commit `6d2abda4`, Transformers 4.48.2 a nastavenie bez chat template. Pole `task_hashes` je však v zdrojových JSON súboroch prázdne, takže nezávislé overenie obsahu benchmarkových dát pomocou hashov nie je možné. Staršie výsledky používali samostatný PEFT adaptér, nový výsledok zlúčený model v BF16; spôsob načítania a zaokrúhlenie pri zlúčení preto nie sú úplne identické. Podrobnosti sú v [audite porovnateľnosti](aya_300k_lr_experiments/benchmarks/lmeval/provenance/comparability_audit.json).
 
 ### SK-bench
 
-Dokončil sa **čiastkový slovenský zero-shot beh s 222 položkami**. Položka predstavuje kombináciu úlohy, podúlohy a variantu promptu, nie jeden hodnotený príklad. Zahrnuté boli tri slovenské formulácie promptov dostupných úloh. Použila sa Alpaca šablóna zlúčeného modelu, greedy dekódovanie, seed 1234, `bf16`, jedna GPU a maximálny kontext 32 768 tokenov.
+Dokončil sa **čiastkový slovenský zero-shot beh s 222 položkami**. Zahrnuté boli tri slovenské formulácie promptov dostupných úloh. Použila sa Alpaca šablóna zlúčeného modelu, greedy dekódovanie, seed 1234, `bf16`, jedna GPU a maximálny kontext 32 768 tokenov.
 
-Z pôvodných 412 slovenských položiek sa vynechalo 187 položiek pre 12 nedostupných repozitárov: Chiby, Autoškola, e-testy, IFEval-SK, OneRuler-SK, Scio, Slohry, Slovak school exams, Slovak Trivia, SlovakSum, SME-Sum a slovné analógie. Ich kontrola vracala `RepositoryNotFoundError`; tento stav sám osebe nerozlišuje neexistujúci repozitár od súkromného repozitára bez oprávnenia. Ďalšie tri položky NLI-SK-annotated sa vynechali, pretože prístup vyžadoval schválenie autorkou a pri spúšťaní behu ešte nebol udelený. Ostatné NLI úlohy zo skLEP v behu zostali.
+Z pôvodných 412 slovenských položiek sa vynechalo 187 položiek pre 12 nedostupných repozitárov: Chiby, Autoškola, e-testy, IFEval-SK, OneRuler-SK, Scio, Slohry, Slovak school exams, Slovak Trivia, SlovakSum, SME-Sum a slovné analógie. Ich kontrola vracala `RepositoryNotFoundError`. Ďalšie tri položky NLI-SK-annotated sa vynechali, pretože prístup vyžadoval schválenie autorkou a pri spúšťaní behu ešte nebol udelený. Ostatné NLI úlohy zo skLEP v behu zostali.
 
-Uložené sú [vykonané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt), [vynechané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt), [report dostupnosti](aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json), úplný výsledný JSON a úspešný log. Audit 222 detailných parquet výstupov overil **159 426 hodnotení** (53 142 pre každý z troch promptov). **Log zároveň potvrdzuje orezanie príliš dlhého vstupu.** Všetky uložené `model_response.truncated_tokens_count` sú síce nulové, ale tieto počítadlá nezachytili orezanie hlásené enginom. Pri MultiWikiQA má 69 vstupov dĺžku presne na hranici použitého rozpočtu (23 pri každom prompte); ide o kandidátov konzistentných s orezaním, nie o spoľahlivo dokázaný počet orezaných príkladov. Výsledky dlhých QA vstupov preto treba interpretovať s týmto obmedzením. Podrobnosti sú v [audite](aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json) a [počítadlách po úlohách](aya_300k_lr_experiments/benchmarks/skbench/results/truncation_audit_per_task.csv).
+Uložené sú [vykonané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt), [vynechané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt), [report dostupnosti](aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json), úplný výsledný JSON a úspešný log. Audit 222 detailných parquet výstupov overil **159 426 hodnotení** (53 142 pre každý z troch promptov). **Log zároveň potvrdzuje orezanie príliš dlhého vstupu.**
 
-Výber opisných výsledkov z dostupných úloh. Uvedený priemer je vážený počtom hodnotených príkladov v príslušných variantoch; všetky tri prompty majú v danej úlohe rovnaký počet príkladov. Pri MMLU zahŕňa všetkých 57 predmetov a tri prompty. Toto nie je oficiálna agregácia úplného SK-bench.
 
 | Úloha | Metrika | Počet variantov | Priemer (%) |
 |---|---|---:|---:|
@@ -981,14 +970,8 @@ Výber opisných výsledkov z dostupných úloh. Uvedený priemer je vážený p
 
 Úplné metriky sú v [per_task_metrics.csv](aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv); rodinné a promptové súhrny vrátane nevážených priemerov sú v [family_summary.csv](aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv) a [family_prompt_summary.csv](aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv).
 
-Výsledky treba čítať spolu s definíciou metriky a promptu. Napríklad pri ARC je priemer presnej metriky `em` približne 2.37 %, zatiaľ čo `em_relaxed` dosahuje 19.08 %. Rozdiel ukazuje citlivosť generatívneho hodnotenia na spôsob zápisu odpovede. Úplný surový JSON zachováva aj agregát `all`, ktorý sa tu neinterpretuje ako celkové skóre: mieša rôzne metriky a pokrýva neúplnú množinu úloh. Samotné MMLU tvorí 171 z 222 položiek, preto jednoduché spriemerovanie všetkých riadkov nie je vyváženým súhrnom schopností modelu.
 
-**Tento výsledok nie je oficiálne celkové skóre úplného SK-bench.** Pri porovnávaní ďalších modelov treba použiť rovnaký zmrazený zoznam úloh a rovnaký protokol. SK-bench používa generovanie a následné porovnávanie textovej odpovede, zatiaľ čo väčšina uvedených lm-eval úloh vyhodnocuje pravdepodobnosti možností. Odlišné sú aj konkrétne datasety, prompty a metriky; hodnoty medzi týmito dvoma frameworkmi preto nie sú priamo zameniteľné.
+**Tento výsledok nie je oficiálne celkové skóre úplného SK-bench.** 
 
-Pri príprave sa sampler FlashInfer pokúšal kompilovať cez starší systémový CUDA toolkit. Úspešný beh používal `VLLM_USE_FLASHINFER_SAMPLER=0` a `--enforce-eager`. Podrobné verzie a parametre sú súčasťou benchmarkových metadát.
+Pri príprave sa sampler FlashInfer pokúšal kompilovať cez starší systémový CUDA toolkit. Úspešný beh používal `VLLM_USE_FLASHINFER_SAMPLER=0` a `--enforce-eager`.
 
-## Uložené podklady
-
-Priečinok [aya_300k_lr_experiments/](aya_300k_lr_experiments/) obsahuje pôvodné konfigurácie, tréningové a benchmarkové logy, výsledné JSON/CSV súbory, grafy, vybrané metadáta a skripty. Veľké váhy, checkpointy, `trainer_state.json`, kompletné per-sample parquet súbory a duplicitné inference cache zostávajú v pôvodnom archíve. Grafy a odvodené tabuľky tréningu možno obnoviť z malých uložených histórií bez prístupu k váham modelov.
-
-Zdrojový archív a jeho SHA-256 sú zaznamenané v [source_archive.json](aya_300k_lr_experiments/provenance/source_archive.json).

aya_300k_lr_experiments/benchmarks/export/chat_template.jinja

@@ -1,9 +0,0 @@
-{% set system_message = 'Below is an instruction that describes a task. Write a response that appropriately completes the request.
-
-' %}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in loop_messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '### Instruction:
-' + content + '
-
-### Response:
-' }}{% elif message['role'] == 'assistant' %}{{ content + '</s>' + '
-
-' }}{% endif %}{% endfor %}

aya_300k_lr_experiments/benchmarks/export/config.json

@@ -1,30 +0,0 @@
-{
-  "architectures": [
-    "MistralForCausalLM"
-  ],
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "bfloat16",
-  "eos_token_id": 2,
-  "head_dim": 128,
-  "hidden_act": "silu",
-  "hidden_size": 4096,
-  "initializer_range": 0.02,
-  "intermediate_size": 14336,
-  "max_position_embeddings": 32768,
-  "model_type": "mistral",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 32,
-  "num_key_value_heads": 8,
-  "pad_token_id": 2,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "sliding_window": 4096,
-  "tie_word_embeddings": false,
-  "transformers_version": "5.5.0",
-  "use_cache": true,
-  "vocab_size": 32000
-}

aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml

@@ -1,12 +0,0 @@
-model_name_or_path: slovak-nlp/mistral-sk-7b
-model_revision: 089497ae0d72018e9591895c54f774bf0b4a83a4
-adapter_name_or_path: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010
-
-template: alpaca
-finetuning_type: lora
-infer_dtype: bfloat16
-
-export_dir: /home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
-export_size: 5
-export_device: cpu
-export_legacy_format: false

aya_300k_lr_experiments/benchmarks/export/generation_config.json

@@ -1,6 +0,0 @@
-{
-  "_from_model_config": true,
-  "bos_token_id": 1,
-  "eos_token_id": 2,
-  "transformers_version": "5.5.0"
-}

aya_300k_lr_experiments/benchmarks/export/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": true,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "extra_special_tokens": [],
-  "is_local": false,
-  "legacy": true,
-  "model_max_length": 1000000000000000019884624838656,
-  "pad_token": "</s>",
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "spaces_between_special_tokens": false,
-  "split_special_tokens": false,
-  "tokenizer_class": "TokenizersBackend",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

aya_300k_lr_experiments/benchmarks/lmeval/provenance/SHA256SUMS.txt

@@ -1,18 +0,0 @@
-499e639455dc6bb9c79228b54b82c66b509f8fe6f2d33513b482b90dbf990674  comparison_with_100k.csv
-b587953b0ce136fea20edf50105005c645a2abfad2cd4f6bfa58ee17cf36b34c  full_metrics.csv
-fd33cc64e660d4d996bd1b6af966e900c72b0c3b5c4bdb4e6915ac0756814e94  logs/lr_0p00010_20261023_224332.log
-2e959d2b2052ecc4dc5f571ac674ccaa39119149c58e6ac0da467aa0335b8248  provenance/comparability_audit.json
-f491f734663154901f0b5a4bb44aaed69b5d95448415291107fea0c703b055e8  provenance/custom_tasks/arc_sk.yaml
-f2feb3ae7dc6e8ae3db09a91747119d7256d1ef33b9a1e417a40cb2a949ce5b9  provenance/custom_tasks/hellaswag_sk.yaml
-d2f8c18839d15d365eed6997564e11aab43173eef91470e8a5834a9471772c2a  provenance/custom_tasks/m_mmlu_sk.yaml
-9443e2d5e398c82f921d8e714ff01693bfce793eef9bf3b0ad581a8bbf137c4a  provenance/custom_tasks/sklegal.yaml
-759c8cb35d27e50dd4dc4e81b34932d8bc96e36e309ed666858a904624a61131  provenance/custom_tasks/skquad.yaml
-de1b8b08f998ebb3dcdd881c3e2aaba89dd53f6ea2a51b0529eadf7936ef2ed3  provenance/custom_tasks/task.py
-fa1e7075c4715a4465be66041d36ef3f9b4dc8a7e41898554078294d2510f525  provenance/custom_tasks/task_configs_from_wandb.json
-cf922b04a755762060430ef37a18a860017cc9c79e89cf23ac7465fa97244faa  provenance/custom_tasks/truthfulqa_sk_mc1.yaml
-b9209889fdf0e38915e29b83911213010f6c49d474836f474491da085ab26e01  provenance/custom_tasks/truthfulqa_sk_mc2.yaml
-205bd6b1be478ed9cb2a104bf47214cf75debe485fb43c6b78d6584e638283de  provenance/lm_eval_commit.txt
-b4dcc9d0c2e7b5f5036ddc38a7aad4111934e39d84019b2122ad6561b780b5a4  provenance/package_versions.json
-44d8afff7e8d918ef9e53404a8fa17fe19d6dcfaf782519f01be44a2408da898  provenance/run_settings.txt
-f6f8d8470b78daf35d80f5a7315cc9f38d867daaead08df39a165cbcf98afd4e  provenance/tasks.txt
-5774cad7440ab807be44dc5bfa82ac7cbb4c0281204fc397a10c2bbfe7b8f452  results/results_2026-10-24T02-28-32.987157.json

aya_300k_lr_experiments/benchmarks/lmeval/provenance/comparability_audit.json

@@ -1,379 +0,0 @@
-{
-  "comparison_scope": "All ten Hruska Aya 100k runs; Titan excluded.",
-  "units": "Accuracy metrics use 0..1; SKQuad scores use 0..100. N/A remains unavailable.",
-  "new_result_sha256": "5774cad7440ab807be44dc5bfa82ac7cbb4c0281204fc397a10c2bbfe7b8f452",
-  "task_hashes_caveat": "task_hashes is empty in all eleven JSONs; equality is not a nonempty content-hash verification.",
-  "model_loading_difference": "Older runs load base model plus PEFT adapter; 300k loads a merged model with explicit base tokenizer. Model arguments, parameter counts and model SHA therefore differ.",
-  "new_n_samples": {
-    "truthfulqa_sk_mc2": {
-      "original": 778,
-      "effective": 778
-    },
-    "truthfulqa_sk_mc1": {
-      "original": 778,
-      "effective": 778
-    },
-    "skquad": {
-      "original": 9583,
-      "effective": 9583
-    },
-    "sklegal": {
-      "original": 1334,
-      "effective": 1334
-    },
-    "m_mmlu_sk": {
-      "original": 13062,
-      "effective": 13062
-    },
-    "hellaswag_sk": {
-      "original": 9485,
-      "effective": 9485
-    },
-    "arc_sk": {
-      "original": 1169,
-      "effective": 1169
-    }
-  },
-  "comparisons": [
-    {
-      "model_id": "01_baseline_r16_a16_lr5e5_adamw",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_01_baseline_r16_a16_lr5e5_adamw,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "02_optimizer_adamw8bit_r16_a16_lr5e5",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_02_optimizer_adamw8bit_r16_a16_lr5e5,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "03_rank8_a16_lr5e5",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_03_rank8_a16_lr5e5,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7262703616,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "04_rank32_a32_lr5e5",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_04_rank32_a32_lr5e5,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7325618176,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "05_r16_alpha32_lr5e5",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_05_r16_alpha32_lr5e5,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "06_lr1e4_r16_a16",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_06_lr1e4_r16_a16,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "07_lr2e5_r16_a16",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_07_lr2e5_r16_a16,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "08_attention_only_r16_a16_lr5e5",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_08_attention_only_r16_a16_lr5e5,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7255363584,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "09_batch2_accum4_r16_a16_lr5e5",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_09_batch2_accum4_r16_a16_lr5e5,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    },
-    {
-      "model_id": "10_r16_a32_lr1e4",
-      "equal_to_300k": {
-        "versions": true,
-        "configs": true,
-        "task_hashes": true,
-        "n-samples": true,
-        "n-shot": true,
-        "git_hash": true,
-        "transformers_version": true,
-        "chat_template": true,
-        "chat_template_sha": true,
-        "system_instruction": true,
-        "fewshot_as_multiturn": true
-      },
-      "evaluation_config_differences": {
-        "model_args": {
-          "older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_10_r16_a32_lr1e4,dtype=bfloat16",
-          "new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
-        },
-        "model_num_parameters": {
-          "older_100k": 7283675136,
-          "new_300k": 7241732096
-        },
-        "model_sha": {
-          "older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-          "new_300k": ""
-        },
-        "peft_sha": {
-          "older_100k": "",
-          "new_300k": null
-        }
-      }
-    }
-  ]
-}

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/arc_sk.yaml

@@ -1,42 +0,0 @@
-tag:
-- arc_multilingual
-task: arc_sk
-repeats: 1
-metadata:
-  version: 2
-test_split: test
-description: ''
-doc_to_text: query
-metric_list:
-- metric: acc
-  aggregation: mean
-  higher_is_better: true
-- metric: acc_norm
-  aggregation: mean
-  higher_is_better: true
-num_fewshot: 0
-output_type: multiple_choice
-unsafe_code: false
-dataset_name: sk
-dataset_path: alexandrainst/m_arc
-process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
-  \    def _process_doc(doc):\n        # breakpoint()\n        out_doc = {\n     \
-  \       \"id\": doc[\"id\"],\n            \"query\": \"Question: \" + preprocess(doc[\"\
-  instruction\"]) + \"\\nAnswer:\",\n            \"choices\": [\n                preprocess(option)\n\
-  \                for option in [\n                    doc[\"option_a\"],\n     \
-  \               doc[\"option_b\"],\n                    doc[\"option_c\"],\n   \
-  \                 doc[\"option_d\"],\n                    doc[\"option_e\"],\n \
-  \               ]\n                if option\n            ],\n            \"gold\"\
-  : [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n        }\n     \
-  \   return out_doc\n\n    return dataset.map(_process_doc)\n"
-doc_to_choice: choices
-doc_to_target: gold
-training_split: train
-target_delimiter: ' '
-validation_split: validation
-fewshot_delimiter: '
-
-
-  '
-should_decontaminate: true
-doc_to_decontamination_query: query

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/hellaswag_sk.yaml

@@ -1,35 +0,0 @@
-tag:
-- hellaswag_multilingual
-task: hellaswag_sk
-repeats: 1
-metadata:
-  version: 1
-description: ''
-doc_to_text: query
-metric_list:
-- metric: acc
-  aggregation: mean
-  higher_is_better: true
-- metric: acc_norm
-  aggregation: mean
-  higher_is_better: true
-num_fewshot: 0
-output_type: multiple_choice
-unsafe_code: false
-dataset_name: sk
-dataset_path: alexandrainst/m_hellaswag
-process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
-  \    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"\
-  ].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"\
-  ] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"\
-  endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return\
-  \ out_doc\n\n    return dataset.map(_process_doc)\n"
-doc_to_choice: choices
-doc_to_target: '{{label.lstrip()}}'
-target_delimiter: ' '
-validation_split: val
-fewshot_delimiter: '
-
-
-  '
-should_decontaminate: false

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/m_mmlu_sk.yaml

@@ -1,43 +0,0 @@
-tag:
-- m_mmlu
-task: m_mmlu_sk
-repeats: 1
-metadata:
-  version: 0
-test_split: test
-description: ''
-doc_to_text: '{{instruction.strip()}}
-
-  A. {{option_a}}
-
-  B. {{option_b}}
-
-  C. {{option_c}}
-
-  D. {{option_d}}
-
-  Answer:'
-metric_list:
-- metric: acc
-  aggregation: mean
-  higher_is_better: true
-num_fewshot: 0
-output_type: multiple_choice
-unsafe_code: false
-dataset_name: sk
-dataset_path: alexandrainst/m_mmlu
-doc_to_choice:
-- A
-- B
-- C
-- D
-doc_to_target: answer
-fewshot_split: train
-fewshot_config:
-  sampler: first_n
-target_delimiter: ' '
-fewshot_delimiter: '
-
-
-  '
-should_decontaminate: false

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/sklegal.yaml

@@ -1,23 +0,0 @@
-task: sklegal
-repeats: 1
-metadata:
-  version: 1
-test_split: test
-description: ''
-doc_to_text: '{{prompt}} '
-metric_list:
-  - metric: acc
-    aggregation: mean
-    higher_is_better: true
-  - metric: acc_norm
-    aggregation: mean
-    higher_is_better: true
-num_fewshot: 0
-output_type: multiple_choice
-unsafe_code: false
-dataset_path: TUKE-KEMT/slovak-legal-exam
-doc_to_choice: '{{answers}}'
-doc_to_target: label
-target_delimiter: ' '
-fewshot_delimiter: "\n\n"
-should_decontaminate: false

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/skquad.yaml

@@ -1,2 +0,0 @@
-task: skquad
-class: !function task.SQuAD2

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task.py

@@ -1,245 +0,0 @@
-"""
-Know What You Don’t Know: Unanswerable Questions for SQuAD
-https://arxiv.org/pdf/1806.03822.pdf
-
-Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset,
-consisting of questions posed by crowdworkers on a set of Wikipedia articles,
-where the answer to every question is a segment of text, or span, from the
-corresponding reading passage, or the question might be unanswerable.
-SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable
-questions written adversarially by crowdworkers to look similar to answerable ones.
-To do well on SQuAD2.0, systems must not only answer questions when possible, but
-also determine when no answer is supported by the paragraph and abstain from answering.
-
-Homepage: https://rajpurkar.github.io/SQuAD-explorer/
-"""
-
-from functools import partial
-from math import exp
-
-import datasets
-from packaging import version
-
-from lm_eval.api.instance import Instance
-from lm_eval.api.task import ConfigurableTask
-
-
-_CITATION = """
-@misc{rajpurkar2018know,
-    title={Know What You Don't Know: Unanswerable Questions for SQuAD},
-    author={Pranav Rajpurkar and Robin Jia and Percy Liang},
-    year={2018},
-    eprint={1806.03822},
-    archivePrefix={arXiv},
-    primaryClass={cs.CL}
-}
-"""
-
-
-def _squad_metric(predictions, references):
-    import evaluate
-
-    squad_metric = evaluate.load("squad_v2")
-    return squad_metric.compute(predictions=predictions, references=references)
-
-
-def _squad_agg(key, items):
-    predictions, references = zip(*items)
-
-    return _squad_metric(predictions=predictions, references=references).get(key, 0)
-
-
-class SQuAD2(ConfigurableTask):
-    VERSION = 3
-    DATASET_PATH = "TUKE-DeutscheTelekom/skquad"
-    DATASET_NAME = None
-
-    def __init__(self, config=None):
-        super().__init__(config={"metadata": {"version": self.VERSION}})
-
-    # HF changed squad on us so we have to make sure we aren't running the old one
-    assert version.parse(datasets.__version__) >= version.parse("1.11.0"), (
-        "datasets v1.11.0 or later required for SQuAD"
-    )
-
-    def has_training_docs(self):
-        return True
-
-    def has_validation_docs(self):
-        return True
-
-    def has_test_docs(self):
-        return False
-
-    def training_docs(self):
-        return self.dataset["train"]
-
-    def validation_docs(self):
-        return self.dataset["validation"]
-
-    def doc_to_text(self, doc):
-        return (
-            "Title: "
-            + doc["title"]
-            + "\n\n"
-            + "Background: "
-            + doc["context"]
-            + "\n\n"
-            + "Question: "
-            + doc["question"]
-            + "\n\n"
-            + "Answer:"
-        )
-
-    def should_decontaminate(self):
-        return True
-
-    def doc_to_decontamination_query(self, doc):
-        return doc["context"]
-
-    def doc_to_target(self, doc):
-        answer_list = doc["answers"]["text"]
-        if len(answer_list) > 0:
-            answer = answer_list[0]
-        else:
-            answer = "unanswerable"
-        return " " + answer
-
-    def construct_requests(
-        self, doc, ctx, chat_template=None, apply_chat_template=False, **kwargs
-    ):
-        """Uses RequestFactory to construct Requests and returns an iterable of
-        Requests which will be sent to the LM.
-
-        :param doc:
-            The document as returned from training_docs, validation_docs, or test_docs.
-        :param ctx: str
-            The context string, generated by fewshot_context. This includes the natural
-            language description, as well as the few shot examples, and the question
-            part of the document for `doc`.
-        """
-
-        return [
-            Instance(
-                request_type="generate_until",
-                doc=doc,
-                arguments=(ctx, {"until": ["\n"]}),
-                idx=0,
-                **kwargs,
-            ),
-            Instance(
-                request_type="loglikelihood",
-                doc=doc,
-                arguments=(ctx, " " + "unanswerable"),
-                idx=0,
-                **kwargs,
-            ),
-        ]
-
-    def process_results(self, doc, results):
-        """Take a single document and the LM results and evaluates, returning a
-        dict where keys are the names of submetrics and values are the values of
-        the metric for that one document
-
-        :param doc:
-            The document as returned from training_docs, validation_docs, or test_docs.
-        :param results:
-            The results of the requests created in construct_requests.
-        """
-
-        continuation, (logprob_unanswerable, _) = results
-
-        no_answer_probability = exp(logprob_unanswerable)
-
-        predictions = {
-            "id": doc["id"],
-            "prediction_text": continuation,
-            "no_answer_probability": no_answer_probability,
-        }
-
-        references = {
-            "id": doc["id"],
-            "answers": doc["answers"],
-        }
-
-        return {
-            "exact": (
-                predictions,
-                references,
-            ),  # Exact match (the normalized answer exactly match the gold answer)
-            "f1": (
-                predictions,
-                references,
-            ),  # The F-score of predicted tokens versus the gold answer
-            "HasAns_exact": (
-                predictions,
-                references,
-            ),  # Exact match (the normalized answer exactly match the gold answer)
-            "HasAns_f1": (
-                predictions,
-                references,
-            ),  # The F-score of predicted tokens versus the gold answer
-            "NoAns_exact": (
-                predictions,
-                references,
-            ),  # Exact match (the normalized answer exactly match the gold answer)
-            "NoAns_f1": (
-                predictions,
-                references,
-            ),  # The F-score of predicted tokens versus the gold answer
-            "best_exact": (
-                predictions,
-                references,
-            ),  # Best exact match (with varying threshold)
-            "best_f1": (predictions, references),  # Best F1 (with varying threshold)
-        }
-
-    def aggregation(self):
-        """
-        :returns: {str: [float] -> float}
-            A dictionary where keys are the names of submetrics and values are
-            functions that aggregate a list of metrics
-        """
-        return {
-            "exact": partial(
-                _squad_agg, "exact"
-            ),  # Exact match (the normalized answer exactly match the gold answer)
-            "f1": partial(
-                _squad_agg, "f1"
-            ),  # The F-score of predicted tokens versus the gold answer
-            "HasAns_exact": partial(
-                _squad_agg, "HasAns_exact"
-            ),  # Exact match (the normalized answer exactly match the gold answer)
-            "HasAns_f1": partial(
-                _squad_agg, "HasAns_f1"
-            ),  # The F-score of predicted tokens versus the gold answer
-            "NoAns_exact": partial(
-                _squad_agg, "NoAns_exact"
-            ),  # Exact match (the normalized answer exactly match the gold answer)
-            "NoAns_f1": partial(
-                _squad_agg, "NoAns_f1"
-            ),  # The F-score of predicted tokens versus the gold answer
-            "best_exact": partial(
-                _squad_agg, "best_exact"
-            ),  # Best exact match (with varying threshold)
-            "best_f1": partial(
-                _squad_agg, "best_f1"
-            ),  # Best F1 (with varying threshold)
-        }
-
-    def higher_is_better(self):
-        """
-        :returns: {str: bool}
-            A dictionary where keys are the names of submetrics and values are
-            whether a higher value of the submetric is better
-        """
-        return {
-            "exact": True,  # Exact match (the normalized answer exactly match the gold answer)
-            "f1": True,  # The F-score of predicted tokens versus the gold answer
-            "HasAns_exact": True,  # Exact match (the normalized answer exactly match the gold answer)
-            "HasAns_f1": True,  # The F-score of predicted tokens versus the gold answer
-            "NoAns_exact": True,  # Exact match (the normalized answer exactly match the gold answer)
-            "NoAns_f1": True,  # The F-score of predicted tokens versus the gold answer
-            "best_exact": True,  # Best exact match (with varying threshold)
-            "best_f1": True,  # Best F1 (with varying threshold)
-        }

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task_configs_from_wandb.json

@@ -1,232 +0,0 @@
-{
-  "arc_sk": {
-    "tag": [
-      "arc_multilingual"
-    ],
-    "task": "arc_sk",
-    "repeats": 1,
-    "metadata": {
-      "version": 2
-    },
-    "test_split": "test",
-    "description": "",
-    "doc_to_text": "query",
-    "metric_list": [
-      {
-        "metric": "acc",
-        "aggregation": "mean",
-        "higher_is_better": true
-      },
-      {
-        "metric": "acc_norm",
-        "aggregation": "mean",
-        "higher_is_better": true
-      }
-    ],
-    "num_fewshot": 0,
-    "output_type": "multiple_choice",
-    "unsafe_code": false,
-    "dataset_name": "sk",
-    "dataset_path": "alexandrainst/m_arc",
-    "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        # breakpoint()\n        out_doc = {\n            \"id\": doc[\"id\"],\n            \"query\": \"Question: \" + preprocess(doc[\"instruction\"]) + \"\\nAnswer:\",\n            \"choices\": [\n                preprocess(option)\n                for option in [\n                    doc[\"option_a\"],\n                    doc[\"option_b\"],\n                    doc[\"option_c\"],\n                    doc[\"option_d\"],\n                    doc[\"option_e\"],\n                ]\n                if option\n            ],\n            \"gold\": [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
-    "doc_to_choice": "choices",
-    "doc_to_target": "gold",
-    "training_split": "train",
-    "target_delimiter": " ",
-    "validation_split": "validation",
-    "fewshot_delimiter": "\n\n",
-    "should_decontaminate": true,
-    "doc_to_decontamination_query": "query"
-  },
-  "skquad": {
-    "task": "skquad",
-    "repeats": 1,
-    "metadata": {
-      "version": 3
-    },
-    "description": "",
-    "num_fewshot": 0,
-    "output_type": "generate_until",
-    "unsafe_code": false,
-    "target_delimiter": " ",
-    "fewshot_delimiter": "\n\n",
-    "generation_kwargs": {
-      "until": [
-        "\n\n"
-      ],
-      "do_sample": false
-    },
-    "should_decontaminate": false
-  },
-  "sklegal": {
-    "task": "sklegal",
-    "repeats": 1,
-    "metadata": {
-      "version": 1
-    },
-    "test_split": "test",
-    "description": "",
-    "doc_to_text": "{{prompt}} ",
-    "metric_list": [
-      {
-        "metric": "acc",
-        "aggregation": "mean",
-        "higher_is_better": true
-      },
-      {
-        "metric": "acc_norm",
-        "aggregation": "mean",
-        "higher_is_better": true
-      }
-    ],
-    "num_fewshot": 0,
-    "output_type": "multiple_choice",
-    "unsafe_code": false,
-    "dataset_path": "TUKE-KEMT/slovak-legal-exam",
-    "doc_to_choice": "{{answers}}",
-    "doc_to_target": "label",
-    "target_delimiter": " ",
-    "fewshot_delimiter": "\n\n",
-    "should_decontaminate": false
-  },
-  "m_mmlu_sk": {
-    "tag": [
-      "m_mmlu"
-    ],
-    "task": "m_mmlu_sk",
-    "repeats": 1,
-    "metadata": {
-      "version": 0
-    },
-    "test_split": "test",
-    "description": "",
-    "doc_to_text": "{{instruction.strip()}}\nA. {{option_a}}\nB. {{option_b}}\nC. {{option_c}}\nD. {{option_d}}\nAnswer:",
-    "metric_list": [
-      {
-        "metric": "acc",
-        "aggregation": "mean",
-        "higher_is_better": true
-      }
-    ],
-    "num_fewshot": 0,
-    "output_type": "multiple_choice",
-    "unsafe_code": false,
-    "dataset_name": "sk",
-    "dataset_path": "alexandrainst/m_mmlu",
-    "doc_to_choice": [
-      "A",
-      "B",
-      "C",
-      "D"
-    ],
-    "doc_to_target": "answer",
-    "fewshot_split": "train",
-    "fewshot_config": {
-      "sampler": "first_n"
-    },
-    "target_delimiter": " ",
-    "fewshot_delimiter": "\n\n",
-    "should_decontaminate": false
-  },
-  "hellaswag_sk": {
-    "tag": [
-      "hellaswag_multilingual"
-    ],
-    "task": "hellaswag_sk",
-    "repeats": 1,
-    "metadata": {
-      "version": 1
-    },
-    "description": "",
-    "doc_to_text": "query",
-    "metric_list": [
-      {
-        "metric": "acc",
-        "aggregation": "mean",
-        "higher_is_better": true
-      },
-      {
-        "metric": "acc_norm",
-        "aggregation": "mean",
-        "higher_is_better": true
-      }
-    ],
-    "num_fewshot": 0,
-    "output_type": "multiple_choice",
-    "unsafe_code": false,
-    "dataset_name": "sk",
-    "dataset_path": "alexandrainst/m_hellaswag",
-    "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
-    "doc_to_choice": "choices",
-    "doc_to_target": "{{label.lstrip()}}",
-    "target_delimiter": " ",
-    "validation_split": "val",
-    "fewshot_delimiter": "\n\n",
-    "should_decontaminate": false
-  },
-  "truthfulqa_sk_mc1": {
-    "tag": [
-      "truthfulqa_multilingual"
-    ],
-    "task": "truthfulqa_sk_mc1",
-    "repeats": 1,
-    "metadata": {
-      "version": 1
-    },
-    "description": "",
-    "doc_to_text": "query",
-    "metric_list": [
-      {
-        "metric": "acc",
-        "aggregation": "mean",
-        "higher_is_better": true
-      }
-    ],
-    "num_fewshot": 0,
-    "output_type": "multiple_choice",
-    "unsafe_code": false,
-    "dataset_name": "sk",
-    "dataset_path": "alexandrainst/m_truthfulqa",
-    "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        out_doc = {\n            \"question\": preprocess(doc[\"question\"]),\n            \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n            \"mc1_choices\": doc[\"mc1_targets_choices\"],\n            \"mc2_choices\": doc[\"mc2_targets_choices\"],\n            \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n            \"gold\": \" \",\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
-    "doc_to_choice": "mc1_choices",
-    "doc_to_target": 0,
-    "target_delimiter": " ",
-    "validation_split": "val",
-    "fewshot_delimiter": "\n\n",
-    "should_decontaminate": true,
-    "doc_to_decontamination_query": "question"
-  },
-  "truthfulqa_sk_mc2": {
-    "tag": [
-      "truthfulqa_multilingual"
-    ],
-    "task": "truthfulqa_sk_mc2",
-    "repeats": 1,
-    "metadata": {
-      "version": 1
-    },
-    "description": "",
-    "doc_to_text": "query",
-    "metric_list": [
-      {
-        "metric": "acc",
-        "aggregation": "mean",
-        "higher_is_better": true
-      }
-    ],
-    "num_fewshot": 0,
-    "output_type": "multiple_choice",
-    "unsafe_code": false,
-    "dataset_name": "sk",
-    "dataset_path": "alexandrainst/m_truthfulqa",
-    "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        out_doc = {\n            \"question\": preprocess(doc[\"question\"]),\n            \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n            \"mc1_choices\": doc[\"mc1_targets_choices\"],\n            \"mc2_choices\": doc[\"mc2_targets_choices\"],\n            \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n            \"gold\": \" \",\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
-    "doc_to_choice": "mc2_choices",
-    "doc_to_target": 0,
-    "process_results": "def process_results_mc2(doc, results):\n    lls, is_greedy = zip(*results)\n\n    # Split on the first `0` as everything before it is true (`1`).\n    split_idx = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n    # Compute the normalized probability mass for the correct answer.\n    ll_true, ll_false = lls[:split_idx], lls[split_idx:]\n    p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n    p_true = p_true / (sum(p_true) + sum(p_false))\n\n    return {\"acc\": sum(p_true)}\n",
-    "target_delimiter": " ",
-    "validation_split": "val",
-    "fewshot_delimiter": "\n\n",
-    "should_decontaminate": true,
-    "doc_to_decontamination_query": "question"
-  }
-}

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc1.yaml

@@ -1,34 +0,0 @@
-tag:
-- truthfulqa_multilingual
-task: truthfulqa_sk_mc1
-repeats: 1
-metadata:
-  version: 1
-description: ''
-doc_to_text: query
-metric_list:
-- metric: acc
-  aggregation: mean
-  higher_is_better: true
-num_fewshot: 0
-output_type: multiple_choice
-unsafe_code: false
-dataset_name: sk
-dataset_path: alexandrainst/m_truthfulqa
-process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
-  \    def _process_doc(doc):\n        out_doc = {\n            \"question\": preprocess(doc[\"\
-  question\"]),\n            \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"\
-  question\"]) + \"\\nA:\",\n            \"mc1_choices\": doc[\"mc1_targets_choices\"\
-  ],\n            \"mc2_choices\": doc[\"mc2_targets_choices\"],\n            \"mc2_targets\"\
-  : {\"labels\": doc[\"mc2_targets_labels\"]},\n            \"gold\": \" \",\n   \
-  \     }\n        return out_doc\n\n    return dataset.map(_process_doc)\n"
-doc_to_choice: mc1_choices
-doc_to_target: 0
-target_delimiter: ' '
-validation_split: val
-fewshot_delimiter: '
-
-
-  '
-should_decontaminate: true
-doc_to_decontamination_query: question

aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc2.yaml

@@ -1,40 +0,0 @@
-tag:
-- truthfulqa_multilingual
-task: truthfulqa_sk_mc2
-repeats: 1
-metadata:
-  version: 1
-description: ''
-doc_to_text: query
-metric_list:
-- metric: acc
-  aggregation: mean
-  higher_is_better: true
-num_fewshot: 0
-output_type: multiple_choice
-unsafe_code: false
-dataset_name: sk
-dataset_path: alexandrainst/m_truthfulqa
-process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
-  \    def _process_doc(doc):\n        out_doc = {\n            \"question\": preprocess(doc[\"\
-  question\"]),\n            \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"\
-  question\"]) + \"\\nA:\",\n            \"mc1_choices\": doc[\"mc1_targets_choices\"\
-  ],\n            \"mc2_choices\": doc[\"mc2_targets_choices\"],\n            \"mc2_targets\"\
-  : {\"labels\": doc[\"mc2_targets_labels\"]},\n            \"gold\": \" \",\n   \
-  \     }\n        return out_doc\n\n    return dataset.map(_process_doc)\n"
-doc_to_choice: mc2_choices
-doc_to_target: 0
-process_results: "def process_results_mc2(doc, results):\n    lls, is_greedy = zip(*results)\n\
-  \n    # Split on the first `0` as everything before it is true (`1`).\n    split_idx\
-  \ = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n    # Compute the normalized\
-  \ probability mass for the correct answer.\n    ll_true, ll_false = lls[:split_idx],\
-  \ lls[split_idx:]\n    p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n\
-  \    p_true = p_true / (sum(p_true) + sum(p_false))\n\n    return {\"acc\": sum(p_true)}\n"
-target_delimiter: ' '
-validation_split: val
-fewshot_delimiter: '
-
-
-  '
-should_decontaminate: true
-doc_to_decontamination_query: question

aya_300k_lr_experiments/benchmarks/lmeval/provenance/lm_eval_commit.txt

@@ -1 +0,0 @@
-6d2abda4fd171e68a8789330c4149e37c1ca0bda

aya_300k_lr_experiments/benchmarks/lmeval/provenance/package_versions.json

@@ -1,8 +0,0 @@
-{
-  "lm_eval": "0.4.8",
-  "transformers": "4.48.2",
-  "torch": "2.6.0+cu124",
-  "datasets": "5.0.1",
-  "peft": "0.14.0",
-  "evaluate": "0.4.6"
-}

aya_300k_lr_experiments/benchmarks/lmeval/provenance/run_settings.txt

@@ -1,8 +0,0 @@
-model=hf
-pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
-tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4
-dtype=bfloat16
-num_fewshot=0
-batch_size=1
-apply_chat_template=false
-CUDA_VISIBLE_DEVICES=1

aya_300k_lr_experiments/benchmarks/lmeval/provenance/tasks.txt

@@ -1 +0,0 @@
-arc_sk,hellaswag_sk,m_mmlu_sk,truthfulqa_sk_mc1,truthfulqa_sk_mc2,sklegal,skquad

aya_300k_lr_experiments/benchmarks/skbench/audit.md

@@ -1,82 +0,0 @@
-# Audit úspešného čiastočného SK-bench behu
-
-Vybraný model po tréningu na 300k vzorke Aya s LR `0.00010` dokončil beh `sk_zero_available_without_nli_lr_0p00010_20261022_185255`. Log končí hláškou `Benchmarking completed.` a obsahuje výsledné metriky. Benchmarkovalo sa iba toto LR, vybrané podľa najnižšieho finálneho validačného loss zo šiestich tréningových behov; výsledky preto neporovnávajú benchmarkovú kvalitu šiestich LR ani budúceho tréningu na celej Aya množine.
-
-## Pokrytie a počty
-
-- Zoznam `configs/tasks.txt`, 222 konfigurácií úloh vo výslednom JSON a 222 súborov DETAILS sa zhodujú. Názvy sa pri porovnaní dekódujú z Windows `%3A`/`%7C` a normalizuje sa koncové označenie zero-shot variantu.
-- Vyhodnotených bolo **222 zo 412 pôvodných task variantov**: 74 kombinácií úlohy/subtasku, každá s tromi promptmi. Ide o 13 rodín úloh. MMLU tvorí 171 variantov (57 predmetov × 3 prompty).
-- DETAILS obsahujú **159 426 vyhodnotených inštancií**, po **53 142 na každý prompt**. Tri prompty opakovane hodnotia tie isté základné testové otázky; 159 426 nie je počet nezávislých otázok ani dôkaz unikátnosti medzi rôznymi datasetmi.
-- `configs/omitted_tasks.txt` obsahuje 190 vynechaných variantov. Report dostupnosti eviduje 12 nedostupných repozitárov so 187 variantmi; tri ďalšie vynechané varianty patria samostatnému NLI-SK. Historický access report uvádza NLI-SK ako `OK`, ale jeho tri varianty v konečnom behu nie sú. NLI úlohy z rodiny SKLEP zostali zahrnuté.
-- `max_samples: null` znamená, že beh nemal globálny limit vzoriek. Počty sa získali z metadát jednotlivých DETAILS súborov, nie zo zaokrúhlených metrík.
-
-Tento dostupný subset sa nesmie označovať za oficiálne kompletné SK-bench skóre. Chýbajúce úlohy ani chýbajúce rodiny sa neimputovali.
-
-## Zaznamenané nastavenie
-
-Výsledný JSON potvrdzuje `bfloat16`, kontext `32768`, tensor/data/pipeline parallelism `1/1/1`, využitie GPU pamäte `0.85`, seed modelu `1234`, teplotu `0` a `enforce_eager: true`. Pole `generation_parameters.seed` je `null`; konkrétny seed je uložený v `model_config.seed`. Všetky úlohy majú `num_fewshots: 0`.
-
-216 task variantov používa najviac 32 generovaných tokenov a šesť QA variantov 126 tokenov; stop sekvencia je nový riadok. Tieto limity výstupu treba pri porovnaní výsledkov zachovať. Záznam neobsahuje spoľahlivý počet odpovedí ukončených výstupným limitom.
-
-Zmrazený commit SK-bench je `15823bbcc2d995e69441fcb7d34abb61bd8180c5`. Pôvodné `lighteval_sha` vo výslednom JSON je `?`; tento údaj sa dodatočne nevymýšľa. `uv.lock`, commit, modelová konfigurácia, per-task konfigurácie a revízie datasetov sú uchované v `provenance/` a `configs/`. Log zaznamenáva aj runtime patchovanie; samotný commit a lock preto bez rovnakého postupu nie sú úplným dôkazom identického prostredia.
-
-## Orezanie dlhých vstupov
-
-Kontrola projektovala `model_response.truncated_tokens_count` zo **všetkých 222 DETAILS súborov**, bez čítania duplicitného generation cache. Všetkých 159 426 hodnôt existuje, žiadna nie je `null` a všetky sú nula. Aj pôvodné súhrnné počítadlá sú nulové vrátane `non_truncated`, takže samy osebe nepotvrdzujú neprítomnosť orezania.
-
-**Log priamo potvrdzuje orezanie kontextu.** Vyhľadanie `truncat` bez rozlíšenia veľkosti písmen našlo presne jednu hlášku, na LF riadku 16033:
-
-```text
-context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens.
-```
-
-Následná diagnostika dĺžok zo všetkých DETAILS našla **69 tokenových polí presne na vstupnom limite 32 642 tokenov**, po 23 v každom z troch variantov `multi_wiki_qa`. Vo všetkých troch sa opakujú tie isté indexy riadkov. Žiadne uložené pole neprekročilo príslušný limit `32768 − generation_size`; ostatné úlohy mali vstupy pod limitom. Súvisiace uložené reťazce vstupu mali pri hraničných riadkoch až 121 852 znakov.
-
-Týchto 69 riadkov sú **hraniční kandidáti zodpovedajúci orezaniu**, nie spoľahlivo dokázaný počet orezaných vstupov. Log neuvádza identifikátor konkrétneho tasku ani počet dotknutých vzoriek. Bez opätovnej tokenizácie pôvodných vstupov nemožno z dĺžky uloženého reťazca určiť pôvodnú dĺžku v tokenoch alebo počet odstránených tokenov. Overený záver je: **orezanie nastalo aspoň raz; jeho presný rozsah nie je z dostupných počítadiel známy**.
-
-Strojový súhrn preto obsahuje `context_truncation_observed_in_log: true`, `actual_truncated_sample_count: null` a nulové počítadlá označuje prefixom `details_reported_`. Podrobnosti sú v `results/truncation_audit_per_task.csv`, `results/log_truncation_evidence.json`, `results/input_length_diagnostics_per_task.csv` a `results/input_length_budget_boundary_rows.csv`.
-
-## Metriky a agregácia
-
-`results/per_task_metrics.csv` obsahuje iba 222 reálne vykonaných task variantov, ich skutočné metriky a štandardné chyby priamo z výsledného JSON. Prioritná metrika na čítanie je `em_relaxed`, prípadne `em_norm_relaxed`, ak ju daná úloha skutočne používa. QA má vlastné metriky `exact_match_slk_prefix` a `f1_slk`; nepremenúvajú sa na všeobecnú accuracy.
-
-Odvodené súhrny nikdy nemiešajú odlišné metriky do jedného skóre:
-
-- `family_summary.csv`: priemer v rodine, vždy osobitne pre konkrétnu metriku, cez všetky dostupné subtasky a tri prompty.
-- `family_prompt_summary.csv`: tá istá agregácia osobitne pre prompt 1, 2 a 3.
-- `three_prompt_summary.csv`: priemer troch promptov pre každú zo 74 konkrétnych kombinácií úlohy/subtasku, opäť pre každú metriku osobitne.
-- `task_macro_mean` je aritmetický priemer task metrík s rovnakou váhou každého variantu. `sample_weighted_mean` váži tieto metriky počtom riadkov v DETAILS. Prompty sa v oboch prípadoch považujú za opakované vyhodnotenia, nie za nové nezávislé otázky. Súhrnné intervaly spoľahlivosti sa z per-task chýb nevymýšľajú.
-
-Pôvodný JSON má 226 kľúčov výsledkov: 222 úloh, tri vstavané MMLU promptové priemery a `all`. Štyri agregované riadky sú zachované v pôvodnom JSON, ale v odvodených súhrnoch sa znovu nepripočítavajú. `all` obsahuje viac metrík s rôznymi sadami úloh a netvorí oficiálne celkové skóre.
-
-Napríklad MMLU `em_relaxed` má task macro priemer `0.1600941253` a sample-weighted priemer `0.1572245190`. Rozdiel vzniká rôznou veľkosťou 57 predmetov. SKLEP klasifikačné úlohy s `em_norm_relaxed` majú macro `0.3507994510`, ale sample-weighted `0.2144044321`; výber váh tu zásadne mení opisný výsledok.
-
-## Obnovenie auditu
-
-Audit používa Python a `pyarrow==26.0.0`. Knižnica sa pri spracovaní nainštalovala len do pomocného priečinka pracovného projektu; nie je súčasťou tohto stromu. Pôvodné parquety, cache ani váhy sa do Git podkladov nekopírujú. Na opätovný audit treba samostatne rozbalený zdrojový archív a jeho `archive_manifest.json`.
-
-Archív treba najprv rozbaliť pomocou uloženého pomocného skriptu (Python 3.11+), ktorý vytvorí aj `archive_manifest.json` a `archive_summary.json`. Názvy s `:` a `|` prevedie na prenosné percent-encoded názvy a zachová mapovanie na originál. Priečinok určený cez `--output` musí byť nový alebo prázdny:
-
-```bash
-python scripts/unpack_results_archive.py \
-  --archive /path/to/aya_300k_benchmarky_20261024_215708.tar.gz \
-  --output /path/to/extracted/archive
-```
-
-Z koreňa experimentu možno potom spustiť:
-
-```bash
-python -m pip install --target .audit-deps -r benchmarks/skbench/provenance/requirements-audit.txt
-python benchmarks/skbench/provenance/audit_skbench.py \
-  --source /path/to/extracted/archive \
-  --output benchmarks/skbench \
-  --pyarrow-path .audit-deps
-python benchmarks/skbench/provenance/input_length_diagnostics.py \
-  --source /path/to/extracted/archive \
-  --output benchmarks/skbench/results \
-  --pyarrow-path .audit-deps
-```
-
-Prvý skript kontroluje pokrytie, kopíruje malé pôvodné artefakty, počíta metrikové súhrny a overuje truncation counter projekciou jediného stĺpca. Druhý skript projektuje uložené tokenové polia a vstupné reťazce, ale exportuje iba ich dĺžky a indexy hraničných riadkov. Texty otázok a odpovedí v týchto odvodených diagnostikách nie sú.
-
-SHA-256 presných kópií pôvodných súborov je v `provenance/copied_files_sha256.csv`; zdrojový archív identifikuje `provenance/archive_summary.json`.

aya_300k_lr_experiments/benchmarks/skbench/provenance/archive_summary.json

@@ -1,87 +0,0 @@
-{
-  "source": "C:\\Users\\Jakub\\Desktop\\aya_300k_benchmarky_20261024_215708.tar.gz",
-  "compressed_bytes": 349555561,
-  "sha256": "b45e431cbc63bc73ef953f8aa3837695ccec43783e3da2dfccb0d99f7d1fe79c",
-  "files": 581,
-  "uncompressed_bytes": 1782181826,
-  "skipped_special": [],
-  "top_level_files": {
-    "aya_300k_lr_validation": 110,
-    "skbench_evaluation": 464,
-    "scripts": 7
-  },
-  "largest_files": [
-    {
-      "original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk|0_2026-10-22T21-55-04.268099.parquet",
-      "local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet",
-      "size": 127542209,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk|0_2026-10-22T21-55-04.268099.parquet",
-      "local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet",
-      "size": 127233608,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk|0_2026-10-22T21-55-04.268099.parquet",
-      "local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet",
-      "size": 125437876,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt2_qa_sk|0/82e1a28fce479a47/GENERATIVE.parquet",
-      "local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt2_qa_sk%7C0/82e1a28fce479a47/GENERATIVE.parquet",
-      "size": 103508951,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt1_qa_sk|0/f0737eb9e82c262f/GENERATIVE.parquet",
-      "local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt1_qa_sk%7C0/f0737eb9e82c262f/GENERATIVE.parquet",
-      "size": 103246110,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt3_qa_sk|0/579d61d41168b7f2/GENERATIVE.parquet",
-      "local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt3_qa_sk%7C0/579d61d41168b7f2/GENERATIVE.parquet",
-      "size": 101814277,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet",
-      "local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet",
-      "size": 66791884,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet",
-      "local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet",
-      "size": 66012897,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet",
-      "local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet",
-      "size": 64169583,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt2_qa_sk:question-answering|0/46d75c2a99a95ed9/GENERATIVE.parquet",
-      "local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt2_qa_sk%3Aquestion-answering%7C0/46d75c2a99a95ed9/GENERATIVE.parquet",
-      "size": 53746461,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt1_qa_sk:question-answering|0/b09ec46c034c8de4/GENERATIVE.parquet",
-      "local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt1_qa_sk%3Aquestion-answering%7C0/b09ec46c034c8de4/GENERATIVE.parquet",
-      "size": 53089807,
-      "type": "file"
-    },
-    {
-      "original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt3_qa_sk:question-answering|0/06ebedbab75fe49b/GENERATIVE.parquet",
-      "local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt3_qa_sk%3Aquestion-answering%7C0/06ebedbab75fe49b/GENERATIVE.parquet",
-      "size": 51637093,
-      "type": "file"
-    }
-  ]
-}

aya_300k_lr_experiments/benchmarks/skbench/provenance/audit_skbench.py

@@ -1,234 +0,0 @@
-"""Curate the completed 300k learning-rate SK-bench run without copying raw data.
-
-Requires Python 3.10+ and pyarrow (installed separately). The only read of the
-large details parquets projects model_response.truncated_tokens_count; row counts
-come from Parquet metadata. Cached GENERATIVE.parquet files are never read.
-"""
-from __future__ import annotations
-
-import argparse
-import collections
-import csv
-import hashlib
-import json
-import pathlib
-import re
-import shutil
-import statistics
-import sys
-from urllib.parse import unquote
-
-
-def write_csv(path, rows):
-    if not rows:
-        return
-    columns = list(dict.fromkeys(k for row in rows for k in row))
-    with path.open('w', newline='', encoding='utf-8') as stream:
-        writer = csv.DictWriter(stream, fieldnames=columns)
-        writer.writeheader()
-        writer.writerows(rows)
-
-
-def normalize(name):
-    name = unquote(name.strip())
-    return re.sub(r'\|([0-9]+)$', r':\1', name)
-
-
-def get_lines(path):
-    return [line.strip() for line in path.read_text(encoding='utf-8').splitlines() if line.strip() and not line.startswith('#')]
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('--source', type=pathlib.Path, required=True, help='Extracted archive root containing skbench_evaluation/')
-    parser.add_argument('--output', type=pathlib.Path, required=True)
-    parser.add_argument('--pyarrow-path', type=pathlib.Path, help='Optional project-local pip --target directory')
-    args = parser.parse_args()
-    if args.pyarrow_path:
-        sys.path.insert(0, str(args.pyarrow_path.resolve()))
-    import pyarrow
-    import pyarrow.parquet as pq
-
-    source = args.source.resolve()
-    out = args.output.resolve()
-    base = source / 'skbench_evaluation'
-    run_name = 'sk_zero_available_without_nli_lr_0p00010_20261022_185255'
-    run = base / 'results' / run_name
-    for name in ['results', 'logs', 'configs', 'provenance']:
-        (out/name).mkdir(parents=True, exist_ok=True)
-    result_path = run/'results/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/results_2026-10-22T21-55-04.268099.json'
-    d = json.loads(result_path.read_text(encoding='utf-8'))
-    tasks = get_lines(run/'tasks.txt')
-    omitted = get_lines(run/'omitted_tasks.txt')
-    configs = d['config_tasks']
-    normalized_expected = {normalize(x) for x in tasks}
-    assert len(tasks) == len(normalized_expected) == 222
-    assert normalized_expected == {normalize(x) for x in configs}, 'Config task coverage mismatch'
-    assert normalized_expected <= {normalize(x) for x in d['results']}, 'Result task coverage mismatch'
-    assert len(omitted) == 190
-    all_tasks = get_lines(base/'configs/tasks_sk_zero.txt')
-    assert normalized_expected.isdisjoint({normalize(x) for x in omitted})
-    assert normalized_expected | {normalize(x) for x in omitted} == {normalize(x) for x in all_tasks}
-
-    copies = {result_path: out/'results'/result_path.name,
-              base/'logs'/f'{run_name}.log': out/'logs'/f'{run_name}.log',
-              run/'tasks.txt': out/'configs/tasks.txt',
-              run/'omitted_tasks.txt': out/'configs/omitted_tasks.txt',
-              run/'uv.lock': out/'provenance/uv.lock',
-              run/'skbench_commit.txt': out/'provenance/skbench_commit.txt',
-              source/'archive_summary.json': out/'provenance/archive_summary.json'}
-    for path in sorted((base/'configs').iterdir()):
-        if path.is_file():
-            copies[path] = out/'configs'/path.name
-    provenance = []
-    for src, dst in copies.items():
-        shutil.copyfile(src, dst)
-        data = src.read_bytes()
-        provenance.append({'source_archive_member': src.relative_to(source).as_posix(),
-                           'curated_path': dst.relative_to(out).as_posix(),
-                           'bytes': len(data), 'sha256': hashlib.sha256(data).hexdigest()})
-    write_csv(out/'provenance/copied_files_sha256.csv', provenance)
-    (out/'configs/model_config.json').write_text(json.dumps(d['config_general'], indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
-    (out/'configs/task_configs.json').write_text(json.dumps(configs, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
-
-    # Resolve the original names from the extraction manifest; Windows paths
-    # encode ':' and '|' as %3A and %7C, while payload bytes were unchanged.
-    manifest = json.loads((source/'archive_manifest.json').read_text(encoding='utf-8'))
-    if isinstance(manifest, dict):
-        manifest = manifest.get('entries', manifest.get('files', []))
-    original_paths = {x['local']: x['original'] for x in manifest if isinstance(x, dict) and 'local' in x}
-    details = []
-    lookup = {}
-    files = sorted((run/'details').rglob('*.parquet'))
-    assert len(files) == 222, f'Expected 222 DETAILS files, got {len(files)}'
-    for file in files:
-        decoded = unquote(file.name)
-        task = re.match(r'^details_(.+)_2026-10-22T21-55-04\.268099\.parquet$', decoded).group(1)
-        pf = pq.ParquetFile(file)
-        projected = pf.read(columns=['model_response.truncated_tokens_count'])
-        values = projected.column('model_response').combine_chunks().field('truncated_tokens_count').to_pylist()
-        assert len(values) == pf.metadata.num_rows
-        assert all(v is None or (isinstance(v, int) and v >= 0) for v in values)
-        row = {'task': task, 'sample_count': pf.metadata.num_rows,
-               'truncation_values_present': sum(v is not None for v in values),
-               'truncation_null_values': sum(v is None for v in values),
-               'samples_with_truncation': sum(v is not None and v > 0 for v in values),
-               'truncated_tokens_total': sum(v or 0 for v in values),
-               'max_truncated_tokens_per_sample': max((v or 0 for v in values), default=0),
-               'parquet_bytes': file.stat().st_size,
-               'source_archive_member': original_paths.get(file.relative_to(source).as_posix(), file.relative_to(source).as_posix()),
-               'extracted_relative_path': file.relative_to(source).as_posix()}
-        assert normalize(task) not in lookup
-        lookup[normalize(task)] = row
-        details.append(row)
-    assert set(lookup) == normalized_expected, 'DETAILS task coverage mismatch'
-    write_csv(out/'results/truncation_audit_per_task.csv', details)
-
-    metric_names = sorted({key for task in configs for key in d['results'][task]})
-    per_task = []
-    for task in sorted(configs):
-        metric = d['results'][task]
-        config = configs[task]
-        detail = lookup[normalize(task)]
-        primary = next((name for name in ['em_relaxed', 'em_norm_relaxed', 'exact_match_slk_prefix'] if name in metric), None)
-        family = task.split('.prompt')[0]
-        prompt = int(re.search(r'\.prompt([123])_', task).group(1))
-        task_group = re.sub(r'\.prompt[123]_', '.', task).removesuffix('|0')
-        row = {'task': task, 'family': family, 'task_group_without_prompt': task_group,
-               'prompt': prompt, 'num_fewshots': config['num_fewshots'], 'sample_count': detail['sample_count'],
-               'primary_metric': primary, 'primary_value': metric[primary],
-               'generation_size': config['generation_size'], 'evaluation_splits': '|'.join(config['evaluation_splits']),
-               'hf_repo': config['hf_repo'], 'hf_subset': config['hf_subset'], 'hf_revision': config['hf_revision']}
-        row.update({key: metric.get(key, '') for key in metric_names})
-        per_task.append(row)
-    write_csv(out/'results/per_task_metrics.csv', per_task)
-
-    # Each summary groups only identical metrics; values remain 0..1.
-    # Macro means weight task variants equally. Sample-weighted means weight
-    # their actual DETAILS rows. Repeated prompts are evaluation instances,
-    # never additional independent underlying test questions.
-    def summarize(group_columns, filename):
-        groups = collections.defaultdict(list)
-        for row in per_task:
-            for metric_name in metric_names:
-                if not metric_name.endswith('_stderr') and row[metric_name] != '':
-                    groups[tuple(row[x] for x in group_columns)+(metric_name,)].append(row)
-        result = []
-        for key, rows in sorted(groups.items()):
-            metric_name = key[-1]
-            total = sum(x['sample_count'] for x in rows)
-            item = dict(zip(group_columns, key[:-1]))
-            item.update({'metric': metric_name, 'task_variants': len(rows),
-                         'prompt_count': len({x['prompt'] for x in rows}),
-                         'evaluated_instances': total,
-                         'task_macro_mean': statistics.mean(x[metric_name] for x in rows),
-                         'sample_weighted_mean': sum(x['sample_count']*x[metric_name] for x in rows)/total,
-                         'min_task_value': min(x[metric_name] for x in rows),
-                         'max_task_value': max(x[metric_name] for x in rows)})
-            result.append(item)
-        write_csv(out/'results'/filename, result)
-        return result
-    family_summary = summarize(['family'], 'family_summary.csv')
-    summarize(['family', 'prompt'], 'family_prompt_summary.csv')
-    three_prompt = summarize(['family', 'task_group_without_prompt'], 'three_prompt_summary.csv')
-    assert all(x['task_variants'] == x['prompt_count'] == 3 for x in three_prompt)
-
-    access = json.loads((base/'configs/dataset_access_report.json').read_text(encoding='utf-8'))
-    inaccessible = [x for x in access if x['status'] != 'OK']
-    with (base/'logs'/f'{run_name}.log').open(encoding='utf-8', newline='') as stream:
-        log_text = stream.read()
-    truncation_warnings = re.findall(r'context_size \+ max_new_tokens=\d+[^\r\n]*Truncating context to \d+ tokens\.', log_text)
-    warning_evidence = []
-    for line_number, line in enumerate(log_text.split('\n'), 1):
-        if 'Truncating context' in line:
-            warning_evidence.append({'line_number': line_number, 'message': line[line.index('context_size'):], 'source': f'logs/{run_name}.log'})
-    (out/'results/log_truncation_evidence.json').write_text(json.dumps(warning_evidence, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
-    summary = {'run_name': run_name, 'result_file': result_path.name,
-               'task_list_count': len(tasks), 'configured_tasks': len(configs), 'scored_task_variants': len(per_task),
-               'result_json_keys': len(d['results']),
-               'extra_result_keys_excluded_from_task_summaries': sorted(set(d['results'])-set(configs)),
-               'all_original_task_variants': len(all_tasks),
-               'omitted_task_variants': len(omitted), 'inaccessible_repositories': len(inaccessible),
-               'inaccessible_task_variants': sum(len(x['tasks']) for x in inaccessible),
-               'additionally_excluded_nli_task_variants': 3,
-               'families': dict(sorted(collections.Counter(row['family'] for row in per_task).items())),
-               'prompt_counts': dict(sorted(collections.Counter(row['prompt'] for row in per_task).items())),
-               'evaluated_instances': sum(row['sample_count'] for row in per_task),
-               'instances_by_prompt': {str(prompt): sum(x['sample_count'] for x in per_task if x['prompt'] == prompt) for prompt in [1,2,3]},
-               'details_files_audited': len(details), 'truncation_column': 'model_response.truncated_tokens_count',
-               'truncation_values_present': sum(x['truncation_values_present'] for x in details),
-               'truncation_null_values': sum(x['truncation_null_values'] for x in details),
-               'details_reported_samples_with_truncation': sum(x['samples_with_truncation'] for x in details),
-               'details_reported_truncated_tokens_total': sum(x['truncated_tokens_total'] for x in details),
-               'details_reported_max_truncated_tokens_per_sample': max(x['max_truncated_tokens_per_sample'] for x in details),
-               'context_truncation_observed_in_log': bool(truncation_warnings),
-               'actual_truncated_sample_count': None,
-               'console_context_truncation_warning_count': len(truncation_warnings),
-               'console_context_truncation_warnings': truncation_warnings,
-               'truncation_interpretation': 'All DETAILS counters are zero, but the console explicitly reports context truncation. Do not infer no truncation or an affected-sample count from these counters.',
-               'model_config': d['config_general']['model_config'],
-               'max_samples': d['config_general']['max_samples'],
-               'generation_size_counts': dict(collections.Counter(str(x['generation_size']) for x in per_task)),
-               'original_summary_general': d['summary_general'],
-               'pyarrow_version': pyarrow.__version__, 'python_version': sys.version,
-               'aggregation': 'Descriptive metric-specific task macro means and sample-weighted means; no official full-SK-bench overall score.',
-               'limitations': ['Partial available zero-shot subset: 222 of 412 task variants.',
-                               'Only the validation-selected LR=0.00010 was benchmarked.',
-                               'Sample counts are evaluation instances; prompt variants reuse underlying questions.',
-                               'One console warning reports context truncation; all DETAILS counters are zero, so the recorded counters do not reliably enumerate affected inputs.',
-                               'The audit cannot recover the number of affected tasks/samples or quantify output generation-size termination.',
-                               'All-zero summary counters include non_truncated=0 and cannot prove absence of truncation.',
-                               'Raw DETAILS and generation-cache parquets are excluded from this curated tree.']}
-    (out/'results/audit_summary.json').write_text(json.dumps(summary, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
-    script_destination = out/'provenance/audit_skbench.py'
-    if pathlib.Path(__file__).resolve() != script_destination.resolve():
-        shutil.copyfile(pathlib.Path(__file__), script_destination)
-    print(json.dumps({key: summary[key] for key in ['scored_task_variants', 'evaluated_instances', 'instances_by_prompt', 'details_reported_samples_with_truncation', 'context_truncation_observed_in_log', 'actual_truncated_sample_count', 'truncation_null_values', 'families']}, indent=2))
-    print('Primary family macro means (metrics kept separate):')
-    for row in family_summary:
-        if row['metric'] in ['em_relaxed', 'em_norm_relaxed', 'exact_match_slk_prefix', 'f1_slk']:
-            print(row)
-
-
-if __name__ == '__main__':
-    main()

aya_300k_lr_experiments/benchmarks/skbench/provenance/copied_files_sha256.csv

@@ -1,14 +0,0 @@
-source_archive_member,curated_path,bytes,sha256
-skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/results/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/results_2026-10-22T21-55-04.268099.json,results/results_2026-10-22T21-55-04.268099.json,638180,966123dc20b547860d04f3434f65752c681ca2d83baebee22873e553161ff0df
-skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log,logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log,4537535,23a857210201913c5506ac39553a86fd803ed0c057e1226fcd978030a40b6c78
-skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/tasks.txt,configs/tasks.txt,11415,297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
-skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/omitted_tasks.txt,configs/omitted_tasks.txt,8629,613b78472e6788eab826f31da1100c00b597ed9bbc8151af38ebb8ef5bf830af
-skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/uv.lock,provenance/uv.lock,690556,045d2840fda141981afba93bd4a1b9ddb59f0f0791c162cf66b2c88e6491fb41
-skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/skbench_commit.txt,provenance/skbench_commit.txt,41,12655509d516c6a57225a072f708e5ca854600e7daa3bf5cdc95e6b36e94fe70
-archive_summary.json,provenance/archive_summary.json,6529,535390044cd67477ea1631527ac736f72108935f25a11a613b1709caf8154957
-skbench_evaluation/configs/dataset_access_report.json,configs/dataset_access_report.json,29828,8eea8c082d1eb914850128d48021a39c7e1fb7189578c5f1e496004af6ff593b
-skbench_evaluation/configs/export_lr_0p00010.yaml,configs/export_lr_0p00010.yaml,391,bff3e0b8abd130122c8d88be71647e0ea67e4a836b3caaed4e3bc307ef267dca
-skbench_evaluation/configs/tasks_sk_zero.txt,configs/tasks_sk_zero.txt,20044,25c3cdfeb1f46adb86b68be24a9d8e22439733501e2fffda1ce5034e610660a4
-skbench_evaluation/configs/tasks_sk_zero_available.txt,configs/tasks_sk_zero_available.txt,11517,5ce7f1666b2073afd93aab03a116937f48954cf51d313c567c198b31f0dfcaf2
-skbench_evaluation/configs/tasks_sk_zero_available_without_nli.txt,configs/tasks_sk_zero_available_without_nli.txt,11415,297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
-skbench_evaluation/configs/tasks_sk_zero_unavailable.txt,configs/tasks_sk_zero_unavailable.txt,8527,0071cffec8ab2446a5594d0b3a3116151e7fc375bb300394e9ea7cb91ed7df10

aya_300k_lr_experiments/benchmarks/skbench/provenance/input_length_diagnostics.py

@@ -1,62 +0,0 @@
-"""Inspect token-array and raw-prompt lengths; never export prompt content."""
-import argparse, csv, json, pathlib, re, shutil, statistics, sys
-from urllib.parse import unquote
-
-parser=argparse.ArgumentParser(description=__doc__)
-parser.add_argument('--source',type=pathlib.Path,required=True)
-parser.add_argument('--output',type=pathlib.Path,required=True)
-parser.add_argument('--pyarrow-path',type=pathlib.Path)
-args=parser.parse_args()
-if args.pyarrow_path: sys.path.insert(0,str(args.pyarrow_path.resolve()))
-import pyarrow.parquet as pq
-run=args.source/'skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255'
-d=json.loads(next((run/'results').rglob('*.json')).read_text())
-limit=d['config_general']['model_config']['max_model_length']
-rows=[]
-outliers=[]
-for file in sorted((run/'details').rglob('*.parquet')):
-    task=re.match(r'details_(.+)_2026-10-22T21-55-04\.268099\.parquet',unquote(file.name)).group(1)
-    cfg=d['config_tasks'][task]
-    budget=limit-cfg['generation_size']
-    data=pq.ParquetFile(file).read(columns=['model_response.input_tokens','model_response.input'])
-    data=data.column('model_response').combine_chunks()
-    tokens=data.field('input_tokens').to_pylist()
-    texts=data.field('input').to_pylist()
-    lens=[None if value is None else len(value) for value in tokens]
-    chars=[None if value is None else len(value) for value in texts]
-    row={'task':task,'generation_size':cfg['generation_size'],'context_limit':limit,'input_budget':budget,
-         'samples':len(lens),'null_token_arrays':sum(x is None for x in lens),'null_input_strings':sum(x is None for x in chars),
-         'min_recorded_input_tokens':min(x for x in lens if x is not None),
-         'max_recorded_input_tokens':max(x for x in lens if x is not None),
-         'mean_recorded_input_tokens':statistics.mean(x for x in lens if x is not None),
-         'arrays_at_input_budget':sum(x==budget for x in lens),
-         'arrays_over_input_budget':sum(x is not None and x>budget for x in lens),
-         'max_recorded_input_characters':max(x for x in chars if x is not None)}
-    rows.append(row)
-    for index,length in enumerate(lens):
-        if length is not None and length>=budget:
-            outliers.append({'task':task,'details_row_index_zero_based':index,'input_token_array_length':length,
-                             'generation_size':cfg['generation_size'],'input_budget':budget,
-                             'recorded_input_characters':chars[index]})
-for name,values in [('input_length_diagnostics_per_task.csv',rows),('input_length_budget_boundary_rows.csv',outliers)]:
-    with (args.output/name).open('w',newline='',encoding='utf-8') as f:
-        writer=csv.DictWriter(f,fieldnames=list(values[0]) if values else ['task'])
-        writer.writeheader();writer.writerows(values)
-with (args.source/'skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log').open(encoding='utf-8',newline='') as stream:
-    log=stream.read()
-messages=[]
-for number,line in enumerate(log.split('\n'),1):
-    if re.search(r'truncat',line,re.I):
-        messages.append({'log_lf_line_number':number,'message':re.sub(r'\x1b\[[0-9;]*[A-Za-z]','',line).strip()[-1500:]})
-summary={'task_count':len(rows),'evaluated_instances':sum(x['samples'] for x in rows),
-         'max_recorded_input_tokens':max(x['max_recorded_input_tokens'] for x in rows),
-         'arrays_at_input_budget':sum(x['arrays_at_input_budget'] for x in rows),
-         'arrays_over_input_budget':sum(x['arrays_over_input_budget'] for x in rows),
-         'log_messages_matching_truncat_case_insensitive':messages,
-         'limitation':'Input strings and token arrays are recorded artifacts, not a separately tokenized original source prompt. Their lengths cannot recover how much text was discarded or establish the number of truly truncated inputs.'}
-(args.output/'input_length_diagnostics_summary.json').write_text(json.dumps(summary,indent=2,ensure_ascii=False)+'\n',encoding='utf-8')
-print(json.dumps(summary,indent=2,ensure_ascii=False))
-print('BOUNDARY TASK COUNTS',json.dumps({task:sum(row['task']==task for row in outliers) for task in sorted({row['task'] for row in outliers})}))
-script_destination=args.output.parent/'provenance/input_length_diagnostics.py'
-if pathlib.Path(__file__).resolve()!=script_destination.resolve():
-    shutil.copyfile(pathlib.Path(__file__),script_destination)

aya_300k_lr_experiments/benchmarks/skbench/provenance/requirements-audit.txt

@@ -1 +0,0 @@
-pyarrow==26.0.0

aya_300k_lr_experiments/benchmarks/skbench/provenance/skbench_commit.txt

@@ -1 +0,0 @@
-15823bbcc2d995e69441fcb7d34abb61bd8180c5

aya_300k_lr_experiments/provenance/source_archive.json

@@ -1,12 +0,0 @@
-{
-  "source_archive": "aya_300k_benchmarky_20261024_215708.tar.gz",
-  "sha256": "b45e431cbc63bc73ef953f8aa3837695ccec43783e3da2dfccb0d99f7d1fe79c",
-  "compressed_bytes": 349555561,
-  "scope": "Hruska Aya 300k learning-rate selection and evaluation of selected LR 0.00010; Titan excluded",
-  "base_model": "slovak-nlp/mistral-sk-7b",
-  "base_model_revision": "089497ae0d72018e9591895c54f774bf0b4a83a4",
-  "selection_criterion": "lowest final validation loss among the six completed 300k runs",
-  "intended_next_stage": "planned training on the full Aya dataset on Perun; not executed in this experiment",
-  "data_reproducibility_limit": "No source dataset revision, data-file checksum, or held-out sample IDs were included in this archive",
-  "curation": "Model weights, checkpoints, trainer_state.json, duplicated inference caches and full parquet details excluded from the Git subset. Original archive retained separately."
-}

aya_300k_lr_experiments/results/eval_loss_history.csv

@@ -1,115 +0,0 @@
-learning_rate,run,step,eval_loss
-8e-05,lr_0p00008,1000,1.0020490884780884
-8e-05,lr_0p00008,2000,0.9672194719314575
-8e-05,lr_0p00008,3000,0.9505742192268372
-8e-05,lr_0p00008,4000,0.9352602362632751
-8e-05,lr_0p00008,5000,0.9294734597206116
-8e-05,lr_0p00008,6000,0.9264867305755615
-8e-05,lr_0p00008,7000,0.9184143543243408
-8e-05,lr_0p00008,8000,0.9120934009552002
-8e-05,lr_0p00008,9000,0.9065558910369873
-8e-05,lr_0p00008,10000,0.9004696011543274
-8e-05,lr_0p00008,11000,0.8929902911186218
-8e-05,lr_0p00008,12000,0.8897570371627808
-8e-05,lr_0p00008,13000,0.8844991326332092
-8e-05,lr_0p00008,14000,0.8800909519195557
-8e-05,lr_0p00008,15000,0.8777939677238464
-8e-05,lr_0p00008,16000,0.8767354488372803
-8e-05,lr_0p00008,17000,0.875752329826355
-8e-05,lr_0p00008,18000,0.875270426273346
-8e-05,lr_0p00008,18688,0.875216543674469
-0.0001,lr_0p00010,1000,1.0004310607910156
-0.0001,lr_0p00010,2000,0.9658941626548767
-0.0001,lr_0p00010,3000,0.9526628255844116
-0.0001,lr_0p00010,4000,0.9393344521522522
-0.0001,lr_0p00010,5000,0.9336668848991394
-0.0001,lr_0p00010,6000,0.931794285774231
-0.0001,lr_0p00010,7000,0.9227241277694702
-0.0001,lr_0p00010,8000,0.9158084392547607
-0.0001,lr_0p00010,9000,0.9100468158721924
-0.0001,lr_0p00010,10000,0.9019826650619507
-0.0001,lr_0p00010,11000,0.8941853642463684
-0.0001,lr_0p00010,12000,0.8896107077598572
-0.0001,lr_0p00010,13000,0.8851001262664795
-0.0001,lr_0p00010,14000,0.8795706033706665
-0.0001,lr_0p00010,15000,0.8770407438278198
-0.0001,lr_0p00010,16000,0.8748454451560974
-0.0001,lr_0p00010,17000,0.8731834888458252
-0.0001,lr_0p00010,18000,0.8727287650108337
-0.0001,lr_0p00010,18688,0.87261962890625
-0.00015,lr_0p00015,1000,1.0024861097335815
-0.00015,lr_0p00015,2000,0.9763674139976501
-0.00015,lr_0p00015,3000,0.97147536277771
-0.00015,lr_0p00015,4000,0.9657827019691467
-0.00015,lr_0p00015,5000,0.9659448266029358
-0.00015,lr_0p00015,6000,0.9597828388214111
-0.00015,lr_0p00015,7000,0.951335608959198
-0.00015,lr_0p00015,8000,0.9469588994979858
-0.00015,lr_0p00015,9000,0.9383289813995361
-0.00015,lr_0p00015,10000,0.9309172630310059
-0.00015,lr_0p00015,11000,0.9210461974143982
-0.00015,lr_0p00015,12000,0.9106600284576416
-0.00015,lr_0p00015,13000,0.9036438465118408
-0.00015,lr_0p00015,14000,0.8952069878578186
-0.00015,lr_0p00015,15000,0.8909939527511597
-0.00015,lr_0p00015,16000,0.8871906995773315
-0.00015,lr_0p00015,17000,0.8851967453956604
-0.00015,lr_0p00015,18000,0.8842737674713135
-0.00015,lr_0p00015,18688,0.8842697143554688
-0.00018,lr_0p00018,1000,0.9958739280700684
-0.00018,lr_0p00018,2000,0.9904810786247253
-0.00018,lr_0p00018,3000,0.9867064356803894
-0.00018,lr_0p00018,4000,0.9833324551582336
-0.00018,lr_0p00018,5000,0.9868490099906921
-0.00018,lr_0p00018,6000,0.9755926728248596
-0.00018,lr_0p00018,7000,0.9768977761268616
-0.00018,lr_0p00018,8000,0.9687609076499939
-0.00018,lr_0p00018,9000,0.9562658071517944
-0.00018,lr_0p00018,10000,0.9435049891471863
-0.00018,lr_0p00018,11000,0.9342814683914185
-0.00018,lr_0p00018,12000,0.922635018825531
-0.00018,lr_0p00018,13000,0.9130367636680603
-0.00018,lr_0p00018,14000,0.904951810836792
-0.00018,lr_0p00018,15000,0.8994976878166199
-0.00018,lr_0p00018,16000,0.8953551650047302
-0.00018,lr_0p00018,17000,0.8925029039382935
-0.00018,lr_0p00018,18000,0.891387939453125
-0.00018,lr_0p00018,18688,0.8914602994918823
-0.0002,lr_0p00020,1000,0.9974449276924133
-0.0002,lr_0p00020,2000,0.9973769783973694
-0.0002,lr_0p00020,3000,1.0056378841400146
-0.0002,lr_0p00020,4000,0.9973607063293457
-0.0002,lr_0p00020,5000,1.0010013580322266
-0.0002,lr_0p00020,6000,1.000449776649475
-0.0002,lr_0p00020,7000,0.9865806698799133
-0.0002,lr_0p00020,8000,0.9824715852737427
-0.0002,lr_0p00020,9000,0.9666008353233337
-0.0002,lr_0p00020,10000,0.9573416709899902
-0.0002,lr_0p00020,11000,0.9432744383811951
-0.0002,lr_0p00020,12000,0.9326162338256836
-0.0002,lr_0p00020,13000,0.9201604127883911
-0.0002,lr_0p00020,14000,0.9126585721969604
-0.0002,lr_0p00020,15000,0.907073438167572
-0.0002,lr_0p00020,16000,0.9014313817024231
-0.0002,lr_0p00020,17000,0.8986518979072571
-0.0002,lr_0p00020,18000,0.897506594657898
-0.0002,lr_0p00020,18688,0.8976690173149109
-0.00023,lr_0p00023,1000,1.0002912282943726
-0.00023,lr_0p00023,2000,1.0145149230957031
-0.00023,lr_0p00023,3000,1.0228914022445679
-0.00023,lr_0p00023,4000,1.0323290824890137
-0.00023,lr_0p00023,5000,1.0345656871795654
-0.00023,lr_0p00023,6000,1.0246530771255493
-0.00023,lr_0p00023,7000,1.0160919427871704
-0.00023,lr_0p00023,8000,1.0105783939361572
-0.00023,lr_0p00023,9000,0.9934014678001404
-0.00023,lr_0p00023,10000,0.980603039264679
-0.00023,lr_0p00023,11000,0.963624119758606
-0.00023,lr_0p00023,12000,0.951417088508606
-0.00023,lr_0p00023,13000,0.9340505599975586
-0.00023,lr_0p00023,14000,0.9227956533432007
-0.00023,lr_0p00023,15000,0.9158667325973511
-0.00023,lr_0p00023,16000,0.908878743648529
-0.00023,lr_0p00023,17000,0.9046502709388733
-0.00023,lr_0p00023,18000,0.9031931161880493
-0.00023,lr_0p00023,18688,0.9032362699508667

aya_300k_lr_experiments/scripts/analyze_aya_300k_lr.py

@@ -1,338 +0,0 @@
-from pathlib import Path
-import csv
-import json
-import math
-
-
-ROOT = Path(
-    "/home/schwarc/diplomovka/"
-    "aya_300k_lr_validation"
-)
-
-RESULT_DIR = ROOT / "results"
-RESULT_DIR.mkdir(
-    parents=True,
-    exist_ok=True
-)
-
-
-RUNS = [
-    ("0.00018", ROOT / "lr_0p00018"),
-    ("0.00020", ROOT / "lr_0p00020"),
-    ("0.00023", ROOT / "lr_0p00023"),
-]
-
-
-def load_best_state(output):
-
-    files = list(
-        output.rglob(
-            "trainer_state.json"
-        )
-    )
-
-    if not files:
-        return None
-
-    candidates = []
-
-    for p in files:
-        try:
-            data = json.loads(
-                p.read_text(
-                    encoding="utf-8"
-                )
-            )
-
-            candidates.append(
-                (
-                    int(
-                        data.get(
-                            "global_step",
-                            0
-                        )
-                    ),
-                    data,
-                    p,
-                )
-            )
-
-        except Exception:
-            pass
-
-    if not candidates:
-        return None
-
-    return max(
-        candidates,
-        key=lambda x: x[0]
-    )
-
-
-rows = []
-eval_curves = {}
-grad_curves = {}
-
-
-for lr, output in RUNS:
-
-    result = load_best_state(
-        output
-    )
-
-    if result is None:
-        continue
-
-    global_step, state, state_file = result
-
-    history = state.get(
-        "log_history",
-        []
-    )
-
-    evals = []
-    grads = []
-
-    for item in history:
-
-        step = int(
-            item.get("step", 0)
-        )
-
-        if "eval_loss" in item:
-            try:
-                value = float(
-                    item["eval_loss"]
-                )
-                if math.isfinite(value):
-                    evals.append(
-                        (step, value)
-                    )
-            except Exception:
-                pass
-
-        if "grad_norm" in item:
-            try:
-                value = float(
-                    item["grad_norm"]
-                )
-                if math.isfinite(value):
-                    grads.append(
-                        (step, value)
-                    )
-            except Exception:
-                pass
-
-
-    if evals:
-        final_eval_step, final_eval = (
-            evals[-1]
-        )
-
-        min_eval_step, min_eval = min(
-            evals,
-            key=lambda x: x[1]
-        )
-    else:
-        final_eval_step = ""
-        final_eval = ""
-        min_eval_step = ""
-        min_eval = ""
-
-
-    grad_values = [
-        value
-        for _, value in grads
-    ]
-
-    if grad_values:
-
-        max_grad = max(
-            grad_values
-        )
-
-        ordered = sorted(
-            grad_values
-        )
-
-        p95_index = int(
-            0.95
-            * (len(ordered) - 1)
-        )
-
-        p95_grad = (
-            ordered[p95_index]
-        )
-
-    else:
-        max_grad = ""
-        p95_grad = ""
-
-
-    rows.append(
-        {
-            "learning_rate": lr,
-            "global_step": global_step,
-            "final_eval_step":
-                final_eval_step,
-            "final_eval_loss":
-                final_eval,
-            "min_eval_step":
-                min_eval_step,
-            "min_eval_loss":
-                min_eval,
-            "max_grad_norm":
-                max_grad,
-            "p95_grad_norm":
-                p95_grad,
-            "trainer_state":
-                str(state_file),
-        }
-    )
-
-    eval_curves[lr] = evals
-    grad_curves[lr] = grads
-
-
-csv_path = (
-    RESULT_DIR
-    / "aya_300k_lr_summary.csv"
-)
-
-with csv_path.open(
-    "w",
-    encoding="utf-8",
-    newline=""
-) as f:
-
-    fields = [
-        "learning_rate",
-        "global_step",
-        "final_eval_step",
-        "final_eval_loss",
-        "min_eval_step",
-        "min_eval_loss",
-        "max_grad_norm",
-        "p95_grad_norm",
-        "trainer_state",
-    ]
-
-    writer = csv.DictWriter(
-        f,
-        fieldnames=fields
-    )
-
-    writer.writeheader()
-    writer.writerows(rows)
-
-
-print()
-print("===== 300K LR SUMMARY =====")
-
-for row in rows:
-    print(
-        f"LR={row['learning_rate']} | "
-        f"step={row['global_step']} | "
-        f"final_eval={row['final_eval_loss']} | "
-        f"min_eval={row['min_eval_loss']} "
-        f"@ {row['min_eval_step']} | "
-        f"max_grad={row['max_grad_norm']} | "
-        f"p95_grad={row['p95_grad_norm']}"
-    )
-
-print()
-print("CSV:", csv_path)
-
-
-try:
-    import matplotlib.pyplot as plt
-
-    if eval_curves:
-
-        plt.figure(
-            figsize=(12, 7)
-        )
-
-        for lr, points in (
-            eval_curves.items()
-        ):
-            if not points:
-                continue
-
-            plt.plot(
-                [x for x, _ in points],
-                [y for _, y in points],
-                marker="o",
-                label=f"LR {lr}",
-            )
-
-        plt.xlabel(
-            "optimizer step"
-        )
-        plt.ylabel(
-            "eval loss"
-        )
-        plt.title(
-            "Aya 300k - eval loss"
-        )
-        plt.legend()
-        plt.grid(alpha=0.25)
-        plt.tight_layout()
-
-        plt.savefig(
-            RESULT_DIR
-            / "eval_loss_over_steps.png",
-            dpi=200,
-            bbox_inches="tight"
-        )
-
-        plt.close()
-
-
-    if grad_curves:
-
-        plt.figure(
-            figsize=(12, 7)
-        )
-
-        for lr, points in (
-            grad_curves.items()
-        ):
-            if not points:
-                continue
-
-            plt.plot(
-                [x for x, _ in points],
-                [y for _, y in points],
-                label=f"LR {lr}",
-            )
-
-        plt.xlabel(
-            "optimizer step"
-        )
-        plt.ylabel(
-            "grad norm"
-        )
-        plt.title(
-            "Aya 300k - gradient norm"
-        )
-        plt.yscale("log")
-        plt.legend()
-        plt.grid(alpha=0.25)
-        plt.tight_layout()
-
-        plt.savefig(
-            RESULT_DIR
-            / "grad_norm_over_steps.png",
-            dpi=200,
-            bbox_inches="tight"
-        )
-
-        plt.close()
-
-
-except Exception as exc:
-    print(
-        "Grafy sa nepodarilo vytvorit:",
-        exc
-    )

aya_300k_lr_experiments/scripts/analyze_aya_300k_lr_all.py

@@ -1,341 +0,0 @@
-from pathlib import Path
-import csv
-import json
-import math
-
-
-ROOT = Path(
-    "/home/schwarc/diplomovka/"
-    "aya_300k_lr_validation"
-)
-
-RESULT_DIR = ROOT / "results"
-RESULT_DIR.mkdir(
-    parents=True,
-    exist_ok=True
-)
-
-
-RUNS = [
-    ("0.00008", ROOT / "lr_0p00008"),
-    ("0.00010", ROOT / "lr_0p00010"),
-    ("0.00015", ROOT / "lr_0p00015"),
-    ("0.00018", ROOT / "lr_0p00018"),
-    ("0.00020", ROOT / "lr_0p00020"),
-    ("0.00023", ROOT / "lr_0p00023"),
-]
-
-
-def load_best_state(output):
-
-    files = list(
-        output.rglob(
-            "trainer_state.json"
-        )
-    )
-
-    if not files:
-        return None
-
-    candidates = []
-
-    for p in files:
-        try:
-            data = json.loads(
-                p.read_text(
-                    encoding="utf-8"
-                )
-            )
-
-            candidates.append(
-                (
-                    int(
-                        data.get(
-                            "global_step",
-                            0
-                        )
-                    ),
-                    data,
-                    p,
-                )
-            )
-
-        except Exception:
-            pass
-
-    if not candidates:
-        return None
-
-    return max(
-        candidates,
-        key=lambda x: x[0]
-    )
-
-
-rows = []
-eval_curves = {}
-grad_curves = {}
-
-
-for lr, output in RUNS:
-
-    result = load_best_state(
-        output
-    )
-
-    if result is None:
-        continue
-
-    global_step, state, state_file = result
-
-    history = state.get(
-        "log_history",
-        []
-    )
-
-    evals = []
-    grads = []
-
-    for item in history:
-
-        step = int(
-            item.get("step", 0)
-        )
-
-        if "eval_loss" in item:
-            try:
-                value = float(
-                    item["eval_loss"]
-                )
-                if math.isfinite(value):
-                    evals.append(
-                        (step, value)
-                    )
-            except Exception:
-                pass
-
-        if "grad_norm" in item:
-            try:
-                value = float(
-                    item["grad_norm"]
-                )
-                if math.isfinite(value):
-                    grads.append(
-                        (step, value)
-                    )
-            except Exception:
-                pass
-
-
-    if evals:
-        final_eval_step, final_eval = (
-            evals[-1]
-        )
-
-        min_eval_step, min_eval = min(
-            evals,
-            key=lambda x: x[1]
-        )
-    else:
-        final_eval_step = ""
-        final_eval = ""
-        min_eval_step = ""
-        min_eval = ""
-
-
-    grad_values = [
-        value
-        for _, value in grads
-    ]
-
-    if grad_values:
-
-        max_grad = max(
-            grad_values
-        )
-
-        ordered = sorted(
-            grad_values
-        )
-
-        p95_index = int(
-            0.95
-            * (len(ordered) - 1)
-        )
-
-        p95_grad = (
-            ordered[p95_index]
-        )
-
-    else:
-        max_grad = ""
-        p95_grad = ""
-
-
-    rows.append(
-        {
-            "learning_rate": lr,
-            "global_step": global_step,
-            "final_eval_step":
-                final_eval_step,
-            "final_eval_loss":
-                final_eval,
-            "min_eval_step":
-                min_eval_step,
-            "min_eval_loss":
-                min_eval,
-            "max_grad_norm":
-                max_grad,
-            "p95_grad_norm":
-                p95_grad,
-            "trainer_state":
-                str(state_file),
-        }
-    )
-
-    eval_curves[lr] = evals
-    grad_curves[lr] = grads
-
-
-csv_path = (
-    RESULT_DIR
-    / "aya_300k_lr_summary.csv"
-)
-
-with csv_path.open(
-    "w",
-    encoding="utf-8",
-    newline=""
-) as f:
-
-    fields = [
-        "learning_rate",
-        "global_step",
-        "final_eval_step",
-        "final_eval_loss",
-        "min_eval_step",
-        "min_eval_loss",
-        "max_grad_norm",
-        "p95_grad_norm",
-        "trainer_state",
-    ]
-
-    writer = csv.DictWriter(
-        f,
-        fieldnames=fields
-    )
-
-    writer.writeheader()
-    writer.writerows(rows)
-
-
-print()
-print("===== 300K LR SUMMARY =====")
-
-for row in rows:
-    print(
-        f"LR={row['learning_rate']} | "
-        f"step={row['global_step']} | "
-        f"final_eval={row['final_eval_loss']} | "
-        f"min_eval={row['min_eval_loss']} "
-        f"@ {row['min_eval_step']} | "
-        f"max_grad={row['max_grad_norm']} | "
-        f"p95_grad={row['p95_grad_norm']}"
-    )
-
-print()
-print("CSV:", csv_path)
-
-
-try:
-    import matplotlib.pyplot as plt
-
-    if eval_curves:
-
-        plt.figure(
-            figsize=(12, 7)
-        )
-
-        for lr, points in (
-            eval_curves.items()
-        ):
-            if not points:
-                continue
-
-            plt.plot(
-                [x for x, _ in points],
-                [y for _, y in points],
-                marker="o",
-                label=f"LR {lr}",
-            )
-
-        plt.xlabel(
-            "optimizer step"
-        )
-        plt.ylabel(
-            "eval loss"
-        )
-        plt.title(
-            "Aya 300k - eval loss"
-        )
-        plt.legend()
-        plt.grid(alpha=0.25)
-        plt.tight_layout()
-
-        plt.savefig(
-            RESULT_DIR
-            / "eval_loss_over_steps.png",
-            dpi=200,
-            bbox_inches="tight"
-        )
-
-        plt.close()
-
-
-    if grad_curves:
-
-        plt.figure(
-            figsize=(12, 7)
-        )
-
-        for lr, points in (
-            grad_curves.items()
-        ):
-            if not points:
-                continue
-
-            plt.plot(
-                [x for x, _ in points],
-                [y for _, y in points],
-                label=f"LR {lr}",
-            )
-
-        plt.xlabel(
-            "optimizer step"
-        )
-        plt.ylabel(
-            "grad norm"
-        )
-        plt.title(
-            "Aya 300k - gradient norm"
-        )
-        plt.yscale("log")
-        plt.legend()
-        plt.grid(alpha=0.25)
-        plt.tight_layout()
-
-        plt.savefig(
-            RESULT_DIR
-            / "grad_norm_over_steps.png",
-            dpi=200,
-            bbox_inches="tight"
-        )
-
-        plt.close()
-
-
-except Exception as exc:
-    print(
-        "Grafy sa nepodarilo vytvorit:",
-        exc
-    )

aya_300k_lr_experiments/scripts/analyze_training_results.py

@@ -1,119 +0,0 @@
-#!/usr/bin/env python3
-"""Recompute run summaries and the LR comparison from compact curated evidence.
-
-Run with no arguments from any directory. The standard library produces the CSV
-and JSON summaries; matplotlib is required only for the PNG chart. No trainer
-state, checkpoints, model weights, or tokenizers are needed.
-
-The gradient P95 matches the archived summary: sort N observed norms and choose
-the zero-based index floor(0.95 * (N - 1)), without interpolation.
-"""
-import csv
-import json
-import math
-from collections import defaultdict
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-RESULTS = ROOT / "results"
-
-
-def read_rows(filename):
-    with (RESULTS / filename).open(newline="", encoding="utf-8") as handle:
-        return list(csv.DictReader(handle))
-
-
-def summarize():
-    evaluations = defaultdict(list)
-    gradients = defaultdict(list)
-    for row in read_rows("eval_loss_history.csv"):
-        evaluations[row["run"]].append(row)
-    for row in read_rows("gradient_history.csv"):
-        gradients[row["run"]].append(float(row["grad_norm"]))
-    raw_summary = {float(row["learning_rate"]): row for row in read_rows("aya_300k_lr_summary.csv")}
-    summaries = []
-    for run, rows in sorted(evaluations.items()):
-        rows.sort(key=lambda row: int(row["step"]))
-        lr = float(rows[0]["learning_rate"])
-        if any(float(row["learning_rate"]) != lr for row in rows):
-            raise ValueError(f"Inconsistent learning rate in {run}")
-        trial = ROOT / "trials" / run
-        train = json.loads((trial / "train_results.json").read_text(encoding="utf-8"))
-        final_eval = json.loads((trial / "eval_results.json").read_text(encoding="utf-8"))
-        norms = sorted(gradients[run])
-        if not norms or not (trial / "TRAINING_COMPLETE").is_file():
-            raise ValueError(f"Incomplete evidence for {run}")
-        final = rows[-1]
-        best = min(rows, key=lambda row: float(row["eval_loss"]))
-        summary = {
-            "learning_rate": lr, "run": run,
-            "global_step": int(raw_summary[lr]["global_step"]),
-            "epoch": float(train["epoch"]),
-            "final_eval_step": int(final["step"]),
-            "final_eval_loss": float(final["eval_loss"]),
-            "min_eval_step": int(best["step"]),
-            "min_eval_loss": float(best["eval_loss"]),
-            "max_grad_norm": max(norms),
-            "p95_grad_norm": norms[math.floor(0.95 * (len(norms) - 1))],
-            "train_runtime": float(train["train_runtime"]),
-            "eval_observations": len(rows),
-            "gradient_observations": len(norms),
-        }
-        if summary["final_eval_loss"] != float(final_eval["eval_loss"]):
-            raise ValueError(f"Final evaluation disagrees with eval_results.json: {run}")
-        for field in ("global_step", "final_eval_step", "final_eval_loss", "min_eval_step",
-                      "min_eval_loss", "max_grad_norm", "p95_grad_norm"):
-            if float(summary[field]) != float(raw_summary[lr][field]):
-                raise ValueError(f"Archived summary disagrees at {run}: {field}")
-        summaries.append(summary)
-    if len(summaries) != 6 or any(row["global_step"] != 18688 or row["epoch"] != 1.0
-                                  or row["eval_observations"] != 19 or row["gradient_observations"] != 1868
-                                  for row in summaries):
-        raise ValueError("Expected six completed one-epoch runs with 19 eval and 1,868 gradient observations each")
-    with (RESULTS / "training_run_summary.csv").open("w", newline="", encoding="utf-8") as handle:
-        writer = csv.DictWriter(handle, fieldnames=list(summaries[0]))
-        writer.writeheader()
-        writer.writerows(summaries)
-    (RESULTS / "training_run_summary.json").write_text(json.dumps(summaries, indent=2) + "\n", encoding="utf-8")
-    return summaries
-
-
-def plot(summaries):
-    try:
-        import matplotlib
-        matplotlib.use("Agg")
-        import matplotlib.pyplot as plt
-    except ImportError as exc:
-        raise SystemExit("Summaries recomputed. Install matplotlib to regenerate the PNG chart.") from exc
-    plt.rcParams.update({"font.family": "DejaVu Sans", "font.size": 11})
-    fig, ax = plt.subplots(figsize=(8, 4.8), layout="constrained")
-    x = [row["learning_rate"] * 1e4 for row in summaries]
-    y = [row["final_eval_loss"] for row in summaries]
-    ax.plot(x, y, color="#335c81", marker="o", linewidth=1.6, markersize=6)
-    selected = min(summaries, key=lambda row: row["final_eval_loss"])
-    sx, sy = selected["learning_rate"] * 1e4, selected["final_eval_loss"]
-    ax.scatter([sx], [sy], color="#ba4a00", s=75, zorder=3)
-    for xi, yi in zip(x, y):
-        ax.annotate(f"{yi:.5f}", (xi, yi), xytext=(0, 10), textcoords="offset points", ha="center", fontsize=9)
-    ax.annotate("Selected LR = 0.00010", (sx, sy), xytext=(20, -24),
-                textcoords="offset points", color="#8a3700", fontsize=10)
-    ax.set(title="Aya 300k: final validation loss by learning rate",
-           xlabel="Learning rate (×10⁻⁴)", ylabel="Final validation loss (lower is better)")
-    ax.set_xticks(x, [f"{value:.1f}" for value in x])
-    ax.set_ylim(min(y) - 0.009, max(y) + 0.012)
-    ax.grid(axis="y", color="#d8dee5", linewidth=0.7)
-    ax.spines[["top", "right"]].set_visible(False)
-    fig.savefig(RESULTS / "lr_vs_final_eval_loss.png", dpi=200, metadata={"Description": "Six manually chosen learning rates; final evaluation at step 18688 after one epoch."})
-    plt.close(fig)
-
-
-def main():
-    summaries = summarize()
-    plot(summaries)
-    best = min(summaries, key=lambda row: row["final_eval_loss"])
-    print(f"Verified {len(summaries)} runs. Selected {best['run']}: final eval loss {best['final_eval_loss']:.15g}")
-    print(f"Saved summaries and chart in {RESULTS}")
-
-
-if __name__ == "__main__":
-    main()

aya_300k_lr_experiments/scripts/rerun_skbench_lr_0p00010.sh

@@ -1,37 +0,0 @@
-#!/usr/bin/env bash
-# Reconstructed from the completed run's saved parameters and frozen task list.
-set -euo pipefail
-
-PROJECT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)
-SKBENCH_DIR=${SKBENCH_DIR:-/home/schwarc/diplomovka/sk-bench}
-MODEL_DIR=${MODEL_DIR:-/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010}
-OUTPUT_ROOT=${OUTPUT_ROOT:-/home/schwarc/diplomovka/skbench_evaluation/results}
-GPU_ID=${GPU_ID:-1}
-TASKS="$PROJECT/benchmarks/skbench/configs/tasks.txt"
-OMITTED="$PROJECT/benchmarks/skbench/configs/omitted_tasks.txt"
-
-test -f "$MODEL_DIR/config.json"
-test -f "$TASKS"
-test -f "$OMITTED"
-cd "$SKBENCH_DIR"
-
-RUN="$OUTPUT_ROOT/sk_zero_available_without_nli_lr_0p00010_$(date +%Y%m%d_%H%M%S)"
-mkdir -p "$RUN"
-cp "$TASKS" "$RUN/tasks.txt"
-cp "$OMITTED" "$RUN/omitted_tasks.txt"
-cp uv.lock "$RUN/uv.lock"
-git rev-parse HEAD > "$RUN/skbench_commit.txt"
-
-CUDA_VISIBLE_DEVICES="$GPU_ID" VLLM_USE_FLASHINFER_SAMPLER=0 \
-uv run --locked python eval_vllm.py \
-  --model-name "$MODEL_DIR" \
-  --tasks-file "$RUN/tasks.txt" \
-  --output-dir "$RUN" \
-  --tensor-parallel-size 1 \
-  --dtype bfloat16 \
-  --gpu-memory-utilization 0.85 \
-  --max-model-length 32768 \
-  --max-batched-tokens 2048 \
-  --enforce-eager \
-  --save-details \
-  2>&1 | tee "$RUN/run.log"

aya_300k_lr_experiments/scripts/run_aya_300k_lr_validation.sh

@@ -1,76 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-source /home/schwarc/miniconda3/etc/profile.d/conda.sh
-conda activate llama_factory_unsloth
-
-ROOT="/home/schwarc/diplomovka"
-EXP="$ROOT/aya_300k_lr_validation"
-
-cd /home/schwarc/LLaMA-Factory
-
-export CUDA_VISIBLE_DEVICES=0
-unset FORCE_TORCHRUN || true
-export PYTHONUNBUFFERED=1
-
-
-RUNS=(
-    "lr_0p00018"
-    "lr_0p00020"
-    "lr_0p00023"
-)
-
-
-for NAME in "${RUNS[@]}"; do
-
-    CFG="$EXP/configs/$NAME.yaml"
-    OUT="$EXP/$NAME"
-    LOG="$EXP/logs/$NAME.log"
-    DONE="$OUT/TRAINING_COMPLETE"
-
-    echo
-    echo "========================================"
-    echo "RUN: $NAME"
-    echo "========================================"
-
-    if [ -f "$DONE" ]; then
-        echo "Uz dokoncene - preskakujem."
-        continue
-    fi
-
-    # Ak existuje checkpoint po predoslom pade servera,
-    # radsej automaticky nic neprepiseme.
-    if [ -d "$OUT" ] && \
-       find "$OUT" -maxdepth 1 -type d \
-       -name 'checkpoint-*' | grep -q .; then
-
-        echo
-        echo "POZOR:"
-        echo "Nasiel som nedokonceny run s checkpointom:"
-        echo "$OUT"
-        echo
-        echo "Nebudem ho automaticky prepisovat."
-        echo "Treba ho najprv resume-nut."
-        exit 2
-    fi
-
-
-    llamafactory-cli train "$CFG" \
-        2>&1 | tee "$LOG"
-
-    mkdir -p "$OUT"
-    touch "$DONE"
-
-    python \
-        "$ROOT/scripts/analyze_aya_300k_lr.py"
-
-done
-
-
-echo
-echo "========================================"
-echo "VSETKY 300K LR BEHY HOTOVE"
-echo "========================================"
-
-python \
-    "$ROOT/scripts/analyze_aya_300k_lr.py"

aya_300k_lr_experiments/scripts/run_aya_300k_lr_validation_lower.sh

@@ -1,76 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-source /home/schwarc/miniconda3/etc/profile.d/conda.sh
-conda activate llama_factory_unsloth
-
-ROOT="/home/schwarc/diplomovka"
-EXP="$ROOT/aya_300k_lr_validation"
-
-cd /home/schwarc/LLaMA-Factory
-
-export CUDA_VISIBLE_DEVICES=0
-unset FORCE_TORCHRUN || true
-export PYTHONUNBUFFERED=1
-
-
-RUNS=(
-    "lr_0p00015"
-    "lr_0p00010"
-    "lr_0p00008"
-)
-
-
-for NAME in "${RUNS[@]}"; do
-
-    CFG="$EXP/configs/$NAME.yaml"
-    OUT="$EXP/$NAME"
-    LOG="$EXP/logs/$NAME.log"
-    DONE="$OUT/TRAINING_COMPLETE"
-
-    echo
-    echo "========================================"
-    echo "RUN: $NAME"
-    echo "========================================"
-
-    if [ -f "$DONE" ]; then
-        echo "Uz dokoncene - preskakujem."
-        continue
-    fi
-
-    # Ak existuje checkpoint po predoslom pade servera,
-    # radsej automaticky nic neprepiseme.
-    if [ -d "$OUT" ] && \
-       find "$OUT" -maxdepth 1 -type d \
-       -name 'checkpoint-*' | grep -q .; then
-
-        echo
-        echo "POZOR:"
-        echo "Nasiel som nedokonceny run s checkpointom:"
-        echo "$OUT"
-        echo
-        echo "Nebudem ho automaticky prepisovat."
-        echo "Treba ho najprv resume-nut."
-        exit 2
-    fi
-
-
-    llamafactory-cli train "$CFG" \
-        2>&1 | tee "$LOG"
-
-    mkdir -p "$OUT"
-    touch "$DONE"
-
-    python \
-        "$ROOT/scripts/analyze_aya_300k_lr_all.py"
-
-done
-
-
-echo
-echo "========================================"
-echo "VSETKY 300K LR BEHY HOTOVE"
-echo "========================================"
-
-python \
-    "$ROOT/scripts/analyze_aya_300k_lr_all.py"

aya_300k_lr_experiments/scripts/run_lmeval_aya_300k_lr_0p00010.sh

@@ -1,74 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-eval "$(/home/schwarc/miniconda3/bin/conda shell.bash hook)"
-conda activate lm_eval_hruska
-cd /home/schwarc/lm-evaluation-harness
-
-# This directory already contains the base model merged with the Aya adapter.
-MODEL="/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010"
-TASKS="arc_sk,hellaswag_sk,m_mmlu_sk,truthfulqa_sk_mc1,truthfulqa_sk_mc2,sklegal,skquad"
-INCLUDE_PATH="/home/schwarc/diplomovka/lmeval_slovak_tasks"
-ROOT="/home/schwarc/diplomovka/aya_300k_lr_validation/lmeval"
-NAME="lr_0p00010_$(date +%Y%m%d_%H%M%S)"
-OUT="$ROOT/results/$NAME"
-LOG="$ROOT/logs/$NAME.log"
-SUMMARY="$ROOT/summary.csv"
-
-test -f "$MODEL/config.json"
-test -f "$MODEL/model.safetensors.index.json"
-test -d "$INCLUDE_PATH"
-mkdir -p "$OUT" "$ROOT/logs"
-cp -a "$INCLUDE_PATH" "$OUT/custom_tasks"
-git rev-parse HEAD > "$OUT/lm_eval_commit.txt"
-printf '%s\n' "$TASKS" > "$OUT/tasks.txt"
-printf '%s\n' "model=hf" "pretrained=$MODEL" "tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4" "dtype=bfloat16" \
-  "num_fewshot=0" "batch_size=1" "apply_chat_template=false" \
-  "CUDA_VISIBLE_DEVICES=1" > "$OUT/run_settings.txt"
-
-python - <<'PY' > "$OUT/package_versions.json"
-import importlib.metadata as md
-import json
-
-versions = {}
-for package in ("lm_eval", "transformers", "torch", "datasets", "peft", "evaluate"):
-    try:
-        versions[package] = md.version(package)
-    except md.PackageNotFoundError:
-        versions[package] = None
-print(json.dumps(versions, indent=2))
-PY
-
-if [[ ! -f "$SUMMARY" ]]; then
-    printf '%s\n' 'model,exit_code,start_time,end_time,log_file' > "$SUMMARY"
-fi
-
-START_TIME=$(date '+%Y-%m-%d %H:%M:%S')
-printf 'Model: %s\nResults: %s\nLog: %s\n' "$MODEL" "$OUT" "$LOG"
-
-set +e
-CUDA_VISIBLE_DEVICES=1 lm_eval \
-  --model hf \
-  --model_args "pretrained=$MODEL,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16" \
-  --tasks "$TASKS" \
-  --include_path "$INCLUDE_PATH" \
-  --device cuda:0 \
-  --batch_size 1 \
-  --num_fewshot 0 \
-  --output_path "$OUT" \
-  2>&1 | tee "$LOG"
-EXIT_CODES=("${PIPESTATUS[@]}")
-set -e
-EXIT_CODE=${EXIT_CODES[0]}
-if (( EXIT_CODE == 0 )); then
-    EXIT_CODE=${EXIT_CODES[1]}
-fi
-
-END_TIME=$(date '+%Y-%m-%d %H:%M:%S')
-printf '%s,%s,%s,%s,%s\n' "$NAME" "$EXIT_CODE" "$START_TIME" "$END_TIME" "$LOG" >> "$SUMMARY"
-if (( EXIT_CODE == 0 )); then
-    printf 'Evaluation completed: %s\n' "$OUT"
-else
-    printf 'Evaluation failed (exit code %s). Log: %s\n' "$EXIT_CODE" "$LOG" >&2
-fi
-exit "$EXIT_CODE"

aya_300k_lr_experiments/scripts/setup_aya_300k_lr_validation.sh

@@ -1,899 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-ROOT="/home/schwarc/diplomovka"
-LF="/home/schwarc/LLaMA-Factory"
-
-EXP="$ROOT/aya_300k_lr_validation"
-CONFIG_DIR="$EXP/configs"
-LOG_DIR="$EXP/logs"
-RESULT_DIR="$EXP/results"
-
-DATA_FILE="$LF/data/aya_slovak_300k_fixed.jsonl"
-DATASET_INFO="$LF/data/dataset_info.json"
-
-BASE_CONFIG="$ROOT/optuna_lr_hpo_1gpu/base_hpo_config.yaml"
-
-mkdir -p "$EXP" "$CONFIG_DIR" "$LOG_DIR" "$RESULT_DIR" "$ROOT/scripts"
-
-source /home/schwarc/miniconda3/etc/profile.d/conda.sh
-conda activate llama_factory_unsloth
-
-
-echo
-echo "========================================"
-echo "1. KONTROLA PROSTREDIA"
-echo "========================================"
-
-python - <<'PY'
-import torch
-import datasets
-import yaml
-
-print("Torch:", torch.__version__)
-print("CUDA:", torch.cuda.is_available())
-print("Datasets:", datasets.__version__)
-
-if torch.cuda.is_available():
-    print("GPU 0:", torch.cuda.get_device_name(0))
-PY
-
-
-# ============================================================
-# 2. VYTVORENIE FIXNEHO 300K DATASETU
-# ============================================================
-
-echo
-echo "========================================"
-echo "2. AYA SLOVAK 300K"
-echo "========================================"
-
-python - <<'PY'
-from pathlib import Path
-from datasets import load_dataset
-import json
-
-TARGET = Path(
-    "/home/schwarc/LLaMA-Factory/data/"
-    "aya_slovak_300k_fixed.jsonl"
-)
-
-EXPECTED = 300_000
-SEED = 42
-
-
-def count_lines(path):
-    with path.open("r", encoding="utf-8") as f:
-        return sum(1 for _ in f)
-
-
-if TARGET.exists():
-    count = count_lines(TARGET)
-
-    if count == EXPECTED:
-        print(
-            f"Dataset uz existuje a ma spravne "
-            f"{count} riadkov."
-        )
-        raise SystemExit(0)
-
-    else:
-        raise RuntimeError(
-            f"{TARGET} existuje, ale ma "
-            f"{count} riadkov namiesto {EXPECTED}. "
-            "Nebudem ho automaticky prepisovat."
-        )
-
-
-print("Nacitavam Aya Collection - slovak/train...")
-
-ds = load_dataset(
-    "CohereLabs/aya_collection_language_split",
-    "slovak",
-    split="train",
-)
-
-print("Povodny train split:", len(ds))
-
-
-def valid_batch(batch):
-    result = []
-
-    for inp, target in zip(
-        batch["inputs"],
-        batch["targets"],
-    ):
-        ok = (
-            isinstance(inp, str)
-            and isinstance(target, str)
-            and bool(inp.strip())
-            and bool(target.strip())
-        )
-
-        result.append(ok)
-
-    return result
-
-
-print("Filtrujem prazdne inputs/targets...")
-
-ds = ds.filter(
-    valid_batch,
-    batched=True,
-    batch_size=10000,
-    num_proc=4,
-)
-
-print("Platne zaznamy:", len(ds))
-
-if len(ds) < EXPECTED:
-    raise RuntimeError(
-        f"Po filtrovani ostalo iba {len(ds)} zaznamov."
-    )
-
-
-print("Shuffle seed =", SEED)
-
-ds = ds.shuffle(seed=SEED)
-
-ds = ds.select(
-    range(EXPECTED)
-)
-
-
-print("Zapisujem:", TARGET)
-
-with TARGET.open(
-    "w",
-    encoding="utf-8"
-) as f:
-
-    for row in ds:
-
-        item = {
-            "instruction": row["inputs"].strip(),
-            "input": "",
-            "output": row["targets"].strip(),
-        }
-
-        f.write(
-            json.dumps(
-                item,
-                ensure_ascii=False
-            )
-            + "\n"
-        )
-
-
-count = count_lines(TARGET)
-
-if count != EXPECTED:
-    raise RuntimeError(
-        f"Finalny dataset ma {count} riadkov."
-    )
-
-print("HOTOVO.")
-print("Pocet:", count)
-print("Dataset:", TARGET)
-PY
-
-
-# ============================================================
-# 3. DATASET_INFO.JSON
-# ============================================================
-
-echo
-echo "========================================"
-echo "3. DATASET_INFO"
-echo "========================================"
-
-python - <<'PY'
-from pathlib import Path
-import json
-import copy
-import shutil
-
-p = Path(
-    "/home/schwarc/LLaMA-Factory/data/"
-    "dataset_info.json"
-)
-
-backup = p.parent / (
-    "dataset_info_before_aya300k_backup.json"
-)
-
-if not backup.exists():
-    shutil.copy2(p, backup)
-
-
-with p.open(
-    "r",
-    encoding="utf-8"
-) as f:
-    info = json.load(f)
-
-
-source_name = "aya_slovak_100k_fixed"
-new_name = "aya_slovak_300k_fixed"
-
-if source_name not in info:
-    raise RuntimeError(
-        f"V dataset_info chyba {source_name}"
-    )
-
-
-entry = copy.deepcopy(
-    info[source_name]
-)
-
-entry["file_name"] = (
-    "aya_slovak_300k_fixed.jsonl"
-)
-
-info[new_name] = entry
-
-
-with p.open(
-    "w",
-    encoding="utf-8"
-) as f:
-
-    json.dump(
-        info,
-        f,
-        ensure_ascii=False,
-        indent=2
-    )
-
-
-print(
-    "Registrovany dataset:",
-    new_name
-)
-
-print(
-    json.dumps(
-        info[new_name],
-        ensure_ascii=False,
-        indent=2
-    )
-)
-PY
-
-
-# ============================================================
-# 4. VYTVORENIE 3 CONFIGOV
-# ============================================================
-
-echo
-echo "========================================"
-echo "4. CONFIGY"
-echo "========================================"
-
-python - <<'PY'
-from pathlib import Path
-import yaml
-
-ROOT = Path(
-    "/home/schwarc/diplomovka"
-)
-
-EXP = ROOT / "aya_300k_lr_validation"
-
-BASE = (
-    ROOT
-    / "optuna_lr_hpo_1gpu"
-    / "base_hpo_config.yaml"
-)
-
-if not BASE.exists():
-    BASE = (
-        ROOT
-        / "configs"
-        / "hruska_compare9_100k_fixed"
-        / "10_r16_a32_lr1e4.yaml"
-    )
-
-if not BASE.exists():
-    raise RuntimeError(
-        "Nenasiel som base config."
-    )
-
-
-with BASE.open(
-    "r",
-    encoding="utf-8"
-) as f:
-    base = yaml.safe_load(f)
-
-
-# ------------------------------------------------------------
-# FIXNE PARAMETRE PRE VSETKY 3 BEHY
-# ------------------------------------------------------------
-
-base["dataset"] = "aya_slovak_300k_fixed"
-base["streaming"] = False
-
-# presne 1 cely epoch
-base.pop("max_steps", None)
-base["num_train_epochs"] = 1.0
-
-# 300k -> 1000 validation -> cca 299k train
-base["val_size"] = 1000
-
-# 1 GPU:
-# 1 * 16 * 1 = effective batch 16
-base["per_device_train_batch_size"] = 1
-base["per_device_eval_batch_size"] = 1
-base["gradient_accumulation_steps"] = 16
-
-# cca 18 688 optimizer krokov
-# ~8 % warmup
-base["warmup_steps"] = 1500
-base.pop("warmup_ratio", None)
-
-# chceme sledovat priebeh
-base["eval_strategy"] = "steps"
-base["eval_steps"] = 1000
-base["logging_steps"] = 10
-
-# checkpointy pre pripad restartu servera
-base["save_steps"] = 2000
-base["save_total_limit"] = 2
-
-base["seed"] = 42
-
-base["overwrite_output_dir"] = True
-base["report_to"] = "none"
-
-# explicitne zachovame Unsloth + bf16
-base["use_unsloth"] = True
-base["use_unsloth_gc"] = True
-base["bf16"] = True
-base["fp16"] = False
-
-# chceme rovnaky optimizer/scheduler
-base["optim"] = "adamw_torch"
-base["lr_scheduler_type"] = "cosine"
-
-# rovnaka gradient clipping hranica
-base["max_grad_norm"] = 0.5
-
-
-runs = [
-    ("lr_0p00018", 0.00018),
-    ("lr_0p00020", 0.00020),
-    ("lr_0p00023", 0.00023),
-]
-
-
-for name, lr in runs:
-
-    cfg = dict(base)
-
-    cfg["learning_rate"] = lr
-
-    cfg["output_dir"] = str(
-        EXP / name
-    )
-
-    path = (
-        EXP
-        / "configs"
-        / f"{name}.yaml"
-    )
-
-    with path.open(
-        "w",
-        encoding="utf-8"
-    ) as f:
-
-        yaml.safe_dump(
-            cfg,
-            f,
-            sort_keys=False,
-            allow_unicode=True
-        )
-
-    print()
-    print(path)
-    print("  LR:", lr)
-    print(
-        "  output:",
-        cfg["output_dir"]
-    )
-
-
-print()
-print("Base kontrola:")
-print("dataset:", base["dataset"])
-print("epochs:", base["num_train_epochs"])
-print(
-    "batch:",
-    base["per_device_train_batch_size"]
-)
-print(
-    "grad_accum:",
-    base["gradient_accumulation_steps"]
-)
-print(
-    "warmup:",
-    base["warmup_steps"]
-)
-print(
-    "eval_steps:",
-    base["eval_steps"]
-)
-print(
-    "save_steps:",
-    base["save_steps"]
-)
-print(
-    "LoRA rank:",
-    base.get("lora_rank")
-)
-print(
-    "LoRA alpha:",
-    base.get("lora_alpha")
-)
-print(
-    "LoRA dropout:",
-    base.get("lora_dropout")
-)
-print(
-    "LoRA target:",
-    base.get("lora_target")
-)
-PY
-
-
-# ============================================================
-# 5. ANALYZATOR
-# ============================================================
-
-cat > "$ROOT/scripts/analyze_aya_300k_lr.py" <<'PY'
-from pathlib import Path
-import csv
-import json
-import math
-
-
-ROOT = Path(
-    "/home/schwarc/diplomovka/"
-    "aya_300k_lr_validation"
-)
-
-RESULT_DIR = ROOT / "results"
-RESULT_DIR.mkdir(
-    parents=True,
-    exist_ok=True
-)
-
-
-RUNS = [
-    ("0.00018", ROOT / "lr_0p00018"),
-    ("0.00020", ROOT / "lr_0p00020"),
-    ("0.00023", ROOT / "lr_0p00023"),
-]
-
-
-def load_best_state(output):
-
-    files = list(
-        output.rglob(
-            "trainer_state.json"
-        )
-    )
-
-    if not files:
-        return None
-
-    candidates = []
-
-    for p in files:
-        try:
-            data = json.loads(
-                p.read_text(
-                    encoding="utf-8"
-                )
-            )
-
-            candidates.append(
-                (
-                    int(
-                        data.get(
-                            "global_step",
-                            0
-                        )
-                    ),
-                    data,
-                    p,
-                )
-            )
-
-        except Exception:
-            pass
-
-    if not candidates:
-        return None
-
-    return max(
-        candidates,
-        key=lambda x: x[0]
-    )
-
-
-rows = []
-eval_curves = {}
-grad_curves = {}
-
-
-for lr, output in RUNS:
-
-    result = load_best_state(
-        output
-    )
-
-    if result is None:
-        continue
-
-    global_step, state, state_file = result
-
-    history = state.get(
-        "log_history",
-        []
-    )
-
-    evals = []
-    grads = []
-
-    for item in history:
-
-        step = int(
-            item.get("step", 0)
-        )
-
-        if "eval_loss" in item:
-            try:
-                value = float(
-                    item["eval_loss"]
-                )
-                if math.isfinite(value):
-                    evals.append(
-                        (step, value)
-                    )
-            except Exception:
-                pass
-
-        if "grad_norm" in item:
-            try:
-                value = float(
-                    item["grad_norm"]
-                )
-                if math.isfinite(value):
-                    grads.append(
-                        (step, value)
-                    )
-            except Exception:
-                pass
-
-
-    if evals:
-        final_eval_step, final_eval = (
-            evals[-1]
-        )
-
-        min_eval_step, min_eval = min(
-            evals,
-            key=lambda x: x[1]
-        )
-    else:
-        final_eval_step = ""
-        final_eval = ""
-        min_eval_step = ""
-        min_eval = ""
-
-
-    grad_values = [
-        value
-        for _, value in grads
-    ]
-
-    if grad_values:
-
-        max_grad = max(
-            grad_values
-        )
-
-        ordered = sorted(
-            grad_values
-        )
-
-        p95_index = int(
-            0.95
-            * (len(ordered) - 1)
-        )
-
-        p95_grad = (
-            ordered[p95_index]
-        )
-
-    else:
-        max_grad = ""
-        p95_grad = ""
-
-
-    rows.append(
-        {
-            "learning_rate": lr,
-            "global_step": global_step,
-            "final_eval_step":
-                final_eval_step,
-            "final_eval_loss":
-                final_eval,
-            "min_eval_step":
-                min_eval_step,
-            "min_eval_loss":
-                min_eval,
-            "max_grad_norm":
-                max_grad,
-            "p95_grad_norm":
-                p95_grad,
-            "trainer_state":
-                str(state_file),
-        }
-    )
-
-    eval_curves[lr] = evals
-    grad_curves[lr] = grads
-
-
-csv_path = (
-    RESULT_DIR
-    / "aya_300k_lr_summary.csv"
-)
-
-with csv_path.open(
-    "w",
-    encoding="utf-8",
-    newline=""
-) as f:
-
-    fields = [
-        "learning_rate",
-        "global_step",
-        "final_eval_step",
-        "final_eval_loss",
-        "min_eval_step",
-        "min_eval_loss",
-        "max_grad_norm",
-        "p95_grad_norm",
-        "trainer_state",
-    ]
-
-    writer = csv.DictWriter(
-        f,
-        fieldnames=fields
-    )
-
-    writer.writeheader()
-    writer.writerows(rows)
-
-
-print()
-print("===== 300K LR SUMMARY =====")
-
-for row in rows:
-    print(
-        f"LR={row['learning_rate']} | "
-        f"step={row['global_step']} | "
-        f"final_eval={row['final_eval_loss']} | "
-        f"min_eval={row['min_eval_loss']} "
-        f"@ {row['min_eval_step']} | "
-        f"max_grad={row['max_grad_norm']} | "
-        f"p95_grad={row['p95_grad_norm']}"
-    )
-
-print()
-print("CSV:", csv_path)
-
-
-try:
-    import matplotlib.pyplot as plt
-
-    if eval_curves:
-
-        plt.figure(
-            figsize=(12, 7)
-        )
-
-        for lr, points in (
-            eval_curves.items()
-        ):
-            if not points:
-                continue
-
-            plt.plot(
-                [x for x, _ in points],
-                [y for _, y in points],
-                marker="o",
-                label=f"LR {lr}",
-            )
-
-        plt.xlabel(
-            "optimizer step"
-        )
-        plt.ylabel(
-            "eval loss"
-        )
-        plt.title(
-            "Aya 300k - eval loss"
-        )
-        plt.legend()
-        plt.grid(alpha=0.25)
-        plt.tight_layout()
-
-        plt.savefig(
-            RESULT_DIR
-            / "eval_loss_over_steps.png",
-            dpi=200,
-            bbox_inches="tight"
-        )
-
-        plt.close()
-
-
-    if grad_curves:
-
-        plt.figure(
-            figsize=(12, 7)
-        )
-
-        for lr, points in (
-            grad_curves.items()
-        ):
-            if not points:
-                continue
-
-            plt.plot(
-                [x for x, _ in points],
-                [y for _, y in points],
-                label=f"LR {lr}",
-            )
-
-        plt.xlabel(
-            "optimizer step"
-        )
-        plt.ylabel(
-            "grad norm"
-        )
-        plt.title(
-            "Aya 300k - gradient norm"
-        )
-        plt.yscale("log")
-        plt.legend()
-        plt.grid(alpha=0.25)
-        plt.tight_layout()
-
-        plt.savefig(
-            RESULT_DIR
-            / "grad_norm_over_steps.png",
-            dpi=200,
-            bbox_inches="tight"
-        )
-
-        plt.close()
-
-
-except Exception as exc:
-    print(
-        "Grafy sa nepodarilo vytvorit:",
-        exc
-    )
-PY
-
-
-# ============================================================
-# 6. RUNNER
-# ============================================================
-
-cat > "$ROOT/scripts/run_aya_300k_lr_validation.sh" <<'BASH2'
-#!/usr/bin/env bash
-set -euo pipefail
-
-source /home/schwarc/miniconda3/etc/profile.d/conda.sh
-conda activate llama_factory_unsloth
-
-ROOT="/home/schwarc/diplomovka"
-EXP="$ROOT/aya_300k_lr_validation"
-
-cd /home/schwarc/LLaMA-Factory
-
-export CUDA_VISIBLE_DEVICES=0
-unset FORCE_TORCHRUN || true
-export PYTHONUNBUFFERED=1
-
-
-RUNS=(
-    "lr_0p00018"
-    "lr_0p00020"
-    "lr_0p00023"
-)
-
-
-for NAME in "${RUNS[@]}"; do
-
-    CFG="$EXP/configs/$NAME.yaml"
-    OUT="$EXP/$NAME"
-    LOG="$EXP/logs/$NAME.log"
-    DONE="$OUT/TRAINING_COMPLETE"
-
-    echo
-    echo "========================================"
-    echo "RUN: $NAME"
-    echo "========================================"
-
-    if [ -f "$DONE" ]; then
-        echo "Uz dokoncene - preskakujem."
-        continue
-    fi
-
-    # Ak existuje checkpoint po predoslom pade servera,
-    # radsej automaticky nic neprepiseme.
-    if [ -d "$OUT" ] && \
-       find "$OUT" -maxdepth 1 -type d \
-       -name 'checkpoint-*' | grep -q .; then
-
-        echo
-        echo "POZOR:"
-        echo "Nasiel som nedokonceny run s checkpointom:"
-        echo "$OUT"
-        echo
-        echo "Nebudem ho automaticky prepisovat."
-        echo "Treba ho najprv resume-nut."
-        exit 2
-    fi
-
-
-    llamafactory-cli train "$CFG" \
-        2>&1 | tee "$LOG"
-
-    mkdir -p "$OUT"
-    touch "$DONE"
-
-    python \
-        "$ROOT/scripts/analyze_aya_300k_lr.py"
-
-done
-
-
-echo
-echo "========================================"
-echo "VSETKY 300K LR BEHY HOTOVE"
-echo "========================================"
-
-python \
-    "$ROOT/scripts/analyze_aya_300k_lr.py"
-BASH2
-
-
-chmod +x \
-"$ROOT/scripts/run_aya_300k_lr_validation.sh"
-
-
-echo
-echo "========================================"
-echo "SETUP HOTOVY"
-echo "========================================"
-
-echo
-echo "Dataset:"
-wc -l "$DATA_FILE"
-
-echo
-echo "Configy:"
-ls -1 "$CONFIG_DIR"
-
-echo
-echo "Runner:"
-echo "$ROOT/scripts/run_aya_300k_lr_validation.sh"

aya_300k_lr_experiments/scripts/unpack_results_archive.py

@@ -1,74 +0,0 @@
-from pathlib import Path, PurePosixPath
-import argparse
-import collections
-import hashlib
-import json
-import re
-import tarfile
-
-parser = argparse.ArgumentParser(description="Safely extract an experiment archive and create audit path manifests (Python 3.11+).")
-parser.add_argument("--archive", type=Path, required=True)
-parser.add_argument("--output", type=Path, required=True)
-args = parser.parse_args()
-SOURCE = args.archive.resolve()
-DEST = args.output.resolve()
-if DEST.exists() and any(DEST.iterdir()):
-    raise SystemExit(f"Output directory must be empty: {DEST}")
-DEST.mkdir(parents=True, exist_ok=True)
-start_size = SOURCE.stat().st_size
-
-def safe_component(value):
-    value = value.replace('%', '%25')
-    value = re.sub(r'[<>:"\\|?*\x00-\x1f]', lambda m: f'%{ord(m.group()):02X}', value)
-    while value.endswith((' ', '.')):
-        value = value[:-1] + f'%{ord(value[-1]):02X}'
-    if value.split('.')[0].upper() in {'CON', 'PRN', 'AUX', 'NUL', *(f'COM{i}' for i in range(1, 10)), *(f'LPT{i}' for i in range(1, 10))}:
-        value = '%5F' + value
-    return value
-
-manifest = []
-seen = set()
-with tarfile.open(SOURCE, 'r:gz') as archive:
-    for member in archive:
-        original = PurePosixPath(member.name)
-        if original.is_absolute() or '..' in original.parts or '\\' in member.name:
-            raise RuntimeError(f'Unsafe archive path: {member.name!r}')
-        relative = Path(*(safe_component(x) for x in original.parts if x not in ('', '.')))
-        target = DEST / relative
-        if not target.resolve().is_relative_to(DEST.resolve()):
-            raise RuntimeError(f'Path escapes extraction root: {member.name!r}')
-        row = {'original': member.name, 'local': relative.as_posix(), 'size': member.size, 'type': 'other'}
-        if member.isdir():
-            target.mkdir(parents=True, exist_ok=True)
-            row['type'] = 'directory'
-        elif member.isfile():
-            key = relative.as_posix().casefold()
-            if key in seen:
-                raise RuntimeError(f'Archive filename collision: {member.name!r}')
-            seen.add(key)
-            target.parent.mkdir(parents=True, exist_ok=True)
-            extracted = archive.extractfile(member)
-            assert extracted is not None
-            with extracted, target.open('wb') as handle:
-                while chunk := extracted.read(1024 * 1024):
-                    handle.write(chunk)
-            if target.stat().st_size != member.size:
-                raise RuntimeError(f'Incomplete extraction: {member.name!r}')
-            row['type'] = 'file'
-        else:
-            row['type'] = 'skipped_link_or_special'
-        manifest.append(row)
-
-if SOURCE.stat().st_size != start_size:
-    raise RuntimeError('Archive changed during extraction')
-with SOURCE.open('rb') as handle:
-    digest = hashlib.file_digest(handle, 'sha256').hexdigest()
-(DEST / 'archive_manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding='utf-8')
-files = [x for x in manifest if x['type'] == 'file']
-summary = {'source': str(SOURCE), 'compressed_bytes': start_size, 'sha256': digest,
-           'files': len(files), 'uncompressed_bytes': sum(x['size'] for x in files),
-           'skipped_special': [x for x in manifest if x['type'] == 'skipped_link_or_special'],
-           'top_level_files': dict(collections.Counter(PurePosixPath(x['original']).parts[0] for x in files)),
-           'largest_files': sorted(files, key=lambda x: x['size'], reverse=True)[:12]}
-(DEST / 'archive_summary.json').write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding='utf-8')
-print(json.dumps(summary, ensure_ascii=False, indent=2))