js886io/Diplomovka
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Aya 300k learning-rate study and benchmark results

Browse Source 
Document six completed LR runs as preparation for full Aya training on Perun. Record selection of LR 0.00010 by lowest final validation loss, selected-model lm-eval and partial SK-bench results, reproducible provenance, and dataset-access/context-truncation limitations. Preserve existing 30k and Titan material.
This commit is contained in:
Jakub Schwarc 2026-10-25 00:33:24 +02:00
parent 29f94a93d9
commit 67061cf373
130 changed files with 163908 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

126
README.md
View File

@ -866,3 +866,129 @@ Presné hodnoty LR, finálne aj minimálne eval loss, kroky, training loss a run
- Na tomto 30k tréningu mali nižšie skúšané LR vyšší finálny eval loss.
- Najlepší skúšaný LR leží na hornej hranici rozsahu.
---
# Hruška – hľadanie learning rate na 300k Aya príkladoch
Po skúšobnom hľadaní LR pomocou Optuny na 30k príkladoch nasledovalo **overenie šiestich ručne zvolených hodnôt LR na väčšej, fixnej 300k vzorke**. Cieľom bolo nájsť vhodné východiskové nastavenie pre plánovaný tréning na celej slovenskej Aya množine na **Perune**. Tento 300k experiment ešte nebol tréningom na celej množine.
Podklady k tréningu a následným benchmarkom vybraného modelu sú v priečinku [aya_300k_lr_experiments/](aya_300k_lr_experiments/). Postup opakovania a technické podmienky sú v [reproduction.md](aya_300k_lr_experiments/reproduction.md).
## Dáta a spoločná konfigurácia
Zo slovenského `train` splitu `CohereLabs/aya_collection_language_split` sa vyfiltrovali prázdne alebo neplatné `inputs`/`targets`, údaje sa premiešali so seed 42 a vybralo sa 300 000 záznamov. Polia sa previedli na Alpaca formát: `inputs` na `instruction`, prázdny `input` a `targets` na `output`.
| Parameter | Hodnota |
|---|---|
| Základný model | `slovak-nlp/mistral-sk-7b` |
| Použitý modelový snapshot | `089497ae0d72018e9591895c54f774bf0b4a83a4` |
| Dataset | fixný `aya_slovak_300k_fixed`, bez streamingu |
| Rozdelenie | 299 000 tréningových + 1 000 validačných príkladov |
| Hardvér | Hruška, 1× RTX A4500, GPU 0 |
| Framework a presnosť | LLaMA-Factory + Unsloth, `bf16` |
| Fine-tuning | LoRA, rank 16, alpha 32, dropout 0.0 |
| LoRA moduly | `q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj` |
| Template / cutoff | `alpaca` / 1024 tokenov |
| Packing / train on prompt | `false` / `false` |
| Batch / gradient accumulation | 1 / 16; effective batch 16 |
| Tréning | 1 epocha, 18 688 optimizer krokov |
| Optimizer / scheduler | `adamw_torch` / cosine |
| Warmup / max grad norm | 1500 krokov / 0.5 |
| Evaluácia / logovanie | každých 1000 / 10 krokov + finálna evaluácia |
| Seed | 42 |
Pôvodné [konfigurácie](aya_300k_lr_experiments/configs/) a [tréningové logy](aya_300k_lr_experiments/logs/) sú zachované. Konfigurácie sa líšia hodnotou LR, výstupnou cestou a prevádzkovým príznakom `overwrite_output_dir`. Archív neobsahuje checksum vstupného datasetu ani identifikátory validačných vzoriek; úplnú identitu dát preto nemožno nezávisle overiť iba z uložených konfigurácií.
## Výsledky a výber LR pre ďalší tréning
Prvá séria obsahovala LR `0.00018`, `0.00020` a `0.00023`. Po jej vyhodnotení sa rozsah rozšíril smerom nadol o `0.00015`, `0.00010` a `0.00008`. Všetkých šesť behov dokončilo jednu epochu.
| Learning rate | Finálny eval loss | Minimálny eval loss | Krok minima |
|---:|---:|---:|---:|
| 0.00008 | 0.875217 | 0.875217 | 18 688 |
| **0.00010** | **0.872620** | **0.872620** | **18 688** |
| 0.00015 | 0.884270 | 0.884270 | 18 688 |
| 0.00018 | 0.891460 | 0.891388 | 18 000 |
| 0.00020 | 0.897669 | 0.897507 | 18 000 |
| 0.00023 | 0.903236 | 0.903193 | 18 000 |
**LR `0.00010` bol vybraný preto, že dosiahol najnižší finálny validačný loss zo všetkých šiestich 300k behov: `0.87261962890625`.** Rozdiel oproti LR `0.00008` je približne 0.002597. Ide o najlepší bodový výsledok v tejto sérii; neprebehli opakovania s viacerými seedmi ani overenie optimálneho LR na celej Aya množine. Pre Perun slúži ako východisko.
Na 30k skúške bol najlepší skúšaný LR `0.0002`. Výsledky 300k série preto ukazujú význam overenia LR na väčšej vzorke. Absolútne hodnoty eval loss medzi 30k, 100k a 300k sériami nie sú priamym porovnaním kvality modelov, keďže nemožno predpokladať zhodné validačné vzorky.
Presné údaje sú v [pôvodnom LR súhrne](aya_300k_lr_experiments/results/aya_300k_lr_summary.csv), [rozšírenom súhrne behov](aya_300k_lr_experiments/results/training_run_summary.csv) a [histórii validačného loss](aya_300k_lr_experiments/results/eval_loss_history.csv).
<p align="center">
<img src="aya_300k_lr_experiments/results/lr_vs_final_eval_loss.png" alt="Finálny eval loss podľa learning rate na 300k Aya príkladoch" width="50%">
</p>
<p align="center">
<img src="aya_300k_lr_experiments/results/eval_loss_over_steps.png" alt="Priebeh validačného loss pre šesť learning rate" width="50%">
</p>
<p align="center">
<img src="aya_300k_lr_experiments/results/grad_norm_over_steps.png" alt="Priebeh zaznamenaného gradient norm pre šesť learning rate" width="50%">
</p>
## Export vybraného modelu a benchmarky
Na následné benchmarky sa vybral model s LR `0.00010`. Jeho LoRA adaptér sa zlúčil s pôvodným základným modelom do samostatného modelu v `bfloat16`. Export nepredstavoval ďalší tréning. [Exportná konfigurácia](aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml) používa rovnaký snapshot základného modelu a template `alpaca`.
Benchmarky skúmajú schopnosti tohto vybraného modelu. **Ostatných päť LR zatiaľ týmito benchmarkmi nebolo porovnaných**, takže z nich nemožno vyhlásiť LR `0.00010` za najlepší LR podľa downstream metrík.
### lm-evaluation-harness
Použilo sa pôvodné prostredie `lm_eval_hruska`, HF backend, `bf16`, batch size 1 a zero-shot nastavenie bez aplikovania chat template. Zlúčený model sa načítal samostatne; tokenizer pochádzal z pôvodného modelového snapshotu. Tento postup obchádza nekompatibilitu exportovaného označenia `TokenizersBackend` so starším Transformers 4.48.2.
Vyhodnotenie dokončilo všetkých sedem pôvodných úloh a skončilo s návratovým kódom 0. Zachované sú [výsledky](aya_300k_lr_experiments/benchmarks/lmeval/results/), [log](aya_300k_lr_experiments/benchmarks/lmeval/logs/), custom úlohy a metadáta prostredia. V nasledujúcej tabuľke sú accuracy metriky prevedené na percentá; SKQuad F1 je už v zdroji v škále 0–100.
| Model | ARC acc_norm | HellaSwag acc_norm | M-MMLU acc | TQA MC1 | TQA MC2 | SKLegal acc_norm | SKQuad F1 |
|---|---:|---:|---:|---:|---:|---:|---:|
| 100k – experiment 01, baseline | 34.05 | 52.09 | 35.50 | 17.99 | 34.39 | 46.85 | 36.68 |
| 100k – experiment 08, attention-only | 35.24 | 52.18 | 34.48 | 18.77 | 36.74 | 47.68 | 43.38 |
| 100k – experiment 10, r16/a32/LR 0.0001 | 32.93 | 51.57 | 34.48 | 20.57 | 34.80 | 46.18 | 37.76 |
| **300k – vybraný LR 0.00010** | 34.73 | 51.64 | 31.03 | 19.67 | 35.07 | 46.10 | 38.73 |
Oproti 100k experimentu 10 dosiahol vybraný 300k model bodovo vyššie ARC acc_norm (+1.80 percentuálneho bodu) a SKQuad F1 (+0.97 bodu), ale nižšie M-MMLU acc (−3.45 percentuálneho bodu). Ostatné zmeny sú menšie. V žiadnej zo siedmich hlavných metrík neprekonal maximum z celej desiatky 100k behov; tieto maximá navyše pochádzajú z rôznych konfigurácií. Výber najnižšieho validačného loss teda neznamená automaticky najlepší výsledok v každej downstream úlohe. Nejde o vyhodnotenie všetkých šiestich 300k LR ani o test štatistickej významnosti rozdielov.
Úplné porovnanie so všetkými desiatimi staršími Hruška experimentmi je v [comparison_with_100k.csv](aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv). Vo všetkých porovnávaných Hruška behoch sa zhodujú konfigurácie a verzie úloh, počty vzoriek, seeds, evaluator commit `6d2abda4`, Transformers 4.48.2 a nastavenie bez chat template. Pole `task_hashes` je však v zdrojových JSON súboroch prázdne, takže nezávislé overenie obsahu benchmarkových dát pomocou hashov nie je možné. Staršie výsledky používali samostatný PEFT adaptér, nový výsledok zlúčený model v BF16; spôsob načítania a zaokrúhlenie pri zlúčení preto nie sú úplne identické. Podrobnosti sú v [audite porovnateľnosti](aya_300k_lr_experiments/benchmarks/lmeval/provenance/comparability_audit.json).
### SK-bench
Dokončil sa **čiastkový slovenský zero-shot beh s 222 položkami**. Položka predstavuje kombináciu úlohy, podúlohy a variantu promptu, nie jeden hodnotený príklad. Zahrnuté boli tri slovenské formulácie promptov dostupných úloh. Použila sa Alpaca šablóna zlúčeného modelu, greedy dekódovanie, seed 1234, `bf16`, jedna GPU a maximálny kontext 32 768 tokenov.
Z pôvodných 412 slovenských položiek sa vynechalo 187 položiek pre 12 nedostupných repozitárov: Chiby, Autoškola, e-testy, IFEval-SK, OneRuler-SK, Scio, Slohry, Slovak school exams, Slovak Trivia, SlovakSum, SME-Sum a slovné analógie. Ich kontrola vracala `RepositoryNotFoundError`; tento stav sám osebe nerozlišuje neexistujúci repozitár od súkromného repozitára bez oprávnenia. Ďalšie tri položky NLI-SK-annotated sa vynechali, pretože prístup vyžadoval schválenie autorkou a pri spúšťaní behu ešte nebol udelený. Ostatné NLI úlohy zo skLEP v behu zostali.
Uložené sú [vykonané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt), [vynechané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt), [report dostupnosti](aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json), úplný výsledný JSON a úspešný log. Audit 222 detailných parquet výstupov overil **159 426 hodnotení** (53 142 pre každý z troch promptov). **Log zároveň potvrdzuje orezanie príliš dlhého vstupu.** Všetky uložené `model_response.truncated_tokens_count` sú síce nulové, ale tieto počítadlá nezachytili orezanie hlásené enginom. Pri MultiWikiQA má 69 vstupov dĺžku presne na hranici použitého rozpočtu (23 pri každom prompte); ide o kandidátov konzistentných s orezaním, nie o spoľahlivo dokázaný počet orezaných príkladov. Výsledky dlhých QA vstupov preto treba interpretovať s týmto obmedzením. Podrobnosti sú v [audite](aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json) a [počítadlách po úlohách](aya_300k_lr_experiments/benchmarks/skbench/results/truncation_audit_per_task.csv).
Výber opisných výsledkov z dostupných úloh. Uvedený priemer je vážený počtom hodnotených príkladov v príslušných variantoch; všetky tri prompty majú v danej úlohe rovnaký počet príkladov. Pri MMLU zahŕňa všetkých 57 predmetov a tri prompty. Toto nie je oficiálna agregácia úplného SK-bench.
| Úloha | Metrika | Počet variantov | Priemer (%) |
|---|---|---:|---:|
| ARC | `em_relaxed` | 3 | 19.08 |
| Belebele | `em_relaxed` | 3 | 21.33 |
| Financial Exams | `em_relaxed` | 3 | 22.94 |
| Global-MMLU-Lite | `em_relaxed` | 3 | 19.83 |
| MMLU (57 predmetov) | `em_relaxed` | 171 | 15.72 |
| SlovakCOPA | `em_relaxed` | 3 | 40.47 |
| SKJ1 | `em_relaxed` | 3 | 4.33 |
| TruthfulQA MC1 | `em_relaxed` | 3 | 1.20 |
| MovieReview | `em_norm_relaxed` | 3 | 40.90 |
| ToxicSK | `em_norm_relaxed` | 3 | 47.60 |
| MultiWikiQA | `f1_slk` | 3 | 50.99 |
| skLEP-QA | `f1_slk` | 3 | 37.50 |
Úplné metriky sú v [per_task_metrics.csv](aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv); rodinné a promptové súhrny vrátane nevážených priemerov sú v [family_summary.csv](aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv) a [family_prompt_summary.csv](aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv).
Výsledky treba čítať spolu s definíciou metriky a promptu. Napríklad pri ARC je priemer presnej metriky `em` približne 2.37 %, zatiaľ čo `em_relaxed` dosahuje 19.08 %. Rozdiel ukazuje citlivosť generatívneho hodnotenia na spôsob zápisu odpovede. Úplný surový JSON zachováva aj agregát `all`, ktorý sa tu neinterpretuje ako celkové skóre: mieša rôzne metriky a pokrýva neúplnú množinu úloh. Samotné MMLU tvorí 171 z 222 položiek, preto jednoduché spriemerovanie všetkých riadkov nie je vyváženým súhrnom schopností modelu.
**Tento výsledok nie je oficiálne celkové skóre úplného SK-bench.** Pri porovnávaní ďalších modelov treba použiť rovnaký zmrazený zoznam úloh a rovnaký protokol. SK-bench používa generovanie a následné porovnávanie textovej odpovede, zatiaľ čo väčšina uvedených lm-eval úloh vyhodnocuje pravdepodobnosti možností. Odlišné sú aj konkrétne datasety, prompty a metriky; hodnoty medzi týmito dvoma frameworkmi preto nie sú priamo zameniteľné.
Pri príprave sa sampler FlashInfer pokúšal kompilovať cez starší systémový CUDA toolkit. Úspešný beh používal `VLLM_USE_FLASHINFER_SAMPLER=0` a `--enforce-eager`. Podrobné verzie a parametre sú súčasťou benchmarkových metadát.
## Uložené podklady
Priečinok [aya_300k_lr_experiments/](aya_300k_lr_experiments/) obsahuje pôvodné konfigurácie, tréningové a benchmarkové logy, výsledné JSON/CSV súbory, grafy, vybrané metadáta a skripty. Veľké váhy, checkpointy, `trainer_state.json`, kompletné per-sample parquet súbory a duplicitné inference cache zostávajú v pôvodnom archíve. Grafy a odvodené tabuľky tréningu možno obnoviť z malých uložených histórií bez prístupu k váham modelov.
Zdrojový archív a jeho SHA-256 sú zaznamenané v [source_archive.json](aya_300k_lr_experiments/provenance/source_archive.json).

1
aya_300k_lr_experiments/.gitattributes vendored Normal file
View File

@ -0,0 +1 @@
*.sh text eol=lf

9
aya_300k_lr_experiments/.gitignore vendored Normal file
View File

@ -0,0 +1,9 @@
__pycache__/
*.pyc
*.safetensors
*.bin
*.pt
*.pth
*.parquet
checkpoint-*/
trainer_state.json

9
aya_300k_lr_experiments/benchmarks/export/chat_template.jinja Normal file
View File

@ -0,0 +1,9 @@
{% set system_message = 'Below is an instruction that describes a task. Write a response that appropriately completes the request.
' %}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in loop_messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '### Instruction:
' + content + '
### Response:
' }}{% elif message['role'] == 'assistant' %}{{ content + '</s>' + '
' }}{% endif %}{% endfor %}

30
aya_300k_lr_experiments/benchmarks/export/config.json Normal file
View File

@ -0,0 +1,30 @@
{
"architectures": [
"MistralForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 1,
"dtype": "bfloat16",
"eos_token_id": 2,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "mistral",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pad_token_id": 2,
"rms_norm_eps": 1e-05,
"rope_parameters": {
"rope_theta": 10000.0,
"rope_type": "default"
},
"sliding_window": 4096,
"tie_word_embeddings": false,
"transformers_version": "5.5.0",
"use_cache": true,
"vocab_size": 32000
}

12
aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml Normal file
View File

@ -0,0 +1,12 @@
model_name_or_path: slovak-nlp/mistral-sk-7b
model_revision: 089497ae0d72018e9591895c54f774bf0b4a83a4
adapter_name_or_path: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010
template: alpaca
finetuning_type: lora
infer_dtype: bfloat16
export_dir: /home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
export_size: 5
export_device: cpu
export_legacy_format: false

6
aya_300k_lr_experiments/benchmarks/export/generation_config.json Normal file
View File

@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"transformers_version": "5.5.0"
}

19
aya_300k_lr_experiments/benchmarks/export/tokenizer_config.json Normal file
View File

@ -0,0 +1,19 @@
{
"add_prefix_space": true,
"backend": "tokenizers",
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"extra_special_tokens": [],
"is_local": false,
"legacy": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "</s>",
"padding_side": "left",
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"split_special_tokens": false,
"tokenizer_class": "TokenizersBackend",
"unk_token": "<unk>",
"use_default_system_prompt": false
}

12
aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv Normal file
View File

@ -0,0 +1,12 @@
model_id,training_samples,model_name,source_result_json,arc_acc_norm,hellaswag_acc_norm,mmlu_acc,truthfulqa_mc1_acc,truthfulqa_mc2_acc,sklegal_acc_norm,skquad_f1
01_baseline_r16_a16_lr5e5_adamw,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_01_baseline_r16_a16_lr5e5_adamw,lmeval/mistral_aya(hruska)/01_baseline_r16_a16_lr5e5_adamw/results_2026-09-30T21-23-33.435952.json,0.34046193327630453,0.5209277807063785,0.3549992344204563,0.17994858611825193,0.34391089322085056,0.46851574212893554,36.6797346829585
02_optimizer_adamw8bit_r16_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_02_optimizer_adamw8bit_r16_a16_lr5e5,lmeval/mistral_aya(hruska)/02_optimizer_adamw8bit_r16_a16_lr5e5/results_2026-10-01T04-23-23.791749.json,0.3378956372968349,0.521244069583553,0.35438677078548464,0.1928020565552699,0.35373417730948353,0.4715142428785607,36.21782844597198
03_rank8_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_03_rank8_a16_lr5e5,lmeval/mistral_aya(hruska)/03_rank8_a16_lr5e5/results_2026-10-01T10-25-51.531765.json,0.33105218135158254,0.5167105956773853,0.34550604807839536,0.18380462724935734,0.3424520772406517,0.4715142428785607,36.36378811056779
04_rank32_a32_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_04_rank32_a32_lr5e5,lmeval/mistral_aya(hruska)/04_rank32_a32_lr5e5/results_2026-10-01T17-22-48.146390.json,0.33532934131736525,0.5200843437005799,0.3449701423977951,0.19023136246786632,0.3367013908863517,0.47301349325337333,36.02282635500946
05_r16_alpha32_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_05_r16_alpha32_lr5e5,lmeval/mistral_aya(hruska)/05_r16_alpha32_lr5e5/results_2026-10-02T00-19-55.725179.json,0.33361847733105215,0.5190300474433316,0.35714285714285715,0.19665809768637532,0.3555125811399826,0.46701649175412296,37.873456685535025
06_lr1e4_r16_a16,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_06_lr1e4_r16_a16,lmeval/mistral_aya(hruska)/06_lr1e4_r16_a16/results_2026-10-02T06-59-24.237777.json,0.33105218135158254,0.5207169214549289,0.3549992344204563,0.19023136246786632,0.34733281230911506,0.46551724137931033,37.57962398015711
07_lr2e5_r16_a16,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_07_lr2e5_r16_a16,lmeval/mistral_aya(hruska)/07_lr2e5_r16_a16/results_2026-10-02T14-27-49.730611.json,0.32763045337895635,0.522509225092251,0.3379268105956209,0.16709511568123395,0.32463255972324384,0.47226386806596704,33.83807391933327
08_attention_only_r16_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_08_attention_only_r16_a16_lr5e5,lmeval/mistral_aya(hruska)/08_attention_only_r16_a16_lr5e5/results_2026-10-02T19-20-02.720498.json,0.35243798118049613,0.5217712177121772,0.3448170264890522,0.18766066838046272,0.3673805917101876,0.4767616191904048,43.37517960476381
09_batch2_accum4_r16_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_09_batch2_accum4_r16_a16_lr5e5,lmeval/mistral_aya(hruska)/09_batch2_accum4_r16_a16_lr5e5/results_2026-10-03T01-58-48.046423.json,0.3378956372968349,0.5209277807063785,0.35140101056499773,0.18637532133676094,0.340221777957301,0.46926536731634183,35.3128443275733
10_r16_a32_lr1e4,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_10_r16_a32_lr1e4,lmeval/mistral_aya(hruska)/10_r16_a32_lr1e4/results_2026-10-03T07-58-30.464881.json,0.32934131736526945,0.515656299420137,0.3448170264890522,0.20565552699228792,0.34795284436431645,0.4617691154422789,37.75612784500817
aya_300k_lr_0p00010,300000,/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json,0.3473053892215569,0.5163943068002108,0.3102893890675241,0.19665809768637532,0.3506542164316134,0.46101949025487254,38.72920807013749
1 model_id training_samples model_name source_result_json arc_acc_norm hellaswag_acc_norm mmlu_acc truthfulqa_mc1_acc truthfulqa_mc2_acc sklegal_acc_norm skquad_f1
2 01_baseline_r16_a16_lr5e5_adamw 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_01_baseline_r16_a16_lr5e5_adamw lmeval/mistral_aya(hruska)/01_baseline_r16_a16_lr5e5_adamw/results_2026-09-30T21-23-33.435952.json 0.34046193327630453 0.5209277807063785 0.3549992344204563 0.17994858611825193 0.34391089322085056 0.46851574212893554 36.6797346829585
3 02_optimizer_adamw8bit_r16_a16_lr5e5 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_02_optimizer_adamw8bit_r16_a16_lr5e5 lmeval/mistral_aya(hruska)/02_optimizer_adamw8bit_r16_a16_lr5e5/results_2026-10-01T04-23-23.791749.json 0.3378956372968349 0.521244069583553 0.35438677078548464 0.1928020565552699 0.35373417730948353 0.4715142428785607 36.21782844597198
4 03_rank8_a16_lr5e5 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_03_rank8_a16_lr5e5 lmeval/mistral_aya(hruska)/03_rank8_a16_lr5e5/results_2026-10-01T10-25-51.531765.json 0.33105218135158254 0.5167105956773853 0.34550604807839536 0.18380462724935734 0.3424520772406517 0.4715142428785607 36.36378811056779
5 04_rank32_a32_lr5e5 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_04_rank32_a32_lr5e5 lmeval/mistral_aya(hruska)/04_rank32_a32_lr5e5/results_2026-10-01T17-22-48.146390.json 0.33532934131736525 0.5200843437005799 0.3449701423977951 0.19023136246786632 0.3367013908863517 0.47301349325337333 36.02282635500946
6 05_r16_alpha32_lr5e5 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_05_r16_alpha32_lr5e5 lmeval/mistral_aya(hruska)/05_r16_alpha32_lr5e5/results_2026-10-02T00-19-55.725179.json 0.33361847733105215 0.5190300474433316 0.35714285714285715 0.19665809768637532 0.3555125811399826 0.46701649175412296 37.873456685535025
7 06_lr1e4_r16_a16 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_06_lr1e4_r16_a16 lmeval/mistral_aya(hruska)/06_lr1e4_r16_a16/results_2026-10-02T06-59-24.237777.json 0.33105218135158254 0.5207169214549289 0.3549992344204563 0.19023136246786632 0.34733281230911506 0.46551724137931033 37.57962398015711
8 07_lr2e5_r16_a16 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_07_lr2e5_r16_a16 lmeval/mistral_aya(hruska)/07_lr2e5_r16_a16/results_2026-10-02T14-27-49.730611.json 0.32763045337895635 0.522509225092251 0.3379268105956209 0.16709511568123395 0.32463255972324384 0.47226386806596704 33.83807391933327
9 08_attention_only_r16_a16_lr5e5 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_08_attention_only_r16_a16_lr5e5 lmeval/mistral_aya(hruska)/08_attention_only_r16_a16_lr5e5/results_2026-10-02T19-20-02.720498.json 0.35243798118049613 0.5217712177121772 0.3448170264890522 0.18766066838046272 0.3673805917101876 0.4767616191904048 43.37517960476381
10 09_batch2_accum4_r16_a16_lr5e5 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_09_batch2_accum4_r16_a16_lr5e5 lmeval/mistral_aya(hruska)/09_batch2_accum4_r16_a16_lr5e5/results_2026-10-03T01-58-48.046423.json 0.3378956372968349 0.5209277807063785 0.35140101056499773 0.18637532133676094 0.340221777957301 0.46926536731634183 35.3128443275733
11 10_r16_a32_lr1e4 100000 /home/schwarc/diplomovka/hruska_compare9_100k_fixed_10_r16_a32_lr1e4 lmeval/mistral_aya(hruska)/10_r16_a32_lr1e4/results_2026-10-03T07-58-30.464881.json 0.32934131736526945 0.515656299420137 0.3448170264890522 0.20565552699228792 0.34795284436431645 0.4617691154422789 37.75612784500817
12 aya_300k_lr_0p00010 300000 /home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010 aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json 0.3473053892215569 0.5163943068002108 0.3102893890675241 0.19665809768637532 0.3506542164316134 0.46101949025487254 38.72920807013749

35
aya_300k_lr_experiments/benchmarks/lmeval/full_metrics.csv Normal file
View File

@ -0,0 +1,35 @@
model_id,training_samples,task,metric,value,unit,status
aya_300k_lr_0p00010,300000,arc_sk,"acc,none",0.30025662959794697,0..1,reported
aya_300k_lr_0p00010,300000,arc_sk,"acc_stderr,none",0.013412022629366669,0..1,reported
aya_300k_lr_0p00010,300000,arc_sk,"acc_norm,none",0.3473053892215569,0..1,reported
aya_300k_lr_0p00010,300000,arc_sk,"acc_norm_stderr,none",0.013931226499492353,0..1,reported
aya_300k_lr_0p00010,300000,hellaswag_sk,"acc,none",0.4083289404322615,0..1,reported
aya_300k_lr_0p00010,300000,hellaswag_sk,"acc_stderr,none",0.005047187883274763,0..1,reported
aya_300k_lr_0p00010,300000,hellaswag_sk,"acc_norm,none",0.5163943068002108,0..1,reported
aya_300k_lr_0p00010,300000,hellaswag_sk,"acc_norm_stderr,none",0.005131456510575215,0..1,reported
aya_300k_lr_0p00010,300000,m_mmlu_sk,"acc,none",0.3102893890675241,0..1,reported
aya_300k_lr_0p00010,300000,m_mmlu_sk,"acc_stderr,none",0.0040478899905589515,0..1,reported
aya_300k_lr_0p00010,300000,sklegal,"acc,none",0.2856071964017991,0..1,reported
aya_300k_lr_0p00010,300000,sklegal,"acc_stderr,none",0.012371946515772182,0..1,reported
aya_300k_lr_0p00010,300000,sklegal,"acc_norm,none",0.46101949025487254,0..1,reported
aya_300k_lr_0p00010,300000,sklegal,"acc_norm_stderr,none",0.013653094581189941,0..1,reported
aya_300k_lr_0p00010,300000,skquad,"exact,none",15.120525931336742,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"exact_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"f1,none",38.72920807013749,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"f1_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"HasAns_exact,none",18.540838569047313,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"HasAns_exact_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"HasAns_f1,none",47.549942420326644,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"HasAns_f1_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"NoAns_exact,none",0.1681614349775785,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"NoAns_exact_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"NoAns_f1,none",0.1681614349775785,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"NoAns_f1_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"best_exact,none",18.75195658979443,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"best_exact_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,skquad,"best_f1,none",38.7292080701372,0..100,reported
aya_300k_lr_0p00010,300000,skquad,"best_f1_stderr,none",N/A,0..100,unavailable
aya_300k_lr_0p00010,300000,truthfulqa_sk_mc1,"acc,none",0.19665809768637532,0..1,reported
aya_300k_lr_0p00010,300000,truthfulqa_sk_mc1,"acc_stderr,none",0.014259215011433178,0..1,reported
aya_300k_lr_0p00010,300000,truthfulqa_sk_mc2,"acc,none",0.3506542164316134,0..1,reported
aya_300k_lr_0p00010,300000,truthfulqa_sk_mc2,"acc_stderr,none",0.014662940445997986,0..1,reported
1 model_id training_samples task metric value unit status
2 aya_300k_lr_0p00010 300000 arc_sk acc,none 0.30025662959794697 0..1 reported
3 aya_300k_lr_0p00010 300000 arc_sk acc_stderr,none 0.013412022629366669 0..1 reported
4 aya_300k_lr_0p00010 300000 arc_sk acc_norm,none 0.3473053892215569 0..1 reported
5 aya_300k_lr_0p00010 300000 arc_sk acc_norm_stderr,none 0.013931226499492353 0..1 reported
6 aya_300k_lr_0p00010 300000 hellaswag_sk acc,none 0.4083289404322615 0..1 reported
7 aya_300k_lr_0p00010 300000 hellaswag_sk acc_stderr,none 0.005047187883274763 0..1 reported
8 aya_300k_lr_0p00010 300000 hellaswag_sk acc_norm,none 0.5163943068002108 0..1 reported
9 aya_300k_lr_0p00010 300000 hellaswag_sk acc_norm_stderr,none 0.005131456510575215 0..1 reported
10 aya_300k_lr_0p00010 300000 m_mmlu_sk acc,none 0.3102893890675241 0..1 reported
11 aya_300k_lr_0p00010 300000 m_mmlu_sk acc_stderr,none 0.0040478899905589515 0..1 reported
12 aya_300k_lr_0p00010 300000 sklegal acc,none 0.2856071964017991 0..1 reported
13 aya_300k_lr_0p00010 300000 sklegal acc_stderr,none 0.012371946515772182 0..1 reported
14 aya_300k_lr_0p00010 300000 sklegal acc_norm,none 0.46101949025487254 0..1 reported
15 aya_300k_lr_0p00010 300000 sklegal acc_norm_stderr,none 0.013653094581189941 0..1 reported
16 aya_300k_lr_0p00010 300000 skquad exact,none 15.120525931336742 0..100 reported
17 aya_300k_lr_0p00010 300000 skquad exact_stderr,none N/A 0..100 unavailable
18 aya_300k_lr_0p00010 300000 skquad f1,none 38.72920807013749 0..100 reported
19 aya_300k_lr_0p00010 300000 skquad f1_stderr,none N/A 0..100 unavailable
20 aya_300k_lr_0p00010 300000 skquad HasAns_exact,none 18.540838569047313 0..100 reported
21 aya_300k_lr_0p00010 300000 skquad HasAns_exact_stderr,none N/A 0..100 unavailable
22 aya_300k_lr_0p00010 300000 skquad HasAns_f1,none 47.549942420326644 0..100 reported
23 aya_300k_lr_0p00010 300000 skquad HasAns_f1_stderr,none N/A 0..100 unavailable
24 aya_300k_lr_0p00010 300000 skquad NoAns_exact,none 0.1681614349775785 0..100 reported
25 aya_300k_lr_0p00010 300000 skquad NoAns_exact_stderr,none N/A 0..100 unavailable
26 aya_300k_lr_0p00010 300000 skquad NoAns_f1,none 0.1681614349775785 0..100 reported
27 aya_300k_lr_0p00010 300000 skquad NoAns_f1_stderr,none N/A 0..100 unavailable
28 aya_300k_lr_0p00010 300000 skquad best_exact,none 18.75195658979443 0..100 reported
29 aya_300k_lr_0p00010 300000 skquad best_exact_stderr,none N/A 0..100 unavailable
30 aya_300k_lr_0p00010 300000 skquad best_f1,none 38.7292080701372 0..100 reported
31 aya_300k_lr_0p00010 300000 skquad best_f1_stderr,none N/A 0..100 unavailable
32 aya_300k_lr_0p00010 300000 truthfulqa_sk_mc1 acc,none 0.19665809768637532 0..1 reported
33 aya_300k_lr_0p00010 300000 truthfulqa_sk_mc1 acc_stderr,none 0.014259215011433178 0..1 reported
34 aya_300k_lr_0p00010 300000 truthfulqa_sk_mc2 acc,none 0.3506542164316134 0..1 reported
35 aya_300k_lr_0p00010 300000 truthfulqa_sk_mc2 acc_stderr,none 0.014662940445997986 0..1 reported

58
aya_300k_lr_experiments/benchmarks/lmeval/logs/lr_0p00010_20261023_224332.log Normal file
View File

File diff suppressed because one or more lines are too long

18
aya_300k_lr_experiments/benchmarks/lmeval/provenance/SHA256SUMS.txt Normal file
View File

@ -0,0 +1,18 @@
499e639455dc6bb9c79228b54b82c66b509f8fe6f2d33513b482b90dbf990674 comparison_with_100k.csv
b587953b0ce136fea20edf50105005c645a2abfad2cd4f6bfa58ee17cf36b34c full_metrics.csv
fd33cc64e660d4d996bd1b6af966e900c72b0c3b5c4bdb4e6915ac0756814e94 logs/lr_0p00010_20261023_224332.log
2e959d2b2052ecc4dc5f571ac674ccaa39119149c58e6ac0da467aa0335b8248 provenance/comparability_audit.json
f491f734663154901f0b5a4bb44aaed69b5d95448415291107fea0c703b055e8 provenance/custom_tasks/arc_sk.yaml
f2feb3ae7dc6e8ae3db09a91747119d7256d1ef33b9a1e417a40cb2a949ce5b9 provenance/custom_tasks/hellaswag_sk.yaml
d2f8c18839d15d365eed6997564e11aab43173eef91470e8a5834a9471772c2a provenance/custom_tasks/m_mmlu_sk.yaml
9443e2d5e398c82f921d8e714ff01693bfce793eef9bf3b0ad581a8bbf137c4a provenance/custom_tasks/sklegal.yaml
759c8cb35d27e50dd4dc4e81b34932d8bc96e36e309ed666858a904624a61131 provenance/custom_tasks/skquad.yaml
de1b8b08f998ebb3dcdd881c3e2aaba89dd53f6ea2a51b0529eadf7936ef2ed3 provenance/custom_tasks/task.py
fa1e7075c4715a4465be66041d36ef3f9b4dc8a7e41898554078294d2510f525 provenance/custom_tasks/task_configs_from_wandb.json
cf922b04a755762060430ef37a18a860017cc9c79e89cf23ac7465fa97244faa provenance/custom_tasks/truthfulqa_sk_mc1.yaml
b9209889fdf0e38915e29b83911213010f6c49d474836f474491da085ab26e01 provenance/custom_tasks/truthfulqa_sk_mc2.yaml
205bd6b1be478ed9cb2a104bf47214cf75debe485fb43c6b78d6584e638283de provenance/lm_eval_commit.txt
b4dcc9d0c2e7b5f5036ddc38a7aad4111934e39d84019b2122ad6561b780b5a4 provenance/package_versions.json
44d8afff7e8d918ef9e53404a8fa17fe19d6dcfaf782519f01be44a2408da898 provenance/run_settings.txt
f6f8d8470b78daf35d80f5a7315cc9f38d867daaead08df39a165cbcf98afd4e provenance/tasks.txt
5774cad7440ab807be44dc5bfa82ac7cbb4c0281204fc397a10c2bbfe7b8f452 results/results_2026-10-24T02-28-32.987157.json

379
aya_300k_lr_experiments/benchmarks/lmeval/provenance/comparability_audit.json Normal file
View File

@ -0,0 +1,379 @@
{
"comparison_scope": "All ten Hruska Aya 100k runs; Titan excluded.",
"units": "Accuracy metrics use 0..1; SKQuad scores use 0..100. N/A remains unavailable.",
"new_result_sha256": "5774cad7440ab807be44dc5bfa82ac7cbb4c0281204fc397a10c2bbfe7b8f452",
"task_hashes_caveat": "task_hashes is empty in all eleven JSONs; equality is not a nonempty content-hash verification.",
"model_loading_difference": "Older runs load base model plus PEFT adapter; 300k loads a merged model with explicit base tokenizer. Model arguments, parameter counts and model SHA therefore differ.",
"new_n_samples": {
"truthfulqa_sk_mc2": {
"original": 778,
"effective": 778
},
"truthfulqa_sk_mc1": {
"original": 778,
"effective": 778
},
"skquad": {
"original": 9583,
"effective": 9583
},
"sklegal": {
"original": 1334,
"effective": 1334
},
"m_mmlu_sk": {
"original": 13062,
"effective": 13062
},
"hellaswag_sk": {
"original": 9485,
"effective": 9485
},
"arc_sk": {
"original": 1169,
"effective": 1169
}
},
"comparisons": [
{
"model_id": "01_baseline_r16_a16_lr5e5_adamw",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_01_baseline_r16_a16_lr5e5_adamw,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "02_optimizer_adamw8bit_r16_a16_lr5e5",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_02_optimizer_adamw8bit_r16_a16_lr5e5,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "03_rank8_a16_lr5e5",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_03_rank8_a16_lr5e5,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7262703616,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "04_rank32_a32_lr5e5",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_04_rank32_a32_lr5e5,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7325618176,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "05_r16_alpha32_lr5e5",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_05_r16_alpha32_lr5e5,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "06_lr1e4_r16_a16",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_06_lr1e4_r16_a16,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "07_lr2e5_r16_a16",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_07_lr2e5_r16_a16,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "08_attention_only_r16_a16_lr5e5",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_08_attention_only_r16_a16_lr5e5,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7255363584,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "09_batch2_accum4_r16_a16_lr5e5",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_09_batch2_accum4_r16_a16_lr5e5,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
},
{
"model_id": "10_r16_a32_lr1e4",
"equal_to_300k": {
"versions": true,
"configs": true,
"task_hashes": true,
"n-samples": true,
"n-shot": true,
"git_hash": true,
"transformers_version": true,
"chat_template": true,
"chat_template_sha": true,
"system_instruction": true,
"fewshot_as_multiturn": true
},
"evaluation_config_differences": {
"model_args": {
"older_100k": "pretrained=slovak-nlp/mistral-sk-7b,peft=/home/schwarc/diplomovka/hruska_compare9_100k_fixed_10_r16_a32_lr1e4,dtype=bfloat16",
"new_300k": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16"
},
"model_num_parameters": {
"older_100k": 7283675136,
"new_300k": 7241732096
},
"model_sha": {
"older_100k": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"new_300k": ""
},
"peft_sha": {
"older_100k": "",
"new_300k": null
}
}
}
]
}

42
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/arc_sk.yaml Normal file
View File

@ -0,0 +1,42 @@
tag:
- arc_multilingual
task: arc_sk
repeats: 1
metadata:
version: 2
test_split: test
description: ''
doc_to_text: query
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
num_fewshot: 0
output_type: multiple_choice
unsafe_code: false
dataset_name: sk
dataset_path: alexandrainst/m_arc
process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
\ def _process_doc(doc):\n # breakpoint()\n out_doc = {\n \
\ \"id\": doc[\"id\"],\n \"query\": \"Question: \" + preprocess(doc[\"\
instruction\"]) + \"\\nAnswer:\",\n \"choices\": [\n preprocess(option)\n\
\ for option in [\n doc[\"option_a\"],\n \
\ doc[\"option_b\"],\n doc[\"option_c\"],\n \
\ doc[\"option_d\"],\n doc[\"option_e\"],\n \
\ ]\n if option\n ],\n \"gold\"\
: [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n }\n \
\ return out_doc\n\n return dataset.map(_process_doc)\n"
doc_to_choice: choices
doc_to_target: gold
training_split: train
target_delimiter: ' '
validation_split: validation
fewshot_delimiter: '
'
should_decontaminate: true
doc_to_decontamination_query: query

35
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/hellaswag_sk.yaml Normal file
View File

@ -0,0 +1,35 @@
tag:
- hellaswag_multilingual
task: hellaswag_sk
repeats: 1
metadata:
version: 1
description: ''
doc_to_text: query
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
num_fewshot: 0
output_type: multiple_choice
unsafe_code: false
dataset_name: sk
dataset_path: alexandrainst/m_hellaswag
process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
\ def _process_doc(doc):\n ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"\
].capitalize()\n out_doc = {\n \"query\": preprocess(doc[\"activity_label\"\
] + \": \" + ctx),\n \"choices\": [preprocess(ending) for ending in doc[\"\
endings\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return\
\ out_doc\n\n return dataset.map(_process_doc)\n"
doc_to_choice: choices
doc_to_target: '{{label.lstrip()}}'
target_delimiter: ' '
validation_split: val
fewshot_delimiter: '
'
should_decontaminate: false

43
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/m_mmlu_sk.yaml Normal file
View File

@ -0,0 +1,43 @@
tag:
- m_mmlu
task: m_mmlu_sk
repeats: 1
metadata:
version: 0
test_split: test
description: ''
doc_to_text: '{{instruction.strip()}}
A. {{option_a}}
B. {{option_b}}
C. {{option_c}}
D. {{option_d}}
Answer:'
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
num_fewshot: 0
output_type: multiple_choice
unsafe_code: false
dataset_name: sk
dataset_path: alexandrainst/m_mmlu
doc_to_choice:
- A
- B
- C
- D
doc_to_target: answer
fewshot_split: train
fewshot_config:
sampler: first_n
target_delimiter: ' '
fewshot_delimiter: '
'
should_decontaminate: false

23
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/sklegal.yaml Normal file
View File

@ -0,0 +1,23 @@
task: sklegal
repeats: 1
metadata:
version: 1
test_split: test
description: ''
doc_to_text: '{{prompt}} '
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
num_fewshot: 0
output_type: multiple_choice
unsafe_code: false
dataset_path: TUKE-KEMT/slovak-legal-exam
doc_to_choice: '{{answers}}'
doc_to_target: label
target_delimiter: ' '
fewshot_delimiter: "\n\n"
should_decontaminate: false

2
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/skquad.yaml Normal file
View File

@ -0,0 +1,2 @@
task: skquad
class: !function task.SQuAD2

245
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task.py Normal file
View File

@ -0,0 +1,245 @@
"""
Know What You Don’t Know: Unanswerable Questions for SQuAD
https://arxiv.org/pdf/1806.03822.pdf
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset,
consisting of questions posed by crowdworkers on a set of Wikipedia articles,
where the answer to every question is a segment of text, or span, from the
corresponding reading passage, or the question might be unanswerable.
SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable
questions written adversarially by crowdworkers to look similar to answerable ones.
To do well on SQuAD2.0, systems must not only answer questions when possible, but
also determine when no answer is supported by the paragraph and abstain from answering.
Homepage: https://rajpurkar.github.io/SQuAD-explorer/
"""
from functools import partial
from math import exp
import datasets
from packaging import version
from lm_eval.api.instance import Instance
from lm_eval.api.task import ConfigurableTask
_CITATION = """
@misc{rajpurkar2018know,
title={Know What You Don't Know: Unanswerable Questions for SQuAD},
author={Pranav Rajpurkar and Robin Jia and Percy Liang},
year={2018},
eprint={1806.03822},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
"""
def _squad_metric(predictions, references):
import evaluate
squad_metric = evaluate.load("squad_v2")
return squad_metric.compute(predictions=predictions, references=references)
def _squad_agg(key, items):
predictions, references = zip(*items)
return _squad_metric(predictions=predictions, references=references).get(key, 0)
class SQuAD2(ConfigurableTask):
VERSION = 3
DATASET_PATH = "TUKE-DeutscheTelekom/skquad"
DATASET_NAME = None
def __init__(self, config=None):
super().__init__(config={"metadata": {"version": self.VERSION}})
# HF changed squad on us so we have to make sure we aren't running the old one
assert version.parse(datasets.__version__) >= version.parse("1.11.0"), (
"datasets v1.11.0 or later required for SQuAD"
)
def has_training_docs(self):
return True
def has_validation_docs(self):
return True
def has_test_docs(self):
return False
def training_docs(self):
return self.dataset["train"]
def validation_docs(self):
return self.dataset["validation"]
def doc_to_text(self, doc):
return (
"Title: "
+ doc["title"]
+ "\n\n"
+ "Background: "
+ doc["context"]
+ "\n\n"
+ "Question: "
+ doc["question"]
+ "\n\n"
+ "Answer:"
)
def should_decontaminate(self):
return True
def doc_to_decontamination_query(self, doc):
return doc["context"]
def doc_to_target(self, doc):
answer_list = doc["answers"]["text"]
if len(answer_list) > 0:
answer = answer_list[0]
else:
answer = "unanswerable"
return " " + answer
def construct_requests(
self, doc, ctx, chat_template=None, apply_chat_template=False, **kwargs
):
"""Uses RequestFactory to construct Requests and returns an iterable of
Requests which will be sent to the LM.
:param doc:
The document as returned from training_docs, validation_docs, or test_docs.
:param ctx: str
The context string, generated by fewshot_context. This includes the natural
language description, as well as the few shot examples, and the question
part of the document for `doc`.
"""
return [
Instance(
request_type="generate_until",
doc=doc,
arguments=(ctx, {"until": ["\n"]}),
idx=0,
**kwargs,
),
Instance(
request_type="loglikelihood",
doc=doc,
arguments=(ctx, " " + "unanswerable"),
idx=0,
**kwargs,
),
]
def process_results(self, doc, results):
"""Take a single document and the LM results and evaluates, returning a
dict where keys are the names of submetrics and values are the values of
the metric for that one document
:param doc:
The document as returned from training_docs, validation_docs, or test_docs.
:param results:
The results of the requests created in construct_requests.
"""
continuation, (logprob_unanswerable, _) = results
no_answer_probability = exp(logprob_unanswerable)
predictions = {
"id": doc["id"],
"prediction_text": continuation,
"no_answer_probability": no_answer_probability,
}
references = {
"id": doc["id"],
"answers": doc["answers"],
}
return {
"exact": (
predictions,
references,
), # Exact match (the normalized answer exactly match the gold answer)
"f1": (
predictions,
references,
), # The F-score of predicted tokens versus the gold answer
"HasAns_exact": (
predictions,
references,
), # Exact match (the normalized answer exactly match the gold answer)
"HasAns_f1": (
predictions,
references,
), # The F-score of predicted tokens versus the gold answer
"NoAns_exact": (
predictions,
references,
), # Exact match (the normalized answer exactly match the gold answer)
"NoAns_f1": (
predictions,
references,
), # The F-score of predicted tokens versus the gold answer
"best_exact": (
predictions,
references,
), # Best exact match (with varying threshold)
"best_f1": (predictions, references), # Best F1 (with varying threshold)
}
def aggregation(self):
"""
:returns: {str: [float] -> float}
A dictionary where keys are the names of submetrics and values are
functions that aggregate a list of metrics
"""
return {
"exact": partial(
_squad_agg, "exact"
), # Exact match (the normalized answer exactly match the gold answer)
"f1": partial(
_squad_agg, "f1"
), # The F-score of predicted tokens versus the gold answer
"HasAns_exact": partial(
_squad_agg, "HasAns_exact"
), # Exact match (the normalized answer exactly match the gold answer)
"HasAns_f1": partial(
_squad_agg, "HasAns_f1"
), # The F-score of predicted tokens versus the gold answer
"NoAns_exact": partial(
_squad_agg, "NoAns_exact"
), # Exact match (the normalized answer exactly match the gold answer)
"NoAns_f1": partial(
_squad_agg, "NoAns_f1"
), # The F-score of predicted tokens versus the gold answer
"best_exact": partial(
_squad_agg, "best_exact"
), # Best exact match (with varying threshold)
"best_f1": partial(
_squad_agg, "best_f1"
), # Best F1 (with varying threshold)
}
def higher_is_better(self):
"""
:returns: {str: bool}
A dictionary where keys are the names of submetrics and values are
whether a higher value of the submetric is better
"""
return {
"exact": True, # Exact match (the normalized answer exactly match the gold answer)
"f1": True, # The F-score of predicted tokens versus the gold answer
"HasAns_exact": True, # Exact match (the normalized answer exactly match the gold answer)
"HasAns_f1": True, # The F-score of predicted tokens versus the gold answer
"NoAns_exact": True, # Exact match (the normalized answer exactly match the gold answer)
"NoAns_f1": True, # The F-score of predicted tokens versus the gold answer
"best_exact": True, # Best exact match (with varying threshold)
"best_f1": True, # Best F1 (with varying threshold)
}

232
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task_configs_from_wandb.json Normal file
View File

@ -0,0 +1,232 @@
{
"arc_sk": {
"tag": [
"arc_multilingual"
],
"task": "arc_sk",
"repeats": 1,
"metadata": {
"version": 2
},
"test_split": "test",
"description": "",
"doc_to_text": "query",
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"num_fewshot": 0,
"output_type": "multiple_choice",
"unsafe_code": false,
"dataset_name": "sk",
"dataset_path": "alexandrainst/m_arc",
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n # breakpoint()\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"Question: \" + preprocess(doc[\"instruction\"]) + \"\\nAnswer:\",\n \"choices\": [\n preprocess(option)\n for option in [\n doc[\"option_a\"],\n doc[\"option_b\"],\n doc[\"option_c\"],\n doc[\"option_d\"],\n doc[\"option_e\"],\n ]\n if option\n ],\n \"gold\": [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
"doc_to_choice": "choices",
"doc_to_target": "gold",
"training_split": "train",
"target_delimiter": " ",
"validation_split": "validation",
"fewshot_delimiter": "\n\n",
"should_decontaminate": true,
"doc_to_decontamination_query": "query"
},
"skquad": {
"task": "skquad",
"repeats": 1,
"metadata": {
"version": 3
},
"description": "",
"num_fewshot": 0,
"output_type": "generate_until",
"unsafe_code": false,
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"generation_kwargs": {
"until": [
"\n\n"
],
"do_sample": false
},
"should_decontaminate": false
},
"sklegal": {
"task": "sklegal",
"repeats": 1,
"metadata": {
"version": 1
},
"test_split": "test",
"description": "",
"doc_to_text": "{{prompt}} ",
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"num_fewshot": 0,
"output_type": "multiple_choice",
"unsafe_code": false,
"dataset_path": "TUKE-KEMT/slovak-legal-exam",
"doc_to_choice": "{{answers}}",
"doc_to_target": "label",
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"should_decontaminate": false
},
"m_mmlu_sk": {
"tag": [
"m_mmlu"
],
"task": "m_mmlu_sk",
"repeats": 1,
"metadata": {
"version": 0
},
"test_split": "test",
"description": "",
"doc_to_text": "{{instruction.strip()}}\nA. {{option_a}}\nB. {{option_b}}\nC. {{option_c}}\nD. {{option_d}}\nAnswer:",
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
}
],
"num_fewshot": 0,
"output_type": "multiple_choice",
"unsafe_code": false,
"dataset_name": "sk",
"dataset_path": "alexandrainst/m_mmlu",
"doc_to_choice": [
"A",
"B",
"C",
"D"
],
"doc_to_target": "answer",
"fewshot_split": "train",
"fewshot_config": {
"sampler": "first_n"
},
"target_delimiter": " ",
"fewshot_delimiter": "\n\n",
"should_decontaminate": false
},
"hellaswag_sk": {
"tag": [
"hellaswag_multilingual"
],
"task": "hellaswag_sk",
"repeats": 1,
"metadata": {
"version": 1
},
"description": "",
"doc_to_text": "query",
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
},
{
"metric": "acc_norm",
"aggregation": "mean",
"higher_is_better": true
}
],
"num_fewshot": 0,
"output_type": "multiple_choice",
"unsafe_code": false,
"dataset_name": "sk",
"dataset_path": "alexandrainst/m_hellaswag",
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n out_doc = {\n \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
"doc_to_choice": "choices",
"doc_to_target": "{{label.lstrip()}}",
"target_delimiter": " ",
"validation_split": "val",
"fewshot_delimiter": "\n\n",
"should_decontaminate": false
},
"truthfulqa_sk_mc1": {
"tag": [
"truthfulqa_multilingual"
],
"task": "truthfulqa_sk_mc1",
"repeats": 1,
"metadata": {
"version": 1
},
"description": "",
"doc_to_text": "query",
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
}
],
"num_fewshot": 0,
"output_type": "multiple_choice",
"unsafe_code": false,
"dataset_name": "sk",
"dataset_path": "alexandrainst/m_truthfulqa",
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
"doc_to_choice": "mc1_choices",
"doc_to_target": 0,
"target_delimiter": " ",
"validation_split": "val",
"fewshot_delimiter": "\n\n",
"should_decontaminate": true,
"doc_to_decontamination_query": "question"
},
"truthfulqa_sk_mc2": {
"tag": [
"truthfulqa_multilingual"
],
"task": "truthfulqa_sk_mc2",
"repeats": 1,
"metadata": {
"version": 1
},
"description": "",
"doc_to_text": "query",
"metric_list": [
{
"metric": "acc",
"aggregation": "mean",
"higher_is_better": true
}
],
"num_fewshot": 0,
"output_type": "multiple_choice",
"unsafe_code": false,
"dataset_name": "sk",
"dataset_path": "alexandrainst/m_truthfulqa",
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
"doc_to_choice": "mc2_choices",
"doc_to_target": 0,
"process_results": "def process_results_mc2(doc, results):\n lls, is_greedy = zip(*results)\n\n # Split on the first `0` as everything before it is true (`1`).\n split_idx = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n # Compute the normalized probability mass for the correct answer.\n ll_true, ll_false = lls[:split_idx], lls[split_idx:]\n p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n p_true = p_true / (sum(p_true) + sum(p_false))\n\n return {\"acc\": sum(p_true)}\n",
"target_delimiter": " ",
"validation_split": "val",
"fewshot_delimiter": "\n\n",
"should_decontaminate": true,
"doc_to_decontamination_query": "question"
}
}

34
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc1.yaml Normal file
View File

@ -0,0 +1,34 @@
tag:
- truthfulqa_multilingual
task: truthfulqa_sk_mc1
repeats: 1
metadata:
version: 1
description: ''
doc_to_text: query
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
num_fewshot: 0
output_type: multiple_choice
unsafe_code: false
dataset_name: sk
dataset_path: alexandrainst/m_truthfulqa
process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
\ def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"\
question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"\
question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"\
],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\"\
: {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n \
\ }\n return out_doc\n\n return dataset.map(_process_doc)\n"
doc_to_choice: mc1_choices
doc_to_target: 0
target_delimiter: ' '
validation_split: val
fewshot_delimiter: '
'
should_decontaminate: true
doc_to_decontamination_query: question

40
aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc2.yaml Normal file
View File

@ -0,0 +1,40 @@
tag:
- truthfulqa_multilingual
task: truthfulqa_sk_mc2
repeats: 1
metadata:
version: 1
description: ''
doc_to_text: query
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
num_fewshot: 0
output_type: multiple_choice
unsafe_code: false
dataset_name: sk
dataset_path: alexandrainst/m_truthfulqa
process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
\ def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"\
question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"\
question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"\
],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\"\
: {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n \
\ }\n return out_doc\n\n return dataset.map(_process_doc)\n"
doc_to_choice: mc2_choices
doc_to_target: 0
process_results: "def process_results_mc2(doc, results):\n lls, is_greedy = zip(*results)\n\
\n # Split on the first `0` as everything before it is true (`1`).\n split_idx\
\ = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n # Compute the normalized\
\ probability mass for the correct answer.\n ll_true, ll_false = lls[:split_idx],\
\ lls[split_idx:]\n p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n\
\ p_true = p_true / (sum(p_true) + sum(p_false))\n\n return {\"acc\": sum(p_true)}\n"
target_delimiter: ' '
validation_split: val
fewshot_delimiter: '
'
should_decontaminate: true
doc_to_decontamination_query: question

1
aya_300k_lr_experiments/benchmarks/lmeval/provenance/lm_eval_commit.txt Normal file
View File

@ -0,0 +1 @@
6d2abda4fd171e68a8789330c4149e37c1ca0bda

8
aya_300k_lr_experiments/benchmarks/lmeval/provenance/package_versions.json Normal file
View File

@ -0,0 +1,8 @@
{
"lm_eval": "0.4.8",
"transformers": "4.48.2",
"torch": "2.6.0+cu124",
"datasets": "5.0.1",
"peft": "0.14.0",
"evaluate": "0.4.6"
}

8
aya_300k_lr_experiments/benchmarks/lmeval/provenance/run_settings.txt Normal file
View File

@ -0,0 +1,8 @@
model=hf
pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4
dtype=bfloat16
num_fewshot=0
batch_size=1
apply_chat_template=false
CUDA_VISIBLE_DEVICES=1

1
aya_300k_lr_experiments/benchmarks/lmeval/provenance/tasks.txt Normal file
View File

@ -0,0 +1 @@
arc_sk,hellaswag_sk,m_mmlu_sk,truthfulqa_sk_mc1,truthfulqa_sk_mc2,sklegal,skquad

431
aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json Normal file
View File

File diff suppressed because one or more lines are too long

82
aya_300k_lr_experiments/benchmarks/skbench/audit.md Normal file
View File

@ -0,0 +1,82 @@
# Audit úspešného čiastočného SK-bench behu
Vybraný model po tréningu na 300k vzorke Aya s LR `0.00010` dokončil beh `sk_zero_available_without_nli_lr_0p00010_20261022_185255`. Log končí hláškou `Benchmarking completed.` a obsahuje výsledné metriky. Benchmarkovalo sa iba toto LR, vybrané podľa najnižšieho finálneho validačného loss zo šiestich tréningových behov; výsledky preto neporovnávajú benchmarkovú kvalitu šiestich LR ani budúceho tréningu na celej Aya množine.
## Pokrytie a počty
- Zoznam `configs/tasks.txt`, 222 konfigurácií úloh vo výslednom JSON a 222 súborov DETAILS sa zhodujú. Názvy sa pri porovnaní dekódujú z Windows `%3A`/`%7C` a normalizuje sa koncové označenie zero-shot variantu.
- Vyhodnotených bolo **222 zo 412 pôvodných task variantov**: 74 kombinácií úlohy/subtasku, každá s tromi promptmi. Ide o 13 rodín úloh. MMLU tvorí 171 variantov (57 predmetov × 3 prompty).
- DETAILS obsahujú **159 426 vyhodnotených inštancií**, po **53 142 na každý prompt**. Tri prompty opakovane hodnotia tie isté základné testové otázky; 159 426 nie je počet nezávislých otázok ani dôkaz unikátnosti medzi rôznymi datasetmi.
- `configs/omitted_tasks.txt` obsahuje 190 vynechaných variantov. Report dostupnosti eviduje 12 nedostupných repozitárov so 187 variantmi; tri ďalšie vynechané varianty patria samostatnému NLI-SK. Historický access report uvádza NLI-SK ako `OK`, ale jeho tri varianty v konečnom behu nie sú. NLI úlohy z rodiny SKLEP zostali zahrnuté.
- `max_samples: null` znamená, že beh nemal globálny limit vzoriek. Počty sa získali z metadát jednotlivých DETAILS súborov, nie zo zaokrúhlených metrík.
Tento dostupný subset sa nesmie označovať za oficiálne kompletné SK-bench skóre. Chýbajúce úlohy ani chýbajúce rodiny sa neimputovali.
## Zaznamenané nastavenie
Výsledný JSON potvrdzuje `bfloat16`, kontext `32768`, tensor/data/pipeline parallelism `1/1/1`, využitie GPU pamäte `0.85`, seed modelu `1234`, teplotu `0` a `enforce_eager: true`. Pole `generation_parameters.seed` je `null`; konkrétny seed je uložený v `model_config.seed`. Všetky úlohy majú `num_fewshots: 0`.
216 task variantov používa najviac 32 generovaných tokenov a šesť QA variantov 126 tokenov; stop sekvencia je nový riadok. Tieto limity výstupu treba pri porovnaní výsledkov zachovať. Záznam neobsahuje spoľahlivý počet odpovedí ukončených výstupným limitom.
Zmrazený commit SK-bench je `15823bbcc2d995e69441fcb7d34abb61bd8180c5`. Pôvodné `lighteval_sha` vo výslednom JSON je `?`; tento údaj sa dodatočne nevymýšľa. `uv.lock`, commit, modelová konfigurácia, per-task konfigurácie a revízie datasetov sú uchované v `provenance/` a `configs/`. Log zaznamenáva aj runtime patchovanie; samotný commit a lock preto bez rovnakého postupu nie sú úplným dôkazom identického prostredia.
## Orezanie dlhých vstupov
Kontrola projektovala `model_response.truncated_tokens_count` zo **všetkých 222 DETAILS súborov**, bez čítania duplicitného generation cache. Všetkých 159 426 hodnôt existuje, žiadna nie je `null` a všetky sú nula. Aj pôvodné súhrnné počítadlá sú nulové vrátane `non_truncated`, takže samy osebe nepotvrdzujú neprítomnosť orezania.
**Log priamo potvrdzuje orezanie kontextu.** Vyhľadanie `truncat` bez rozlíšenia veľkosti písmen našlo presne jednu hlášku, na LF riadku 16033:
```text
context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens.
```
Následná diagnostika dĺžok zo všetkých DETAILS našla **69 tokenových polí presne na vstupnom limite 32 642 tokenov**, po 23 v každom z troch variantov `multi_wiki_qa`. Vo všetkých troch sa opakujú tie isté indexy riadkov. Žiadne uložené pole neprekročilo príslušný limit `32768 − generation_size`; ostatné úlohy mali vstupy pod limitom. Súvisiace uložené reťazce vstupu mali pri hraničných riadkoch až 121 852 znakov.
Týchto 69 riadkov sú **hraniční kandidáti zodpovedajúci orezaniu**, nie spoľahlivo dokázaný počet orezaných vstupov. Log neuvádza identifikátor konkrétneho tasku ani počet dotknutých vzoriek. Bez opätovnej tokenizácie pôvodných vstupov nemožno z dĺžky uloženého reťazca určiť pôvodnú dĺžku v tokenoch alebo počet odstránených tokenov. Overený záver je: **orezanie nastalo aspoň raz; jeho presný rozsah nie je z dostupných počítadiel známy**.
Strojový súhrn preto obsahuje `context_truncation_observed_in_log: true`, `actual_truncated_sample_count: null` a nulové počítadlá označuje prefixom `details_reported_`. Podrobnosti sú v `results/truncation_audit_per_task.csv`, `results/log_truncation_evidence.json`, `results/input_length_diagnostics_per_task.csv` a `results/input_length_budget_boundary_rows.csv`.
## Metriky a agregácia
`results/per_task_metrics.csv` obsahuje iba 222 reálne vykonaných task variantov, ich skutočné metriky a štandardné chyby priamo z výsledného JSON. Prioritná metrika na čítanie je `em_relaxed`, prípadne `em_norm_relaxed`, ak ju daná úloha skutočne používa. QA má vlastné metriky `exact_match_slk_prefix` a `f1_slk`; nepremenúvajú sa na všeobecnú accuracy.
Odvodené súhrny nikdy nemiešajú odlišné metriky do jedného skóre:
- `family_summary.csv`: priemer v rodine, vždy osobitne pre konkrétnu metriku, cez všetky dostupné subtasky a tri prompty.
- `family_prompt_summary.csv`: tá istá agregácia osobitne pre prompt 1, 2 a 3.
- `three_prompt_summary.csv`: priemer troch promptov pre každú zo 74 konkrétnych kombinácií úlohy/subtasku, opäť pre každú metriku osobitne.
- `task_macro_mean` je aritmetický priemer task metrík s rovnakou váhou každého variantu. `sample_weighted_mean` váži tieto metriky počtom riadkov v DETAILS. Prompty sa v oboch prípadoch považujú za opakované vyhodnotenia, nie za nové nezávislé otázky. Súhrnné intervaly spoľahlivosti sa z per-task chýb nevymýšľajú.
Pôvodný JSON má 226 kľúčov výsledkov: 222 úloh, tri vstavané MMLU promptové priemery a `all`. Štyri agregované riadky sú zachované v pôvodnom JSON, ale v odvodených súhrnoch sa znovu nepripočítavajú. `all` obsahuje viac metrík s rôznymi sadami úloh a netvorí oficiálne celkové skóre.
Napríklad MMLU `em_relaxed` má task macro priemer `0.1600941253` a sample-weighted priemer `0.1572245190`. Rozdiel vzniká rôznou veľkosťou 57 predmetov. SKLEP klasifikačné úlohy s `em_norm_relaxed` majú macro `0.3507994510`, ale sample-weighted `0.2144044321`; výber váh tu zásadne mení opisný výsledok.
## Obnovenie auditu
Audit používa Python a `pyarrow==26.0.0`. Knižnica sa pri spracovaní nainštalovala len do pomocného priečinka pracovného projektu; nie je súčasťou tohto stromu. Pôvodné parquety, cache ani váhy sa do Git podkladov nekopírujú. Na opätovný audit treba samostatne rozbalený zdrojový archív a jeho `archive_manifest.json`.
Archív treba najprv rozbaliť pomocou uloženého pomocného skriptu (Python 3.11+), ktorý vytvorí aj `archive_manifest.json` a `archive_summary.json`. Názvy s `:` a `|` prevedie na prenosné percent-encoded názvy a zachová mapovanie na originál. Priečinok určený cez `--output` musí byť nový alebo prázdny:
```bash
python scripts/unpack_results_archive.py \
--archive /path/to/aya_300k_benchmarky_20261024_215708.tar.gz \
--output /path/to/extracted/archive
```
Z koreňa experimentu možno potom spustiť:
```bash
python -m pip install --target .audit-deps -r benchmarks/skbench/provenance/requirements-audit.txt
python benchmarks/skbench/provenance/audit_skbench.py \
--source /path/to/extracted/archive \
--output benchmarks/skbench \
--pyarrow-path .audit-deps
python benchmarks/skbench/provenance/input_length_diagnostics.py \
--source /path/to/extracted/archive \
--output benchmarks/skbench/results \
--pyarrow-path .audit-deps
```
Prvý skript kontroluje pokrytie, kopíruje malé pôvodné artefakty, počíta metrikové súhrny a overuje truncation counter projekciou jediného stĺpca. Druhý skript projektuje uložené tokenové polia a vstupné reťazce, ale exportuje iba ich dĺžky a indexy hraničných riadkov. Texty otázok a odpovedí v týchto odvodených diagnostikách nie sú.
SHA-256 presných kópií pôvodných súborov je v `provenance/copied_files_sha256.csv`; zdrojový archív identifikuje `provenance/archive_summary.json`.

622
aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json Normal file
View File

@ -0,0 +1,622 @@
[
{
"repo": "CohereLabs/Global-MMLU-Lite",
"revision": "600af1bff2d447c7fc3aee22f11f607560d5fcb7",
"status": "OK",
"error": "",
"tasks": [
"globalmmlu-lite.prompt3_mqa_single_correct_sk|0",
"globalmmlu-lite.prompt1_mqa_single_correct_sk|0",
"globalmmlu-lite.prompt2_mqa_single_correct_sk|0"
]
},
{
"repo": "NaiveNeuron/SlovakCOPA",
"revision": "5c61e4cd9343f9b4fab623bdc50bb895cc170771",
"status": "OK",
"error": "",
"tasks": [
"slovakcopa.prompt3_copa_sk|0",
"slovakcopa.prompt1_copa_sk|0",
"slovakcopa.prompt2_copa_sk|0"
]
},
{
"repo": "TUKE-KEMT/senti-sk",
"revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
"status": "OK",
"error": "",
"tasks": [
"senti_sk.prompt2_sentiment_binary_sk|0",
"senti_sk.prompt3_sentiment_binary_sk|0",
"senti_sk.prompt1_sentiment_binary_sk|0",
"senti_sk.prompt3_sentiment_3_class_sk|0",
"senti_sk.prompt1_sentiment_3_class_sk|0",
"senti_sk.prompt2_sentiment_3_class_sk|0"
]
},
{
"repo": "TUKE-KEMT/slovak-financial-exam",
"revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
"status": "OK",
"error": "",
"tasks": [
"financial_exams.prompt1_completion_sk|0",
"financial_exams.prompt3_completion_sk|0",
"financial_exams.prompt2_completion_sk|0"
]
},
{
"repo": "TUKE-KEMT/toxic-sk",
"revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
"status": "OK",
"error": "",
"tasks": [
"toxic_sk.prompt2_toxicity_sk|0",
"toxic_sk.prompt3_toxicity_sk|0",
"toxic_sk.prompt1_toxicity_sk|0"
]
},
{
"repo": "alexandrainst/m_truthfulqa",
"revision": "f0445d470f1925882b990f5f247fdcf288972f60",
"status": "OK",
"error": "",
"tasks": [
"truthfulqa.prompt3_mqa_single_correct_sk:mc1|0",
"truthfulqa.prompt1_mqa_single_correct_sk:mc1|0",
"truthfulqa.prompt2_mqa_single_correct_sk:mc1|0"
]
},
{
"repo": "alexandrainst/multi-wiki-qa",
"revision": "9bdb3350487e399bd1cd873e1395290d636db5ce",
"status": "OK",
"error": "",
"tasks": [
"multi_wiki_qa.prompt3_qa_sk|0",
"multi_wiki_qa.prompt2_qa_sk|0",
"multi_wiki_qa.prompt1_qa_sk|0"
]
},
{
"repo": "facebook/belebele",
"revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
"status": "OK",
"error": "",
"tasks": [
"belebele.prompt3_mqa_single_correct_sk|0",
"belebele.prompt1_mqa_single_correct_sk|0",
"belebele.prompt2_mqa_single_correct_sk|0"
]
},
{
"repo": "jon-tow/okapi_arc_challenge",
"revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
"status": "OK",
"error": "",
"tasks": [
"arc.prompt3_mqa_single_correct_sk|0",
"arc.prompt1_mqa_single_correct_sk|0",
"arc.prompt2_mqa_single_correct_sk|0"
]
},
{
"repo": "lighteval/okapi_mmlu",
"revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
"status": "OK",
"error": "",
"tasks": [
"mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0",
"mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0",
"mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0",
"mmlu.prompt3_mqa_single_correct_sk:anatomy|0",
"mmlu.prompt1_mqa_single_correct_sk:anatomy|0",
"mmlu.prompt2_mqa_single_correct_sk:anatomy|0",
"mmlu.prompt3_mqa_single_correct_sk:astronomy|0",
"mmlu.prompt1_mqa_single_correct_sk:astronomy|0",
"mmlu.prompt2_mqa_single_correct_sk:astronomy|0",
"mmlu.prompt3_mqa_single_correct_sk:business_ethics|0",
"mmlu.prompt1_mqa_single_correct_sk:business_ethics|0",
"mmlu.prompt2_mqa_single_correct_sk:business_ethics|0",
"mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0",
"mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0",
"mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0",
"mmlu.prompt3_mqa_single_correct_sk:college_biology|0",
"mmlu.prompt1_mqa_single_correct_sk:college_biology|0",
"mmlu.prompt2_mqa_single_correct_sk:college_biology|0",
"mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0",
"mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0",
"mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0",
"mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0",
"mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0",
"mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0",
"mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0",
"mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0",
"mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0",
"mmlu.prompt3_mqa_single_correct_sk:college_medicine|0",
"mmlu.prompt1_mqa_single_correct_sk:college_medicine|0",
"mmlu.prompt2_mqa_single_correct_sk:college_medicine|0",
"mmlu.prompt3_mqa_single_correct_sk:college_physics|0",
"mmlu.prompt1_mqa_single_correct_sk:college_physics|0",
"mmlu.prompt2_mqa_single_correct_sk:college_physics|0",
"mmlu.prompt3_mqa_single_correct_sk:computer_security|0",
"mmlu.prompt1_mqa_single_correct_sk:computer_security|0",
"mmlu.prompt2_mqa_single_correct_sk:computer_security|0",
"mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0",
"mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0",
"mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0",
"mmlu.prompt3_mqa_single_correct_sk:econometrics|0",
"mmlu.prompt1_mqa_single_correct_sk:econometrics|0",
"mmlu.prompt2_mqa_single_correct_sk:econometrics|0",
"mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0",
"mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0",
"mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0",
"mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0",
"mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0",
"mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0",
"mmlu.prompt3_mqa_single_correct_sk:formal_logic|0",
"mmlu.prompt1_mqa_single_correct_sk:formal_logic|0",
"mmlu.prompt2_mqa_single_correct_sk:formal_logic|0",
"mmlu.prompt3_mqa_single_correct_sk:global_facts|0",
"mmlu.prompt1_mqa_single_correct_sk:global_facts|0",
"mmlu.prompt2_mqa_single_correct_sk:global_facts|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0",
"mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0",
"mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0",
"mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0",
"mmlu.prompt3_mqa_single_correct_sk:human_aging|0",
"mmlu.prompt1_mqa_single_correct_sk:human_aging|0",
"mmlu.prompt2_mqa_single_correct_sk:human_aging|0",
"mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0",
"mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0",
"mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0",
"mmlu.prompt3_mqa_single_correct_sk:international_law|0",
"mmlu.prompt1_mqa_single_correct_sk:international_law|0",
"mmlu.prompt2_mqa_single_correct_sk:international_law|0",
"mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0",
"mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0",
"mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0",
"mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0",
"mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0",
"mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0",
"mmlu.prompt3_mqa_single_correct_sk:machine_learning|0",
"mmlu.prompt1_mqa_single_correct_sk:machine_learning|0",
"mmlu.prompt2_mqa_single_correct_sk:machine_learning|0",
"mmlu.prompt3_mqa_single_correct_sk:management|0",
"mmlu.prompt1_mqa_single_correct_sk:management|0",
"mmlu.prompt2_mqa_single_correct_sk:management|0",
"mmlu.prompt3_mqa_single_correct_sk:marketing|0",
"mmlu.prompt1_mqa_single_correct_sk:marketing|0",
"mmlu.prompt2_mqa_single_correct_sk:marketing|0",
"mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0",
"mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0",
"mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0",
"mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0",
"mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0",
"mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0",
"mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0",
"mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0",
"mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0",
"mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0",
"mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0",
"mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0",
"mmlu.prompt3_mqa_single_correct_sk:nutrition|0",
"mmlu.prompt1_mqa_single_correct_sk:nutrition|0",
"mmlu.prompt2_mqa_single_correct_sk:nutrition|0",
"mmlu.prompt3_mqa_single_correct_sk:philosophy|0",
"mmlu.prompt1_mqa_single_correct_sk:philosophy|0",
"mmlu.prompt2_mqa_single_correct_sk:philosophy|0",
"mmlu.prompt3_mqa_single_correct_sk:prehistory|0",
"mmlu.prompt1_mqa_single_correct_sk:prehistory|0",
"mmlu.prompt2_mqa_single_correct_sk:prehistory|0",
"mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0",
"mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0",
"mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0",
"mmlu.prompt3_mqa_single_correct_sk:professional_law|0",
"mmlu.prompt1_mqa_single_correct_sk:professional_law|0",
"mmlu.prompt2_mqa_single_correct_sk:professional_law|0",
"mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0",
"mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0",
"mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0",
"mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0",
"mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0",
"mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0",
"mmlu.prompt3_mqa_single_correct_sk:public_relations|0",
"mmlu.prompt1_mqa_single_correct_sk:public_relations|0",
"mmlu.prompt2_mqa_single_correct_sk:public_relations|0",
"mmlu.prompt3_mqa_single_correct_sk:security_studies|0",
"mmlu.prompt1_mqa_single_correct_sk:security_studies|0",
"mmlu.prompt2_mqa_single_correct_sk:security_studies|0",
"mmlu.prompt3_mqa_single_correct_sk:sociology|0",
"mmlu.prompt1_mqa_single_correct_sk:sociology|0",
"mmlu.prompt2_mqa_single_correct_sk:sociology|0",
"mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0",
"mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0",
"mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0",
"mmlu.prompt3_mqa_single_correct_sk:virology|0",
"mmlu.prompt1_mqa_single_correct_sk:virology|0",
"mmlu.prompt2_mqa_single_correct_sk:virology|0",
"mmlu.prompt3_mqa_single_correct_sk:world_religions|0",
"mmlu.prompt1_mqa_single_correct_sk:world_religions|0",
"mmlu.prompt2_mqa_single_correct_sk:world_religions|0"
]
},
{
"repo": "mteb/slovak_movie_review_sentiment",
"revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
"status": "OK",
"error": "",
"tasks": [
"movie_review.prompt3_sentiment_binary_sk|0",
"movie_review.prompt2_sentiment_binary_sk|0",
"movie_review.prompt1_sentiment_binary_sk|0"
]
},
{
"repo": "natalia-nk/NLI-SK-annotated",
"revision": "79914f425b59d8b9fabb6d38c37f6d81f9723f46",
"status": "OK",
"error": "",
"tasks": [
"nli_sk.prompt1_nli_3_classes_sk|0",
"nli_sk.prompt2_nli_3_classes_sk|0",
"nli_sk.prompt3_nli_3_classes_sk|0"
]
},
{
"repo": "slovak-nlp/SKJ1",
"revision": "e615c60b415389b629703f82d3a79810dbde6734",
"status": "OK",
"error": "",
"tasks": [
"skj1.prompt3_skj1_sk|0",
"skj1.prompt1_skj1_sk|0",
"skj1.prompt2_skj1_sk|0"
]
},
{
"repo": "slovak-nlp/chiby-classification",
"revision": "ce8533398057ace6da817758335633067648ca93",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-1e7a56917b7b19192d552e13;119d40c0-7b4d-43f3-8cd4-82370a8ee77e)",
"tasks": [
"chiby.prompt2_chiby_sk|0",
"chiby.prompt1_chiby_sk|0",
"chiby.prompt3_chiby_sk|0"
]
},
{
"repo": "slovak-nlp/driving-test-dataset-slovak",
"revision": "37064890fbc00bfca59c035d38f52d265949361e",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-3129bdcb539f88e7672e2027;105ec634-cecc-4558-b3c9-e83508219914)",
"tasks": [
"driving_test.prompt3_mqa_single_correct_sk|0",
"driving_test.prompt1_mqa_single_correct_sk|0",
"driving_test.prompt2_mqa_single_correct_sk|0"
]
},
{
"repo": "slovak-nlp/e-testy-sk",
"revision": "1a5e798e0473241097c4a26c511426167522adf5",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-4f270b5d6ba17a181d7897ee;9fecd2d5-dc7c-4a4c-8755-61fd6bcda935)",
"tasks": [
"e_tests.prompt3_mqa_single_correct_sk:biologia|0",
"e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0",
"e_tests.prompt3_mqa_single_correct_sk:informatika|0",
"e_tests.prompt3_mqa_single_correct_sk:geografia|0",
"e_tests.prompt3_mqa_single_correct_sk:matematika|0",
"e_tests.prompt3_mqa_single_correct_sk:chemia|0",
"e_tests.prompt3_mqa_single_correct_sk:dejepis|0",
"e_tests.prompt3_mqa_single_correct_sk:fyzika|0",
"e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0",
"e_tests.prompt1_mqa_single_correct_sk:biologia|0",
"e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0",
"e_tests.prompt1_mqa_single_correct_sk:informatika|0",
"e_tests.prompt1_mqa_single_correct_sk:geografia|0",
"e_tests.prompt1_mqa_single_correct_sk:matematika|0",
"e_tests.prompt1_mqa_single_correct_sk:chemia|0",
"e_tests.prompt1_mqa_single_correct_sk:dejepis|0",
"e_tests.prompt1_mqa_single_correct_sk:fyzika|0",
"e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0",
"e_tests.prompt2_mqa_single_correct_sk:biologia|0",
"e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0",
"e_tests.prompt2_mqa_single_correct_sk:informatika|0",
"e_tests.prompt2_mqa_single_correct_sk:geografia|0",
"e_tests.prompt2_mqa_single_correct_sk:matematika|0",
"e_tests.prompt2_mqa_single_correct_sk:chemia|0",
"e_tests.prompt2_mqa_single_correct_sk:dejepis|0",
"e_tests.prompt2_mqa_single_correct_sk:fyzika|0",
"e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0",
"e_tests.prompt2_mqa_multiple_correct_sk:biologia|0",
"e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0",
"e_tests.prompt2_mqa_multiple_correct_sk:informatika|0",
"e_tests.prompt2_mqa_multiple_correct_sk:geografia|0",
"e_tests.prompt2_mqa_multiple_correct_sk:matematika|0",
"e_tests.prompt2_mqa_multiple_correct_sk:chemia|0",
"e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0",
"e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0",
"e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0",
"e_tests.prompt1_mqa_multiple_correct_sk:biologia|0",
"e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0",
"e_tests.prompt1_mqa_multiple_correct_sk:informatika|0",
"e_tests.prompt1_mqa_multiple_correct_sk:geografia|0",
"e_tests.prompt1_mqa_multiple_correct_sk:matematika|0",
"e_tests.prompt1_mqa_multiple_correct_sk:chemia|0",
"e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0",
"e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0",
"e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0",
"e_tests.prompt3_mqa_multiple_correct_sk:biologia|0",
"e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0",
"e_tests.prompt3_mqa_multiple_correct_sk:informatika|0",
"e_tests.prompt3_mqa_multiple_correct_sk:geografia|0",
"e_tests.prompt3_mqa_multiple_correct_sk:matematika|0",
"e_tests.prompt3_mqa_multiple_correct_sk:chemia|0",
"e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0",
"e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0",
"e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0",
"e_tests.prompt1_qa_sk:biologia|0",
"e_tests.prompt1_qa_sk:obcianska-nauka|0",
"e_tests.prompt1_qa_sk:informatika|0",
"e_tests.prompt1_qa_sk:geografia|0",
"e_tests.prompt1_qa_sk:chemia|0",
"e_tests.prompt1_qa_sk:dejepis|0",
"e_tests.prompt1_qa_sk:fyzika|0",
"e_tests.prompt1_qa_sk:slovensky-jazyk|0",
"e_tests.prompt3_qa_sk:biologia|0",
"e_tests.prompt3_qa_sk:obcianska-nauka|0",
"e_tests.prompt3_qa_sk:informatika|0",
"e_tests.prompt3_qa_sk:geografia|0",
"e_tests.prompt3_qa_sk:chemia|0",
"e_tests.prompt3_qa_sk:dejepis|0",
"e_tests.prompt3_qa_sk:fyzika|0",
"e_tests.prompt3_qa_sk:slovensky-jazyk|0",
"e_tests.prompt2_qa_sk:biologia|0",
"e_tests.prompt2_qa_sk:obcianska-nauka|0",
"e_tests.prompt2_qa_sk:informatika|0",
"e_tests.prompt2_qa_sk:geografia|0",
"e_tests.prompt2_qa_sk:chemia|0",
"e_tests.prompt2_qa_sk:dejepis|0",
"e_tests.prompt2_qa_sk:fyzika|0",
"e_tests.prompt2_qa_sk:slovensky-jazyk|0"
]
},
{
"repo": "slovak-nlp/ifeval-sk",
"revision": "2114c70fdcd22e8e3e5ea11f6b77067740959bf4",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-5666942d3a4121b676a928ff;cbd91d44-6ee2-44b0-b74b-3aca15012a31)",
"tasks": [
"ifeval_sk.ifeval|0"
]
},
{
"repo": "slovak-nlp/oneruler-sk-processed",
"revision": "c3a160bc7ce2dada86803f26ee699ca75ee26e41",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-342e828a639d5d19427fae4f;91605400-c335-4c30-8e40-78bdf0e40b3c)",
"tasks": [
"oneruler_sk.oneruler_slovak:cwe_4096|0",
"oneruler_sk.oneruler_slovak:cwe_8192|0",
"oneruler_sk.oneruler_slovak:cwe_16384|0",
"oneruler_sk.oneruler_slovak:niah_multikey_4096|0",
"oneruler_sk.oneruler_slovak:niah_multikey_8192|0",
"oneruler_sk.oneruler_slovak:niah_multikey_16384|0",
"oneruler_sk.oneruler_slovak:niah_multiquery_4096|0",
"oneruler_sk.oneruler_slovak:niah_multiquery_8192|0",
"oneruler_sk.oneruler_slovak:niah_multiquery_16384|0",
"oneruler_sk.oneruler_slovak:niah_multivalue_4096|0",
"oneruler_sk.oneruler_slovak:niah_multivalue_8192|0",
"oneruler_sk.oneruler_slovak:niah_multivalue_16384|0",
"oneruler_sk.oneruler_slovak:niah_none_4096|0",
"oneruler_sk.oneruler_slovak:niah_none_8192|0",
"oneruler_sk.oneruler_slovak:niah_none_16384|0",
"oneruler_sk.oneruler_slovak:niah_single_4096|0",
"oneruler_sk.oneruler_slovak:niah_single_8192|0",
"oneruler_sk.oneruler_slovak:niah_single_16384|0"
]
},
{
"repo": "slovak-nlp/scio-testy",
"revision": "78c67eeae5d48b92590945a1c589b7730bfce8df",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-203c6cfb4d094ab1703aa786;3f87f5ab-83e3-4b52-b012-e6e26eed3d43)",
"tasks": [
"scio.prompt2_mqa_single_correct_sk:aj|0",
"scio.prompt2_mqa_single_correct_sk:mat|0",
"scio.prompt2_mqa_single_correct_sk:bio|0",
"scio.prompt2_mqa_single_correct_sk:che|0",
"scio.prompt2_mqa_single_correct_sk:vsp|0",
"scio.prompt3_mqa_single_correct_sk:aj|0",
"scio.prompt3_mqa_single_correct_sk:mat|0",
"scio.prompt3_mqa_single_correct_sk:bio|0",
"scio.prompt3_mqa_single_correct_sk:che|0",
"scio.prompt3_mqa_single_correct_sk:vsp|0",
"scio.prompt1_mqa_single_correct_sk:aj|0",
"scio.prompt1_mqa_single_correct_sk:mat|0",
"scio.prompt1_mqa_single_correct_sk:bio|0",
"scio.prompt1_mqa_single_correct_sk:che|0",
"scio.prompt1_mqa_single_correct_sk:vsp|0"
]
},
{
"repo": "slovak-nlp/sklep",
"revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
"status": "OK",
"error": "",
"tasks": [
"sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0",
"sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0",
"sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0",
"sklep.prompt1_hate_speech_sk:hate-speech|0",
"sklep.prompt3_hate_speech_sk:hate-speech|0",
"sklep.prompt2_hate_speech_sk:hate-speech|0",
"sklep.prompt1_nli_3_classes_sk:nli|0",
"sklep.prompt2_nli_3_classes_sk:nli|0",
"sklep.prompt3_nli_3_classes_sk:nli|0",
"sklep.prompt1_nli_binary_sk:rte|0",
"sklep.prompt3_nli_binary_sk:rte|0",
"sklep.prompt2_nli_binary_sk:rte|0",
"sklep.prompt3_qa_sk:question-answering|0",
"sklep.prompt2_qa_sk:question-answering|0",
"sklep.prompt1_qa_sk:question-answering|0"
]
},
{
"repo": "slovak-nlp/slohry",
"revision": "3b47acd8a549c65ec6ab34aacd538f3ebcbfa32f",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-18026919100903a2056ca192;a073f7f3-9b00-49ca-982c-c03741ca2364)",
"tasks": [
"slohry.prompt2_declension_sk:declension_high|0",
"slohry.prompt2_declension_sk:declension_mid|0",
"slohry.prompt2_declension_sk:declension_low|0",
"slohry.prompt1_declension_sk:declension_high|0",
"slohry.prompt1_declension_sk:declension_mid|0",
"slohry.prompt1_declension_sk:declension_low|0",
"slohry.prompt3_declension_sk:declension_high|0",
"slohry.prompt3_declension_sk:declension_mid|0",
"slohry.prompt3_declension_sk:declension_low|0",
"slohry.prompt3_gradation_sk:gradation_high|0",
"slohry.prompt3_gradation_sk:gradation_mid|0",
"slohry.prompt3_gradation_sk:gradation_low|0",
"slohry.prompt1_gradation_sk:gradation_high|0",
"slohry.prompt1_gradation_sk:gradation_mid|0",
"slohry.prompt1_gradation_sk:gradation_low|0",
"slohry.prompt2_gradation_sk:gradation_high|0",
"slohry.prompt2_gradation_sk:gradation_mid|0",
"slohry.prompt2_gradation_sk:gradation_low|0",
"slohry.prompt1_identity_form_sk:identity_form_high|0",
"slohry.prompt1_identity_form_sk:identity_form_mid|0",
"slohry.prompt1_identity_form_sk:identity_form_low|0",
"slohry.prompt2_identity_form_sk:identity_form_high|0",
"slohry.prompt2_identity_form_sk:identity_form_mid|0",
"slohry.prompt2_identity_form_sk:identity_form_low|0",
"slohry.prompt3_identity_form_sk:identity_form_high|0",
"slohry.prompt3_identity_form_sk:identity_form_mid|0",
"slohry.prompt3_identity_form_sk:identity_form_low|0",
"slohry.prompt2_inflection_sk:inflection_high|0",
"slohry.prompt2_inflection_sk:inflection_mid|0",
"slohry.prompt2_inflection_sk:inflection_low|0",
"slohry.prompt1_inflection_sk:inflection_high|0",
"slohry.prompt1_inflection_sk:inflection_mid|0",
"slohry.prompt1_inflection_sk:inflection_low|0",
"slohry.prompt3_inflection_sk:inflection_high|0",
"slohry.prompt3_inflection_sk:inflection_mid|0",
"slohry.prompt3_inflection_sk:inflection_low|0"
]
},
{
"repo": "slovak-nlp/slovak-school-exams-test-preprocessed",
"revision": "1d51f8205ee2341d11f05273d9c6d4ba3532a1df",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-7d40d0951ba637dd705fbdd4;c5ed7cb3-8e89-455e-af3d-699ae41c323d)",
"tasks": [
"slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0",
"slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0",
"slovak_tests.prompt3_mqa_single_correct_sk:maturita|0",
"slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0",
"slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0",
"slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0",
"slovak_tests.prompt1_mqa_single_correct_sk:maturita|0",
"slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0",
"slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0",
"slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0",
"slovak_tests.prompt2_mqa_single_correct_sk:maturita|0",
"slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0",
"slovak_tests.prompt1_qa_sk:monitor9|0",
"slovak_tests.prompt1_qa_sk:maturita|0",
"slovak_tests.prompt1_qa_sk:prijmacie_skusky|0",
"slovak_tests.prompt3_qa_sk:monitor9|0",
"slovak_tests.prompt3_qa_sk:maturita|0",
"slovak_tests.prompt3_qa_sk:prijmacie_skusky|0",
"slovak_tests.prompt2_qa_sk:monitor9|0",
"slovak_tests.prompt2_qa_sk:maturita|0",
"slovak_tests.prompt2_qa_sk:prijmacie_skusky|0"
]
},
{
"repo": "slovak-nlp/slovak_trivia",
"revision": "2f27ff5a3c266a379c164d3cb43fc9cf8e45b37a",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-60bbc6ed1f90512f2a40bf37;5a9d9ee9-412d-4cdd-a8a3-66accb5e4d3e)",
"tasks": [
"slovak_trivia.prompt2_mqa_single_correct_sk|0",
"slovak_trivia.prompt3_mqa_single_correct_sk|0",
"slovak_trivia.prompt1_mqa_single_correct_sk|0"
]
},
{
"repo": "slovak-nlp/slovaksum-downsampled",
"revision": "2e4cdabe972ac040114701ea9e885e0a648a33fb",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-3266710308ee05503220c4de;bd8592d6-e022-40ef-a2dd-c9bd490e957f)",
"tasks": [
"slovaksum.prompt2_summarization_sk|0",
"slovaksum.prompt3_summarization_sk|0",
"slovaksum.prompt1_summarization_sk|0"
]
},
{
"repo": "slovak-nlp/smesum-downsampled",
"revision": "cb427bdc96e444637508fb982915f706c19d81d4",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-4c1e6f5602eda70b00239325;126975ea-e381-456f-8fa6-0ffff90f3938)",
"tasks": [
"smesum.prompt2_summarization_sk|0",
"smesum.prompt3_summarization_sk|0",
"smesum.prompt1_summarization_sk|0"
]
},
{
"repo": "slovak-nlp/word-analogies-sk",
"revision": "d6564d49c07b29ea93028f55a57c25eb231762f9",
"status": "NEDOSTUPNÉ",
"error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-18fc3a1223dc59e9238fa860;c8c94f13-fd96-4630-9510-c1ab0db7a366)",
"tasks": [
"analogies_sk.prompt1_analogy_sk|0",
"analogies_sk.prompt2_analogy_sk|0",
"analogies_sk.prompt3_analogy_sk|0"
]
}
]

12
aya_300k_lr_experiments/benchmarks/skbench/configs/export_lr_0p00010.yaml Normal file
View File

@ -0,0 +1,12 @@
model_name_or_path: slovak-nlp/mistral-sk-7b
model_revision: 089497ae0d72018e9591895c54f774bf0b4a83a4
adapter_name_or_path: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010
template: alpaca
finetuning_type: lora
infer_dtype: bfloat16
export_dir: /home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
export_size: 5
export_device: cpu
export_legacy_format: false

59
aya_300k_lr_experiments/benchmarks/skbench/configs/model_config.json Normal file
View File

@ -0,0 +1,59 @@
{
"lighteval_sha": "?",
"num_fewshot_seeds": 1,
"max_samples": null,
"job_id": "0",
"start_time": 796668.90423246,
"end_time": 807588.555648873,
"total_evaluation_time_secondes": "10919.65141641302",
"model_config": {
"model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010",
"generation_parameters": {
"num_blocks": null,
"block_size": null,
"early_stopping": null,
"repetition_penalty": null,
"frequency_penalty": null,
"length_penalty": null,
"presence_penalty": null,
"max_new_tokens": null,
"min_new_tokens": null,
"seed": null,
"stop_tokens": null,
"temperature": 0,
"top_k": null,
"min_p": null,
"top_p": null,
"truncate_prompt": null,
"cache_implementation": null,
"response_format": null,
"skip_special_tokens": true
},
"system_prompt": null,
"cache_dir": "~/.cache/huggingface/lighteval",
"tokenizer": null,
"revision": "main",
"dtype": "bfloat16",
"tensor_parallel_size": 1,
"data_parallel_size": 1,
"pipeline_parallel_size": 1,
"gpu_memory_utilization": 0.85,
"enable_prefix_caching": null,
"max_model_length": 32768,
"quantization": null,
"load_format": null,
"swap_space": 4,
"seed": 1234,
"trust_remote_code": false,
"add_special_tokens": true,
"multichoice_continuations_start_space": true,
"pairwise_tokenization": false,
"max_num_seqs": 128,
"max_num_batched_tokens": 2048,
"subfolder": null,
"is_async": false,
"override_chat_template": null,
"enforce_eager": true
},
"model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010"
}

190
aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt Normal file
View File

@ -0,0 +1,190 @@
chiby.prompt2_chiby_sk|0
chiby.prompt1_chiby_sk|0
chiby.prompt3_chiby_sk|0
ifeval_sk.ifeval|0
oneruler_sk.oneruler_slovak:cwe_4096|0
oneruler_sk.oneruler_slovak:cwe_8192|0
oneruler_sk.oneruler_slovak:cwe_16384|0
oneruler_sk.oneruler_slovak:niah_multikey_4096|0
oneruler_sk.oneruler_slovak:niah_multikey_8192|0
oneruler_sk.oneruler_slovak:niah_multikey_16384|0
oneruler_sk.oneruler_slovak:niah_multiquery_4096|0
oneruler_sk.oneruler_slovak:niah_multiquery_8192|0
oneruler_sk.oneruler_slovak:niah_multiquery_16384|0
oneruler_sk.oneruler_slovak:niah_multivalue_4096|0
oneruler_sk.oneruler_slovak:niah_multivalue_8192|0
oneruler_sk.oneruler_slovak:niah_multivalue_16384|0
oneruler_sk.oneruler_slovak:niah_none_4096|0
oneruler_sk.oneruler_slovak:niah_none_8192|0
oneruler_sk.oneruler_slovak:niah_none_16384|0
oneruler_sk.oneruler_slovak:niah_single_4096|0
oneruler_sk.oneruler_slovak:niah_single_8192|0
oneruler_sk.oneruler_slovak:niah_single_16384|0
slovaksum.prompt2_summarization_sk|0
slovaksum.prompt3_summarization_sk|0
slovaksum.prompt1_summarization_sk|0
smesum.prompt2_summarization_sk|0
smesum.prompt3_summarization_sk|0
smesum.prompt1_summarization_sk|0
slovak_trivia.prompt2_mqa_single_correct_sk|0
slovak_trivia.prompt3_mqa_single_correct_sk|0
slovak_trivia.prompt1_mqa_single_correct_sk|0
analogies_sk.prompt1_analogy_sk|0
analogies_sk.prompt2_analogy_sk|0
analogies_sk.prompt3_analogy_sk|0
driving_test.prompt3_mqa_single_correct_sk|0
driving_test.prompt1_mqa_single_correct_sk|0
driving_test.prompt2_mqa_single_correct_sk|0
e_tests.prompt3_mqa_single_correct_sk:biologia|0
e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_single_correct_sk:informatika|0
e_tests.prompt3_mqa_single_correct_sk:geografia|0
e_tests.prompt3_mqa_single_correct_sk:matematika|0
e_tests.prompt3_mqa_single_correct_sk:chemia|0
e_tests.prompt3_mqa_single_correct_sk:dejepis|0
e_tests.prompt3_mqa_single_correct_sk:fyzika|0
e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_single_correct_sk:biologia|0
e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_single_correct_sk:informatika|0
e_tests.prompt1_mqa_single_correct_sk:geografia|0
e_tests.prompt1_mqa_single_correct_sk:matematika|0
e_tests.prompt1_mqa_single_correct_sk:chemia|0
e_tests.prompt1_mqa_single_correct_sk:dejepis|0
e_tests.prompt1_mqa_single_correct_sk:fyzika|0
e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_single_correct_sk:biologia|0
e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_single_correct_sk:informatika|0
e_tests.prompt2_mqa_single_correct_sk:geografia|0
e_tests.prompt2_mqa_single_correct_sk:matematika|0
e_tests.prompt2_mqa_single_correct_sk:chemia|0
e_tests.prompt2_mqa_single_correct_sk:dejepis|0
e_tests.prompt2_mqa_single_correct_sk:fyzika|0
e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
e_tests.prompt1_qa_sk:biologia|0
e_tests.prompt1_qa_sk:obcianska-nauka|0
e_tests.prompt1_qa_sk:informatika|0
e_tests.prompt1_qa_sk:geografia|0
e_tests.prompt1_qa_sk:chemia|0
e_tests.prompt1_qa_sk:dejepis|0
e_tests.prompt1_qa_sk:fyzika|0
e_tests.prompt1_qa_sk:slovensky-jazyk|0
e_tests.prompt3_qa_sk:biologia|0
e_tests.prompt3_qa_sk:obcianska-nauka|0
e_tests.prompt3_qa_sk:informatika|0
e_tests.prompt3_qa_sk:geografia|0
e_tests.prompt3_qa_sk:chemia|0
e_tests.prompt3_qa_sk:dejepis|0
e_tests.prompt3_qa_sk:fyzika|0
e_tests.prompt3_qa_sk:slovensky-jazyk|0
e_tests.prompt2_qa_sk:biologia|0
e_tests.prompt2_qa_sk:obcianska-nauka|0
e_tests.prompt2_qa_sk:informatika|0
e_tests.prompt2_qa_sk:geografia|0
e_tests.prompt2_qa_sk:chemia|0
e_tests.prompt2_qa_sk:dejepis|0
e_tests.prompt2_qa_sk:fyzika|0
e_tests.prompt2_qa_sk:slovensky-jazyk|0
slovak_tests.prompt1_qa_sk:monitor9|0
slovak_tests.prompt1_qa_sk:maturita|0
slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
slovak_tests.prompt3_qa_sk:monitor9|0
slovak_tests.prompt3_qa_sk:maturita|0
slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
slovak_tests.prompt2_qa_sk:monitor9|0
slovak_tests.prompt2_qa_sk:maturita|0
slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
slohry.prompt2_declension_sk:declension_high|0
slohry.prompt2_declension_sk:declension_mid|0
slohry.prompt2_declension_sk:declension_low|0
slohry.prompt1_declension_sk:declension_high|0
slohry.prompt1_declension_sk:declension_mid|0
slohry.prompt1_declension_sk:declension_low|0
slohry.prompt3_declension_sk:declension_high|0
slohry.prompt3_declension_sk:declension_mid|0
slohry.prompt3_declension_sk:declension_low|0
slohry.prompt3_gradation_sk:gradation_high|0
slohry.prompt3_gradation_sk:gradation_mid|0
slohry.prompt3_gradation_sk:gradation_low|0
slohry.prompt1_gradation_sk:gradation_high|0
slohry.prompt1_gradation_sk:gradation_mid|0
slohry.prompt1_gradation_sk:gradation_low|0
slohry.prompt2_gradation_sk:gradation_high|0
slohry.prompt2_gradation_sk:gradation_mid|0
slohry.prompt2_gradation_sk:gradation_low|0
slohry.prompt1_identity_form_sk:identity_form_high|0
slohry.prompt1_identity_form_sk:identity_form_mid|0
slohry.prompt1_identity_form_sk:identity_form_low|0
slohry.prompt2_identity_form_sk:identity_form_high|0
slohry.prompt2_identity_form_sk:identity_form_mid|0
slohry.prompt2_identity_form_sk:identity_form_low|0
slohry.prompt3_identity_form_sk:identity_form_high|0
slohry.prompt3_identity_form_sk:identity_form_mid|0
slohry.prompt3_identity_form_sk:identity_form_low|0
slohry.prompt2_inflection_sk:inflection_high|0
slohry.prompt2_inflection_sk:inflection_mid|0
slohry.prompt2_inflection_sk:inflection_low|0
slohry.prompt1_inflection_sk:inflection_high|0
slohry.prompt1_inflection_sk:inflection_mid|0
slohry.prompt1_inflection_sk:inflection_low|0
slohry.prompt3_inflection_sk:inflection_high|0
slohry.prompt3_inflection_sk:inflection_mid|0
slohry.prompt3_inflection_sk:inflection_low|0
scio.prompt2_mqa_single_correct_sk:aj|0
scio.prompt2_mqa_single_correct_sk:mat|0
scio.prompt2_mqa_single_correct_sk:bio|0
scio.prompt2_mqa_single_correct_sk:che|0
scio.prompt2_mqa_single_correct_sk:vsp|0
scio.prompt3_mqa_single_correct_sk:aj|0
scio.prompt3_mqa_single_correct_sk:mat|0
scio.prompt3_mqa_single_correct_sk:bio|0
scio.prompt3_mqa_single_correct_sk:che|0
scio.prompt3_mqa_single_correct_sk:vsp|0
scio.prompt1_mqa_single_correct_sk:aj|0
scio.prompt1_mqa_single_correct_sk:mat|0
scio.prompt1_mqa_single_correct_sk:bio|0
scio.prompt1_mqa_single_correct_sk:che|0
scio.prompt1_mqa_single_correct_sk:vsp|0
nli_sk.prompt1_nli_3_classes_sk|0
nli_sk.prompt2_nli_3_classes_sk|0
nli_sk.prompt3_nli_3_classes_sk|0

13475
aya_300k_lr_experiments/benchmarks/skbench/configs/task_configs.json Normal file
View File

File diff suppressed because it is too large Load Diff

222
aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt Normal file
View File

@ -0,0 +1,222 @@
arc.prompt3_mqa_single_correct_sk|0
arc.prompt1_mqa_single_correct_sk|0
arc.prompt2_mqa_single_correct_sk|0
belebele.prompt3_mqa_single_correct_sk|0
belebele.prompt1_mqa_single_correct_sk|0
belebele.prompt2_mqa_single_correct_sk|0
financial_exams.prompt1_completion_sk|0
financial_exams.prompt3_completion_sk|0
financial_exams.prompt2_completion_sk|0
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt3_mqa_single_correct_sk:anatomy|0
mmlu.prompt1_mqa_single_correct_sk:anatomy|0
mmlu.prompt2_mqa_single_correct_sk:anatomy|0
mmlu.prompt3_mqa_single_correct_sk:astronomy|0
mmlu.prompt1_mqa_single_correct_sk:astronomy|0
mmlu.prompt2_mqa_single_correct_sk:astronomy|0
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt3_mqa_single_correct_sk:college_biology|0
mmlu.prompt1_mqa_single_correct_sk:college_biology|0
mmlu.prompt2_mqa_single_correct_sk:college_biology|0
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
mmlu.prompt3_mqa_single_correct_sk:college_physics|0
mmlu.prompt1_mqa_single_correct_sk:college_physics|0
mmlu.prompt2_mqa_single_correct_sk:college_physics|0
mmlu.prompt3_mqa_single_correct_sk:computer_security|0
mmlu.prompt1_mqa_single_correct_sk:computer_security|0
mmlu.prompt2_mqa_single_correct_sk:computer_security|0
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt3_mqa_single_correct_sk:econometrics|0
mmlu.prompt1_mqa_single_correct_sk:econometrics|0
mmlu.prompt2_mqa_single_correct_sk:econometrics|0
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
mmlu.prompt3_mqa_single_correct_sk:global_facts|0
mmlu.prompt1_mqa_single_correct_sk:global_facts|0
mmlu.prompt2_mqa_single_correct_sk:global_facts|0
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt3_mqa_single_correct_sk:human_aging|0
mmlu.prompt1_mqa_single_correct_sk:human_aging|0
mmlu.prompt2_mqa_single_correct_sk:human_aging|0
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt3_mqa_single_correct_sk:international_law|0
mmlu.prompt1_mqa_single_correct_sk:international_law|0
mmlu.prompt2_mqa_single_correct_sk:international_law|0
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
mmlu.prompt3_mqa_single_correct_sk:management|0
mmlu.prompt1_mqa_single_correct_sk:management|0
mmlu.prompt2_mqa_single_correct_sk:management|0
mmlu.prompt3_mqa_single_correct_sk:marketing|0
mmlu.prompt1_mqa_single_correct_sk:marketing|0
mmlu.prompt2_mqa_single_correct_sk:marketing|0
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt3_mqa_single_correct_sk:nutrition|0
mmlu.prompt1_mqa_single_correct_sk:nutrition|0
mmlu.prompt2_mqa_single_correct_sk:nutrition|0
mmlu.prompt3_mqa_single_correct_sk:philosophy|0
mmlu.prompt1_mqa_single_correct_sk:philosophy|0
mmlu.prompt2_mqa_single_correct_sk:philosophy|0
mmlu.prompt3_mqa_single_correct_sk:prehistory|0
mmlu.prompt1_mqa_single_correct_sk:prehistory|0
mmlu.prompt2_mqa_single_correct_sk:prehistory|0
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt3_mqa_single_correct_sk:professional_law|0
mmlu.prompt1_mqa_single_correct_sk:professional_law|0
mmlu.prompt2_mqa_single_correct_sk:professional_law|0
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt3_mqa_single_correct_sk:public_relations|0
mmlu.prompt1_mqa_single_correct_sk:public_relations|0
mmlu.prompt2_mqa_single_correct_sk:public_relations|0
mmlu.prompt3_mqa_single_correct_sk:security_studies|0
mmlu.prompt1_mqa_single_correct_sk:security_studies|0
mmlu.prompt2_mqa_single_correct_sk:security_studies|0
mmlu.prompt3_mqa_single_correct_sk:sociology|0
mmlu.prompt1_mqa_single_correct_sk:sociology|0
mmlu.prompt2_mqa_single_correct_sk:sociology|0
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt3_mqa_single_correct_sk:virology|0
mmlu.prompt1_mqa_single_correct_sk:virology|0
mmlu.prompt2_mqa_single_correct_sk:virology|0
mmlu.prompt3_mqa_single_correct_sk:world_religions|0
mmlu.prompt1_mqa_single_correct_sk:world_religions|0
mmlu.prompt2_mqa_single_correct_sk:world_religions|0
multi_wiki_qa.prompt3_qa_sk|0
multi_wiki_qa.prompt2_qa_sk|0
multi_wiki_qa.prompt1_qa_sk|0
senti_sk.prompt2_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_binary_sk|0
senti_sk.prompt1_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_3_class_sk|0
senti_sk.prompt1_sentiment_3_class_sk|0
senti_sk.prompt2_sentiment_3_class_sk|0
skj1.prompt3_skj1_sk|0
skj1.prompt1_skj1_sk|0
skj1.prompt2_skj1_sk|0
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_hate_speech_sk:hate-speech|0
sklep.prompt3_hate_speech_sk:hate-speech|0
sklep.prompt2_hate_speech_sk:hate-speech|0
sklep.prompt1_nli_3_classes_sk:nli|0
sklep.prompt2_nli_3_classes_sk:nli|0
sklep.prompt3_nli_3_classes_sk:nli|0
sklep.prompt1_nli_binary_sk:rte|0
sklep.prompt3_nli_binary_sk:rte|0
sklep.prompt2_nli_binary_sk:rte|0
sklep.prompt3_qa_sk:question-answering|0
sklep.prompt2_qa_sk:question-answering|0
sklep.prompt1_qa_sk:question-answering|0
toxic_sk.prompt2_toxicity_sk|0
toxic_sk.prompt3_toxicity_sk|0
toxic_sk.prompt1_toxicity_sk|0
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
slovakcopa.prompt3_copa_sk|0
slovakcopa.prompt1_copa_sk|0
slovakcopa.prompt2_copa_sk|0
movie_review.prompt3_sentiment_binary_sk|0
movie_review.prompt2_sentiment_binary_sk|0
movie_review.prompt1_sentiment_binary_sk|0
globalmmlu-lite.prompt3_mqa_single_correct_sk|0
globalmmlu-lite.prompt1_mqa_single_correct_sk|0
globalmmlu-lite.prompt2_mqa_single_correct_sk|0

412
aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero.txt Normal file
View File

@ -0,0 +1,412 @@
arc.prompt3_mqa_single_correct_sk|0
arc.prompt1_mqa_single_correct_sk|0
arc.prompt2_mqa_single_correct_sk|0
belebele.prompt3_mqa_single_correct_sk|0
belebele.prompt1_mqa_single_correct_sk|0
belebele.prompt2_mqa_single_correct_sk|0
chiby.prompt2_chiby_sk|0
chiby.prompt1_chiby_sk|0
chiby.prompt3_chiby_sk|0
financial_exams.prompt1_completion_sk|0
financial_exams.prompt3_completion_sk|0
financial_exams.prompt2_completion_sk|0
ifeval_sk.ifeval|0
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt3_mqa_single_correct_sk:anatomy|0
mmlu.prompt1_mqa_single_correct_sk:anatomy|0
mmlu.prompt2_mqa_single_correct_sk:anatomy|0
mmlu.prompt3_mqa_single_correct_sk:astronomy|0
mmlu.prompt1_mqa_single_correct_sk:astronomy|0
mmlu.prompt2_mqa_single_correct_sk:astronomy|0
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt3_mqa_single_correct_sk:college_biology|0
mmlu.prompt1_mqa_single_correct_sk:college_biology|0
mmlu.prompt2_mqa_single_correct_sk:college_biology|0
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
mmlu.prompt3_mqa_single_correct_sk:college_physics|0
mmlu.prompt1_mqa_single_correct_sk:college_physics|0
mmlu.prompt2_mqa_single_correct_sk:college_physics|0
mmlu.prompt3_mqa_single_correct_sk:computer_security|0
mmlu.prompt1_mqa_single_correct_sk:computer_security|0
mmlu.prompt2_mqa_single_correct_sk:computer_security|0
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt3_mqa_single_correct_sk:econometrics|0
mmlu.prompt1_mqa_single_correct_sk:econometrics|0
mmlu.prompt2_mqa_single_correct_sk:econometrics|0
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
mmlu.prompt3_mqa_single_correct_sk:global_facts|0
mmlu.prompt1_mqa_single_correct_sk:global_facts|0
mmlu.prompt2_mqa_single_correct_sk:global_facts|0
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt3_mqa_single_correct_sk:human_aging|0
mmlu.prompt1_mqa_single_correct_sk:human_aging|0
mmlu.prompt2_mqa_single_correct_sk:human_aging|0
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt3_mqa_single_correct_sk:international_law|0
mmlu.prompt1_mqa_single_correct_sk:international_law|0
mmlu.prompt2_mqa_single_correct_sk:international_law|0
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
mmlu.prompt3_mqa_single_correct_sk:management|0
mmlu.prompt1_mqa_single_correct_sk:management|0
mmlu.prompt2_mqa_single_correct_sk:management|0
mmlu.prompt3_mqa_single_correct_sk:marketing|0
mmlu.prompt1_mqa_single_correct_sk:marketing|0
mmlu.prompt2_mqa_single_correct_sk:marketing|0
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt3_mqa_single_correct_sk:nutrition|0
mmlu.prompt1_mqa_single_correct_sk:nutrition|0
mmlu.prompt2_mqa_single_correct_sk:nutrition|0
mmlu.prompt3_mqa_single_correct_sk:philosophy|0
mmlu.prompt1_mqa_single_correct_sk:philosophy|0
mmlu.prompt2_mqa_single_correct_sk:philosophy|0
mmlu.prompt3_mqa_single_correct_sk:prehistory|0
mmlu.prompt1_mqa_single_correct_sk:prehistory|0
mmlu.prompt2_mqa_single_correct_sk:prehistory|0
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt3_mqa_single_correct_sk:professional_law|0
mmlu.prompt1_mqa_single_correct_sk:professional_law|0
mmlu.prompt2_mqa_single_correct_sk:professional_law|0
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt3_mqa_single_correct_sk:public_relations|0
mmlu.prompt1_mqa_single_correct_sk:public_relations|0
mmlu.prompt2_mqa_single_correct_sk:public_relations|0
mmlu.prompt3_mqa_single_correct_sk:security_studies|0
mmlu.prompt1_mqa_single_correct_sk:security_studies|0
mmlu.prompt2_mqa_single_correct_sk:security_studies|0
mmlu.prompt3_mqa_single_correct_sk:sociology|0
mmlu.prompt1_mqa_single_correct_sk:sociology|0
mmlu.prompt2_mqa_single_correct_sk:sociology|0
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt3_mqa_single_correct_sk:virology|0
mmlu.prompt1_mqa_single_correct_sk:virology|0
mmlu.prompt2_mqa_single_correct_sk:virology|0
mmlu.prompt3_mqa_single_correct_sk:world_religions|0
mmlu.prompt1_mqa_single_correct_sk:world_religions|0
mmlu.prompt2_mqa_single_correct_sk:world_religions|0
multi_wiki_qa.prompt3_qa_sk|0
multi_wiki_qa.prompt2_qa_sk|0
multi_wiki_qa.prompt1_qa_sk|0
nli_sk.prompt1_nli_3_classes_sk|0
nli_sk.prompt2_nli_3_classes_sk|0
nli_sk.prompt3_nli_3_classes_sk|0
oneruler_sk.oneruler_slovak:cwe_4096|0
oneruler_sk.oneruler_slovak:cwe_8192|0
oneruler_sk.oneruler_slovak:cwe_16384|0
oneruler_sk.oneruler_slovak:niah_multikey_4096|0
oneruler_sk.oneruler_slovak:niah_multikey_8192|0
oneruler_sk.oneruler_slovak:niah_multikey_16384|0
oneruler_sk.oneruler_slovak:niah_multiquery_4096|0
oneruler_sk.oneruler_slovak:niah_multiquery_8192|0
oneruler_sk.oneruler_slovak:niah_multiquery_16384|0
oneruler_sk.oneruler_slovak:niah_multivalue_4096|0
oneruler_sk.oneruler_slovak:niah_multivalue_8192|0
oneruler_sk.oneruler_slovak:niah_multivalue_16384|0
oneruler_sk.oneruler_slovak:niah_none_4096|0
oneruler_sk.oneruler_slovak:niah_none_8192|0
oneruler_sk.oneruler_slovak:niah_none_16384|0
oneruler_sk.oneruler_slovak:niah_single_4096|0
oneruler_sk.oneruler_slovak:niah_single_8192|0
oneruler_sk.oneruler_slovak:niah_single_16384|0
senti_sk.prompt2_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_binary_sk|0
senti_sk.prompt1_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_3_class_sk|0
senti_sk.prompt1_sentiment_3_class_sk|0
senti_sk.prompt2_sentiment_3_class_sk|0
skj1.prompt3_skj1_sk|0
skj1.prompt1_skj1_sk|0
skj1.prompt2_skj1_sk|0
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_hate_speech_sk:hate-speech|0
sklep.prompt3_hate_speech_sk:hate-speech|0
sklep.prompt2_hate_speech_sk:hate-speech|0
sklep.prompt1_nli_3_classes_sk:nli|0
sklep.prompt2_nli_3_classes_sk:nli|0
sklep.prompt3_nli_3_classes_sk:nli|0
sklep.prompt1_nli_binary_sk:rte|0
sklep.prompt3_nli_binary_sk:rte|0
sklep.prompt2_nli_binary_sk:rte|0
sklep.prompt3_qa_sk:question-answering|0
sklep.prompt2_qa_sk:question-answering|0
sklep.prompt1_qa_sk:question-answering|0
toxic_sk.prompt2_toxicity_sk|0
toxic_sk.prompt3_toxicity_sk|0
toxic_sk.prompt1_toxicity_sk|0
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
slovakcopa.prompt3_copa_sk|0
slovakcopa.prompt1_copa_sk|0
slovakcopa.prompt2_copa_sk|0
slovaksum.prompt2_summarization_sk|0
slovaksum.prompt3_summarization_sk|0
slovaksum.prompt1_summarization_sk|0
smesum.prompt2_summarization_sk|0
smesum.prompt3_summarization_sk|0
smesum.prompt1_summarization_sk|0
movie_review.prompt3_sentiment_binary_sk|0
movie_review.prompt2_sentiment_binary_sk|0
movie_review.prompt1_sentiment_binary_sk|0
slovak_trivia.prompt2_mqa_single_correct_sk|0
slovak_trivia.prompt3_mqa_single_correct_sk|0
slovak_trivia.prompt1_mqa_single_correct_sk|0
analogies_sk.prompt1_analogy_sk|0
analogies_sk.prompt2_analogy_sk|0
analogies_sk.prompt3_analogy_sk|0
driving_test.prompt3_mqa_single_correct_sk|0
driving_test.prompt1_mqa_single_correct_sk|0
driving_test.prompt2_mqa_single_correct_sk|0
e_tests.prompt3_mqa_single_correct_sk:biologia|0
e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_single_correct_sk:informatika|0
e_tests.prompt3_mqa_single_correct_sk:geografia|0
e_tests.prompt3_mqa_single_correct_sk:matematika|0
e_tests.prompt3_mqa_single_correct_sk:chemia|0
e_tests.prompt3_mqa_single_correct_sk:dejepis|0
e_tests.prompt3_mqa_single_correct_sk:fyzika|0
e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_single_correct_sk:biologia|0
e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_single_correct_sk:informatika|0
e_tests.prompt1_mqa_single_correct_sk:geografia|0
e_tests.prompt1_mqa_single_correct_sk:matematika|0
e_tests.prompt1_mqa_single_correct_sk:chemia|0
e_tests.prompt1_mqa_single_correct_sk:dejepis|0
e_tests.prompt1_mqa_single_correct_sk:fyzika|0
e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_single_correct_sk:biologia|0
e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_single_correct_sk:informatika|0
e_tests.prompt2_mqa_single_correct_sk:geografia|0
e_tests.prompt2_mqa_single_correct_sk:matematika|0
e_tests.prompt2_mqa_single_correct_sk:chemia|0
e_tests.prompt2_mqa_single_correct_sk:dejepis|0
e_tests.prompt2_mqa_single_correct_sk:fyzika|0
e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
e_tests.prompt1_qa_sk:biologia|0
e_tests.prompt1_qa_sk:obcianska-nauka|0
e_tests.prompt1_qa_sk:informatika|0
e_tests.prompt1_qa_sk:geografia|0
e_tests.prompt1_qa_sk:chemia|0
e_tests.prompt1_qa_sk:dejepis|0
e_tests.prompt1_qa_sk:fyzika|0
e_tests.prompt1_qa_sk:slovensky-jazyk|0
e_tests.prompt3_qa_sk:biologia|0
e_tests.prompt3_qa_sk:obcianska-nauka|0
e_tests.prompt3_qa_sk:informatika|0
e_tests.prompt3_qa_sk:geografia|0
e_tests.prompt3_qa_sk:chemia|0
e_tests.prompt3_qa_sk:dejepis|0
e_tests.prompt3_qa_sk:fyzika|0
e_tests.prompt3_qa_sk:slovensky-jazyk|0
e_tests.prompt2_qa_sk:biologia|0
e_tests.prompt2_qa_sk:obcianska-nauka|0
e_tests.prompt2_qa_sk:informatika|0
e_tests.prompt2_qa_sk:geografia|0
e_tests.prompt2_qa_sk:chemia|0
e_tests.prompt2_qa_sk:dejepis|0
e_tests.prompt2_qa_sk:fyzika|0
e_tests.prompt2_qa_sk:slovensky-jazyk|0
slovak_tests.prompt1_qa_sk:monitor9|0
slovak_tests.prompt1_qa_sk:maturita|0
slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
slovak_tests.prompt3_qa_sk:monitor9|0
slovak_tests.prompt3_qa_sk:maturita|0
slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
slovak_tests.prompt2_qa_sk:monitor9|0
slovak_tests.prompt2_qa_sk:maturita|0
slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
slohry.prompt2_declension_sk:declension_high|0
slohry.prompt2_declension_sk:declension_mid|0
slohry.prompt2_declension_sk:declension_low|0
slohry.prompt1_declension_sk:declension_high|0
slohry.prompt1_declension_sk:declension_mid|0
slohry.prompt1_declension_sk:declension_low|0
slohry.prompt3_declension_sk:declension_high|0
slohry.prompt3_declension_sk:declension_mid|0
slohry.prompt3_declension_sk:declension_low|0
slohry.prompt3_gradation_sk:gradation_high|0
slohry.prompt3_gradation_sk:gradation_mid|0
slohry.prompt3_gradation_sk:gradation_low|0
slohry.prompt1_gradation_sk:gradation_high|0
slohry.prompt1_gradation_sk:gradation_mid|0
slohry.prompt1_gradation_sk:gradation_low|0
slohry.prompt2_gradation_sk:gradation_high|0
slohry.prompt2_gradation_sk:gradation_mid|0
slohry.prompt2_gradation_sk:gradation_low|0
slohry.prompt1_identity_form_sk:identity_form_high|0
slohry.prompt1_identity_form_sk:identity_form_mid|0
slohry.prompt1_identity_form_sk:identity_form_low|0
slohry.prompt2_identity_form_sk:identity_form_high|0
slohry.prompt2_identity_form_sk:identity_form_mid|0
slohry.prompt2_identity_form_sk:identity_form_low|0
slohry.prompt3_identity_form_sk:identity_form_high|0
slohry.prompt3_identity_form_sk:identity_form_mid|0
slohry.prompt3_identity_form_sk:identity_form_low|0
slohry.prompt2_inflection_sk:inflection_high|0
slohry.prompt2_inflection_sk:inflection_mid|0
slohry.prompt2_inflection_sk:inflection_low|0
slohry.prompt1_inflection_sk:inflection_high|0
slohry.prompt1_inflection_sk:inflection_mid|0
slohry.prompt1_inflection_sk:inflection_low|0
slohry.prompt3_inflection_sk:inflection_high|0
slohry.prompt3_inflection_sk:inflection_mid|0
slohry.prompt3_inflection_sk:inflection_low|0
scio.prompt2_mqa_single_correct_sk:aj|0
scio.prompt2_mqa_single_correct_sk:mat|0
scio.prompt2_mqa_single_correct_sk:bio|0
scio.prompt2_mqa_single_correct_sk:che|0
scio.prompt2_mqa_single_correct_sk:vsp|0
scio.prompt3_mqa_single_correct_sk:aj|0
scio.prompt3_mqa_single_correct_sk:mat|0
scio.prompt3_mqa_single_correct_sk:bio|0
scio.prompt3_mqa_single_correct_sk:che|0
scio.prompt3_mqa_single_correct_sk:vsp|0
scio.prompt1_mqa_single_correct_sk:aj|0
scio.prompt1_mqa_single_correct_sk:mat|0
scio.prompt1_mqa_single_correct_sk:bio|0
scio.prompt1_mqa_single_correct_sk:che|0
scio.prompt1_mqa_single_correct_sk:vsp|0
globalmmlu-lite.prompt3_mqa_single_correct_sk|0
globalmmlu-lite.prompt1_mqa_single_correct_sk|0
globalmmlu-lite.prompt2_mqa_single_correct_sk|0

225
aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available.txt Normal file
View File

@ -0,0 +1,225 @@
arc.prompt3_mqa_single_correct_sk|0
arc.prompt1_mqa_single_correct_sk|0
arc.prompt2_mqa_single_correct_sk|0
belebele.prompt3_mqa_single_correct_sk|0
belebele.prompt1_mqa_single_correct_sk|0
belebele.prompt2_mqa_single_correct_sk|0
financial_exams.prompt1_completion_sk|0
financial_exams.prompt3_completion_sk|0
financial_exams.prompt2_completion_sk|0
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt3_mqa_single_correct_sk:anatomy|0
mmlu.prompt1_mqa_single_correct_sk:anatomy|0
mmlu.prompt2_mqa_single_correct_sk:anatomy|0
mmlu.prompt3_mqa_single_correct_sk:astronomy|0
mmlu.prompt1_mqa_single_correct_sk:astronomy|0
mmlu.prompt2_mqa_single_correct_sk:astronomy|0
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt3_mqa_single_correct_sk:college_biology|0
mmlu.prompt1_mqa_single_correct_sk:college_biology|0
mmlu.prompt2_mqa_single_correct_sk:college_biology|0
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
mmlu.prompt3_mqa_single_correct_sk:college_physics|0
mmlu.prompt1_mqa_single_correct_sk:college_physics|0
mmlu.prompt2_mqa_single_correct_sk:college_physics|0
mmlu.prompt3_mqa_single_correct_sk:computer_security|0
mmlu.prompt1_mqa_single_correct_sk:computer_security|0
mmlu.prompt2_mqa_single_correct_sk:computer_security|0
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt3_mqa_single_correct_sk:econometrics|0
mmlu.prompt1_mqa_single_correct_sk:econometrics|0
mmlu.prompt2_mqa_single_correct_sk:econometrics|0
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
mmlu.prompt3_mqa_single_correct_sk:global_facts|0
mmlu.prompt1_mqa_single_correct_sk:global_facts|0
mmlu.prompt2_mqa_single_correct_sk:global_facts|0
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt3_mqa_single_correct_sk:human_aging|0
mmlu.prompt1_mqa_single_correct_sk:human_aging|0
mmlu.prompt2_mqa_single_correct_sk:human_aging|0
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt3_mqa_single_correct_sk:international_law|0
mmlu.prompt1_mqa_single_correct_sk:international_law|0
mmlu.prompt2_mqa_single_correct_sk:international_law|0
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
mmlu.prompt3_mqa_single_correct_sk:management|0
mmlu.prompt1_mqa_single_correct_sk:management|0
mmlu.prompt2_mqa_single_correct_sk:management|0
mmlu.prompt3_mqa_single_correct_sk:marketing|0
mmlu.prompt1_mqa_single_correct_sk:marketing|0
mmlu.prompt2_mqa_single_correct_sk:marketing|0
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt3_mqa_single_correct_sk:nutrition|0
mmlu.prompt1_mqa_single_correct_sk:nutrition|0
mmlu.prompt2_mqa_single_correct_sk:nutrition|0
mmlu.prompt3_mqa_single_correct_sk:philosophy|0
mmlu.prompt1_mqa_single_correct_sk:philosophy|0
mmlu.prompt2_mqa_single_correct_sk:philosophy|0
mmlu.prompt3_mqa_single_correct_sk:prehistory|0
mmlu.prompt1_mqa_single_correct_sk:prehistory|0
mmlu.prompt2_mqa_single_correct_sk:prehistory|0
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt3_mqa_single_correct_sk:professional_law|0
mmlu.prompt1_mqa_single_correct_sk:professional_law|0
mmlu.prompt2_mqa_single_correct_sk:professional_law|0
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt3_mqa_single_correct_sk:public_relations|0
mmlu.prompt1_mqa_single_correct_sk:public_relations|0
mmlu.prompt2_mqa_single_correct_sk:public_relations|0
mmlu.prompt3_mqa_single_correct_sk:security_studies|0
mmlu.prompt1_mqa_single_correct_sk:security_studies|0
mmlu.prompt2_mqa_single_correct_sk:security_studies|0
mmlu.prompt3_mqa_single_correct_sk:sociology|0
mmlu.prompt1_mqa_single_correct_sk:sociology|0
mmlu.prompt2_mqa_single_correct_sk:sociology|0
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt3_mqa_single_correct_sk:virology|0
mmlu.prompt1_mqa_single_correct_sk:virology|0
mmlu.prompt2_mqa_single_correct_sk:virology|0
mmlu.prompt3_mqa_single_correct_sk:world_religions|0
mmlu.prompt1_mqa_single_correct_sk:world_religions|0
mmlu.prompt2_mqa_single_correct_sk:world_religions|0
multi_wiki_qa.prompt3_qa_sk|0
multi_wiki_qa.prompt2_qa_sk|0
multi_wiki_qa.prompt1_qa_sk|0
nli_sk.prompt1_nli_3_classes_sk|0
nli_sk.prompt2_nli_3_classes_sk|0
nli_sk.prompt3_nli_3_classes_sk|0
senti_sk.prompt2_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_binary_sk|0
senti_sk.prompt1_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_3_class_sk|0
senti_sk.prompt1_sentiment_3_class_sk|0
senti_sk.prompt2_sentiment_3_class_sk|0
skj1.prompt3_skj1_sk|0
skj1.prompt1_skj1_sk|0
skj1.prompt2_skj1_sk|0
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_hate_speech_sk:hate-speech|0
sklep.prompt3_hate_speech_sk:hate-speech|0
sklep.prompt2_hate_speech_sk:hate-speech|0
sklep.prompt1_nli_3_classes_sk:nli|0
sklep.prompt2_nli_3_classes_sk:nli|0
sklep.prompt3_nli_3_classes_sk:nli|0
sklep.prompt1_nli_binary_sk:rte|0
sklep.prompt3_nli_binary_sk:rte|0
sklep.prompt2_nli_binary_sk:rte|0
sklep.prompt3_qa_sk:question-answering|0
sklep.prompt2_qa_sk:question-answering|0
sklep.prompt1_qa_sk:question-answering|0
toxic_sk.prompt2_toxicity_sk|0
toxic_sk.prompt3_toxicity_sk|0
toxic_sk.prompt1_toxicity_sk|0
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
slovakcopa.prompt3_copa_sk|0
slovakcopa.prompt1_copa_sk|0
slovakcopa.prompt2_copa_sk|0
movie_review.prompt3_sentiment_binary_sk|0
movie_review.prompt2_sentiment_binary_sk|0
movie_review.prompt1_sentiment_binary_sk|0
globalmmlu-lite.prompt3_mqa_single_correct_sk|0
globalmmlu-lite.prompt1_mqa_single_correct_sk|0
globalmmlu-lite.prompt2_mqa_single_correct_sk|0

222
aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available_without_nli.txt Normal file
View File

@ -0,0 +1,222 @@
arc.prompt3_mqa_single_correct_sk|0
arc.prompt1_mqa_single_correct_sk|0
arc.prompt2_mqa_single_correct_sk|0
belebele.prompt3_mqa_single_correct_sk|0
belebele.prompt1_mqa_single_correct_sk|0
belebele.prompt2_mqa_single_correct_sk|0
financial_exams.prompt1_completion_sk|0
financial_exams.prompt3_completion_sk|0
financial_exams.prompt2_completion_sk|0
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
mmlu.prompt3_mqa_single_correct_sk:anatomy|0
mmlu.prompt1_mqa_single_correct_sk:anatomy|0
mmlu.prompt2_mqa_single_correct_sk:anatomy|0
mmlu.prompt3_mqa_single_correct_sk:astronomy|0
mmlu.prompt1_mqa_single_correct_sk:astronomy|0
mmlu.prompt2_mqa_single_correct_sk:astronomy|0
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
mmlu.prompt3_mqa_single_correct_sk:college_biology|0
mmlu.prompt1_mqa_single_correct_sk:college_biology|0
mmlu.prompt2_mqa_single_correct_sk:college_biology|0
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
mmlu.prompt3_mqa_single_correct_sk:college_physics|0
mmlu.prompt1_mqa_single_correct_sk:college_physics|0
mmlu.prompt2_mqa_single_correct_sk:college_physics|0
mmlu.prompt3_mqa_single_correct_sk:computer_security|0
mmlu.prompt1_mqa_single_correct_sk:computer_security|0
mmlu.prompt2_mqa_single_correct_sk:computer_security|0
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
mmlu.prompt3_mqa_single_correct_sk:econometrics|0
mmlu.prompt1_mqa_single_correct_sk:econometrics|0
mmlu.prompt2_mqa_single_correct_sk:econometrics|0
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
mmlu.prompt3_mqa_single_correct_sk:global_facts|0
mmlu.prompt1_mqa_single_correct_sk:global_facts|0
mmlu.prompt2_mqa_single_correct_sk:global_facts|0
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
mmlu.prompt3_mqa_single_correct_sk:human_aging|0
mmlu.prompt1_mqa_single_correct_sk:human_aging|0
mmlu.prompt2_mqa_single_correct_sk:human_aging|0
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
mmlu.prompt3_mqa_single_correct_sk:international_law|0
mmlu.prompt1_mqa_single_correct_sk:international_law|0
mmlu.prompt2_mqa_single_correct_sk:international_law|0
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
mmlu.prompt3_mqa_single_correct_sk:management|0
mmlu.prompt1_mqa_single_correct_sk:management|0
mmlu.prompt2_mqa_single_correct_sk:management|0
mmlu.prompt3_mqa_single_correct_sk:marketing|0
mmlu.prompt1_mqa_single_correct_sk:marketing|0
mmlu.prompt2_mqa_single_correct_sk:marketing|0
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
mmlu.prompt3_mqa_single_correct_sk:nutrition|0
mmlu.prompt1_mqa_single_correct_sk:nutrition|0
mmlu.prompt2_mqa_single_correct_sk:nutrition|0
mmlu.prompt3_mqa_single_correct_sk:philosophy|0
mmlu.prompt1_mqa_single_correct_sk:philosophy|0
mmlu.prompt2_mqa_single_correct_sk:philosophy|0
mmlu.prompt3_mqa_single_correct_sk:prehistory|0
mmlu.prompt1_mqa_single_correct_sk:prehistory|0
mmlu.prompt2_mqa_single_correct_sk:prehistory|0
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
mmlu.prompt3_mqa_single_correct_sk:professional_law|0
mmlu.prompt1_mqa_single_correct_sk:professional_law|0
mmlu.prompt2_mqa_single_correct_sk:professional_law|0
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
mmlu.prompt3_mqa_single_correct_sk:public_relations|0
mmlu.prompt1_mqa_single_correct_sk:public_relations|0
mmlu.prompt2_mqa_single_correct_sk:public_relations|0
mmlu.prompt3_mqa_single_correct_sk:security_studies|0
mmlu.prompt1_mqa_single_correct_sk:security_studies|0
mmlu.prompt2_mqa_single_correct_sk:security_studies|0
mmlu.prompt3_mqa_single_correct_sk:sociology|0
mmlu.prompt1_mqa_single_correct_sk:sociology|0
mmlu.prompt2_mqa_single_correct_sk:sociology|0
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
mmlu.prompt3_mqa_single_correct_sk:virology|0
mmlu.prompt1_mqa_single_correct_sk:virology|0
mmlu.prompt2_mqa_single_correct_sk:virology|0
mmlu.prompt3_mqa_single_correct_sk:world_religions|0
mmlu.prompt1_mqa_single_correct_sk:world_religions|0
mmlu.prompt2_mqa_single_correct_sk:world_religions|0
multi_wiki_qa.prompt3_qa_sk|0
multi_wiki_qa.prompt2_qa_sk|0
multi_wiki_qa.prompt1_qa_sk|0
senti_sk.prompt2_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_binary_sk|0
senti_sk.prompt1_sentiment_binary_sk|0
senti_sk.prompt3_sentiment_3_class_sk|0
senti_sk.prompt1_sentiment_3_class_sk|0
senti_sk.prompt2_sentiment_3_class_sk|0
skj1.prompt3_skj1_sk|0
skj1.prompt1_skj1_sk|0
skj1.prompt2_skj1_sk|0
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
sklep.prompt1_hate_speech_sk:hate-speech|0
sklep.prompt3_hate_speech_sk:hate-speech|0
sklep.prompt2_hate_speech_sk:hate-speech|0
sklep.prompt1_nli_3_classes_sk:nli|0
sklep.prompt2_nli_3_classes_sk:nli|0
sklep.prompt3_nli_3_classes_sk:nli|0
sklep.prompt1_nli_binary_sk:rte|0
sklep.prompt3_nli_binary_sk:rte|0
sklep.prompt2_nli_binary_sk:rte|0
sklep.prompt3_qa_sk:question-answering|0
sklep.prompt2_qa_sk:question-answering|0
sklep.prompt1_qa_sk:question-answering|0
toxic_sk.prompt2_toxicity_sk|0
toxic_sk.prompt3_toxicity_sk|0
toxic_sk.prompt1_toxicity_sk|0
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
slovakcopa.prompt3_copa_sk|0
slovakcopa.prompt1_copa_sk|0
slovakcopa.prompt2_copa_sk|0
movie_review.prompt3_sentiment_binary_sk|0
movie_review.prompt2_sentiment_binary_sk|0
movie_review.prompt1_sentiment_binary_sk|0
globalmmlu-lite.prompt3_mqa_single_correct_sk|0
globalmmlu-lite.prompt1_mqa_single_correct_sk|0
globalmmlu-lite.prompt2_mqa_single_correct_sk|0

187
aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_unavailable.txt Normal file
View File

@ -0,0 +1,187 @@
chiby.prompt2_chiby_sk|0
chiby.prompt1_chiby_sk|0
chiby.prompt3_chiby_sk|0
ifeval_sk.ifeval|0
oneruler_sk.oneruler_slovak:cwe_4096|0
oneruler_sk.oneruler_slovak:cwe_8192|0
oneruler_sk.oneruler_slovak:cwe_16384|0
oneruler_sk.oneruler_slovak:niah_multikey_4096|0
oneruler_sk.oneruler_slovak:niah_multikey_8192|0
oneruler_sk.oneruler_slovak:niah_multikey_16384|0
oneruler_sk.oneruler_slovak:niah_multiquery_4096|0
oneruler_sk.oneruler_slovak:niah_multiquery_8192|0
oneruler_sk.oneruler_slovak:niah_multiquery_16384|0
oneruler_sk.oneruler_slovak:niah_multivalue_4096|0
oneruler_sk.oneruler_slovak:niah_multivalue_8192|0
oneruler_sk.oneruler_slovak:niah_multivalue_16384|0
oneruler_sk.oneruler_slovak:niah_none_4096|0
oneruler_sk.oneruler_slovak:niah_none_8192|0
oneruler_sk.oneruler_slovak:niah_none_16384|0
oneruler_sk.oneruler_slovak:niah_single_4096|0
oneruler_sk.oneruler_slovak:niah_single_8192|0
oneruler_sk.oneruler_slovak:niah_single_16384|0
slovaksum.prompt2_summarization_sk|0
slovaksum.prompt3_summarization_sk|0
slovaksum.prompt1_summarization_sk|0
smesum.prompt2_summarization_sk|0
smesum.prompt3_summarization_sk|0
smesum.prompt1_summarization_sk|0
slovak_trivia.prompt2_mqa_single_correct_sk|0
slovak_trivia.prompt3_mqa_single_correct_sk|0
slovak_trivia.prompt1_mqa_single_correct_sk|0
analogies_sk.prompt1_analogy_sk|0
analogies_sk.prompt2_analogy_sk|0
analogies_sk.prompt3_analogy_sk|0
driving_test.prompt3_mqa_single_correct_sk|0
driving_test.prompt1_mqa_single_correct_sk|0
driving_test.prompt2_mqa_single_correct_sk|0
e_tests.prompt3_mqa_single_correct_sk:biologia|0
e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_single_correct_sk:informatika|0
e_tests.prompt3_mqa_single_correct_sk:geografia|0
e_tests.prompt3_mqa_single_correct_sk:matematika|0
e_tests.prompt3_mqa_single_correct_sk:chemia|0
e_tests.prompt3_mqa_single_correct_sk:dejepis|0
e_tests.prompt3_mqa_single_correct_sk:fyzika|0
e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_single_correct_sk:biologia|0
e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_single_correct_sk:informatika|0
e_tests.prompt1_mqa_single_correct_sk:geografia|0
e_tests.prompt1_mqa_single_correct_sk:matematika|0
e_tests.prompt1_mqa_single_correct_sk:chemia|0
e_tests.prompt1_mqa_single_correct_sk:dejepis|0
e_tests.prompt1_mqa_single_correct_sk:fyzika|0
e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_single_correct_sk:biologia|0
e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_single_correct_sk:informatika|0
e_tests.prompt2_mqa_single_correct_sk:geografia|0
e_tests.prompt2_mqa_single_correct_sk:matematika|0
e_tests.prompt2_mqa_single_correct_sk:chemia|0
e_tests.prompt2_mqa_single_correct_sk:dejepis|0
e_tests.prompt2_mqa_single_correct_sk:fyzika|0
e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
e_tests.prompt1_qa_sk:biologia|0
e_tests.prompt1_qa_sk:obcianska-nauka|0
e_tests.prompt1_qa_sk:informatika|0
e_tests.prompt1_qa_sk:geografia|0
e_tests.prompt1_qa_sk:chemia|0
e_tests.prompt1_qa_sk:dejepis|0
e_tests.prompt1_qa_sk:fyzika|0
e_tests.prompt1_qa_sk:slovensky-jazyk|0
e_tests.prompt3_qa_sk:biologia|0
e_tests.prompt3_qa_sk:obcianska-nauka|0
e_tests.prompt3_qa_sk:informatika|0
e_tests.prompt3_qa_sk:geografia|0
e_tests.prompt3_qa_sk:chemia|0
e_tests.prompt3_qa_sk:dejepis|0
e_tests.prompt3_qa_sk:fyzika|0
e_tests.prompt3_qa_sk:slovensky-jazyk|0
e_tests.prompt2_qa_sk:biologia|0
e_tests.prompt2_qa_sk:obcianska-nauka|0
e_tests.prompt2_qa_sk:informatika|0
e_tests.prompt2_qa_sk:geografia|0
e_tests.prompt2_qa_sk:chemia|0
e_tests.prompt2_qa_sk:dejepis|0
e_tests.prompt2_qa_sk:fyzika|0
e_tests.prompt2_qa_sk:slovensky-jazyk|0
slovak_tests.prompt1_qa_sk:monitor9|0
slovak_tests.prompt1_qa_sk:maturita|0
slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
slovak_tests.prompt3_qa_sk:monitor9|0
slovak_tests.prompt3_qa_sk:maturita|0
slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
slovak_tests.prompt2_qa_sk:monitor9|0
slovak_tests.prompt2_qa_sk:maturita|0
slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
slohry.prompt2_declension_sk:declension_high|0
slohry.prompt2_declension_sk:declension_mid|0
slohry.prompt2_declension_sk:declension_low|0
slohry.prompt1_declension_sk:declension_high|0
slohry.prompt1_declension_sk:declension_mid|0
slohry.prompt1_declension_sk:declension_low|0
slohry.prompt3_declension_sk:declension_high|0
slohry.prompt3_declension_sk:declension_mid|0
slohry.prompt3_declension_sk:declension_low|0
slohry.prompt3_gradation_sk:gradation_high|0
slohry.prompt3_gradation_sk:gradation_mid|0
slohry.prompt3_gradation_sk:gradation_low|0
slohry.prompt1_gradation_sk:gradation_high|0
slohry.prompt1_gradation_sk:gradation_mid|0
slohry.prompt1_gradation_sk:gradation_low|0
slohry.prompt2_gradation_sk:gradation_high|0
slohry.prompt2_gradation_sk:gradation_mid|0
slohry.prompt2_gradation_sk:gradation_low|0
slohry.prompt1_identity_form_sk:identity_form_high|0
slohry.prompt1_identity_form_sk:identity_form_mid|0
slohry.prompt1_identity_form_sk:identity_form_low|0
slohry.prompt2_identity_form_sk:identity_form_high|0
slohry.prompt2_identity_form_sk:identity_form_mid|0
slohry.prompt2_identity_form_sk:identity_form_low|0
slohry.prompt3_identity_form_sk:identity_form_high|0
slohry.prompt3_identity_form_sk:identity_form_mid|0
slohry.prompt3_identity_form_sk:identity_form_low|0
slohry.prompt2_inflection_sk:inflection_high|0
slohry.prompt2_inflection_sk:inflection_mid|0
slohry.prompt2_inflection_sk:inflection_low|0
slohry.prompt1_inflection_sk:inflection_high|0
slohry.prompt1_inflection_sk:inflection_mid|0
slohry.prompt1_inflection_sk:inflection_low|0
slohry.prompt3_inflection_sk:inflection_high|0
slohry.prompt3_inflection_sk:inflection_mid|0
slohry.prompt3_inflection_sk:inflection_low|0
scio.prompt2_mqa_single_correct_sk:aj|0
scio.prompt2_mqa_single_correct_sk:mat|0
scio.prompt2_mqa_single_correct_sk:bio|0
scio.prompt2_mqa_single_correct_sk:che|0
scio.prompt2_mqa_single_correct_sk:vsp|0
scio.prompt3_mqa_single_correct_sk:aj|0
scio.prompt3_mqa_single_correct_sk:mat|0
scio.prompt3_mqa_single_correct_sk:bio|0
scio.prompt3_mqa_single_correct_sk:che|0
scio.prompt3_mqa_single_correct_sk:vsp|0
scio.prompt1_mqa_single_correct_sk:aj|0
scio.prompt1_mqa_single_correct_sk:mat|0
scio.prompt1_mqa_single_correct_sk:bio|0
scio.prompt1_mqa_single_correct_sk:che|0
scio.prompt1_mqa_single_correct_sk:vsp|0

31318
aya_300k_lr_experiments/benchmarks/skbench/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log Normal file
View File

File diff suppressed because one or more lines are too long

87
aya_300k_lr_experiments/benchmarks/skbench/provenance/archive_summary.json Normal file
View File

@ -0,0 +1,87 @@
{
"source": "C:\\Users\\Jakub\\Desktop\\aya_300k_benchmarky_20261024_215708.tar.gz",
"compressed_bytes": 349555561,
"sha256": "b45e431cbc63bc73ef953f8aa3837695ccec43783e3da2dfccb0d99f7d1fe79c",
"files": 581,
"uncompressed_bytes": 1782181826,
"skipped_special": [],
"top_level_files": {
"aya_300k_lr_validation": 110,
"skbench_evaluation": 464,
"scripts": 7
},
"largest_files": [
{
"original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk|0_2026-10-22T21-55-04.268099.parquet",
"local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet",
"size": 127542209,
"type": "file"
},
{
"original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk|0_2026-10-22T21-55-04.268099.parquet",
"local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet",
"size": 127233608,
"type": "file"
},
{
"original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk|0_2026-10-22T21-55-04.268099.parquet",
"local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet",
"size": 125437876,
"type": "file"
},
{
"original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt2_qa_sk|0/82e1a28fce479a47/GENERATIVE.parquet",
"local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt2_qa_sk%7C0/82e1a28fce479a47/GENERATIVE.parquet",
"size": 103508951,
"type": "file"
},
{
"original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt1_qa_sk|0/f0737eb9e82c262f/GENERATIVE.parquet",
"local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt1_qa_sk%7C0/f0737eb9e82c262f/GENERATIVE.parquet",
"size": 103246110,
"type": "file"
},
{
"original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt3_qa_sk|0/579d61d41168b7f2/GENERATIVE.parquet",
"local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/multi_wiki_qa.prompt3_qa_sk%7C0/579d61d41168b7f2/GENERATIVE.parquet",
"size": 101814277,
"type": "file"
},
{
"original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet",
"local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet",
"size": 66791884,
"type": "file"
},
{
"original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet",
"local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet",
"size": 66012897,
"type": "file"
},
{
"original": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet",
"local": "skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet",
"size": 64169583,
"type": "file"
},
{
"original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt2_qa_sk:question-answering|0/46d75c2a99a95ed9/GENERATIVE.parquet",
"local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt2_qa_sk%3Aquestion-answering%7C0/46d75c2a99a95ed9/GENERATIVE.parquet",
"size": 53746461,
"type": "file"
},
{
"original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt1_qa_sk:question-answering|0/b09ec46c034c8de4/GENERATIVE.parquet",
"local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt1_qa_sk%3Aquestion-answering%7C0/b09ec46c034c8de4/GENERATIVE.parquet",
"size": 53089807,
"type": "file"
},
{
"original": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt3_qa_sk:question-answering|0/06ebedbab75fe49b/GENERATIVE.parquet",
"local": "skbench_evaluation/models/aya_300k_lr_0p00010/d0e7048bfa3b2274/sklep.prompt3_qa_sk%3Aquestion-answering%7C0/06ebedbab75fe49b/GENERATIVE.parquet",
"size": 51637093,
"type": "file"
}
]
}

234
aya_300k_lr_experiments/benchmarks/skbench/provenance/audit_skbench.py Normal file
View File

@ -0,0 +1,234 @@
"""Curate the completed 300k learning-rate SK-bench run without copying raw data.
Requires Python 3.10+ and pyarrow (installed separately). The only read of the
large details parquets projects model_response.truncated_tokens_count; row counts
come from Parquet metadata. Cached GENERATIVE.parquet files are never read.
"""
from __future__ import annotations
import argparse
import collections
import csv
import hashlib
import json
import pathlib
import re
import shutil
import statistics
import sys
from urllib.parse import unquote
def write_csv(path, rows):
if not rows:
return
columns = list(dict.fromkeys(k for row in rows for k in row))
with path.open('w', newline='', encoding='utf-8') as stream:
writer = csv.DictWriter(stream, fieldnames=columns)
writer.writeheader()
writer.writerows(rows)
def normalize(name):
name = unquote(name.strip())
return re.sub(r'\|([0-9]+)$', r':\1', name)
def get_lines(path):
return [line.strip() for line in path.read_text(encoding='utf-8').splitlines() if line.strip() and not line.startswith('#')]
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--source', type=pathlib.Path, required=True, help='Extracted archive root containing skbench_evaluation/')
parser.add_argument('--output', type=pathlib.Path, required=True)
parser.add_argument('--pyarrow-path', type=pathlib.Path, help='Optional project-local pip --target directory')
args = parser.parse_args()
if args.pyarrow_path:
sys.path.insert(0, str(args.pyarrow_path.resolve()))
import pyarrow
import pyarrow.parquet as pq
source = args.source.resolve()
out = args.output.resolve()
base = source / 'skbench_evaluation'
run_name = 'sk_zero_available_without_nli_lr_0p00010_20261022_185255'
run = base / 'results' / run_name
for name in ['results', 'logs', 'configs', 'provenance']:
(out/name).mkdir(parents=True, exist_ok=True)
result_path = run/'results/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/results_2026-10-22T21-55-04.268099.json'
d = json.loads(result_path.read_text(encoding='utf-8'))
tasks = get_lines(run/'tasks.txt')
omitted = get_lines(run/'omitted_tasks.txt')
configs = d['config_tasks']
normalized_expected = {normalize(x) for x in tasks}
assert len(tasks) == len(normalized_expected) == 222
assert normalized_expected == {normalize(x) for x in configs}, 'Config task coverage mismatch'
assert normalized_expected <= {normalize(x) for x in d['results']}, 'Result task coverage mismatch'
assert len(omitted) == 190
all_tasks = get_lines(base/'configs/tasks_sk_zero.txt')
assert normalized_expected.isdisjoint({normalize(x) for x in omitted})
assert normalized_expected | {normalize(x) for x in omitted} == {normalize(x) for x in all_tasks}
copies = {result_path: out/'results'/result_path.name,
base/'logs'/f'{run_name}.log': out/'logs'/f'{run_name}.log',
run/'tasks.txt': out/'configs/tasks.txt',
run/'omitted_tasks.txt': out/'configs/omitted_tasks.txt',
run/'uv.lock': out/'provenance/uv.lock',
run/'skbench_commit.txt': out/'provenance/skbench_commit.txt',
source/'archive_summary.json': out/'provenance/archive_summary.json'}
for path in sorted((base/'configs').iterdir()):
if path.is_file():
copies[path] = out/'configs'/path.name
provenance = []
for src, dst in copies.items():
shutil.copyfile(src, dst)
data = src.read_bytes()
provenance.append({'source_archive_member': src.relative_to(source).as_posix(),
'curated_path': dst.relative_to(out).as_posix(),
'bytes': len(data), 'sha256': hashlib.sha256(data).hexdigest()})
write_csv(out/'provenance/copied_files_sha256.csv', provenance)
(out/'configs/model_config.json').write_text(json.dumps(d['config_general'], indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
(out/'configs/task_configs.json').write_text(json.dumps(configs, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
# Resolve the original names from the extraction manifest; Windows paths
# encode ':' and '|' as %3A and %7C, while payload bytes were unchanged.
manifest = json.loads((source/'archive_manifest.json').read_text(encoding='utf-8'))
if isinstance(manifest, dict):
manifest = manifest.get('entries', manifest.get('files', []))
original_paths = {x['local']: x['original'] for x in manifest if isinstance(x, dict) and 'local' in x}
details = []
lookup = {}
files = sorted((run/'details').rglob('*.parquet'))
assert len(files) == 222, f'Expected 222 DETAILS files, got {len(files)}'
for file in files:
decoded = unquote(file.name)
task = re.match(r'^details_(.+)_2026-10-22T21-55-04\.268099\.parquet$', decoded).group(1)
pf = pq.ParquetFile(file)
projected = pf.read(columns=['model_response.truncated_tokens_count'])
values = projected.column('model_response').combine_chunks().field('truncated_tokens_count').to_pylist()
assert len(values) == pf.metadata.num_rows
assert all(v is None or (isinstance(v, int) and v >= 0) for v in values)
row = {'task': task, 'sample_count': pf.metadata.num_rows,
'truncation_values_present': sum(v is not None for v in values),
'truncation_null_values': sum(v is None for v in values),
'samples_with_truncation': sum(v is not None and v > 0 for v in values),
'truncated_tokens_total': sum(v or 0 for v in values),
'max_truncated_tokens_per_sample': max((v or 0 for v in values), default=0),
'parquet_bytes': file.stat().st_size,
'source_archive_member': original_paths.get(file.relative_to(source).as_posix(), file.relative_to(source).as_posix()),
'extracted_relative_path': file.relative_to(source).as_posix()}
assert normalize(task) not in lookup
lookup[normalize(task)] = row
details.append(row)
assert set(lookup) == normalized_expected, 'DETAILS task coverage mismatch'
write_csv(out/'results/truncation_audit_per_task.csv', details)
metric_names = sorted({key for task in configs for key in d['results'][task]})
per_task = []
for task in sorted(configs):
metric = d['results'][task]
config = configs[task]
detail = lookup[normalize(task)]
primary = next((name for name in ['em_relaxed', 'em_norm_relaxed', 'exact_match_slk_prefix'] if name in metric), None)
family = task.split('.prompt')[0]
prompt = int(re.search(r'\.prompt([123])_', task).group(1))
task_group = re.sub(r'\.prompt[123]_', '.', task).removesuffix('|0')
row = {'task': task, 'family': family, 'task_group_without_prompt': task_group,
'prompt': prompt, 'num_fewshots': config['num_fewshots'], 'sample_count': detail['sample_count'],
'primary_metric': primary, 'primary_value': metric[primary],
'generation_size': config['generation_size'], 'evaluation_splits': '|'.join(config['evaluation_splits']),
'hf_repo': config['hf_repo'], 'hf_subset': config['hf_subset'], 'hf_revision': config['hf_revision']}
row.update({key: metric.get(key, '') for key in metric_names})
per_task.append(row)
write_csv(out/'results/per_task_metrics.csv', per_task)
# Each summary groups only identical metrics; values remain 0..1.
# Macro means weight task variants equally. Sample-weighted means weight
# their actual DETAILS rows. Repeated prompts are evaluation instances,
# never additional independent underlying test questions.
def summarize(group_columns, filename):
groups = collections.defaultdict(list)
for row in per_task:
for metric_name in metric_names:
if not metric_name.endswith('_stderr') and row[metric_name] != '':
groups[tuple(row[x] for x in group_columns)+(metric_name,)].append(row)
result = []
for key, rows in sorted(groups.items()):
metric_name = key[-1]
total = sum(x['sample_count'] for x in rows)
item = dict(zip(group_columns, key[:-1]))
item.update({'metric': metric_name, 'task_variants': len(rows),
'prompt_count': len({x['prompt'] for x in rows}),
'evaluated_instances': total,
'task_macro_mean': statistics.mean(x[metric_name] for x in rows),
'sample_weighted_mean': sum(x['sample_count']*x[metric_name] for x in rows)/total,
'min_task_value': min(x[metric_name] for x in rows),
'max_task_value': max(x[metric_name] for x in rows)})
result.append(item)
write_csv(out/'results'/filename, result)
return result
family_summary = summarize(['family'], 'family_summary.csv')
summarize(['family', 'prompt'], 'family_prompt_summary.csv')
three_prompt = summarize(['family', 'task_group_without_prompt'], 'three_prompt_summary.csv')
assert all(x['task_variants'] == x['prompt_count'] == 3 for x in three_prompt)
access = json.loads((base/'configs/dataset_access_report.json').read_text(encoding='utf-8'))
inaccessible = [x for x in access if x['status'] != 'OK']
with (base/'logs'/f'{run_name}.log').open(encoding='utf-8', newline='') as stream:
log_text = stream.read()
truncation_warnings = re.findall(r'context_size \+ max_new_tokens=\d+[^\r\n]*Truncating context to \d+ tokens\.', log_text)
warning_evidence = []
for line_number, line in enumerate(log_text.split('\n'), 1):
if 'Truncating context' in line:
warning_evidence.append({'line_number': line_number, 'message': line[line.index('context_size'):], 'source': f'logs/{run_name}.log'})
(out/'results/log_truncation_evidence.json').write_text(json.dumps(warning_evidence, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
summary = {'run_name': run_name, 'result_file': result_path.name,
'task_list_count': len(tasks), 'configured_tasks': len(configs), 'scored_task_variants': len(per_task),
'result_json_keys': len(d['results']),
'extra_result_keys_excluded_from_task_summaries': sorted(set(d['results'])-set(configs)),
'all_original_task_variants': len(all_tasks),
'omitted_task_variants': len(omitted), 'inaccessible_repositories': len(inaccessible),
'inaccessible_task_variants': sum(len(x['tasks']) for x in inaccessible),
'additionally_excluded_nli_task_variants': 3,
'families': dict(sorted(collections.Counter(row['family'] for row in per_task).items())),
'prompt_counts': dict(sorted(collections.Counter(row['prompt'] for row in per_task).items())),
'evaluated_instances': sum(row['sample_count'] for row in per_task),
'instances_by_prompt': {str(prompt): sum(x['sample_count'] for x in per_task if x['prompt'] == prompt) for prompt in [1,2,3]},
'details_files_audited': len(details), 'truncation_column': 'model_response.truncated_tokens_count',
'truncation_values_present': sum(x['truncation_values_present'] for x in details),
'truncation_null_values': sum(x['truncation_null_values'] for x in details),
'details_reported_samples_with_truncation': sum(x['samples_with_truncation'] for x in details),
'details_reported_truncated_tokens_total': sum(x['truncated_tokens_total'] for x in details),
'details_reported_max_truncated_tokens_per_sample': max(x['max_truncated_tokens_per_sample'] for x in details),
'context_truncation_observed_in_log': bool(truncation_warnings),
'actual_truncated_sample_count': None,
'console_context_truncation_warning_count': len(truncation_warnings),
'console_context_truncation_warnings': truncation_warnings,
'truncation_interpretation': 'All DETAILS counters are zero, but the console explicitly reports context truncation. Do not infer no truncation or an affected-sample count from these counters.',
'model_config': d['config_general']['model_config'],
'max_samples': d['config_general']['max_samples'],
'generation_size_counts': dict(collections.Counter(str(x['generation_size']) for x in per_task)),
'original_summary_general': d['summary_general'],
'pyarrow_version': pyarrow.__version__, 'python_version': sys.version,
'aggregation': 'Descriptive metric-specific task macro means and sample-weighted means; no official full-SK-bench overall score.',
'limitations': ['Partial available zero-shot subset: 222 of 412 task variants.',
'Only the validation-selected LR=0.00010 was benchmarked.',
'Sample counts are evaluation instances; prompt variants reuse underlying questions.',
'One console warning reports context truncation; all DETAILS counters are zero, so the recorded counters do not reliably enumerate affected inputs.',
'The audit cannot recover the number of affected tasks/samples or quantify output generation-size termination.',
'All-zero summary counters include non_truncated=0 and cannot prove absence of truncation.',
'Raw DETAILS and generation-cache parquets are excluded from this curated tree.']}
(out/'results/audit_summary.json').write_text(json.dumps(summary, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
script_destination = out/'provenance/audit_skbench.py'
if pathlib.Path(__file__).resolve() != script_destination.resolve():
shutil.copyfile(pathlib.Path(__file__), script_destination)
print(json.dumps({key: summary[key] for key in ['scored_task_variants', 'evaluated_instances', 'instances_by_prompt', 'details_reported_samples_with_truncation', 'context_truncation_observed_in_log', 'actual_truncated_sample_count', 'truncation_null_values', 'families']}, indent=2))
print('Primary family macro means (metrics kept separate):')
for row in family_summary:
if row['metric'] in ['em_relaxed', 'em_norm_relaxed', 'exact_match_slk_prefix', 'f1_slk']:
print(row)
if __name__ == '__main__':
main()

14
aya_300k_lr_experiments/benchmarks/skbench/provenance/copied_files_sha256.csv Normal file
View File

@ -0,0 +1,14 @@
source_archive_member,curated_path,bytes,sha256
skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/results/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/results_2026-10-22T21-55-04.268099.json,results/results_2026-10-22T21-55-04.268099.json,638180,966123dc20b547860d04f3434f65752c681ca2d83baebee22873e553161ff0df
skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log,logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log,4537535,23a857210201913c5506ac39553a86fd803ed0c057e1226fcd978030a40b6c78
skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/tasks.txt,configs/tasks.txt,11415,297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/omitted_tasks.txt,configs/omitted_tasks.txt,8629,613b78472e6788eab826f31da1100c00b597ed9bbc8151af38ebb8ef5bf830af
skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/uv.lock,provenance/uv.lock,690556,045d2840fda141981afba93bd4a1b9ddb59f0f0791c162cf66b2c88e6491fb41
skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/skbench_commit.txt,provenance/skbench_commit.txt,41,12655509d516c6a57225a072f708e5ca854600e7daa3bf5cdc95e6b36e94fe70
archive_summary.json,provenance/archive_summary.json,6529,535390044cd67477ea1631527ac736f72108935f25a11a613b1709caf8154957
skbench_evaluation/configs/dataset_access_report.json,configs/dataset_access_report.json,29828,8eea8c082d1eb914850128d48021a39c7e1fb7189578c5f1e496004af6ff593b
skbench_evaluation/configs/export_lr_0p00010.yaml,configs/export_lr_0p00010.yaml,391,bff3e0b8abd130122c8d88be71647e0ea67e4a836b3caaed4e3bc307ef267dca
skbench_evaluation/configs/tasks_sk_zero.txt,configs/tasks_sk_zero.txt,20044,25c3cdfeb1f46adb86b68be24a9d8e22439733501e2fffda1ce5034e610660a4
skbench_evaluation/configs/tasks_sk_zero_available.txt,configs/tasks_sk_zero_available.txt,11517,5ce7f1666b2073afd93aab03a116937f48954cf51d313c567c198b31f0dfcaf2
skbench_evaluation/configs/tasks_sk_zero_available_without_nli.txt,configs/tasks_sk_zero_available_without_nli.txt,11415,297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
skbench_evaluation/configs/tasks_sk_zero_unavailable.txt,configs/tasks_sk_zero_unavailable.txt,8527,0071cffec8ab2446a5594d0b3a3116151e7fc375bb300394e9ea7cb91ed7df10
1 source_archive_member curated_path bytes sha256
2 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/results/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/results_2026-10-22T21-55-04.268099.json results/results_2026-10-22T21-55-04.268099.json 638180 966123dc20b547860d04f3434f65752c681ca2d83baebee22873e553161ff0df
3 skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log 4537535 23a857210201913c5506ac39553a86fd803ed0c057e1226fcd978030a40b6c78
4 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/tasks.txt configs/tasks.txt 11415 297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
5 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/omitted_tasks.txt configs/omitted_tasks.txt 8629 613b78472e6788eab826f31da1100c00b597ed9bbc8151af38ebb8ef5bf830af
6 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/uv.lock provenance/uv.lock 690556 045d2840fda141981afba93bd4a1b9ddb59f0f0791c162cf66b2c88e6491fb41
7 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/skbench_commit.txt provenance/skbench_commit.txt 41 12655509d516c6a57225a072f708e5ca854600e7daa3bf5cdc95e6b36e94fe70
8 archive_summary.json provenance/archive_summary.json 6529 535390044cd67477ea1631527ac736f72108935f25a11a613b1709caf8154957
9 skbench_evaluation/configs/dataset_access_report.json configs/dataset_access_report.json 29828 8eea8c082d1eb914850128d48021a39c7e1fb7189578c5f1e496004af6ff593b
10 skbench_evaluation/configs/export_lr_0p00010.yaml configs/export_lr_0p00010.yaml 391 bff3e0b8abd130122c8d88be71647e0ea67e4a836b3caaed4e3bc307ef267dca
11 skbench_evaluation/configs/tasks_sk_zero.txt configs/tasks_sk_zero.txt 20044 25c3cdfeb1f46adb86b68be24a9d8e22439733501e2fffda1ce5034e610660a4
12 skbench_evaluation/configs/tasks_sk_zero_available.txt configs/tasks_sk_zero_available.txt 11517 5ce7f1666b2073afd93aab03a116937f48954cf51d313c567c198b31f0dfcaf2
13 skbench_evaluation/configs/tasks_sk_zero_available_without_nli.txt configs/tasks_sk_zero_available_without_nli.txt 11415 297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
14 skbench_evaluation/configs/tasks_sk_zero_unavailable.txt configs/tasks_sk_zero_unavailable.txt 8527 0071cffec8ab2446a5594d0b3a3116151e7fc375bb300394e9ea7cb91ed7df10

62
aya_300k_lr_experiments/benchmarks/skbench/provenance/input_length_diagnostics.py Normal file
View File

@ -0,0 +1,62 @@
"""Inspect token-array and raw-prompt lengths; never export prompt content."""
import argparse, csv, json, pathlib, re, shutil, statistics, sys
from urllib.parse import unquote
parser=argparse.ArgumentParser(description=__doc__)
parser.add_argument('--source',type=pathlib.Path,required=True)
parser.add_argument('--output',type=pathlib.Path,required=True)
parser.add_argument('--pyarrow-path',type=pathlib.Path)
args=parser.parse_args()
if args.pyarrow_path: sys.path.insert(0,str(args.pyarrow_path.resolve()))
import pyarrow.parquet as pq
run=args.source/'skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255'
d=json.loads(next((run/'results').rglob('*.json')).read_text())
limit=d['config_general']['model_config']['max_model_length']
rows=[]
outliers=[]
for file in sorted((run/'details').rglob('*.parquet')):
task=re.match(r'details_(.+)_2026-10-22T21-55-04\.268099\.parquet',unquote(file.name)).group(1)
cfg=d['config_tasks'][task]
budget=limit-cfg['generation_size']
data=pq.ParquetFile(file).read(columns=['model_response.input_tokens','model_response.input'])
data=data.column('model_response').combine_chunks()
tokens=data.field('input_tokens').to_pylist()
texts=data.field('input').to_pylist()
lens=[None if value is None else len(value) for value in tokens]
chars=[None if value is None else len(value) for value in texts]
row={'task':task,'generation_size':cfg['generation_size'],'context_limit':limit,'input_budget':budget,
'samples':len(lens),'null_token_arrays':sum(x is None for x in lens),'null_input_strings':sum(x is None for x in chars),
'min_recorded_input_tokens':min(x for x in lens if x is not None),
'max_recorded_input_tokens':max(x for x in lens if x is not None),
'mean_recorded_input_tokens':statistics.mean(x for x in lens if x is not None),
'arrays_at_input_budget':sum(x==budget for x in lens),
'arrays_over_input_budget':sum(x is not None and x>budget for x in lens),
'max_recorded_input_characters':max(x for x in chars if x is not None)}
rows.append(row)
for index,length in enumerate(lens):
if length is not None and length>=budget:
outliers.append({'task':task,'details_row_index_zero_based':index,'input_token_array_length':length,
'generation_size':cfg['generation_size'],'input_budget':budget,
'recorded_input_characters':chars[index]})
for name,values in [('input_length_diagnostics_per_task.csv',rows),('input_length_budget_boundary_rows.csv',outliers)]:
with (args.output/name).open('w',newline='',encoding='utf-8') as f:
writer=csv.DictWriter(f,fieldnames=list(values[0]) if values else ['task'])
writer.writeheader();writer.writerows(values)
with (args.source/'skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log').open(encoding='utf-8',newline='') as stream:
log=stream.read()
messages=[]
for number,line in enumerate(log.split('\n'),1):
if re.search(r'truncat',line,re.I):
messages.append({'log_lf_line_number':number,'message':re.sub(r'\x1b\[[0-9;]*[A-Za-z]','',line).strip()[-1500:]})
summary={'task_count':len(rows),'evaluated_instances':sum(x['samples'] for x in rows),
'max_recorded_input_tokens':max(x['max_recorded_input_tokens'] for x in rows),
'arrays_at_input_budget':sum(x['arrays_at_input_budget'] for x in rows),
'arrays_over_input_budget':sum(x['arrays_over_input_budget'] for x in rows),
'log_messages_matching_truncat_case_insensitive':messages,
'limitation':'Input strings and token arrays are recorded artifacts, not a separately tokenized original source prompt. Their lengths cannot recover how much text was discarded or establish the number of truly truncated inputs.'}
(args.output/'input_length_diagnostics_summary.json').write_text(json.dumps(summary,indent=2,ensure_ascii=False)+'\n',encoding='utf-8')
print(json.dumps(summary,indent=2,ensure_ascii=False))
print('BOUNDARY TASK COUNTS',json.dumps({task:sum(row['task']==task for row in outliers) for task in sorted({row['task'] for row in outliers})}))
script_destination=args.output.parent/'provenance/input_length_diagnostics.py'
if pathlib.Path(__file__).resolve()!=script_destination.resolve():
shutil.copyfile(pathlib.Path(__file__),script_destination)

1
aya_300k_lr_experiments/benchmarks/skbench/provenance/requirements-audit.txt Normal file
View File

@ -0,0 +1 @@
pyarrow==26.0.0

1
aya_300k_lr_experiments/benchmarks/skbench/provenance/skbench_commit.txt Normal file
View File

@ -0,0 +1 @@
15823bbcc2d995e69441fcb7d34abb61bd8180c5

6274
aya_300k_lr_experiments/benchmarks/skbench/provenance/uv.lock Normal file
View File

File diff suppressed because it is too large Load Diff

137
aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json Normal file
View File

@ -0,0 +1,137 @@
{
"run_name": "sk_zero_available_without_nli_lr_0p00010_20261022_185255",
"result_file": "results_2026-10-22T21-55-04.268099.json",
"task_list_count": 222,
"configured_tasks": 222,
"scored_task_variants": 222,
"result_json_keys": 226,
"extra_result_keys_excluded_from_task_summaries": [
"all",
"mmlu.prompt1_mqa_single_correct_sk:_average|0",
"mmlu.prompt2_mqa_single_correct_sk:_average|0",
"mmlu.prompt3_mqa_single_correct_sk:_average|0"
],
"all_original_task_variants": 412,
"omitted_task_variants": 190,
"inaccessible_repositories": 12,
"inaccessible_task_variants": 187,
"additionally_excluded_nli_task_variants": 3,
"families": {
"arc": 3,
"belebele": 3,
"financial_exams": 3,
"globalmmlu-lite": 3,
"mmlu": 171,
"movie_review": 3,
"multi_wiki_qa": 3,
"senti_sk": 6,
"skj1": 3,
"sklep": 15,
"slovakcopa": 3,
"toxic_sk": 3,
"truthfulqa": 3
},
"prompt_counts": {
"1": 74,
"2": 74,
"3": 74
},
"evaluated_instances": 159426,
"instances_by_prompt": {
"1": 53142,
"2": 53142,
"3": 53142
},
"details_files_audited": 222,
"truncation_column": "model_response.truncated_tokens_count",
"truncation_values_present": 159426,
"truncation_null_values": 0,
"details_reported_samples_with_truncation": 0,
"details_reported_truncated_tokens_total": 0,
"details_reported_max_truncated_tokens_per_sample": 0,
"context_truncation_observed_in_log": true,
"actual_truncated_sample_count": null,
"console_context_truncation_warning_count": 1,
"console_context_truncation_warnings": [
"context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens."
],
"truncation_interpretation": "All DETAILS counters are zero, but the console explicitly reports context truncation. Do not infer no truncation or an affected-sample count from these counters.",
"model_config": {
"model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010",
"generation_parameters": {
"num_blocks": null,
"block_size": null,
"early_stopping": null,
"repetition_penalty": null,
"frequency_penalty": null,
"length_penalty": null,
"presence_penalty": null,
"max_new_tokens": null,
"min_new_tokens": null,
"seed": null,
"stop_tokens": null,
"temperature": 0,
"top_k": null,
"min_p": null,
"top_p": null,
"truncate_prompt": null,
"cache_implementation": null,
"response_format": null,
"skip_special_tokens": true
},
"system_prompt": null,
"cache_dir": "~/.cache/huggingface/lighteval",
"tokenizer": null,
"revision": "main",
"dtype": "bfloat16",
"tensor_parallel_size": 1,
"data_parallel_size": 1,
"pipeline_parallel_size": 1,
"gpu_memory_utilization": 0.85,
"enable_prefix_caching": null,
"max_model_length": 32768,
"quantization": null,
"load_format": null,
"swap_space": 4,
"seed": 1234,
"trust_remote_code": false,
"add_special_tokens": true,
"multichoice_continuations_start_space": true,
"pairwise_tokenization": false,
"max_num_seqs": 128,
"max_num_batched_tokens": 2048,
"subfolder": null,
"is_async": false,
"override_chat_template": null,
"enforce_eager": true
},
"max_samples": null,
"generation_size_counts": {
"32": 216,
"126": 6
},
"original_summary_general": {
"hashes": {
"hash_examples": "316f54894dcb6f0d",
"hash_full_prompts": "ba7f9b257c923f55",
"hash_input_tokens": "8de2f010e9daf7fe",
"hash_cont_tokens": "1f06ad3e4b80f7cc"
},
"truncated": 0,
"non_truncated": 0,
"padded": 0,
"non_padded": 0
},
"pyarrow_version": "26.0.0",
"python_version": "3.12.14 (main, Aug 25 2026, 14:01:42) [MSC v.1944 64 bit (AMD64)]",
"aggregation": "Descriptive metric-specific task macro means and sample-weighted means; no official full-SK-bench overall score.",
"limitations": [
"Partial available zero-shot subset: 222 of 412 task variants.",
"Only the validation-selected LR=0.00010 was benchmarked.",
"Sample counts are evaluation instances; prompt variants reuse underlying questions.",
"One console warning reports context truncation; all DETAILS counters are zero, so the recorded counters do not reliably enumerate affected inputs.",
"The audit cannot recover the number of affected tasks/samples or quantify output generation-size termination.",
"All-zero summary counters include non_truncated=0 and cannot prove absence of truncation.",
"Raw DETAILS and generation-cache parquets are excluded from this curated tree."
]
}

121
aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv Normal file
View File

@ -0,0 +1,121 @@
family,prompt,metric,task_variants,prompt_count,evaluated_instances,task_macro_mean,sample_weighted_mean,min_task_value,max_task_value
arc,1,em,1,1,1169,0.013686911890504704,0.013686911890504704,0.013686911890504704,0.013686911890504704
arc,1,em_conservative,1,1,1169,0.1710863986313088,0.1710863986313088,0.1710863986313088,0.1710863986313088
arc,1,em_relaxed,1,1,1169,0.1710863986313088,0.1710863986313088,0.1710863986313088,0.1710863986313088
arc,2,em,1,1,1169,0.03507271171941831,0.03507271171941831,0.03507271171941831,0.03507271171941831
arc,2,em_conservative,1,1,1169,0.223267750213858,0.223267750213858,0.223267750213858,0.223267750213858
arc,2,em_relaxed,1,1,1169,0.22412318220701455,0.22412318220701455,0.22412318220701455,0.22412318220701455
arc,3,em,1,1,1169,0.022241231822070145,0.022241231822070145,0.022241231822070145,0.022241231822070145
arc,3,em_conservative,1,1,1169,0.17621899059024806,0.17621899059024806,0.17621899059024806,0.17621899059024806
arc,3,em_relaxed,1,1,1169,0.1770744225834046,0.1770744225834046,0.1770744225834046,0.1770744225834046
belebele,1,em,1,1,900,0.02,0.02,0.02,0.02
belebele,1,em_conservative,1,1,900,0.21555555555555556,0.21555555555555556,0.21555555555555556,0.21555555555555556
belebele,1,em_relaxed,1,1,900,0.21555555555555556,0.21555555555555556,0.21555555555555556,0.21555555555555556
belebele,2,em,1,1,900,0.016666666666666666,0.016666666666666666,0.016666666666666666,0.016666666666666666
belebele,2,em_conservative,1,1,900,0.2111111111111111,0.2111111111111111,0.2111111111111111,0.2111111111111111
belebele,2,em_relaxed,1,1,900,0.2111111111111111,0.2111111111111111,0.2111111111111111,0.2111111111111111
belebele,3,em,1,1,900,0.03,0.03,0.03,0.03
belebele,3,em_conservative,1,1,900,0.2111111111111111,0.2111111111111111,0.2111111111111111,0.2111111111111111
belebele,3,em_relaxed,1,1,900,0.21333333333333335,0.21333333333333335,0.21333333333333335,0.21333333333333335
financial_exams,1,em,1,1,1334,0.009745127436281859,0.009745127436281859,0.009745127436281859,0.009745127436281859
financial_exams,1,em_conservative,1,1,1334,0.20089955022488756,0.20089955022488756,0.20089955022488756,0.20089955022488756
financial_exams,1,em_relaxed,1,1,1334,0.20089955022488756,0.20089955022488756,0.20089955022488756,0.20089955022488756
financial_exams,2,em,1,1,1334,0.07571214392803598,0.07571214392803598,0.07571214392803598,0.07571214392803598
financial_exams,2,em_conservative,1,1,1334,0.20539730134932535,0.20539730134932535,0.20539730134932535,0.20539730134932535
financial_exams,2,em_relaxed,1,1,1334,0.2383808095952024,0.2383808095952024,0.2383808095952024,0.2383808095952024
financial_exams,3,em,1,1,1334,0.0037481259370314842,0.0037481259370314842,0.0037481259370314842,0.0037481259370314842
financial_exams,3,em_conservative,1,1,1334,0.24887556221889057,0.24887556221889057,0.24887556221889057,0.24887556221889057
financial_exams,3,em_relaxed,1,1,1334,0.24887556221889057,0.24887556221889057,0.24887556221889057,0.24887556221889057
globalmmlu-lite,1,em,1,1,400,0.025,0.025,0.025,0.025
globalmmlu-lite,1,em_conservative,1,1,400,0.1825,0.1825,0.1825,0.1825
globalmmlu-lite,1,em_relaxed,1,1,400,0.1825,0.1825,0.1825,0.1825
globalmmlu-lite,2,em,1,1,400,0.035,0.035,0.035,0.035
globalmmlu-lite,2,em_conservative,1,1,400,0.21,0.21,0.21,0.21
globalmmlu-lite,2,em_relaxed,1,1,400,0.21,0.21,0.21,0.21
globalmmlu-lite,3,em,1,1,400,0.025,0.025,0.025,0.025
globalmmlu-lite,3,em_conservative,1,1,400,0.2025,0.2025,0.2025,0.2025
globalmmlu-lite,3,em_relaxed,1,1,400,0.2025,0.2025,0.2025,0.2025
mmlu,1,em,57,1,13062,0.023388116537651125,0.02174245904149441,0.0,0.06818181818181818
mmlu,1,em_conservative,57,1,13062,0.1403536981271315,0.13910580309294135,0.0053404539385847796,0.3096446700507614
mmlu,1,em_relaxed,57,1,13062,0.14312818797047927,0.14094319399785638,0.006675567423230975,0.3096446700507614
mmlu,2,em,57,1,13062,0.04274500178923145,0.040116368090644616,0.0,0.13
mmlu,2,em_conservative,57,1,13062,0.17602310250998576,0.17432246210381258,0.0026702269692923898,0.3553299492385787
mmlu,2,em_relaxed,57,1,13062,0.17964687184852454,0.17661920073495635,0.0026702269692923898,0.3553299492385787
mmlu,3,em,57,1,13062,0.028327141879067468,0.026871841984382177,0.0,0.07575757575757576
mmlu,3,em_conservative,57,1,13062,0.15358971887332207,0.15051293829428877,0.0053404539385847796,0.3350253807106599
mmlu,3,em_relaxed,57,1,13062,0.1575073160887174,0.15411116214974735,0.01335113484646195,0.3350253807106599
movie_review,1,em_conservative_norm,1,1,2008,0.6324701195219123,0.6324701195219123,0.6324701195219123,0.6324701195219123
movie_review,1,em_norm,1,1,2008,0.38745019920318724,0.38745019920318724,0.38745019920318724,0.38745019920318724
movie_review,1,em_norm_relaxed,1,1,2008,0.6329681274900398,0.6329681274900398,0.6329681274900398,0.6329681274900398
movie_review,2,em_conservative_norm,1,1,2008,0.47758964143426297,0.47758964143426297,0.47758964143426297,0.47758964143426297
movie_review,2,em_norm,1,1,2008,0.01045816733067729,0.01045816733067729,0.01045816733067729,0.01045816733067729
movie_review,2,em_norm_relaxed,1,1,2008,0.48306772908366535,0.48306772908366535,0.48306772908366535,0.48306772908366535
movie_review,3,em_conservative_norm,1,1,2008,0.09412350597609562,0.09412350597609562,0.09412350597609562,0.09412350597609562
movie_review,3,em_norm,1,1,2008,0.04731075697211155,0.04731075697211155,0.04731075697211155,0.04731075697211155
movie_review,3,em_norm_relaxed,1,1,2008,0.11105577689243028,0.11105577689243028,0.11105577689243028,0.11105577689243028
multi_wiki_qa,1,exact_match_slk_prefix,1,1,4997,0.36401841104662797,0.36401841104662797,0.36401841104662797,0.36401841104662797
multi_wiki_qa,1,f1_slk,1,1,4997,0.5156574509584114,0.5156574509584114,0.5156574509584114,0.5156574509584114
multi_wiki_qa,2,exact_match_slk_prefix,1,1,4997,0.36001600960576347,0.36001600960576347,0.36001600960576347,0.36001600960576347
multi_wiki_qa,2,f1_slk,1,1,4997,0.5090494926903776,0.5090494926903776,0.5090494926903776,0.5090494926903776
multi_wiki_qa,3,exact_match_slk_prefix,1,1,4997,0.34000400240144085,0.34000400240144085,0.34000400240144085,0.34000400240144085
multi_wiki_qa,3,f1_slk,1,1,4997,0.5050071786470199,0.5050071786470199,0.5050071786470199,0.5050071786470199
senti_sk,1,em_conservative_norm,2,1,8405,0.3151902608050904,0.3156454491374182,0.3111587982832618,0.31922172332691895
senti_sk,1,em_norm,2,1,8405,0.21165780446913535,0.2133254015466984,0.1968884120171674,0.22642719692110327
senti_sk,1,em_norm_relaxed,2,1,8405,0.3161543660374346,0.31659726353361095,0.31223175965665234,0.3200769724182168
senti_sk,2,em_conservative_norm,2,1,8405,0.4350953001676545,0.4285544318857823,0.37716484926234767,0.49302575107296137
senti_sk,2,em_norm,2,1,8405,0.011392328544121953,0.01225461035098156,0.0037553648068669528,0.019029292281376953
senti_sk,2,em_norm_relaxed,2,1,8405,0.43993914035536885,0.43355145746579415,0.383365405174257,0.4965128755364807
senti_sk,3,em_conservative_norm,2,1,8405,0.23176005812390282,0.23545508625817965,0.1990343347639485,0.26448578148385715
senti_sk,3,em_norm,2,1,8405,0.06853090550874015,0.0726948245092207,0.03165236051502146,0.10540945050245884
senti_sk,3,em_norm_relaxed,2,1,8405,0.3302092538502268,0.33908387864366446,0.25160944206008584,0.40880906564036773
skj1,1,em,1,1,100,0.0,0.0,0.0,0.0
skj1,1,em_conservative,1,1,100,0.02,0.02,0.02,0.02
skj1,1,em_relaxed,1,1,100,0.02,0.02,0.02,0.02
skj1,2,em,1,1,100,0.0,0.0,0.0,0.0
skj1,2,em_conservative,1,1,100,0.05,0.05,0.05,0.05
skj1,2,em_relaxed,1,1,100,0.06,0.06,0.06,0.06
skj1,3,em,1,1,100,0.0,0.0,0.0,0.0
skj1,3,em_conservative,1,1,100,0.03,0.03,0.03,0.03
skj1,3,em_relaxed,1,1,100,0.05,0.05,0.05,0.05
sklep,1,em_conservative_norm,4,1,9025,0.4046842034899774,0.2173961218836565,0.016386890487609912,0.8800383877159309
sklep,1,em_norm,4,1,9025,0.11229466834690793,0.0520775623268698,0.0,0.44817658349328215
sklep,1,em_norm_relaxed,4,1,9025,0.4050240467784944,0.21772853185595567,0.016786570743405275,0.8809980806142035
sklep,1,exact_match_slk_prefix,1,1,9583,0.16456224564332672,0.16456224564332672,0.16456224564332672,0.16456224564332672
sklep,1,f1_slk,1,1,9583,0.3730151221096053,0.3730151221096053,0.3730151221096053,0.3730151221096053
sklep,2,em_conservative_norm,4,1,9025,0.37570022410310483,0.26204986149584486,0.09952038369304557,0.7247915087187263
sklep,2,em_norm,4,1,9025,0.0006917304534838009,0.00044321329639889195,0.0,0.0018072289156626507
sklep,2,em_norm_relaxed,4,1,9025,0.3764699538087837,0.26249307479224376,0.09972022382094324,0.7247915087187263
sklep,2,exact_match_slk_prefix,1,1,9583,0.16518835437754356,0.16518835437754356,0.16518835437754356,0.16518835437754356
sklep,2,f1_slk,1,1,9583,0.3751701402109962,0.3751701402109962,0.3751701402109962,0.3751701402109962
sklep,3,em_conservative_norm,4,1,9025,0.26202719318429357,0.15889196675900277,0.001398880895283773,0.6967399545109931
sklep,3,em_norm,4,1,9025,0.004139976412367319,0.001994459833795014,0.0,0.015355086372360844
sklep,3,em_norm_relaxed,4,1,9025,0.2709043524933147,0.16299168975069253,0.001398880895283773,0.6967399545109931
sklep,3,exact_match_slk_prefix,1,1,9583,0.16143170197224252,0.16143170197224252,0.16143170197224252,0.16143170197224252
sklep,3,f1_slk,1,1,9583,0.3769392643579436,0.3769392643579436,0.3769392643579436,0.3769392643579436
slovakcopa,1,em,1,1,500,0.0,0.0,0.0,0.0
slovakcopa,1,em_conservative,1,1,500,0.508,0.508,0.508,0.508
slovakcopa,1,em_relaxed,1,1,500,0.618,0.618,0.618,0.618
slovakcopa,2,em,1,1,500,0.0,0.0,0.0,0.0
slovakcopa,2,em_conservative,1,1,500,0.326,0.326,0.326,0.326
slovakcopa,2,em_relaxed,1,1,500,0.51,0.51,0.51,0.51
slovakcopa,3,em,1,1,500,0.0,0.0,0.0,0.0
slovakcopa,3,em_conservative,1,1,500,0.084,0.084,0.084,0.084
slovakcopa,3,em_relaxed,1,1,500,0.086,0.086,0.086,0.086
toxic_sk,1,em_conservative_norm,1,1,881,0.45629965947786605,0.45629965947786605,0.45629965947786605,0.45629965947786605
toxic_sk,1,em_norm,1,1,881,0.0,0.0,0.0,0.0
toxic_sk,1,em_norm_relaxed,1,1,881,0.45629965947786605,0.45629965947786605,0.45629965947786605,0.45629965947786605
toxic_sk,2,em_conservative_norm,1,1,881,0.47559591373439275,0.47559591373439275,0.47559591373439275,0.47559591373439275
toxic_sk,2,em_norm,1,1,881,0.0,0.0,0.0,0.0
toxic_sk,2,em_norm_relaxed,1,1,881,0.47559591373439275,0.47559591373439275,0.47559591373439275,0.47559591373439275
toxic_sk,3,em_conservative_norm,1,1,881,0.4960272417707151,0.4960272417707151,0.4960272417707151,0.4960272417707151
toxic_sk,3,em_norm,1,1,881,0.007945516458569807,0.007945516458569807,0.007945516458569807,0.007945516458569807
toxic_sk,3,em_norm_relaxed,1,1,881,0.4960272417707151,0.4960272417707151,0.4960272417707151,0.4960272417707151
truthfulqa,1,em,1,1,778,0.0,0.0,0.0,0.0
truthfulqa,1,em_conservative,1,1,778,0.007712082262210797,0.007712082262210797,0.007712082262210797,0.007712082262210797
truthfulqa,1,em_relaxed,1,1,778,0.007712082262210797,0.007712082262210797,0.007712082262210797,0.007712082262210797
truthfulqa,2,em,1,1,778,0.0,0.0,0.0,0.0
truthfulqa,2,em_conservative,1,1,778,0.016709511568123392,0.016709511568123392,0.016709511568123392,0.016709511568123392
truthfulqa,2,em_relaxed,1,1,778,0.016709511568123392,0.016709511568123392,0.016709511568123392,0.016709511568123392
truthfulqa,3,em,1,1,778,0.0,0.0,0.0,0.0
truthfulqa,3,em_conservative,1,1,778,0.011568123393316195,0.011568123393316195,0.011568123393316195,0.011568123393316195
truthfulqa,3,em_relaxed,1,1,778,0.011568123393316195,0.011568123393316195,0.011568123393316195,0.011568123393316195
1 family prompt metric task_variants prompt_count evaluated_instances task_macro_mean sample_weighted_mean min_task_value max_task_value
2 arc 1 em 1 1 1169 0.013686911890504704 0.013686911890504704 0.013686911890504704 0.013686911890504704
3 arc 1 em_conservative 1 1 1169 0.1710863986313088 0.1710863986313088 0.1710863986313088 0.1710863986313088
4 arc 1 em_relaxed 1 1 1169 0.1710863986313088 0.1710863986313088 0.1710863986313088 0.1710863986313088
5 arc 2 em 1 1 1169 0.03507271171941831 0.03507271171941831 0.03507271171941831 0.03507271171941831
6 arc 2 em_conservative 1 1 1169 0.223267750213858 0.223267750213858 0.223267750213858 0.223267750213858
7 arc 2 em_relaxed 1 1 1169 0.22412318220701455 0.22412318220701455 0.22412318220701455 0.22412318220701455
8 arc 3 em 1 1 1169 0.022241231822070145 0.022241231822070145 0.022241231822070145 0.022241231822070145
9 arc 3 em_conservative 1 1 1169 0.17621899059024806 0.17621899059024806 0.17621899059024806 0.17621899059024806
10 arc 3 em_relaxed 1 1 1169 0.1770744225834046 0.1770744225834046 0.1770744225834046 0.1770744225834046
11 belebele 1 em 1 1 900 0.02 0.02 0.02 0.02
12 belebele 1 em_conservative 1 1 900 0.21555555555555556 0.21555555555555556 0.21555555555555556 0.21555555555555556
13 belebele 1 em_relaxed 1 1 900 0.21555555555555556 0.21555555555555556 0.21555555555555556 0.21555555555555556
14 belebele 2 em 1 1 900 0.016666666666666666 0.016666666666666666 0.016666666666666666 0.016666666666666666
15 belebele 2 em_conservative 1 1 900 0.2111111111111111 0.2111111111111111 0.2111111111111111 0.2111111111111111
16 belebele 2 em_relaxed 1 1 900 0.2111111111111111 0.2111111111111111 0.2111111111111111 0.2111111111111111
17 belebele 3 em 1 1 900 0.03 0.03 0.03 0.03
18 belebele 3 em_conservative 1 1 900 0.2111111111111111 0.2111111111111111 0.2111111111111111 0.2111111111111111
19 belebele 3 em_relaxed 1 1 900 0.21333333333333335 0.21333333333333335 0.21333333333333335 0.21333333333333335
20 financial_exams 1 em 1 1 1334 0.009745127436281859 0.009745127436281859 0.009745127436281859 0.009745127436281859
21 financial_exams 1 em_conservative 1 1 1334 0.20089955022488756 0.20089955022488756 0.20089955022488756 0.20089955022488756
22 financial_exams 1 em_relaxed 1 1 1334 0.20089955022488756 0.20089955022488756 0.20089955022488756 0.20089955022488756
23 financial_exams 2 em 1 1 1334 0.07571214392803598 0.07571214392803598 0.07571214392803598 0.07571214392803598
24 financial_exams 2 em_conservative 1 1 1334 0.20539730134932535 0.20539730134932535 0.20539730134932535 0.20539730134932535
25 financial_exams 2 em_relaxed 1 1 1334 0.2383808095952024 0.2383808095952024 0.2383808095952024 0.2383808095952024
26 financial_exams 3 em 1 1 1334 0.0037481259370314842 0.0037481259370314842 0.0037481259370314842 0.0037481259370314842
27 financial_exams 3 em_conservative 1 1 1334 0.24887556221889057 0.24887556221889057 0.24887556221889057 0.24887556221889057
28 financial_exams 3 em_relaxed 1 1 1334 0.24887556221889057 0.24887556221889057 0.24887556221889057 0.24887556221889057
29 globalmmlu-lite 1 em 1 1 400 0.025 0.025 0.025 0.025
30 globalmmlu-lite 1 em_conservative 1 1 400 0.1825 0.1825 0.1825 0.1825
31 globalmmlu-lite 1 em_relaxed 1 1 400 0.1825 0.1825 0.1825 0.1825
32 globalmmlu-lite 2 em 1 1 400 0.035 0.035 0.035 0.035
33 globalmmlu-lite 2 em_conservative 1 1 400 0.21 0.21 0.21 0.21
34 globalmmlu-lite 2 em_relaxed 1 1 400 0.21 0.21 0.21 0.21
35 globalmmlu-lite 3 em 1 1 400 0.025 0.025 0.025 0.025
36 globalmmlu-lite 3 em_conservative 1 1 400 0.2025 0.2025 0.2025 0.2025
37 globalmmlu-lite 3 em_relaxed 1 1 400 0.2025 0.2025 0.2025 0.2025
38 mmlu 1 em 57 1 13062 0.023388116537651125 0.02174245904149441 0.0 0.06818181818181818
39 mmlu 1 em_conservative 57 1 13062 0.1403536981271315 0.13910580309294135 0.0053404539385847796 0.3096446700507614
40 mmlu 1 em_relaxed 57 1 13062 0.14312818797047927 0.14094319399785638 0.006675567423230975 0.3096446700507614
41 mmlu 2 em 57 1 13062 0.04274500178923145 0.040116368090644616 0.0 0.13
42 mmlu 2 em_conservative 57 1 13062 0.17602310250998576 0.17432246210381258 0.0026702269692923898 0.3553299492385787
43 mmlu 2 em_relaxed 57 1 13062 0.17964687184852454 0.17661920073495635 0.0026702269692923898 0.3553299492385787
44 mmlu 3 em 57 1 13062 0.028327141879067468 0.026871841984382177 0.0 0.07575757575757576
45 mmlu 3 em_conservative 57 1 13062 0.15358971887332207 0.15051293829428877 0.0053404539385847796 0.3350253807106599
46 mmlu 3 em_relaxed 57 1 13062 0.1575073160887174 0.15411116214974735 0.01335113484646195 0.3350253807106599
47 movie_review 1 em_conservative_norm 1 1 2008 0.6324701195219123 0.6324701195219123 0.6324701195219123 0.6324701195219123
48 movie_review 1 em_norm 1 1 2008 0.38745019920318724 0.38745019920318724 0.38745019920318724 0.38745019920318724
49 movie_review 1 em_norm_relaxed 1 1 2008 0.6329681274900398 0.6329681274900398 0.6329681274900398 0.6329681274900398
50 movie_review 2 em_conservative_norm 1 1 2008 0.47758964143426297 0.47758964143426297 0.47758964143426297 0.47758964143426297
51 movie_review 2 em_norm 1 1 2008 0.01045816733067729 0.01045816733067729 0.01045816733067729 0.01045816733067729
52 movie_review 2 em_norm_relaxed 1 1 2008 0.48306772908366535 0.48306772908366535 0.48306772908366535 0.48306772908366535
53 movie_review 3 em_conservative_norm 1 1 2008 0.09412350597609562 0.09412350597609562 0.09412350597609562 0.09412350597609562
54 movie_review 3 em_norm 1 1 2008 0.04731075697211155 0.04731075697211155 0.04731075697211155 0.04731075697211155
55 movie_review 3 em_norm_relaxed 1 1 2008 0.11105577689243028 0.11105577689243028 0.11105577689243028 0.11105577689243028
56 multi_wiki_qa 1 exact_match_slk_prefix 1 1 4997 0.36401841104662797 0.36401841104662797 0.36401841104662797 0.36401841104662797
57 multi_wiki_qa 1 f1_slk 1 1 4997 0.5156574509584114 0.5156574509584114 0.5156574509584114 0.5156574509584114
58 multi_wiki_qa 2 exact_match_slk_prefix 1 1 4997 0.36001600960576347 0.36001600960576347 0.36001600960576347 0.36001600960576347
59 multi_wiki_qa 2 f1_slk 1 1 4997 0.5090494926903776 0.5090494926903776 0.5090494926903776 0.5090494926903776
60 multi_wiki_qa 3 exact_match_slk_prefix 1 1 4997 0.34000400240144085 0.34000400240144085 0.34000400240144085 0.34000400240144085
61 multi_wiki_qa 3 f1_slk 1 1 4997 0.5050071786470199 0.5050071786470199 0.5050071786470199 0.5050071786470199
62 senti_sk 1 em_conservative_norm 2 1 8405 0.3151902608050904 0.3156454491374182 0.3111587982832618 0.31922172332691895
63 senti_sk 1 em_norm 2 1 8405 0.21165780446913535 0.2133254015466984 0.1968884120171674 0.22642719692110327
64 senti_sk 1 em_norm_relaxed 2 1 8405 0.3161543660374346 0.31659726353361095 0.31223175965665234 0.3200769724182168
65 senti_sk 2 em_conservative_norm 2 1 8405 0.4350953001676545 0.4285544318857823 0.37716484926234767 0.49302575107296137
66 senti_sk 2 em_norm 2 1 8405 0.011392328544121953 0.01225461035098156 0.0037553648068669528 0.019029292281376953
67 senti_sk 2 em_norm_relaxed 2 1 8405 0.43993914035536885 0.43355145746579415 0.383365405174257 0.4965128755364807
68 senti_sk 3 em_conservative_norm 2 1 8405 0.23176005812390282 0.23545508625817965 0.1990343347639485 0.26448578148385715
69 senti_sk 3 em_norm 2 1 8405 0.06853090550874015 0.0726948245092207 0.03165236051502146 0.10540945050245884
70 senti_sk 3 em_norm_relaxed 2 1 8405 0.3302092538502268 0.33908387864366446 0.25160944206008584 0.40880906564036773
71 skj1 1 em 1 1 100 0.0 0.0 0.0 0.0
72 skj1 1 em_conservative 1 1 100 0.02 0.02 0.02 0.02
73 skj1 1 em_relaxed 1 1 100 0.02 0.02 0.02 0.02
74 skj1 2 em 1 1 100 0.0 0.0 0.0 0.0
75 skj1 2 em_conservative 1 1 100 0.05 0.05 0.05 0.05
76 skj1 2 em_relaxed 1 1 100 0.06 0.06 0.06 0.06
77 skj1 3 em 1 1 100 0.0 0.0 0.0 0.0
78 skj1 3 em_conservative 1 1 100 0.03 0.03 0.03 0.03
79 skj1 3 em_relaxed 1 1 100 0.05 0.05 0.05 0.05
80 sklep 1 em_conservative_norm 4 1 9025 0.4046842034899774 0.2173961218836565 0.016386890487609912 0.8800383877159309
81 sklep 1 em_norm 4 1 9025 0.11229466834690793 0.0520775623268698 0.0 0.44817658349328215
82 sklep 1 em_norm_relaxed 4 1 9025 0.4050240467784944 0.21772853185595567 0.016786570743405275 0.8809980806142035
83 sklep 1 exact_match_slk_prefix 1 1 9583 0.16456224564332672 0.16456224564332672 0.16456224564332672 0.16456224564332672
84 sklep 1 f1_slk 1 1 9583 0.3730151221096053 0.3730151221096053 0.3730151221096053 0.3730151221096053
85 sklep 2 em_conservative_norm 4 1 9025 0.37570022410310483 0.26204986149584486 0.09952038369304557 0.7247915087187263
86 sklep 2 em_norm 4 1 9025 0.0006917304534838009 0.00044321329639889195 0.0 0.0018072289156626507
87 sklep 2 em_norm_relaxed 4 1 9025 0.3764699538087837 0.26249307479224376 0.09972022382094324 0.7247915087187263
88 sklep 2 exact_match_slk_prefix 1 1 9583 0.16518835437754356 0.16518835437754356 0.16518835437754356 0.16518835437754356
89 sklep 2 f1_slk 1 1 9583 0.3751701402109962 0.3751701402109962 0.3751701402109962 0.3751701402109962
90 sklep 3 em_conservative_norm 4 1 9025 0.26202719318429357 0.15889196675900277 0.001398880895283773 0.6967399545109931
91 sklep 3 em_norm 4 1 9025 0.004139976412367319 0.001994459833795014 0.0 0.015355086372360844
92 sklep 3 em_norm_relaxed 4 1 9025 0.2709043524933147 0.16299168975069253 0.001398880895283773 0.6967399545109931
93 sklep 3 exact_match_slk_prefix 1 1 9583 0.16143170197224252 0.16143170197224252 0.16143170197224252 0.16143170197224252
94 sklep 3 f1_slk 1 1 9583 0.3769392643579436 0.3769392643579436 0.3769392643579436 0.3769392643579436
95 slovakcopa 1 em 1 1 500 0.0 0.0 0.0 0.0
96 slovakcopa 1 em_conservative 1 1 500 0.508 0.508 0.508 0.508
97 slovakcopa 1 em_relaxed 1 1 500 0.618 0.618 0.618 0.618
98 slovakcopa 2 em 1 1 500 0.0 0.0 0.0 0.0
99 slovakcopa 2 em_conservative 1 1 500 0.326 0.326 0.326 0.326
100 slovakcopa 2 em_relaxed 1 1 500 0.51 0.51 0.51 0.51
101 slovakcopa 3 em 1 1 500 0.0 0.0 0.0 0.0
102 slovakcopa 3 em_conservative 1 1 500 0.084 0.084 0.084 0.084
103 slovakcopa 3 em_relaxed 1 1 500 0.086 0.086 0.086 0.086
104 toxic_sk 1 em_conservative_norm 1 1 881 0.45629965947786605 0.45629965947786605 0.45629965947786605 0.45629965947786605
105 toxic_sk 1 em_norm 1 1 881 0.0 0.0 0.0 0.0
106 toxic_sk 1 em_norm_relaxed 1 1 881 0.45629965947786605 0.45629965947786605 0.45629965947786605 0.45629965947786605
107 toxic_sk 2 em_conservative_norm 1 1 881 0.47559591373439275 0.47559591373439275 0.47559591373439275 0.47559591373439275
108 toxic_sk 2 em_norm 1 1 881 0.0 0.0 0.0 0.0
109 toxic_sk 2 em_norm_relaxed 1 1 881 0.47559591373439275 0.47559591373439275 0.47559591373439275 0.47559591373439275
110 toxic_sk 3 em_conservative_norm 1 1 881 0.4960272417707151 0.4960272417707151 0.4960272417707151 0.4960272417707151
111 toxic_sk 3 em_norm 1 1 881 0.007945516458569807 0.007945516458569807 0.007945516458569807 0.007945516458569807
112 toxic_sk 3 em_norm_relaxed 1 1 881 0.4960272417707151 0.4960272417707151 0.4960272417707151 0.4960272417707151
113 truthfulqa 1 em 1 1 778 0.0 0.0 0.0 0.0
114 truthfulqa 1 em_conservative 1 1 778 0.007712082262210797 0.007712082262210797 0.007712082262210797 0.007712082262210797
115 truthfulqa 1 em_relaxed 1 1 778 0.007712082262210797 0.007712082262210797 0.007712082262210797 0.007712082262210797
116 truthfulqa 2 em 1 1 778 0.0 0.0 0.0 0.0
117 truthfulqa 2 em_conservative 1 1 778 0.016709511568123392 0.016709511568123392 0.016709511568123392 0.016709511568123392
118 truthfulqa 2 em_relaxed 1 1 778 0.016709511568123392 0.016709511568123392 0.016709511568123392 0.016709511568123392
119 truthfulqa 3 em 1 1 778 0.0 0.0 0.0 0.0
120 truthfulqa 3 em_conservative 1 1 778 0.011568123393316195 0.011568123393316195 0.011568123393316195 0.011568123393316195
121 truthfulqa 3 em_relaxed 1 1 778 0.011568123393316195 0.011568123393316195 0.011568123393316195 0.011568123393316195

41
aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv Normal file
View File

@ -0,0 +1,41 @@
family,metric,task_variants,prompt_count,evaluated_instances,task_macro_mean,sample_weighted_mean,min_task_value,max_task_value
arc,em,3,3,3507,0.023666951810664385,0.023666951810664385,0.013686911890504704,0.03507271171941831
arc,em_conservative,3,3,3507,0.19019104647847163,0.19019104647847163,0.1710863986313088,0.223267750213858
arc,em_relaxed,3,3,3507,0.19076133447390933,0.19076133447390933,0.1710863986313088,0.22412318220701455
belebele,em,3,3,2700,0.022222222222222223,0.022222222222222223,0.016666666666666666,0.03
belebele,em_conservative,3,3,2700,0.21259259259259258,0.21259259259259258,0.2111111111111111,0.21555555555555556
belebele,em_relaxed,3,3,2700,0.21333333333333335,0.21333333333333335,0.2111111111111111,0.21555555555555556
financial_exams,em,3,3,4002,0.029735132433783108,0.029735132433783108,0.0037481259370314842,0.07571214392803598
financial_exams,em_conservative,3,3,4002,0.21839080459770116,0.21839080459770116,0.20089955022488756,0.24887556221889057
financial_exams,em_relaxed,3,3,4002,0.22938530734632684,0.22938530734632684,0.20089955022488756,0.24887556221889057
globalmmlu-lite,em,3,3,1200,0.028333333333333335,0.028333333333333332,0.025,0.035
globalmmlu-lite,em_conservative,3,3,1200,0.19833333333333333,0.19833333333333333,0.1825,0.21
globalmmlu-lite,em_relaxed,3,3,1200,0.19833333333333333,0.19833333333333333,0.1825,0.21
mmlu,em,171,3,39186,0.03148675340198335,0.029576889705507068,0.0,0.13
mmlu,em_conservative,171,3,39186,0.15665550650347979,0.15464706783034757,0.0026702269692923898,0.3553299492385787
mmlu,em_relaxed,171,3,39186,0.16009412530257372,0.15722451896085338,0.0026702269692923898,0.3553299492385787
movie_review,em_conservative_norm,3,3,6024,0.40139442231075695,0.40139442231075695,0.09412350597609562,0.6324701195219123
movie_review,em_norm,3,3,6024,0.14840637450199204,0.14840637450199204,0.01045816733067729,0.38745019920318724
movie_review,em_norm_relaxed,3,3,6024,0.40903054448871184,0.40903054448871184,0.11105577689243028,0.6329681274900398
multi_wiki_qa,exact_match_slk_prefix,3,3,14991,0.35467947435127745,0.35467947435127745,0.34000400240144085,0.36401841104662797
multi_wiki_qa,f1_slk,3,3,14991,0.5099047074319363,0.5099047074319363,0.5050071786470199,0.5156574509584114
senti_sk,em_conservative_norm,6,3,25215,0.32734853969888256,0.32655165576046,0.1990343347639485,0.49302575107296137
senti_sk,em_norm,6,3,25215,0.09719367950733247,0.09942494546896688,0.0037553648068669528,0.22642719692110327
senti_sk,em_norm_relaxed,6,3,25215,0.3621009200810101,0.36307753321435654,0.25160944206008584,0.4965128755364807
skj1,em,3,3,300,0.0,0.0,0.0,0.0
skj1,em_conservative,3,3,300,0.03333333333333333,0.03333333333333333,0.02,0.05
skj1,em_relaxed,3,3,300,0.043333333333333335,0.043333333333333335,0.02,0.06
sklep,em_conservative_norm,12,3,27075,0.34747054025912527,0.21277931671283473,0.001398880895283773,0.8800383877159309
sklep,em_norm,12,3,27075,0.039042125070919684,0.01817174515235457,0.0,0.44817658349328215
sklep,em_norm_relaxed,12,3,27075,0.3507994510268643,0.21440443213296398,0.001398880895283773,0.8809980806142035
sklep,exact_match_slk_prefix,3,3,28749,0.16372743399770426,0.16372743399770426,0.16143170197224252,0.16518835437754356
sklep,f1_slk,3,3,28749,0.3750415088928484,0.3750415088928484,0.3730151221096053,0.3769392643579436
slovakcopa,em,3,3,1500,0.0,0.0,0.0,0.0
slovakcopa,em_conservative,3,3,1500,0.306,0.306,0.084,0.508
slovakcopa,em_relaxed,3,3,1500,0.4046666666666667,0.4046666666666667,0.086,0.618
toxic_sk,em_conservative_norm,3,3,2643,0.4759742716609913,0.4759742716609913,0.45629965947786605,0.4960272417707151
toxic_sk,em_norm,3,3,2643,0.0026485054861899358,0.0026485054861899358,0.0,0.007945516458569807
toxic_sk,em_norm_relaxed,3,3,2643,0.4759742716609913,0.4759742716609913,0.45629965947786605,0.4960272417707151
truthfulqa,em,3,3,2334,0.0,0.0,0.0,0.0
truthfulqa,em_conservative,3,3,2334,0.011996572407883462,0.011996572407883462,0.007712082262210797,0.016709511568123392
truthfulqa,em_relaxed,3,3,2334,0.011996572407883462,0.011996572407883462,0.007712082262210797,0.016709511568123392
1 family metric task_variants prompt_count evaluated_instances task_macro_mean sample_weighted_mean min_task_value max_task_value
2 arc em 3 3 3507 0.023666951810664385 0.023666951810664385 0.013686911890504704 0.03507271171941831
3 arc em_conservative 3 3 3507 0.19019104647847163 0.19019104647847163 0.1710863986313088 0.223267750213858
4 arc em_relaxed 3 3 3507 0.19076133447390933 0.19076133447390933 0.1710863986313088 0.22412318220701455
5 belebele em 3 3 2700 0.022222222222222223 0.022222222222222223 0.016666666666666666 0.03
6 belebele em_conservative 3 3 2700 0.21259259259259258 0.21259259259259258 0.2111111111111111 0.21555555555555556
7 belebele em_relaxed 3 3 2700 0.21333333333333335 0.21333333333333335 0.2111111111111111 0.21555555555555556
8 financial_exams em 3 3 4002 0.029735132433783108 0.029735132433783108 0.0037481259370314842 0.07571214392803598
9 financial_exams em_conservative 3 3 4002 0.21839080459770116 0.21839080459770116 0.20089955022488756 0.24887556221889057
10 financial_exams em_relaxed 3 3 4002 0.22938530734632684 0.22938530734632684 0.20089955022488756 0.24887556221889057
11 globalmmlu-lite em 3 3 1200 0.028333333333333335 0.028333333333333332 0.025 0.035
12 globalmmlu-lite em_conservative 3 3 1200 0.19833333333333333 0.19833333333333333 0.1825 0.21
13 globalmmlu-lite em_relaxed 3 3 1200 0.19833333333333333 0.19833333333333333 0.1825 0.21
14 mmlu em 171 3 39186 0.03148675340198335 0.029576889705507068 0.0 0.13
15 mmlu em_conservative 171 3 39186 0.15665550650347979 0.15464706783034757 0.0026702269692923898 0.3553299492385787
16 mmlu em_relaxed 171 3 39186 0.16009412530257372 0.15722451896085338 0.0026702269692923898 0.3553299492385787
17 movie_review em_conservative_norm 3 3 6024 0.40139442231075695 0.40139442231075695 0.09412350597609562 0.6324701195219123
18 movie_review em_norm 3 3 6024 0.14840637450199204 0.14840637450199204 0.01045816733067729 0.38745019920318724
19 movie_review em_norm_relaxed 3 3 6024 0.40903054448871184 0.40903054448871184 0.11105577689243028 0.6329681274900398
20 multi_wiki_qa exact_match_slk_prefix 3 3 14991 0.35467947435127745 0.35467947435127745 0.34000400240144085 0.36401841104662797
21 multi_wiki_qa f1_slk 3 3 14991 0.5099047074319363 0.5099047074319363 0.5050071786470199 0.5156574509584114
22 senti_sk em_conservative_norm 6 3 25215 0.32734853969888256 0.32655165576046 0.1990343347639485 0.49302575107296137
23 senti_sk em_norm 6 3 25215 0.09719367950733247 0.09942494546896688 0.0037553648068669528 0.22642719692110327
24 senti_sk em_norm_relaxed 6 3 25215 0.3621009200810101 0.36307753321435654 0.25160944206008584 0.4965128755364807
25 skj1 em 3 3 300 0.0 0.0 0.0 0.0
26 skj1 em_conservative 3 3 300 0.03333333333333333 0.03333333333333333 0.02 0.05
27 skj1 em_relaxed 3 3 300 0.043333333333333335 0.043333333333333335 0.02 0.06
28 sklep em_conservative_norm 12 3 27075 0.34747054025912527 0.21277931671283473 0.001398880895283773 0.8800383877159309
29 sklep em_norm 12 3 27075 0.039042125070919684 0.01817174515235457 0.0 0.44817658349328215
30 sklep em_norm_relaxed 12 3 27075 0.3507994510268643 0.21440443213296398 0.001398880895283773 0.8809980806142035
31 sklep exact_match_slk_prefix 3 3 28749 0.16372743399770426 0.16372743399770426 0.16143170197224252 0.16518835437754356
32 sklep f1_slk 3 3 28749 0.3750415088928484 0.3750415088928484 0.3730151221096053 0.3769392643579436
33 slovakcopa em 3 3 1500 0.0 0.0 0.0 0.0
34 slovakcopa em_conservative 3 3 1500 0.306 0.306 0.084 0.508
35 slovakcopa em_relaxed 3 3 1500 0.4046666666666667 0.4046666666666667 0.086 0.618
36 toxic_sk em_conservative_norm 3 3 2643 0.4759742716609913 0.4759742716609913 0.45629965947786605 0.4960272417707151
37 toxic_sk em_norm 3 3 2643 0.0026485054861899358 0.0026485054861899358 0.0 0.007945516458569807
38 toxic_sk em_norm_relaxed 3 3 2643 0.4759742716609913 0.4759742716609913 0.45629965947786605 0.4960272417707151
39 truthfulqa em 3 3 2334 0.0 0.0 0.0 0.0
40 truthfulqa em_conservative 3 3 2334 0.011996572407883462 0.011996572407883462 0.007712082262210797 0.016709511568123392
41 truthfulqa em_relaxed 3 3 2334 0.011996572407883462 0.011996572407883462 0.007712082262210797 0.016709511568123392

70
aya_300k_lr_experiments/benchmarks/skbench/results/input_length_budget_boundary_rows.csv Normal file
View File

@ -0,0 +1,70 @@
task,details_row_index_zero_based,input_token_array_length,generation_size,input_budget,recorded_input_characters
multi_wiki_qa.prompt1_qa_sk|0,47,32642,126,32642,97590
multi_wiki_qa.prompt1_qa_sk|0,605,32642,126,32642,71863
multi_wiki_qa.prompt1_qa_sk|0,724,32642,126,32642,97580
multi_wiki_qa.prompt1_qa_sk|0,837,32642,126,32642,121781
multi_wiki_qa.prompt1_qa_sk|0,881,32642,126,32642,71849
multi_wiki_qa.prompt1_qa_sk|0,1155,32642,126,32642,71849
multi_wiki_qa.prompt1_qa_sk|0,1479,32642,126,32642,71827
multi_wiki_qa.prompt1_qa_sk|0,1770,32642,126,32642,121822
multi_wiki_qa.prompt1_qa_sk|0,2200,32642,126,32642,97575
multi_wiki_qa.prompt1_qa_sk|0,2352,32642,126,32642,121852
multi_wiki_qa.prompt1_qa_sk|0,2447,32642,126,32642,71858
multi_wiki_qa.prompt1_qa_sk|0,2507,32642,126,32642,121802
multi_wiki_qa.prompt1_qa_sk|0,2890,32642,126,32642,71841
multi_wiki_qa.prompt1_qa_sk|0,3094,32642,126,32642,121827
multi_wiki_qa.prompt1_qa_sk|0,3098,32642,126,32642,97594
multi_wiki_qa.prompt1_qa_sk|0,3354,32642,126,32642,97576
multi_wiki_qa.prompt1_qa_sk|0,4088,32642,126,32642,121819
multi_wiki_qa.prompt1_qa_sk|0,4511,32642,126,32642,97623
multi_wiki_qa.prompt1_qa_sk|0,4584,32642,126,32642,71833
multi_wiki_qa.prompt1_qa_sk|0,4667,32642,126,32642,71832
multi_wiki_qa.prompt1_qa_sk|0,4688,32642,126,32642,97578
multi_wiki_qa.prompt1_qa_sk|0,4752,32642,126,32642,121811
multi_wiki_qa.prompt1_qa_sk|0,4837,32642,126,32642,71839
multi_wiki_qa.prompt2_qa_sk|0,47,32642,126,32642,97593
multi_wiki_qa.prompt2_qa_sk|0,605,32642,126,32642,71866
multi_wiki_qa.prompt2_qa_sk|0,724,32642,126,32642,97583
multi_wiki_qa.prompt2_qa_sk|0,837,32642,126,32642,121784
multi_wiki_qa.prompt2_qa_sk|0,881,32642,126,32642,71852
multi_wiki_qa.prompt2_qa_sk|0,1155,32642,126,32642,71852
multi_wiki_qa.prompt2_qa_sk|0,1479,32642,126,32642,71830
multi_wiki_qa.prompt2_qa_sk|0,1770,32642,126,32642,121825
multi_wiki_qa.prompt2_qa_sk|0,2200,32642,126,32642,97578
multi_wiki_qa.prompt2_qa_sk|0,2352,32642,126,32642,121855
multi_wiki_qa.prompt2_qa_sk|0,2447,32642,126,32642,71861
multi_wiki_qa.prompt2_qa_sk|0,2507,32642,126,32642,121805
multi_wiki_qa.prompt2_qa_sk|0,2890,32642,126,32642,71844
multi_wiki_qa.prompt2_qa_sk|0,3094,32642,126,32642,121830
multi_wiki_qa.prompt2_qa_sk|0,3098,32642,126,32642,97597
multi_wiki_qa.prompt2_qa_sk|0,3354,32642,126,32642,97579
multi_wiki_qa.prompt2_qa_sk|0,4088,32642,126,32642,121822
multi_wiki_qa.prompt2_qa_sk|0,4511,32642,126,32642,97626
multi_wiki_qa.prompt2_qa_sk|0,4584,32642,126,32642,71836
multi_wiki_qa.prompt2_qa_sk|0,4667,32642,126,32642,71835
multi_wiki_qa.prompt2_qa_sk|0,4688,32642,126,32642,97581
multi_wiki_qa.prompt2_qa_sk|0,4752,32642,126,32642,121814
multi_wiki_qa.prompt2_qa_sk|0,4837,32642,126,32642,71842
multi_wiki_qa.prompt3_qa_sk|0,47,32642,126,32642,97521
multi_wiki_qa.prompt3_qa_sk|0,605,32642,126,32642,71794
multi_wiki_qa.prompt3_qa_sk|0,724,32642,126,32642,97511
multi_wiki_qa.prompt3_qa_sk|0,837,32642,126,32642,121712
multi_wiki_qa.prompt3_qa_sk|0,881,32642,126,32642,71780
multi_wiki_qa.prompt3_qa_sk|0,1155,32642,126,32642,71780
multi_wiki_qa.prompt3_qa_sk|0,1479,32642,126,32642,71758
multi_wiki_qa.prompt3_qa_sk|0,1770,32642,126,32642,121753
multi_wiki_qa.prompt3_qa_sk|0,2200,32642,126,32642,97506
multi_wiki_qa.prompt3_qa_sk|0,2352,32642,126,32642,121783
multi_wiki_qa.prompt3_qa_sk|0,2447,32642,126,32642,71789
multi_wiki_qa.prompt3_qa_sk|0,2507,32642,126,32642,121733
multi_wiki_qa.prompt3_qa_sk|0,2890,32642,126,32642,71772
multi_wiki_qa.prompt3_qa_sk|0,3094,32642,126,32642,121758
multi_wiki_qa.prompt3_qa_sk|0,3098,32642,126,32642,97525
multi_wiki_qa.prompt3_qa_sk|0,3354,32642,126,32642,97507
multi_wiki_qa.prompt3_qa_sk|0,4088,32642,126,32642,121750
multi_wiki_qa.prompt3_qa_sk|0,4511,32642,126,32642,97554
multi_wiki_qa.prompt3_qa_sk|0,4584,32642,126,32642,71764
multi_wiki_qa.prompt3_qa_sk|0,4667,32642,126,32642,71763
multi_wiki_qa.prompt3_qa_sk|0,4688,32642,126,32642,97509
multi_wiki_qa.prompt3_qa_sk|0,4752,32642,126,32642,121742
multi_wiki_qa.prompt3_qa_sk|0,4837,32642,126,32642,71770
1 task details_row_index_zero_based input_token_array_length generation_size input_budget recorded_input_characters
2 multi_wiki_qa.prompt1_qa_sk|0 47 32642 126 32642 97590
3 multi_wiki_qa.prompt1_qa_sk|0 605 32642 126 32642 71863
4 multi_wiki_qa.prompt1_qa_sk|0 724 32642 126 32642 97580
5 multi_wiki_qa.prompt1_qa_sk|0 837 32642 126 32642 121781
6 multi_wiki_qa.prompt1_qa_sk|0 881 32642 126 32642 71849
7 multi_wiki_qa.prompt1_qa_sk|0 1155 32642 126 32642 71849
8 multi_wiki_qa.prompt1_qa_sk|0 1479 32642 126 32642 71827
9 multi_wiki_qa.prompt1_qa_sk|0 1770 32642 126 32642 121822
10 multi_wiki_qa.prompt1_qa_sk|0 2200 32642 126 32642 97575
11 multi_wiki_qa.prompt1_qa_sk|0 2352 32642 126 32642 121852
12 multi_wiki_qa.prompt1_qa_sk|0 2447 32642 126 32642 71858
13 multi_wiki_qa.prompt1_qa_sk|0 2507 32642 126 32642 121802
14 multi_wiki_qa.prompt1_qa_sk|0 2890 32642 126 32642 71841
15 multi_wiki_qa.prompt1_qa_sk|0 3094 32642 126 32642 121827
16 multi_wiki_qa.prompt1_qa_sk|0 3098 32642 126 32642 97594
17 multi_wiki_qa.prompt1_qa_sk|0 3354 32642 126 32642 97576
18 multi_wiki_qa.prompt1_qa_sk|0 4088 32642 126 32642 121819
19 multi_wiki_qa.prompt1_qa_sk|0 4511 32642 126 32642 97623
20 multi_wiki_qa.prompt1_qa_sk|0 4584 32642 126 32642 71833
21 multi_wiki_qa.prompt1_qa_sk|0 4667 32642 126 32642 71832
22 multi_wiki_qa.prompt1_qa_sk|0 4688 32642 126 32642 97578
23 multi_wiki_qa.prompt1_qa_sk|0 4752 32642 126 32642 121811
24 multi_wiki_qa.prompt1_qa_sk|0 4837 32642 126 32642 71839
25 multi_wiki_qa.prompt2_qa_sk|0 47 32642 126 32642 97593
26 multi_wiki_qa.prompt2_qa_sk|0 605 32642 126 32642 71866
27 multi_wiki_qa.prompt2_qa_sk|0 724 32642 126 32642 97583
28 multi_wiki_qa.prompt2_qa_sk|0 837 32642 126 32642 121784
29 multi_wiki_qa.prompt2_qa_sk|0 881 32642 126 32642 71852
30 multi_wiki_qa.prompt2_qa_sk|0 1155 32642 126 32642 71852
31 multi_wiki_qa.prompt2_qa_sk|0 1479 32642 126 32642 71830
32 multi_wiki_qa.prompt2_qa_sk|0 1770 32642 126 32642 121825
33 multi_wiki_qa.prompt2_qa_sk|0 2200 32642 126 32642 97578
34 multi_wiki_qa.prompt2_qa_sk|0 2352 32642 126 32642 121855
35 multi_wiki_qa.prompt2_qa_sk|0 2447 32642 126 32642 71861
36 multi_wiki_qa.prompt2_qa_sk|0 2507 32642 126 32642 121805
37 multi_wiki_qa.prompt2_qa_sk|0 2890 32642 126 32642 71844
38 multi_wiki_qa.prompt2_qa_sk|0 3094 32642 126 32642 121830
39 multi_wiki_qa.prompt2_qa_sk|0 3098 32642 126 32642 97597
40 multi_wiki_qa.prompt2_qa_sk|0 3354 32642 126 32642 97579
41 multi_wiki_qa.prompt2_qa_sk|0 4088 32642 126 32642 121822
42 multi_wiki_qa.prompt2_qa_sk|0 4511 32642 126 32642 97626
43 multi_wiki_qa.prompt2_qa_sk|0 4584 32642 126 32642 71836
44 multi_wiki_qa.prompt2_qa_sk|0 4667 32642 126 32642 71835
45 multi_wiki_qa.prompt2_qa_sk|0 4688 32642 126 32642 97581
46 multi_wiki_qa.prompt2_qa_sk|0 4752 32642 126 32642 121814
47 multi_wiki_qa.prompt2_qa_sk|0 4837 32642 126 32642 71842
48 multi_wiki_qa.prompt3_qa_sk|0 47 32642 126 32642 97521
49 multi_wiki_qa.prompt3_qa_sk|0 605 32642 126 32642 71794
50 multi_wiki_qa.prompt3_qa_sk|0 724 32642 126 32642 97511
51 multi_wiki_qa.prompt3_qa_sk|0 837 32642 126 32642 121712
52 multi_wiki_qa.prompt3_qa_sk|0 881 32642 126 32642 71780
53 multi_wiki_qa.prompt3_qa_sk|0 1155 32642 126 32642 71780
54 multi_wiki_qa.prompt3_qa_sk|0 1479 32642 126 32642 71758
55 multi_wiki_qa.prompt3_qa_sk|0 1770 32642 126 32642 121753
56 multi_wiki_qa.prompt3_qa_sk|0 2200 32642 126 32642 97506
57 multi_wiki_qa.prompt3_qa_sk|0 2352 32642 126 32642 121783
58 multi_wiki_qa.prompt3_qa_sk|0 2447 32642 126 32642 71789
59 multi_wiki_qa.prompt3_qa_sk|0 2507 32642 126 32642 121733
60 multi_wiki_qa.prompt3_qa_sk|0 2890 32642 126 32642 71772
61 multi_wiki_qa.prompt3_qa_sk|0 3094 32642 126 32642 121758
62 multi_wiki_qa.prompt3_qa_sk|0 3098 32642 126 32642 97525
63 multi_wiki_qa.prompt3_qa_sk|0 3354 32642 126 32642 97507
64 multi_wiki_qa.prompt3_qa_sk|0 4088 32642 126 32642 121750
65 multi_wiki_qa.prompt3_qa_sk|0 4511 32642 126 32642 97554
66 multi_wiki_qa.prompt3_qa_sk|0 4584 32642 126 32642 71764
67 multi_wiki_qa.prompt3_qa_sk|0 4667 32642 126 32642 71763
68 multi_wiki_qa.prompt3_qa_sk|0 4688 32642 126 32642 97509
69 multi_wiki_qa.prompt3_qa_sk|0 4752 32642 126 32642 121742
70 multi_wiki_qa.prompt3_qa_sk|0 4837 32642 126 32642 71770

223
aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_per_task.csv Normal file
View File

@ -0,0 +1,223 @@
task,generation_size,context_limit,input_budget,samples,null_token_arrays,null_input_strings,min_recorded_input_tokens,max_recorded_input_tokens,mean_recorded_input_tokens,arrays_at_input_budget,arrays_over_input_budget,max_recorded_input_characters
arc.prompt1_mqa_single_correct_sk|0,32,32768,32736,1169,0,0,147,539,236.47733105218134,0,0,1275
arc.prompt2_mqa_single_correct_sk|0,32,32768,32736,1169,0,0,131,523,220.47733105218134,0,0,1233
arc.prompt3_mqa_single_correct_sk|0,32,32768,32736,1169,0,0,142,534,231.47733105218134,0,0,1261
belebele.prompt1_mqa_single_correct_sk|0,32,32768,32736,900,0,0,237,905,434.35777777777776,0,0,2118
belebele.prompt2_mqa_single_correct_sk|0,32,32768,32736,900,0,0,221,889,418.35777777777776,0,0,2076
belebele.prompt3_mqa_single_correct_sk|0,32,32768,32736,900,0,0,232,900,429.35777777777776,0,0,2104
financial_exams.prompt1_completion_sk|0,32,32768,32736,1334,0,0,152,682,295.84107946026984,0,0,1642
financial_exams.prompt2_completion_sk|0,32,32768,32736,1334,0,0,149,679,292.84107946026984,0,0,1633
financial_exams.prompt3_completion_sk|0,32,32768,32736,1334,0,0,137,667,280.84107946026984,0,0,1610
globalmmlu-lite.prompt1_mqa_single_correct_sk|0,32,32768,32736,400,0,0,153,1165,301.385,0,0,2646
globalmmlu-lite.prompt2_mqa_single_correct_sk|0,32,32768,32736,400,0,0,137,1149,285.385,0,0,2604
globalmmlu-lite.prompt3_mqa_single_correct_sk|0,32,32768,32736,400,0,0,148,1160,296.385,0,0,2632
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0,32,32768,32736,100,0,0,154,266,210.72,0,0,626
mmlu.prompt1_mqa_single_correct_sk:anatomy|0,32,32768,32736,134,0,0,153,384,220.02238805970148,0,0,908
mmlu.prompt1_mqa_single_correct_sk:astronomy|0,32,32768,32736,152,0,0,163,480,249.75,0,0,1099
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0,32,32768,32736,100,0,0,160,419,257.8,0,0,1001
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0,32,32768,32736,259,0,0,151,377,224.1853281853282,0,0,888
mmlu.prompt1_mqa_single_correct_sk:college_biology|0,32,32768,32736,142,0,0,171,479,257.1478873239437,0,0,1074
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0,32,32768,32736,100,0,0,160,389,234.02,0,0,952
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0,32,32768,32736,100,0,0,174,602,307.65,0,0,1433
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0,32,32768,32736,99,0,0,163,356,234.63636363636363,0,0,810
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0,32,32768,32736,172,0,0,166,2217,313.7325581395349,0,0,5040
mmlu.prompt1_mqa_single_correct_sk:college_physics|0,32,32768,32736,102,0,0,160,364,246.8921568627451,0,0,861
mmlu.prompt1_mqa_single_correct_sk:computer_security|0,32,32768,32736,100,0,0,161,486,234.18,0,0,1197
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0,32,32768,32736,235,0,0,155,280,195.20425531914893,0,0,669
mmlu.prompt1_mqa_single_correct_sk:econometrics|0,32,32768,32736,112,0,0,164,484,289.25892857142856,0,0,1078
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0,32,32768,32736,145,0,0,148,270,195.91034482758621,0,0,647
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0,32,32768,32736,373,0,0,152,393,212.76139410187668,0,0,927
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0,32,32768,32736,115,0,0,168,572,295.5304347826087,0,0,1336
mmlu.prompt1_mqa_single_correct_sk:global_facts|0,32,32768,32736,100,0,0,166,376,207.62,0,0,860
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0,32,32768,32736,282,0,0,165,469,264.54609929078015,0,0,1078
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0,32,32768,32736,198,0,0,153,484,247.45454545454547,0,0,1110
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0,32,32768,32736,99,0,0,157,760,311.77777777777777,0,0,1787
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0,32,32768,32736,135,0,0,355,1239,745.1703703703704,0,0,2795
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0,32,32768,32736,198,0,0,158,358,207.76262626262627,0,0,817
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0,32,32768,32736,193,0,0,188,470,259.4974093264249,0,0,1096
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0,32,32768,32736,387,0,0,169,441,248.3798449612403,0,0,984
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0,32,32768,32736,266,0,0,156,430,219.24436090225564,0,0,948
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0,32,32768,32736,217,0,0,171,542,253.58986175115209,0,0,1203
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0,32,32768,32736,150,0,0,179,575,284.43333333333334,0,0,1342
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0,32,32768,32736,536,0,0,159,670,238.3320895522388,0,0,1588
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0,32,32768,32736,214,0,0,187,778,343.33644859813086,0,0,1769
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0,32,32768,32736,57,0,0,370,1075,585.6842105263158,0,0,2520
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0,32,32768,32736,180,0,0,261,1734,729.6333333333333,0,0,4002
mmlu.prompt1_mqa_single_correct_sk:human_aging|0,32,32768,32736,219,0,0,160,341,206.5068493150685,0,0,825
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0,32,32768,32736,109,0,0,159,334,211.24770642201835,0,0,855
mmlu.prompt1_mqa_single_correct_sk:international_law|0,32,32768,32736,118,0,0,171,454,302.9491525423729,0,0,1019
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0,32,32768,32736,107,0,0,158,481,244.4018691588785,0,0,1116
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0,32,32768,32736,158,0,0,163,412,247.30379746835442,0,0,979
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0,32,32768,32736,112,0,0,170,435,255.64285714285714,0,0,909
mmlu.prompt1_mqa_single_correct_sk:management|0,32,32768,32736,100,0,0,154,285,197.01,0,0,728
mmlu.prompt1_mqa_single_correct_sk:marketing|0,32,32768,32736,234,0,0,170,361,230.77777777777777,0,0,853
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0,32,32768,32736,100,0,0,161,304,207.78,0,0,728
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0,32,32768,32736,779,0,0,147,565,196.65340179717586,0,0,1348
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0,32,32768,32736,326,0,0,164,411,249.94171779141104,0,0,992
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0,32,32768,32736,749,0,0,261,382,310.8558077436582,0,0,890
mmlu.prompt1_mqa_single_correct_sk:nutrition|0,32,32768,32736,303,0,0,151,476,250.0990099009901,0,0,1123
mmlu.prompt1_mqa_single_correct_sk:philosophy|0,32,32768,32736,308,0,0,161,548,226.2922077922078,0,0,1306
mmlu.prompt1_mqa_single_correct_sk:prehistory|0,32,32768,32736,319,0,0,150,495,236.16614420062695,0,0,1118
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0,32,32768,32736,276,0,0,184,582,315.0217391304348,0,0,1193
mmlu.prompt1_mqa_single_correct_sk:professional_law|0,32,32768,32736,1171,0,0,168,1660,593.5713065755764,0,0,3903
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0,32,32768,32736,264,0,0,261,855,456.2537878787879,0,0,1946
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0,32,32768,32736,596,0,0,161,595,267.18624161073825,0,0,1449
mmlu.prompt1_mqa_single_correct_sk:public_relations|0,32,32768,32736,109,0,0,152,513,224.1559633027523,0,0,1207
mmlu.prompt1_mqa_single_correct_sk:security_studies|0,32,32768,32736,222,0,0,164,1068,439.8063063063063,0,0,2495
mmlu.prompt1_mqa_single_correct_sk:sociology|0,32,32768,32736,197,0,0,170,440,254.248730964467,0,0,1038
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0,32,32768,32736,99,0,0,163,389,233.3939393939394,0,0,900
mmlu.prompt1_mqa_single_correct_sk:virology|0,32,32768,32736,164,0,0,162,673,219.10365853658536,0,0,1384
mmlu.prompt1_mqa_single_correct_sk:world_religions|0,32,32768,32736,171,0,0,155,255,178.2046783625731,0,0,611
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0,32,32768,32736,100,0,0,138,250,194.72,0,0,584
mmlu.prompt2_mqa_single_correct_sk:anatomy|0,32,32768,32736,134,0,0,137,368,204.02238805970148,0,0,866
mmlu.prompt2_mqa_single_correct_sk:astronomy|0,32,32768,32736,152,0,0,147,464,233.75,0,0,1057
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0,32,32768,32736,100,0,0,144,403,241.8,0,0,959
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0,32,32768,32736,259,0,0,135,361,208.1853281853282,0,0,846
mmlu.prompt2_mqa_single_correct_sk:college_biology|0,32,32768,32736,142,0,0,155,463,241.14788732394365,0,0,1032
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0,32,32768,32736,100,0,0,144,373,218.02,0,0,910
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0,32,32768,32736,100,0,0,158,586,291.65,0,0,1391
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0,32,32768,32736,99,0,0,147,340,218.63636363636363,0,0,768
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0,32,32768,32736,172,0,0,150,2201,297.7325581395349,0,0,4998
mmlu.prompt2_mqa_single_correct_sk:college_physics|0,32,32768,32736,102,0,0,144,348,230.8921568627451,0,0,819
mmlu.prompt2_mqa_single_correct_sk:computer_security|0,32,32768,32736,100,0,0,145,470,218.18,0,0,1155
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0,32,32768,32736,235,0,0,139,264,179.20425531914893,0,0,627
mmlu.prompt2_mqa_single_correct_sk:econometrics|0,32,32768,32736,112,0,0,148,468,273.25892857142856,0,0,1036
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0,32,32768,32736,145,0,0,132,254,179.91034482758621,0,0,605
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0,32,32768,32736,373,0,0,136,377,196.76139410187668,0,0,885
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0,32,32768,32736,115,0,0,152,556,279.5304347826087,0,0,1294
mmlu.prompt2_mqa_single_correct_sk:global_facts|0,32,32768,32736,100,0,0,150,360,191.62,0,0,818
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0,32,32768,32736,282,0,0,149,453,248.54609929078015,0,0,1036
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0,32,32768,32736,198,0,0,137,468,231.45454545454547,0,0,1068
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0,32,32768,32736,99,0,0,141,744,295.77777777777777,0,0,1745
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0,32,32768,32736,135,0,0,339,1223,729.1703703703704,0,0,2753
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0,32,32768,32736,198,0,0,142,342,191.76262626262627,0,0,775
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0,32,32768,32736,193,0,0,172,454,243.49740932642487,0,0,1054
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0,32,32768,32736,387,0,0,153,425,232.3798449612403,0,0,942
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0,32,32768,32736,266,0,0,140,414,203.24436090225564,0,0,906
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0,32,32768,32736,217,0,0,155,526,237.58986175115209,0,0,1161
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0,32,32768,32736,150,0,0,163,559,268.43333333333334,0,0,1300
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0,32,32768,32736,536,0,0,143,654,222.3320895522388,0,0,1546
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0,32,32768,32736,214,0,0,171,762,327.33644859813086,0,0,1727
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0,32,32768,32736,57,0,0,354,1059,569.6842105263158,0,0,2478
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0,32,32768,32736,180,0,0,245,1718,713.6333333333333,0,0,3960
mmlu.prompt2_mqa_single_correct_sk:human_aging|0,32,32768,32736,219,0,0,144,325,190.5068493150685,0,0,783
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0,32,32768,32736,109,0,0,143,318,195.24770642201835,0,0,813
mmlu.prompt2_mqa_single_correct_sk:international_law|0,32,32768,32736,118,0,0,155,438,286.9491525423729,0,0,977
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0,32,32768,32736,107,0,0,142,465,228.4018691588785,0,0,1074
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0,32,32768,32736,158,0,0,147,396,231.30379746835442,0,0,937
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0,32,32768,32736,112,0,0,154,419,239.64285714285714,0,0,867
mmlu.prompt2_mqa_single_correct_sk:management|0,32,32768,32736,100,0,0,138,269,181.01,0,0,686
mmlu.prompt2_mqa_single_correct_sk:marketing|0,32,32768,32736,234,0,0,154,345,214.77777777777777,0,0,811
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0,32,32768,32736,100,0,0,145,288,191.78,0,0,686
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0,32,32768,32736,779,0,0,131,549,180.65340179717586,0,0,1306
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0,32,32768,32736,326,0,0,148,395,233.94171779141104,0,0,950
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0,32,32768,32736,749,0,0,245,366,294.8558077436582,0,0,848
mmlu.prompt2_mqa_single_correct_sk:nutrition|0,32,32768,32736,303,0,0,135,460,234.0990099009901,0,0,1081
mmlu.prompt2_mqa_single_correct_sk:philosophy|0,32,32768,32736,308,0,0,145,532,210.2922077922078,0,0,1264
mmlu.prompt2_mqa_single_correct_sk:prehistory|0,32,32768,32736,319,0,0,134,479,220.16614420062695,0,0,1076
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0,32,32768,32736,276,0,0,168,566,299.0217391304348,0,0,1151
mmlu.prompt2_mqa_single_correct_sk:professional_law|0,32,32768,32736,1171,0,0,152,1644,577.5713065755764,0,0,3861
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0,32,32768,32736,264,0,0,245,839,440.2537878787879,0,0,1904
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0,32,32768,32736,596,0,0,145,579,251.18624161073825,0,0,1407
mmlu.prompt2_mqa_single_correct_sk:public_relations|0,32,32768,32736,109,0,0,136,497,208.1559633027523,0,0,1165
mmlu.prompt2_mqa_single_correct_sk:security_studies|0,32,32768,32736,222,0,0,148,1052,423.8063063063063,0,0,2453
mmlu.prompt2_mqa_single_correct_sk:sociology|0,32,32768,32736,197,0,0,154,424,238.248730964467,0,0,996
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0,32,32768,32736,99,0,0,147,373,217.3939393939394,0,0,858
mmlu.prompt2_mqa_single_correct_sk:virology|0,32,32768,32736,164,0,0,146,657,203.10365853658536,0,0,1342
mmlu.prompt2_mqa_single_correct_sk:world_religions|0,32,32768,32736,171,0,0,139,239,162.2046783625731,0,0,569
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0,32,32768,32736,100,0,0,149,261,205.72,0,0,612
mmlu.prompt3_mqa_single_correct_sk:anatomy|0,32,32768,32736,134,0,0,148,379,215.02238805970148,0,0,894
mmlu.prompt3_mqa_single_correct_sk:astronomy|0,32,32768,32736,152,0,0,158,475,244.75,0,0,1085
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0,32,32768,32736,100,0,0,155,414,252.8,0,0,987
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0,32,32768,32736,259,0,0,146,372,219.1853281853282,0,0,874
mmlu.prompt3_mqa_single_correct_sk:college_biology|0,32,32768,32736,142,0,0,166,474,252.14788732394365,0,0,1060
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0,32,32768,32736,100,0,0,155,384,229.02,0,0,938
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0,32,32768,32736,100,0,0,169,597,302.65,0,0,1419
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0,32,32768,32736,99,0,0,158,351,229.63636363636363,0,0,796
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0,32,32768,32736,172,0,0,161,2212,308.7325581395349,0,0,5026
mmlu.prompt3_mqa_single_correct_sk:college_physics|0,32,32768,32736,102,0,0,155,359,241.8921568627451,0,0,847
mmlu.prompt3_mqa_single_correct_sk:computer_security|0,32,32768,32736,100,0,0,156,481,229.18,0,0,1183
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0,32,32768,32736,235,0,0,150,275,190.20425531914893,0,0,655
mmlu.prompt3_mqa_single_correct_sk:econometrics|0,32,32768,32736,112,0,0,159,479,284.25892857142856,0,0,1064
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0,32,32768,32736,145,0,0,143,265,190.91034482758621,0,0,633
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0,32,32768,32736,373,0,0,147,388,207.76139410187668,0,0,913
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0,32,32768,32736,115,0,0,163,567,290.5304347826087,0,0,1322
mmlu.prompt3_mqa_single_correct_sk:global_facts|0,32,32768,32736,100,0,0,161,371,202.62,0,0,846
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0,32,32768,32736,282,0,0,160,464,259.54609929078015,0,0,1064
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0,32,32768,32736,198,0,0,148,479,242.45454545454547,0,0,1096
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0,32,32768,32736,99,0,0,152,755,306.77777777777777,0,0,1773
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0,32,32768,32736,135,0,0,350,1234,740.1703703703704,0,0,2781
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0,32,32768,32736,198,0,0,153,353,202.76262626262627,0,0,803
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0,32,32768,32736,193,0,0,183,465,254.49740932642487,0,0,1082
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0,32,32768,32736,387,0,0,164,436,243.3798449612403,0,0,970
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0,32,32768,32736,266,0,0,151,425,214.24436090225564,0,0,934
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0,32,32768,32736,217,0,0,166,537,248.58986175115209,0,0,1189
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0,32,32768,32736,150,0,0,174,570,279.43333333333334,0,0,1328
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0,32,32768,32736,536,0,0,154,665,233.3320895522388,0,0,1574
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0,32,32768,32736,214,0,0,182,773,338.33644859813086,0,0,1755
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0,32,32768,32736,57,0,0,365,1070,580.6842105263158,0,0,2506
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0,32,32768,32736,180,0,0,256,1729,724.6333333333333,0,0,3988
mmlu.prompt3_mqa_single_correct_sk:human_aging|0,32,32768,32736,219,0,0,155,336,201.5068493150685,0,0,811
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0,32,32768,32736,109,0,0,154,329,206.24770642201835,0,0,841
mmlu.prompt3_mqa_single_correct_sk:international_law|0,32,32768,32736,118,0,0,166,449,297.9491525423729,0,0,1005
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0,32,32768,32736,107,0,0,153,476,239.4018691588785,0,0,1102
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0,32,32768,32736,158,0,0,158,407,242.30379746835442,0,0,965
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0,32,32768,32736,112,0,0,165,430,250.64285714285714,0,0,895
mmlu.prompt3_mqa_single_correct_sk:management|0,32,32768,32736,100,0,0,149,280,192.01,0,0,714
mmlu.prompt3_mqa_single_correct_sk:marketing|0,32,32768,32736,234,0,0,165,356,225.77777777777777,0,0,839
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0,32,32768,32736,100,0,0,156,299,202.78,0,0,714
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0,32,32768,32736,779,0,0,142,560,191.65340179717586,0,0,1334
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0,32,32768,32736,326,0,0,159,406,244.94171779141104,0,0,978
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0,32,32768,32736,749,0,0,256,377,305.8558077436582,0,0,876
mmlu.prompt3_mqa_single_correct_sk:nutrition|0,32,32768,32736,303,0,0,146,471,245.0990099009901,0,0,1109
mmlu.prompt3_mqa_single_correct_sk:philosophy|0,32,32768,32736,308,0,0,156,543,221.2922077922078,0,0,1292
mmlu.prompt3_mqa_single_correct_sk:prehistory|0,32,32768,32736,319,0,0,145,490,231.16614420062695,0,0,1104
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0,32,32768,32736,276,0,0,179,577,310.0217391304348,0,0,1179
mmlu.prompt3_mqa_single_correct_sk:professional_law|0,32,32768,32736,1171,0,0,163,1655,588.5713065755764,0,0,3889
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0,32,32768,32736,264,0,0,256,850,451.2537878787879,0,0,1932
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0,32,32768,32736,596,0,0,156,590,262.18624161073825,0,0,1435
mmlu.prompt3_mqa_single_correct_sk:public_relations|0,32,32768,32736,109,0,0,147,508,219.1559633027523,0,0,1193
mmlu.prompt3_mqa_single_correct_sk:security_studies|0,32,32768,32736,222,0,0,159,1063,434.8063063063063,0,0,2481
mmlu.prompt3_mqa_single_correct_sk:sociology|0,32,32768,32736,197,0,0,165,435,249.248730964467,0,0,1024
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0,32,32768,32736,99,0,0,158,384,228.3939393939394,0,0,886
mmlu.prompt3_mqa_single_correct_sk:virology|0,32,32768,32736,164,0,0,157,668,214.10365853658536,0,0,1370
mmlu.prompt3_mqa_single_correct_sk:world_religions|0,32,32768,32736,171,0,0,150,250,173.2046783625731,0,0,597
movie_review.prompt1_sentiment_binary_sk|0,32,32768,32736,2008,0,0,91,1799,249.22211155378486,0,0,4162
movie_review.prompt2_sentiment_binary_sk|0,32,32768,32736,2008,0,0,102,1810,260.2221115537849,0,0,4184
movie_review.prompt3_sentiment_binary_sk|0,32,32768,32736,2008,0,0,70,1778,228.22211155378486,0,0,4118
multi_wiki_qa.prompt1_qa_sk|0,126,32768,32642,4997,0,0,512,32642,1964.61997198319,23,0,121852
multi_wiki_qa.prompt2_qa_sk|0,126,32768,32642,4997,0,0,517,32642,1969.596958174905,23,0,121855
multi_wiki_qa.prompt3_qa_sk|0,126,32768,32642,4997,0,0,483,32642,1935.7534520712427,23,0,121783
senti_sk.prompt1_sentiment_3_class_sk|0,32,32768,32736,4677,0,0,94,551,123.0096215522771,0,0,1271
senti_sk.prompt1_sentiment_binary_sk|0,32,32768,32736,3728,0,0,88,545,117.15665236051503,0,0,1258
senti_sk.prompt2_sentiment_3_class_sk|0,32,32768,32736,4677,0,0,109,566,138.0096215522771,0,0,1304
senti_sk.prompt2_sentiment_binary_sk|0,32,32768,32736,3728,0,0,99,556,128.156652360515,0,0,1280
senti_sk.prompt3_sentiment_3_class_sk|0,32,32768,32736,4677,0,0,72,529,101.0096215522771,0,0,1224
senti_sk.prompt3_sentiment_binary_sk|0,32,32768,32736,3728,0,0,67,524,96.15665236051503,0,0,1214
skj1.prompt1_skj1_sk|0,32,32768,32736,100,0,0,175,317,215.89,0,0,721
skj1.prompt2_skj1_sk|0,32,32768,32736,100,0,0,162,304,202.89,0,0,691
skj1.prompt3_skj1_sk|0,32,32768,32736,100,0,0,163,305,203.89,0,0,696
sklep.prompt1_hate_speech_sk:hate-speech|0,32,32768,32736,1319,0,0,94,728,129.9871114480667,0,0,1831
sklep.prompt1_nli_3_classes_sk:nli|0,32,32768,32736,5004,0,0,136,292,192.7312150279776,0,0,698
sklep.prompt1_nli_binary_sk:rte|0,32,32768,32736,1660,0,0,143,622,270.4018072289157,0,0,1500
sklep.prompt1_qa_sk:question-answering|0,126,32768,32642,9583,0,0,330,1724,505.43765000521756,0,0,3895
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0,32,32768,32736,1042,0,0,88,557,126.23800383877159,0,0,1280
sklep.prompt2_hate_speech_sk:hate-speech|0,32,32768,32736,1319,0,0,78,712,113.98711144806671,0,0,1804
sklep.prompt2_nli_3_classes_sk:nli|0,32,32768,32736,5004,0,0,186,342,242.7312150279776,0,0,802
sklep.prompt2_nli_binary_sk:rte|0,32,32768,32736,1660,0,0,117,596,244.40180722891566,0,0,1438
sklep.prompt2_qa_sk:question-answering|0,126,32768,32642,9583,0,0,335,1729,510.43765000521756,0,0,3898
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0,32,32768,32736,1042,0,0,99,568,137.2380038387716,0,0,1302
sklep.prompt3_hate_speech_sk:hate-speech|0,32,32768,32736,1319,0,0,102,736,137.9871114480667,0,0,1861
sklep.prompt3_nli_3_classes_sk:nli|0,32,32768,32736,5004,0,0,130,286,186.7312150279776,0,0,673
sklep.prompt3_nli_binary_sk:rte|0,32,32768,32736,1660,0,0,168,647,295.4018072289157,0,0,1546
sklep.prompt3_qa_sk:question-answering|0,126,32768,32642,9583,0,0,302,1696,477.43765000521756,0,0,3826
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0,32,32768,32736,1042,0,0,67,536,105.23800383877159,0,0,1236
slovakcopa.prompt1_copa_sk|0,32,32768,32736,500,0,0,152,207,172.342,0,0,479
slovakcopa.prompt2_copa_sk|0,32,32768,32736,500,0,0,157,212,177.344,0,0,494
slovakcopa.prompt3_copa_sk|0,32,32768,32736,500,0,0,152,208,172.65,0,0,503
toxic_sk.prompt1_toxicity_sk|0,32,32768,32736,881,0,0,80,360,102.18501702610669,0,0,981
toxic_sk.prompt2_toxicity_sk|0,32,32768,32736,881,0,0,72,352,94.18501702610669,0,0,965
toxic_sk.prompt3_toxicity_sk|0,32,32768,32736,881,0,0,105,385,127.18501702610669,0,0,1034
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0,32,32768,32736,778,0,0,145,544,264.5141388174807,0,0,1275
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0,32,32768,32736,778,0,0,129,528,248.51413881748073,0,0,1233
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0,32,32768,32736,778,0,0,140,539,259.5141388174807,0,0,1261
1 task generation_size context_limit input_budget samples null_token_arrays null_input_strings min_recorded_input_tokens max_recorded_input_tokens mean_recorded_input_tokens arrays_at_input_budget arrays_over_input_budget max_recorded_input_characters
2 arc.prompt1_mqa_single_correct_sk|0 32 32768 32736 1169 0 0 147 539 236.47733105218134 0 0 1275
3 arc.prompt2_mqa_single_correct_sk|0 32 32768 32736 1169 0 0 131 523 220.47733105218134 0 0 1233
4 arc.prompt3_mqa_single_correct_sk|0 32 32768 32736 1169 0 0 142 534 231.47733105218134 0 0 1261
5 belebele.prompt1_mqa_single_correct_sk|0 32 32768 32736 900 0 0 237 905 434.35777777777776 0 0 2118
6 belebele.prompt2_mqa_single_correct_sk|0 32 32768 32736 900 0 0 221 889 418.35777777777776 0 0 2076
7 belebele.prompt3_mqa_single_correct_sk|0 32 32768 32736 900 0 0 232 900 429.35777777777776 0 0 2104
8 financial_exams.prompt1_completion_sk|0 32 32768 32736 1334 0 0 152 682 295.84107946026984 0 0 1642
9 financial_exams.prompt2_completion_sk|0 32 32768 32736 1334 0 0 149 679 292.84107946026984 0 0 1633
10 financial_exams.prompt3_completion_sk|0 32 32768 32736 1334 0 0 137 667 280.84107946026984 0 0 1610
11 globalmmlu-lite.prompt1_mqa_single_correct_sk|0 32 32768 32736 400 0 0 153 1165 301.385 0 0 2646
12 globalmmlu-lite.prompt2_mqa_single_correct_sk|0 32 32768 32736 400 0 0 137 1149 285.385 0 0 2604
13 globalmmlu-lite.prompt3_mqa_single_correct_sk|0 32 32768 32736 400 0 0 148 1160 296.385 0 0 2632
14 mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0 32 32768 32736 100 0 0 154 266 210.72 0 0 626
15 mmlu.prompt1_mqa_single_correct_sk:anatomy|0 32 32768 32736 134 0 0 153 384 220.02238805970148 0 0 908
16 mmlu.prompt1_mqa_single_correct_sk:astronomy|0 32 32768 32736 152 0 0 163 480 249.75 0 0 1099
17 mmlu.prompt1_mqa_single_correct_sk:business_ethics|0 32 32768 32736 100 0 0 160 419 257.8 0 0 1001
18 mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0 32 32768 32736 259 0 0 151 377 224.1853281853282 0 0 888
19 mmlu.prompt1_mqa_single_correct_sk:college_biology|0 32 32768 32736 142 0 0 171 479 257.1478873239437 0 0 1074
20 mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0 32 32768 32736 100 0 0 160 389 234.02 0 0 952
21 mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0 32 32768 32736 100 0 0 174 602 307.65 0 0 1433
22 mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0 32 32768 32736 99 0 0 163 356 234.63636363636363 0 0 810
23 mmlu.prompt1_mqa_single_correct_sk:college_medicine|0 32 32768 32736 172 0 0 166 2217 313.7325581395349 0 0 5040
24 mmlu.prompt1_mqa_single_correct_sk:college_physics|0 32 32768 32736 102 0 0 160 364 246.8921568627451 0 0 861
25 mmlu.prompt1_mqa_single_correct_sk:computer_security|0 32 32768 32736 100 0 0 161 486 234.18 0 0 1197
26 mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0 32 32768 32736 235 0 0 155 280 195.20425531914893 0 0 669
27 mmlu.prompt1_mqa_single_correct_sk:econometrics|0 32 32768 32736 112 0 0 164 484 289.25892857142856 0 0 1078
28 mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0 32 32768 32736 145 0 0 148 270 195.91034482758621 0 0 647
29 mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0 32 32768 32736 373 0 0 152 393 212.76139410187668 0 0 927
30 mmlu.prompt1_mqa_single_correct_sk:formal_logic|0 32 32768 32736 115 0 0 168 572 295.5304347826087 0 0 1336
31 mmlu.prompt1_mqa_single_correct_sk:global_facts|0 32 32768 32736 100 0 0 166 376 207.62 0 0 860
32 mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0 32 32768 32736 282 0 0 165 469 264.54609929078015 0 0 1078
33 mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0 32 32768 32736 198 0 0 153 484 247.45454545454547 0 0 1110
34 mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0 32 32768 32736 99 0 0 157 760 311.77777777777777 0 0 1787
35 mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0 32 32768 32736 135 0 0 355 1239 745.1703703703704 0 0 2795
36 mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0 32 32768 32736 198 0 0 158 358 207.76262626262627 0 0 817
37 mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0 32 32768 32736 193 0 0 188 470 259.4974093264249 0 0 1096
38 mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0 32 32768 32736 387 0 0 169 441 248.3798449612403 0 0 984
39 mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0 32 32768 32736 266 0 0 156 430 219.24436090225564 0 0 948
40 mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0 32 32768 32736 217 0 0 171 542 253.58986175115209 0 0 1203
41 mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0 32 32768 32736 150 0 0 179 575 284.43333333333334 0 0 1342
42 mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0 32 32768 32736 536 0 0 159 670 238.3320895522388 0 0 1588
43 mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0 32 32768 32736 214 0 0 187 778 343.33644859813086 0 0 1769
44 mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0 32 32768 32736 57 0 0 370 1075 585.6842105263158 0 0 2520
45 mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0 32 32768 32736 180 0 0 261 1734 729.6333333333333 0 0 4002
46 mmlu.prompt1_mqa_single_correct_sk:human_aging|0 32 32768 32736 219 0 0 160 341 206.5068493150685 0 0 825
47 mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0 32 32768 32736 109 0 0 159 334 211.24770642201835 0 0 855
48 mmlu.prompt1_mqa_single_correct_sk:international_law|0 32 32768 32736 118 0 0 171 454 302.9491525423729 0 0 1019
49 mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0 32 32768 32736 107 0 0 158 481 244.4018691588785 0 0 1116
50 mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0 32 32768 32736 158 0 0 163 412 247.30379746835442 0 0 979
51 mmlu.prompt1_mqa_single_correct_sk:machine_learning|0 32 32768 32736 112 0 0 170 435 255.64285714285714 0 0 909
52 mmlu.prompt1_mqa_single_correct_sk:management|0 32 32768 32736 100 0 0 154 285 197.01 0 0 728
53 mmlu.prompt1_mqa_single_correct_sk:marketing|0 32 32768 32736 234 0 0 170 361 230.77777777777777 0 0 853
54 mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0 32 32768 32736 100 0 0 161 304 207.78 0 0 728
55 mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0 32 32768 32736 779 0 0 147 565 196.65340179717586 0 0 1348
56 mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0 32 32768 32736 326 0 0 164 411 249.94171779141104 0 0 992
57 mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0 32 32768 32736 749 0 0 261 382 310.8558077436582 0 0 890
58 mmlu.prompt1_mqa_single_correct_sk:nutrition|0 32 32768 32736 303 0 0 151 476 250.0990099009901 0 0 1123
59 mmlu.prompt1_mqa_single_correct_sk:philosophy|0 32 32768 32736 308 0 0 161 548 226.2922077922078 0 0 1306
60 mmlu.prompt1_mqa_single_correct_sk:prehistory|0 32 32768 32736 319 0 0 150 495 236.16614420062695 0 0 1118
61 mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0 32 32768 32736 276 0 0 184 582 315.0217391304348 0 0 1193
62 mmlu.prompt1_mqa_single_correct_sk:professional_law|0 32 32768 32736 1171 0 0 168 1660 593.5713065755764 0 0 3903
63 mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0 32 32768 32736 264 0 0 261 855 456.2537878787879 0 0 1946
64 mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0 32 32768 32736 596 0 0 161 595 267.18624161073825 0 0 1449
65 mmlu.prompt1_mqa_single_correct_sk:public_relations|0 32 32768 32736 109 0 0 152 513 224.1559633027523 0 0 1207
66 mmlu.prompt1_mqa_single_correct_sk:security_studies|0 32 32768 32736 222 0 0 164 1068 439.8063063063063 0 0 2495
67 mmlu.prompt1_mqa_single_correct_sk:sociology|0 32 32768 32736 197 0 0 170 440 254.248730964467 0 0 1038
68 mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0 32 32768 32736 99 0 0 163 389 233.3939393939394 0 0 900
69 mmlu.prompt1_mqa_single_correct_sk:virology|0 32 32768 32736 164 0 0 162 673 219.10365853658536 0 0 1384
70 mmlu.prompt1_mqa_single_correct_sk:world_religions|0 32 32768 32736 171 0 0 155 255 178.2046783625731 0 0 611
71 mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0 32 32768 32736 100 0 0 138 250 194.72 0 0 584
72 mmlu.prompt2_mqa_single_correct_sk:anatomy|0 32 32768 32736 134 0 0 137 368 204.02238805970148 0 0 866
73 mmlu.prompt2_mqa_single_correct_sk:astronomy|0 32 32768 32736 152 0 0 147 464 233.75 0 0 1057
74 mmlu.prompt2_mqa_single_correct_sk:business_ethics|0 32 32768 32736 100 0 0 144 403 241.8 0 0 959
75 mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0 32 32768 32736 259 0 0 135 361 208.1853281853282 0 0 846
76 mmlu.prompt2_mqa_single_correct_sk:college_biology|0 32 32768 32736 142 0 0 155 463 241.14788732394365 0 0 1032
77 mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0 32 32768 32736 100 0 0 144 373 218.02 0 0 910
78 mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0 32 32768 32736 100 0 0 158 586 291.65 0 0 1391
79 mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0 32 32768 32736 99 0 0 147 340 218.63636363636363 0 0 768
80 mmlu.prompt2_mqa_single_correct_sk:college_medicine|0 32 32768 32736 172 0 0 150 2201 297.7325581395349 0 0 4998
81 mmlu.prompt2_mqa_single_correct_sk:college_physics|0 32 32768 32736 102 0 0 144 348 230.8921568627451 0 0 819
82 mmlu.prompt2_mqa_single_correct_sk:computer_security|0 32 32768 32736 100 0 0 145 470 218.18 0 0 1155
83 mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0 32 32768 32736 235 0 0 139 264 179.20425531914893 0 0 627
84 mmlu.prompt2_mqa_single_correct_sk:econometrics|0 32 32768 32736 112 0 0 148 468 273.25892857142856 0 0 1036
85 mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0 32 32768 32736 145 0 0 132 254 179.91034482758621 0 0 605
86 mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0 32 32768 32736 373 0 0 136 377 196.76139410187668 0 0 885
87 mmlu.prompt2_mqa_single_correct_sk:formal_logic|0 32 32768 32736 115 0 0 152 556 279.5304347826087 0 0 1294
88 mmlu.prompt2_mqa_single_correct_sk:global_facts|0 32 32768 32736 100 0 0 150 360 191.62 0 0 818
89 mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0 32 32768 32736 282 0 0 149 453 248.54609929078015 0 0 1036
90 mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0 32 32768 32736 198 0 0 137 468 231.45454545454547 0 0 1068
91 mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0 32 32768 32736 99 0 0 141 744 295.77777777777777 0 0 1745
92 mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0 32 32768 32736 135 0 0 339 1223 729.1703703703704 0 0 2753
93 mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0 32 32768 32736 198 0 0 142 342 191.76262626262627 0 0 775
94 mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0 32 32768 32736 193 0 0 172 454 243.49740932642487 0 0 1054
95 mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0 32 32768 32736 387 0 0 153 425 232.3798449612403 0 0 942
96 mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0 32 32768 32736 266 0 0 140 414 203.24436090225564 0 0 906
97 mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0 32 32768 32736 217 0 0 155 526 237.58986175115209 0 0 1161
98 mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0 32 32768 32736 150 0 0 163 559 268.43333333333334 0 0 1300
99 mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0 32 32768 32736 536 0 0 143 654 222.3320895522388 0 0 1546
100 mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0 32 32768 32736 214 0 0 171 762 327.33644859813086 0 0 1727
101 mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0 32 32768 32736 57 0 0 354 1059 569.6842105263158 0 0 2478
102 mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0 32 32768 32736 180 0 0 245 1718 713.6333333333333 0 0 3960
103 mmlu.prompt2_mqa_single_correct_sk:human_aging|0 32 32768 32736 219 0 0 144 325 190.5068493150685 0 0 783
104 mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0 32 32768 32736 109 0 0 143 318 195.24770642201835 0 0 813
105 mmlu.prompt2_mqa_single_correct_sk:international_law|0 32 32768 32736 118 0 0 155 438 286.9491525423729 0 0 977
106 mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0 32 32768 32736 107 0 0 142 465 228.4018691588785 0 0 1074
107 mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0 32 32768 32736 158 0 0 147 396 231.30379746835442 0 0 937
108 mmlu.prompt2_mqa_single_correct_sk:machine_learning|0 32 32768 32736 112 0 0 154 419 239.64285714285714 0 0 867
109 mmlu.prompt2_mqa_single_correct_sk:management|0 32 32768 32736 100 0 0 138 269 181.01 0 0 686
110 mmlu.prompt2_mqa_single_correct_sk:marketing|0 32 32768 32736 234 0 0 154 345 214.77777777777777 0 0 811
111 mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0 32 32768 32736 100 0 0 145 288 191.78 0 0 686
112 mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0 32 32768 32736 779 0 0 131 549 180.65340179717586 0 0 1306
113 mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0 32 32768 32736 326 0 0 148 395 233.94171779141104 0 0 950
114 mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0 32 32768 32736 749 0 0 245 366 294.8558077436582 0 0 848
115 mmlu.prompt2_mqa_single_correct_sk:nutrition|0 32 32768 32736 303 0 0 135 460 234.0990099009901 0 0 1081
116 mmlu.prompt2_mqa_single_correct_sk:philosophy|0 32 32768 32736 308 0 0 145 532 210.2922077922078 0 0 1264
117 mmlu.prompt2_mqa_single_correct_sk:prehistory|0 32 32768 32736 319 0 0 134 479 220.16614420062695 0 0 1076
118 mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0 32 32768 32736 276 0 0 168 566 299.0217391304348 0 0 1151
119 mmlu.prompt2_mqa_single_correct_sk:professional_law|0 32 32768 32736 1171 0 0 152 1644 577.5713065755764 0 0 3861
120 mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0 32 32768 32736 264 0 0 245 839 440.2537878787879 0 0 1904
121 mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0 32 32768 32736 596 0 0 145 579 251.18624161073825 0 0 1407
122 mmlu.prompt2_mqa_single_correct_sk:public_relations|0 32 32768 32736 109 0 0 136 497 208.1559633027523 0 0 1165
123 mmlu.prompt2_mqa_single_correct_sk:security_studies|0 32 32768 32736 222 0 0 148 1052 423.8063063063063 0 0 2453
124 mmlu.prompt2_mqa_single_correct_sk:sociology|0 32 32768 32736 197 0 0 154 424 238.248730964467 0 0 996
125 mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0 32 32768 32736 99 0 0 147 373 217.3939393939394 0 0 858
126 mmlu.prompt2_mqa_single_correct_sk:virology|0 32 32768 32736 164 0 0 146 657 203.10365853658536 0 0 1342
127 mmlu.prompt2_mqa_single_correct_sk:world_religions|0 32 32768 32736 171 0 0 139 239 162.2046783625731 0 0 569
128 mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0 32 32768 32736 100 0 0 149 261 205.72 0 0 612
129 mmlu.prompt3_mqa_single_correct_sk:anatomy|0 32 32768 32736 134 0 0 148 379 215.02238805970148 0 0 894
130 mmlu.prompt3_mqa_single_correct_sk:astronomy|0 32 32768 32736 152 0 0 158 475 244.75 0 0 1085
131 mmlu.prompt3_mqa_single_correct_sk:business_ethics|0 32 32768 32736 100 0 0 155 414 252.8 0 0 987
132 mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0 32 32768 32736 259 0 0 146 372 219.1853281853282 0 0 874
133 mmlu.prompt3_mqa_single_correct_sk:college_biology|0 32 32768 32736 142 0 0 166 474 252.14788732394365 0 0 1060
134 mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0 32 32768 32736 100 0 0 155 384 229.02 0 0 938
135 mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0 32 32768 32736 100 0 0 169 597 302.65 0 0 1419
136 mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0 32 32768 32736 99 0 0 158 351 229.63636363636363 0 0 796
137 mmlu.prompt3_mqa_single_correct_sk:college_medicine|0 32 32768 32736 172 0 0 161 2212 308.7325581395349 0 0 5026
138 mmlu.prompt3_mqa_single_correct_sk:college_physics|0 32 32768 32736 102 0 0 155 359 241.8921568627451 0 0 847
139 mmlu.prompt3_mqa_single_correct_sk:computer_security|0 32 32768 32736 100 0 0 156 481 229.18 0 0 1183
140 mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0 32 32768 32736 235 0 0 150 275 190.20425531914893 0 0 655
141 mmlu.prompt3_mqa_single_correct_sk:econometrics|0 32 32768 32736 112 0 0 159 479 284.25892857142856 0 0 1064
142 mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0 32 32768 32736 145 0 0 143 265 190.91034482758621 0 0 633
143 mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0 32 32768 32736 373 0 0 147 388 207.76139410187668 0 0 913
144 mmlu.prompt3_mqa_single_correct_sk:formal_logic|0 32 32768 32736 115 0 0 163 567 290.5304347826087 0 0 1322
145 mmlu.prompt3_mqa_single_correct_sk:global_facts|0 32 32768 32736 100 0 0 161 371 202.62 0 0 846
146 mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0 32 32768 32736 282 0 0 160 464 259.54609929078015 0 0 1064
147 mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0 32 32768 32736 198 0 0 148 479 242.45454545454547 0 0 1096
148 mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0 32 32768 32736 99 0 0 152 755 306.77777777777777 0 0 1773
149 mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0 32 32768 32736 135 0 0 350 1234 740.1703703703704 0 0 2781
150 mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0 32 32768 32736 198 0 0 153 353 202.76262626262627 0 0 803
151 mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0 32 32768 32736 193 0 0 183 465 254.49740932642487 0 0 1082
152 mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0 32 32768 32736 387 0 0 164 436 243.3798449612403 0 0 970
153 mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0 32 32768 32736 266 0 0 151 425 214.24436090225564 0 0 934
154 mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0 32 32768 32736 217 0 0 166 537 248.58986175115209 0 0 1189
155 mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0 32 32768 32736 150 0 0 174 570 279.43333333333334 0 0 1328
156 mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0 32 32768 32736 536 0 0 154 665 233.3320895522388 0 0 1574
157 mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0 32 32768 32736 214 0 0 182 773 338.33644859813086 0 0 1755
158 mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0 32 32768 32736 57 0 0 365 1070 580.6842105263158 0 0 2506
159 mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0 32 32768 32736 180 0 0 256 1729 724.6333333333333 0 0 3988
160 mmlu.prompt3_mqa_single_correct_sk:human_aging|0 32 32768 32736 219 0 0 155 336 201.5068493150685 0 0 811
161 mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0 32 32768 32736 109 0 0 154 329 206.24770642201835 0 0 841
162 mmlu.prompt3_mqa_single_correct_sk:international_law|0 32 32768 32736 118 0 0 166 449 297.9491525423729 0 0 1005
163 mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0 32 32768 32736 107 0 0 153 476 239.4018691588785 0 0 1102
164 mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0 32 32768 32736 158 0 0 158 407 242.30379746835442 0 0 965
165 mmlu.prompt3_mqa_single_correct_sk:machine_learning|0 32 32768 32736 112 0 0 165 430 250.64285714285714 0 0 895
166 mmlu.prompt3_mqa_single_correct_sk:management|0 32 32768 32736 100 0 0 149 280 192.01 0 0 714
167 mmlu.prompt3_mqa_single_correct_sk:marketing|0 32 32768 32736 234 0 0 165 356 225.77777777777777 0 0 839
168 mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0 32 32768 32736 100 0 0 156 299 202.78 0 0 714
169 mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0 32 32768 32736 779 0 0 142 560 191.65340179717586 0 0 1334
170 mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0 32 32768 32736 326 0 0 159 406 244.94171779141104 0 0 978
171 mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0 32 32768 32736 749 0 0 256 377 305.8558077436582 0 0 876
172 mmlu.prompt3_mqa_single_correct_sk:nutrition|0 32 32768 32736 303 0 0 146 471 245.0990099009901 0 0 1109
173 mmlu.prompt3_mqa_single_correct_sk:philosophy|0 32 32768 32736 308 0 0 156 543 221.2922077922078 0 0 1292
174 mmlu.prompt3_mqa_single_correct_sk:prehistory|0 32 32768 32736 319 0 0 145 490 231.16614420062695 0 0 1104
175 mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0 32 32768 32736 276 0 0 179 577 310.0217391304348 0 0 1179
176 mmlu.prompt3_mqa_single_correct_sk:professional_law|0 32 32768 32736 1171 0 0 163 1655 588.5713065755764 0 0 3889
177 mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0 32 32768 32736 264 0 0 256 850 451.2537878787879 0 0 1932
178 mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0 32 32768 32736 596 0 0 156 590 262.18624161073825 0 0 1435
179 mmlu.prompt3_mqa_single_correct_sk:public_relations|0 32 32768 32736 109 0 0 147 508 219.1559633027523 0 0 1193
180 mmlu.prompt3_mqa_single_correct_sk:security_studies|0 32 32768 32736 222 0 0 159 1063 434.8063063063063 0 0 2481
181 mmlu.prompt3_mqa_single_correct_sk:sociology|0 32 32768 32736 197 0 0 165 435 249.248730964467 0 0 1024
182 mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0 32 32768 32736 99 0 0 158 384 228.3939393939394 0 0 886
183 mmlu.prompt3_mqa_single_correct_sk:virology|0 32 32768 32736 164 0 0 157 668 214.10365853658536 0 0 1370
184 mmlu.prompt3_mqa_single_correct_sk:world_religions|0 32 32768 32736 171 0 0 150 250 173.2046783625731 0 0 597
185 movie_review.prompt1_sentiment_binary_sk|0 32 32768 32736 2008 0 0 91 1799 249.22211155378486 0 0 4162
186 movie_review.prompt2_sentiment_binary_sk|0 32 32768 32736 2008 0 0 102 1810 260.2221115537849 0 0 4184
187 movie_review.prompt3_sentiment_binary_sk|0 32 32768 32736 2008 0 0 70 1778 228.22211155378486 0 0 4118
188 multi_wiki_qa.prompt1_qa_sk|0 126 32768 32642 4997 0 0 512 32642 1964.61997198319 23 0 121852
189 multi_wiki_qa.prompt2_qa_sk|0 126 32768 32642 4997 0 0 517 32642 1969.596958174905 23 0 121855
190 multi_wiki_qa.prompt3_qa_sk|0 126 32768 32642 4997 0 0 483 32642 1935.7534520712427 23 0 121783
191 senti_sk.prompt1_sentiment_3_class_sk|0 32 32768 32736 4677 0 0 94 551 123.0096215522771 0 0 1271
192 senti_sk.prompt1_sentiment_binary_sk|0 32 32768 32736 3728 0 0 88 545 117.15665236051503 0 0 1258
193 senti_sk.prompt2_sentiment_3_class_sk|0 32 32768 32736 4677 0 0 109 566 138.0096215522771 0 0 1304
194 senti_sk.prompt2_sentiment_binary_sk|0 32 32768 32736 3728 0 0 99 556 128.156652360515 0 0 1280
195 senti_sk.prompt3_sentiment_3_class_sk|0 32 32768 32736 4677 0 0 72 529 101.0096215522771 0 0 1224
196 senti_sk.prompt3_sentiment_binary_sk|0 32 32768 32736 3728 0 0 67 524 96.15665236051503 0 0 1214
197 skj1.prompt1_skj1_sk|0 32 32768 32736 100 0 0 175 317 215.89 0 0 721
198 skj1.prompt2_skj1_sk|0 32 32768 32736 100 0 0 162 304 202.89 0 0 691
199 skj1.prompt3_skj1_sk|0 32 32768 32736 100 0 0 163 305 203.89 0 0 696
200 sklep.prompt1_hate_speech_sk:hate-speech|0 32 32768 32736 1319 0 0 94 728 129.9871114480667 0 0 1831
201 sklep.prompt1_nli_3_classes_sk:nli|0 32 32768 32736 5004 0 0 136 292 192.7312150279776 0 0 698
202 sklep.prompt1_nli_binary_sk:rte|0 32 32768 32736 1660 0 0 143 622 270.4018072289157 0 0 1500
203 sklep.prompt1_qa_sk:question-answering|0 126 32768 32642 9583 0 0 330 1724 505.43765000521756 0 0 3895
204 sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0 32 32768 32736 1042 0 0 88 557 126.23800383877159 0 0 1280
205 sklep.prompt2_hate_speech_sk:hate-speech|0 32 32768 32736 1319 0 0 78 712 113.98711144806671 0 0 1804
206 sklep.prompt2_nli_3_classes_sk:nli|0 32 32768 32736 5004 0 0 186 342 242.7312150279776 0 0 802
207 sklep.prompt2_nli_binary_sk:rte|0 32 32768 32736 1660 0 0 117 596 244.40180722891566 0 0 1438
208 sklep.prompt2_qa_sk:question-answering|0 126 32768 32642 9583 0 0 335 1729 510.43765000521756 0 0 3898
209 sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0 32 32768 32736 1042 0 0 99 568 137.2380038387716 0 0 1302
210 sklep.prompt3_hate_speech_sk:hate-speech|0 32 32768 32736 1319 0 0 102 736 137.9871114480667 0 0 1861
211 sklep.prompt3_nli_3_classes_sk:nli|0 32 32768 32736 5004 0 0 130 286 186.7312150279776 0 0 673
212 sklep.prompt3_nli_binary_sk:rte|0 32 32768 32736 1660 0 0 168 647 295.4018072289157 0 0 1546
213 sklep.prompt3_qa_sk:question-answering|0 126 32768 32642 9583 0 0 302 1696 477.43765000521756 0 0 3826
214 sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0 32 32768 32736 1042 0 0 67 536 105.23800383877159 0 0 1236
215 slovakcopa.prompt1_copa_sk|0 32 32768 32736 500 0 0 152 207 172.342 0 0 479
216 slovakcopa.prompt2_copa_sk|0 32 32768 32736 500 0 0 157 212 177.344 0 0 494
217 slovakcopa.prompt3_copa_sk|0 32 32768 32736 500 0 0 152 208 172.65 0 0 503
218 toxic_sk.prompt1_toxicity_sk|0 32 32768 32736 881 0 0 80 360 102.18501702610669 0 0 981
219 toxic_sk.prompt2_toxicity_sk|0 32 32768 32736 881 0 0 72 352 94.18501702610669 0 0 965
220 toxic_sk.prompt3_toxicity_sk|0 32 32768 32736 881 0 0 105 385 127.18501702610669 0 0 1034
221 truthfulqa.prompt1_mqa_single_correct_sk:mc1|0 32 32768 32736 778 0 0 145 544 264.5141388174807 0 0 1275
222 truthfulqa.prompt2_mqa_single_correct_sk:mc1|0 32 32768 32736 778 0 0 129 528 248.51413881748073 0 0 1233
223 truthfulqa.prompt3_mqa_single_correct_sk:mc1|0 32 32768 32736 778 0 0 140 539 259.5141388174807 0 0 1261

14
aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_summary.json Normal file
View File

@ -0,0 +1,14 @@
{
"task_count": 222,
"evaluated_instances": 159426,
"max_recorded_input_tokens": 32642,
"arrays_at_input_budget": 69,
"arrays_over_input_budget": 0,
"log_messages_matching_truncat_case_insensitive": [
{
"log_lf_line_number": 16033,
"message": "Splits: 50%|█████ | 1/2 [1:14:51<1:14:51, 4491.94s/it]context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens."
}
],
"limitation": "Input strings and token arrays are recorded artifacts, not a separately tokenized original source prompt. Their lengths cannot recover how much text was discarded or establish the number of truly truncated inputs."
}

7
aya_300k_lr_experiments/benchmarks/skbench/results/log_truncation_evidence.json Normal file
View File

@ -0,0 +1,7 @@
[
{
"line_number": 16033,
"message": "context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens.",
"source": "logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log"
}
]

223
aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv Normal file
View File

@ -0,0 +1,223 @@
task,family,task_group_without_prompt,prompt,num_fewshots,sample_count,primary_metric,primary_value,generation_size,evaluation_splits,hf_repo,hf_subset,hf_revision,em,em_conservative,em_conservative_norm,em_conservative_norm_stderr,em_conservative_stderr,em_norm,em_norm_relaxed,em_norm_relaxed_stderr,em_norm_stderr,em_relaxed,em_relaxed_stderr,em_stderr,exact_match_slk_prefix,exact_match_slk_prefix_stderr,f1_slk,f1_slk_stderr
arc.prompt1_mqa_single_correct_sk|0,arc,arc.mqa_single_correct_sk,1,0,1169,em_relaxed,0.1710863986313088,32,test,jon-tow/okapi_arc_challenge,sk,56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e,0.013686911890504704,0.1710863986313088,,,0.011018968811092407,,,,,0.1710863986313088,0.011018968811092407,0.0033996852610928244,,,,
arc.prompt2_mqa_single_correct_sk|0,arc,arc.mqa_single_correct_sk,2,0,1169,em_relaxed,0.22412318220701455,32,test,jon-tow/okapi_arc_challenge,sk,56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e,0.03507271171941831,0.223267750213858,,,0.012185048029719049,,,,,0.22412318220701455,0.012201644195165715,0.005382828530594407,,,,
arc.prompt3_mqa_single_correct_sk|0,arc,arc.mqa_single_correct_sk,3,0,1169,em_relaxed,0.1770744225834046,32,test,jon-tow/okapi_arc_challenge,sk,56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e,0.022241231822070145,0.17621899059024806,,,0.011148355810710043,,,,,0.1770744225834046,0.011169578281264629,0.004314931013145922,,,,
belebele.prompt1_mqa_single_correct_sk|0,belebele,belebele.mqa_single_correct_sk,1,0,900,em_relaxed,0.21555555555555556,32,test,facebook/belebele,slk_Latn,7899cdfa4e1e0d733fd77c848e2c273cb1d32be2,0.02,0.21555555555555556,,,0.01371452783236968,,,,,0.21555555555555556,0.01371452783236968,0.004669261421755523,,,,
belebele.prompt2_mqa_single_correct_sk|0,belebele,belebele.mqa_single_correct_sk,2,0,900,em_relaxed,0.2111111111111111,32,test,facebook/belebele,slk_Latn,7899cdfa4e1e0d733fd77c848e2c273cb1d32be2,0.016666666666666666,0.2111111111111111,,,0.013610798969328482,,,,,0.2111111111111111,0.013610798969328482,0.004269675894690937,,,,
belebele.prompt3_mqa_single_correct_sk|0,belebele,belebele.mqa_single_correct_sk,3,0,900,em_relaxed,0.21333333333333335,32,test,facebook/belebele,slk_Latn,7899cdfa4e1e0d733fd77c848e2c273cb1d32be2,0.03,0.2111111111111111,,,0.013610798969328486,,,,,0.21333333333333335,0.013662962863011163,0.005689402360648914,,,,
financial_exams.prompt1_completion_sk|0,financial_exams,financial_exams.completion_sk,1,0,1334,em_relaxed,0.20089955022488756,32,test,TUKE-KEMT/slovak-financial-exam,default,7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c,0.009745127436281859,0.20089955022488756,,,0.010974256206370726,,,,,0.20089955022488756,0.010974256206370726,0.002690618851324012,,,,
financial_exams.prompt2_completion_sk|0,financial_exams,financial_exams.completion_sk,2,0,1334,em_relaxed,0.2383808095952024,32,test,TUKE-KEMT/slovak-financial-exam,default,7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c,0.07571214392803598,0.20539730134932535,,,0.011065150105060274,,,,,0.2383808095952024,0.011670501255680532,0.007245549384888845,,,,
financial_exams.prompt3_completion_sk|0,financial_exams,financial_exams.completion_sk,3,0,1334,em_relaxed,0.24887556221889057,32,test,TUKE-KEMT/slovak-financial-exam,default,7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c,0.0037481259370314842,0.24887556221889057,,,0.01184218927916494,,,,,0.24887556221889057,0.01184218927916494,0.0016736960391157028,,,,
globalmmlu-lite.prompt1_mqa_single_correct_sk|0,globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,1,0,400,em_relaxed,0.1825,32,test,CohereLabs/Global-MMLU-Lite,sk,600af1bff2d447c7fc3aee22f11f607560d5fcb7,0.025,0.1825,,,0.01933698975881026,,,,,0.1825,0.01933698975881026,0.007816023641463065,,,,
globalmmlu-lite.prompt2_mqa_single_correct_sk|0,globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,2,0,400,em_relaxed,0.21,32,test,CohereLabs/Global-MMLU-Lite,sk,600af1bff2d447c7fc3aee22f11f607560d5fcb7,0.035,0.21,,,0.020390916450326858,,,,,0.21,0.020390916450326858,0.009200495791370004,,,,
globalmmlu-lite.prompt3_mqa_single_correct_sk|0,globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,3,0,400,em_relaxed,0.2025,32,test,CohereLabs/Global-MMLU-Lite,sk,600af1bff2d447c7fc3aee22f11f607560d5fcb7,0.025,0.2025,,,0.02011830610666578,,,,,0.2025,0.02011830610666578,0.007816023641463063,,,,
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0,mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,1,0,100,em_relaxed,0.01,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.01,,,0.009999999999999997,,,,,0.01,0.009999999999999997,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:anatomy|0,mmlu,mmlu.mqa_single_correct_sk:anatomy,1,0,134,em_relaxed,0.12686567164179105,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.022388059701492536,0.12686567164179105,,,0.028859365115889928,,,,,0.12686567164179105,0.028859365115889928,0.012828198053329663,,,,
mmlu.prompt1_mqa_single_correct_sk:astronomy|0,mmlu,mmlu.mqa_single_correct_sk:astronomy,1,0,152,em_relaxed,0.14473684210526316,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03289473684210526,0.13815789473684212,,,0.028081042939576552,,,,,0.14473684210526316,0.028631951845930412,0.014514818445457586,,,,
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0,mmlu,mmlu.mqa_single_correct_sk:business_ethics,1,0,100,em_relaxed,0.14,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.14,,,0.03487350880197772,,,,,0.14,0.03487350880197772,0.014070529413628957,,,,
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0,mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,1,0,259,em_relaxed,0.15444015444015444,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.011583011583011582,0.15444015444015444,,,0.022497923516297567,,,,,0.15444015444015444,0.022497923516297567,0.006661484058899665,,,,
mmlu.prompt1_mqa_single_correct_sk:college_biology|0,mmlu,mmlu.mqa_single_correct_sk:college_biology,1,0,142,em_relaxed,0.15492957746478872,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.028169014084507043,0.15492957746478872,,,0.030472217725828407,,,,,0.15492957746478872,0.030472217725828407,0.013933866272175799,,,,
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:college_chemistry,1,0,100,em_relaxed,0.08,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.08,,,0.027265992434429093,,,,,0.08,0.027265992434429093,0.019694638556693227,,,,
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:college_computer_science,1,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.11,,,0.031446603773522035,,,,,0.15,0.035887028128263714,0.009999999999999998,,,,
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:college_mathematics,1,0,99,em_relaxed,0.06060606060606061,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.050505050505050504,,,0.02212079324207198,,,,,0.06060606060606061,0.024102876029567216,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0,mmlu,mmlu.mqa_single_correct_sk:college_medicine,1,0,172,em_relaxed,0.1511627906976744,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.011627906976744186,0.1511627906976744,,,0.027392810919920886,,,,,0.1511627906976744,0.027392810919920886,0.008198095184581854,,,,
mmlu.prompt1_mqa_single_correct_sk:college_physics|0,mmlu,mmlu.mqa_single_correct_sk:college_physics,1,0,102,em_relaxed,0.0784313725490196,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.00980392156862745,0.0784313725490196,,,0.026751495257385273,,,,,0.0784313725490196,0.026751495257385273,0.009803921568627454,,,,
mmlu.prompt1_mqa_single_correct_sk:computer_security|0,mmlu,mmlu.mqa_single_correct_sk:computer_security,1,0,100,em_relaxed,0.08,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.08,,,0.027265992434429082,,,,,0.08,0.027265992434429082,0.014070529413628966,,,,
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0,mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,1,0,235,em_relaxed,0.09361702127659574,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03829787234042553,0.09361702127659574,,,0.01904256081095342,,,,,0.09361702127659574,0.01904256081095342,0.012545838230269625,,,,
mmlu.prompt1_mqa_single_correct_sk:econometrics|0,mmlu,mmlu.mqa_single_correct_sk:econometrics,1,0,112,em_relaxed,0.125,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.026785714285714284,0.125,,,0.03139045014587016,,,,,0.125,0.03139045014587016,0.015324784082343043,,,,
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0,mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,1,0,145,em_relaxed,0.0896551724137931,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.041379310344827586,0.0896551724137931,,,0.023807258824376967,,,,,0.0896551724137931,0.023807258824376967,0.016597159859992713,,,,
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,1,0,373,em_relaxed,0.06166219839142091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005361930294906166,0.058981233243967826,,,0.012214750400069162,,,,,0.06166219839142091,0.012471469772330128,0.003786357797685454,,,,
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0,mmlu,mmlu.mqa_single_correct_sk:formal_logic,1,0,115,em_relaxed,0.12173913043478261,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0782608695652174,,,0.025154976812194616,,,,,0.12173913043478261,0.030624883036764184,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:global_facts|0,mmlu,mmlu.mqa_single_correct_sk:global_facts,1,0,100,em_relaxed,0.11,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06,0.11,,,0.031446603773522035,,,,,0.11,0.031446603773522035,0.02386832565759416,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_biology,1,0,282,em_relaxed,0.18439716312056736,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02127659574468085,0.1773049645390071,,,0.022783829728850895,,,,,0.18439716312056736,0.023134670670309505,0.00860850820920177,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,1,0,198,em_relaxed,0.08585858585858586,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.010101010101010102,0.08080808080808081,,,0.019417681889724533,,,,,0.08585858585858586,0.019960225563172885,0.007124341521250813,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,1,0,99,em_relaxed,0.10101010101010101,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.10101010101010101,,,0.030440148787487714,,,,,0.10101010101010101,0.030440148787487714,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,1,0,135,em_relaxed,0.24444444444444444,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.014814814814814815,0.24444444444444444,,,0.03712537833614866,,,,,0.24444444444444444,0.03712537833614866,0.010436494549594362,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0,mmlu,mmlu.mqa_single_correct_sk:high_school_geography,1,0,198,em_relaxed,0.15656565656565657,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.15656565656565657,,,0.025890520358141454,,,,,0.15656565656565657,0.025890520358141454,0.012213156893572809,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,1,0,193,em_relaxed,0.30569948186528495,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.025906735751295335,0.30569948186528495,,,0.03324837939758159,,,,,0.30569948186528495,0.03324837939758159,0.011464523356953171,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,1,0,387,em_relaxed,0.12919896640826872,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.012919896640826873,0.12661498708010335,,,0.016925900082204414,,,,,0.12919896640826872,0.017072429785340966,0.005747937884708704,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,1,0,266,em_relaxed,0.03759398496240601,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.03759398496240601,,,0.011684641953170118,,,,,0.03759398496240601,0.011684641953170118,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,1,0,217,em_relaxed,0.14285714285714285,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.004608294930875576,0.14285714285714285,,,0.0238095238095238,,,,,0.14285714285714285,0.0238095238095238,0.004608294930875571,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_physics,1,0,150,em_relaxed,0.13333333333333333,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02666666666666667,0.13333333333333333,,,0.027848530325897826,,,,,0.13333333333333333,0.027848530325897826,0.013198422611557032,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,1,0,536,em_relaxed,0.16791044776119404,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.016791044776119403,0.16791044776119404,,,0.016160210122502162,,,,,0.16791044776119404,0.016160210122502162,0.005555010467354857,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,1,0,214,em_relaxed,0.1822429906542056,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.17289719626168223,,,0.025910984835548666,,,,,0.1822429906542056,0.026451344562470138,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,1,0,57,em_relaxed,0.19298245614035087,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.19298245614035087,,,0.05273590328947574,,,,,0.19298245614035087,0.05273590328947574,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,1,0,180,em_relaxed,0.2777777777777778,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005555555555555556,0.2777777777777778,,,0.033477857599386325,,,,,0.2777777777777778,0.033477857599386325,0.005555555555555546,,,,
mmlu.prompt1_mqa_single_correct_sk:human_aging|0,mmlu,mmlu.mqa_single_correct_sk:human_aging,1,0,219,em_relaxed,0.1187214611872146,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0365296803652968,0.1187214611872146,,,0.021907507389103187,,,,,0.1187214611872146,0.021907507389103187,0.012706146532849703,,,,
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0,mmlu,mmlu.mqa_single_correct_sk:human_sexuality,1,0,109,em_relaxed,0.13761467889908258,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.045871559633027525,0.13761467889908258,,,0.03314905716923366,,,,,0.13761467889908258,0.03314905716923366,0.020130904850286974,,,,
mmlu.prompt1_mqa_single_correct_sk:international_law|0,mmlu,mmlu.mqa_single_correct_sk:international_law,1,0,118,em_relaxed,0.13559322033898305,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03389830508474576,0.13559322033898305,,,0.03165083260501198,,,,,0.13559322033898305,0.03165083260501198,0.016730444637044904,,,,
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0,mmlu,mmlu.mqa_single_correct_sk:jurisprudence,1,0,107,em_relaxed,0.14018691588785046,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.056074766355140186,0.14018691588785046,,,0.03372118539369118,,,,,0.14018691588785046,0.03372118539369118,0.02234599008672909,,,,
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0,mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,1,0,158,em_relaxed,0.13924050632911392,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02531645569620253,0.13924050632911392,,,0.027629554439252756,,,,,0.13924050632911392,0.027629554439252756,0.012536706047327604,,,,
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0,mmlu,mmlu.mqa_single_correct_sk:machine_learning,1,0,112,em_relaxed,0.11607142857142858,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.008928571428571428,0.10714285714285714,,,0.02935696201881377,,,,,0.11607142857142858,0.03040253249894973,0.008928571428571423,,,,
mmlu.prompt1_mqa_single_correct_sk:management|0,mmlu,mmlu.mqa_single_correct_sk:management,1,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.15,,,0.0358870281282637,,,,,0.15,0.0358870281282637,0.01407052941362896,,,,
mmlu.prompt1_mqa_single_correct_sk:marketing|0,mmlu,mmlu.mqa_single_correct_sk:marketing,1,0,234,em_relaxed,0.15384615384615385,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03418803418803419,0.15384615384615385,,,0.023636873317489277,,,,,0.15384615384615385,0.023636873317489277,0.011904341997629792,,,,
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0,mmlu,mmlu.mqa_single_correct_sk:medical_genetics,1,0,100,em_relaxed,0.18,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.18,,,0.038612291966536955,,,,,0.18,0.038612291966536955,0.019694638556693223,,,,
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0,mmlu,mmlu.mqa_single_correct_sk:miscellaneous,1,0,779,em_relaxed,0.14505776636713735,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03209242618741977,0.14505776636713735,,,0.01262551435912769,,,,,0.14505776636713735,0.01262551435912769,0.006318709962464618,,,,
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0,mmlu,mmlu.mqa_single_correct_sk:moral_disputes,1,0,326,em_relaxed,0.20245398773006135,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06134969325153374,0.20245398773006135,,,0.022289450459613447,,,,,0.20245398773006135,0.022289450459613447,0.01331117614747688,,,,
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0,mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,1,0,749,em_relaxed,0.006675567423230975,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0053404539385847796,,,0.0026648668562206292,,,,,0.006675567423230975,0.0029774114441068625,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:nutrition|0,mmlu,mmlu.mqa_single_correct_sk:nutrition,1,0,303,em_relaxed,0.1848184818481848,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.036303630363036306,0.1848184818481848,,,0.022335536574954167,,,,,0.1848184818481848,0.022335536574954167,0.010763206393659719,,,,
mmlu.prompt1_mqa_single_correct_sk:philosophy|0,mmlu,mmlu.mqa_single_correct_sk:philosophy,1,0,308,em_relaxed,0.21103896103896103,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06818181818181818,0.21103896103896103,,,0.023288413887090093,,,,,0.21103896103896103,0.023288413887090093,0.014385689805382164,,,,
mmlu.prompt1_mqa_single_correct_sk:prehistory|0,mmlu,mmlu.mqa_single_correct_sk:prehistory,1,0,319,em_relaxed,0.15987460815047022,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04075235109717868,0.15987460815047022,,,0.02055172319966096,,,,,0.15987460815047022,0.02055172319966096,0.011087359931149773,,,,
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0,mmlu,mmlu.mqa_single_correct_sk:professional_accounting,1,0,276,em_relaxed,0.15217391304347827,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0036231884057971015,0.14130434782608695,,,0.02100540442245809,,,,,0.15217391304347827,0.02165993479655374,0.0036231884057970937,,,,
mmlu.prompt1_mqa_single_correct_sk:professional_law|0,mmlu,mmlu.mqa_single_correct_sk:professional_law,1,0,1171,em_relaxed,0.12724167378309137,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0017079419299743809,0.12724167378309137,,,0.009742462799429212,,,,,0.12724167378309137,0.009742462799429212,0.0012071811002646103,,,,
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0,mmlu,mmlu.mqa_single_correct_sk:professional_medicine,1,0,264,em_relaxed,0.13257575757575757,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.13257575757575757,,,0.020910766968363862,,,,,0.13257575757575757,0.020910766968363862,0.0,,,,
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0,mmlu,mmlu.mqa_single_correct_sk:professional_psychology,1,0,596,em_relaxed,0.20134228187919462,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03355704697986577,0.20134228187919462,,,0.016439528475054862,,,,,0.20134228187919462,0.016439528475054862,0.0073828066874755535,,,,
mmlu.prompt1_mqa_single_correct_sk:public_relations|0,mmlu,mmlu.mqa_single_correct_sk:public_relations,1,0,109,em_relaxed,0.1743119266055046,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.027522935779816515,0.1743119266055046,,,0.03650561586028542,,,,,0.1743119266055046,0.03650561586028542,0.015742553732396008,,,,
mmlu.prompt1_mqa_single_correct_sk:security_studies|0,mmlu,mmlu.mqa_single_correct_sk:security_studies,1,0,222,em_relaxed,0.1981981981981982,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04504504504504504,0.1981981981981982,,,0.026815582053277207,,,,,0.1981981981981982,0.026815582053277207,0.01395143304433832,,,,
mmlu.prompt1_mqa_single_correct_sk:sociology|0,mmlu,mmlu.mqa_single_correct_sk:sociology,1,0,197,em_relaxed,0.3096446700507614,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.050761421319796954,0.3096446700507614,,,0.03302479343505637,,,,,0.3096446700507614,0.03302479343505637,0.015679295366526198,,,,
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0,mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,1,0,99,em_relaxed,0.23232323232323232,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.2222222222222222,,,0.041996052556580815,,,,,0.23232323232323232,0.04266016017054687,0.01731601731601731,,,,
mmlu.prompt1_mqa_single_correct_sk:virology|0,mmlu,mmlu.mqa_single_correct_sk:virology,1,0,164,em_relaxed,0.12195121951219512,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03048780487804878,0.12195121951219512,,,0.025630589418441155,,,,,0.12195121951219512,0.025630589418441155,0.013466226472535378,,,,
mmlu.prompt1_mqa_single_correct_sk:world_religions|0,mmlu,mmlu.mqa_single_correct_sk:world_religions,1,0,171,em_relaxed,0.05263157894736842,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.023391812865497075,0.05263157894736842,,,0.017126088775157082,,,,,0.05263157894736842,0.017126088775157082,0.01159224790573493,,,,
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0,mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,2,0,100,em_relaxed,0.05,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.04,,,0.019694638556693237,,,,,0.05,0.021904291355759026,0.009999999999999997,,,,
mmlu.prompt2_mqa_single_correct_sk:anatomy|0,mmlu,mmlu.mqa_single_correct_sk:anatomy,2,0,134,em_relaxed,0.1791044776119403,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05970149253731343,0.1791044776119403,,,0.03324844546213894,,,,,0.1791044776119403,0.03324844546213894,0.020544692585423446,,,,
mmlu.prompt2_mqa_single_correct_sk:astronomy|0,mmlu,mmlu.mqa_single_correct_sk:astronomy,2,0,152,em_relaxed,0.17763157894736842,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05263157894736842,0.17105263157894737,,,0.03064360707167709,,,,,0.17763157894736842,0.031103182383123398,0.01817164224152954,,,,
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0,mmlu,mmlu.mqa_single_correct_sk:business_ethics,2,0,100,em_relaxed,0.22,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03,0.21,,,0.04093601807403326,,,,,0.22,0.0416333199893227,0.017144660799776522,,,,
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0,mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,2,0,259,em_relaxed,0.24324324324324326,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04633204633204633,0.23938223938223938,,,0.026565582359277095,,,,,0.24324324324324326,0.026710910573068247,0.013086679986931563,,,,
mmlu.prompt2_mqa_single_correct_sk:college_biology|0,mmlu,mmlu.mqa_single_correct_sk:college_biology,2,0,142,em_relaxed,0.19014084507042253,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04225352112676056,0.19014084507042253,,,0.03304706552420638,,,,,0.19014084507042253,0.03304706552420638,0.01694131738100396,,,,
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:college_chemistry,2,0,100,em_relaxed,0.14,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.09,0.14,,,0.03487350880197772,,,,,0.14,0.03487350880197772,0.028762349126466143,,,,
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:college_computer_science,2,0,100,em_relaxed,0.17,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.12,,,0.032659863237109066,,,,,0.17,0.03775251680686371,0.009999999999999995,,,,
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:college_mathematics,2,0,99,em_relaxed,0.04040404040404041,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.030303030303030304,,,0.017316017316017295,,,,,0.04040404040404041,0.019890402239668436,0.0,,,,
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0,mmlu,mmlu.mqa_single_correct_sk:college_medicine,2,0,172,em_relaxed,0.20930232558139536,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06395348837209303,0.20930232558139536,,,0.031109583909764628,,,,,0.20930232558139536,0.031109583909764628,0.018710387348210598,,,,
mmlu.prompt2_mqa_single_correct_sk:college_physics|0,mmlu,mmlu.mqa_single_correct_sk:college_physics,2,0,102,em_relaxed,0.12745098039215685,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0392156862745098,0.12745098039215685,,,0.03318224921942078,,,,,0.12745098039215685,0.03318224921942078,0.01931444240436096,,,,
mmlu.prompt2_mqa_single_correct_sk:computer_security|0,mmlu,mmlu.mqa_single_correct_sk:computer_security,2,0,100,em_relaxed,0.11,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.11,,,0.03144660377352203,,,,,0.11,0.03144660377352203,0.014070529413628963,,,,
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0,mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,2,0,235,em_relaxed,0.14042553191489363,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0851063829787234,0.14042553191489363,,,0.022712077616627854,,,,,0.14042553191489363,0.022712077616627854,0.018241411344572286,,,,
mmlu.prompt2_mqa_single_correct_sk:econometrics|0,mmlu,mmlu.mqa_single_correct_sk:econometrics,2,0,112,em_relaxed,0.11607142857142858,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.017857142857142856,0.11607142857142858,,,0.03040253249894973,,,,,0.11607142857142858,0.03040253249894973,0.012569900164490047,,,,
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0,mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,2,0,145,em_relaxed,0.1310344827586207,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06896551724137931,0.1310344827586207,,,0.028119838672931184,,,,,0.1310344827586207,0.028119838672931184,0.021116290886061897,,,,
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,2,0,373,em_relaxed,0.10187667560321716,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0160857908847185,0.09651474530831099,,,0.015310377502350168,,,,,0.10187667560321716,0.015683171805464465,0.006522714286001283,,,,
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0,mmlu,mmlu.mqa_single_correct_sk:formal_logic,2,0,115,em_relaxed,0.13043478260869565,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.008695652173913044,0.08695652173913043,,,0.026390303420299282,,,,,0.13043478260869565,0.03154244565695702,0.008695652173913031,,,,
mmlu.prompt2_mqa_single_correct_sk:global_facts|0,mmlu,mmlu.mqa_single_correct_sk:global_facts,2,0,100,em_relaxed,0.18,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.13,0.18,,,0.03861229196653694,,,,,0.18,0.03861229196653694,0.03379976689896309,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_biology,2,0,282,em_relaxed,0.24822695035460993,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04964539007092199,0.24113475177304963,,,0.02551873104953778,,,,,0.24822695035460993,0.025770015644290406,0.012957736571882147,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,2,0,198,em_relaxed,0.11616161616161616,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.020202020202020204,0.1111111111111111,,,0.022390787638216763,,,,,0.11616161616161616,0.022828881775249377,0.01002380379631872,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,2,0,99,em_relaxed,0.12121212121212122,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.12121212121212122,,,0.03296871474399962,,,,,0.12121212121212122,0.03296871474399962,0.01731601731601732,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,2,0,135,em_relaxed,0.23703703703703705,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.022222222222222223,0.23703703703703705,,,0.03673731683969506,,,,,0.23703703703703705,0.03673731683969506,0.012733899715059683,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0,mmlu,mmlu.mqa_single_correct_sk:high_school_geography,2,0,198,em_relaxed,0.22727272727272727,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0707070707070707,0.22727272727272727,,,0.02985751567338641,,,,,0.22727272727272727,0.02985751567338641,0.01826310542019951,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,2,0,193,em_relaxed,0.32124352331606215,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.046632124352331605,0.32124352331606215,,,0.033699508685490674,,,,,0.32124352331606215,0.033699508685490674,0.015216761819262577,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,2,0,387,em_relaxed,0.1731266149870801,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.031007751937984496,0.17054263565891473,,,0.019143434938568855,,,,,0.1731266149870801,0.01925784882518206,0.00882270258926331,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,2,0,266,em_relaxed,0.06766917293233082,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.011278195488721804,0.06766917293233082,,,0.015429701322264242,,,,,0.06766917293233082,0.015429701322264242,0.0064868510822409155,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,2,0,217,em_relaxed,0.17511520737327188,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02304147465437788,0.17511520737327188,,,0.025860189581375218,,,,,0.17511520737327188,0.025860189581375218,0.010208603193741356,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_physics,2,0,150,em_relaxed,0.14666666666666667,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.14666666666666667,,,0.02898223889753507,,,,,0.14666666666666667,0.02898223889753507,0.016053601490175616,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,2,0,536,em_relaxed,0.22388059701492538,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04477611940298507,0.22388059701492538,,,0.01802169384796668,,,,,0.22388059701492538,0.01802169384796668,0.008941264053122127,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,2,0,214,em_relaxed,0.18691588785046728,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.004672897196261682,0.17757009345794392,,,0.02618451638732211,,,,,0.18691588785046728,0.026711670245826564,0.004672897196261678,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,2,0,57,em_relaxed,0.21052631578947367,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.21052631578947367,,,0.05447885994808057,,,,,0.21052631578947367,0.05447885994808057,0.0,,,,
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,2,0,180,em_relaxed,0.2777777777777778,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005555555555555556,0.2777777777777778,,,0.03347785759938633,,,,,0.2777777777777778,0.03347785759938633,0.005555555555555554,,,,
mmlu.prompt2_mqa_single_correct_sk:human_aging|0,mmlu,mmlu.mqa_single_correct_sk:human_aging,2,0,219,em_relaxed,0.1735159817351598,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0867579908675799,0.1735159817351598,,,0.02564830822412722,,,,,0.1735159817351598,0.02564830822412722,0.01906423736413682,,,,
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0,mmlu,mmlu.mqa_single_correct_sk:human_sexuality,2,0,109,em_relaxed,0.21100917431192662,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05504587155963303,0.21100917431192662,,,0.039262212380368526,,,,,0.21100917431192662,0.039262212380368526,0.0219460246091855,,,,
mmlu.prompt2_mqa_single_correct_sk:international_law|0,mmlu,mmlu.mqa_single_correct_sk:international_law,2,0,118,em_relaxed,0.16101694915254236,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05084745762711865,0.16101694915254236,,,0.03397969368949715,,,,,0.16101694915254236,0.03397969368949715,0.020309989475094187,,,,
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0,mmlu,mmlu.mqa_single_correct_sk:jurisprudence,2,0,107,em_relaxed,0.19626168224299065,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.056074766355140186,0.19626168224299065,,,0.038576441428227824,,,,,0.19626168224299065,0.038576441428227824,0.02234599008672909,,,,
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0,mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,2,0,158,em_relaxed,0.25316455696202533,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05063291139240506,0.25316455696202533,,,0.0347027655481443,,,,,0.25316455696202533,0.0347027655481443,0.017497810541168043,,,,
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0,mmlu,mmlu.mqa_single_correct_sk:machine_learning,2,0,112,em_relaxed,0.14285714285714285,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.026785714285714284,0.125,,,0.03139045014587016,,,,,0.14285714285714285,0.033213611069662675,0.015324784082343059,,,,
mmlu.prompt2_mqa_single_correct_sk:management|0,mmlu,mmlu.mqa_single_correct_sk:management,2,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.15,,,0.035887028128263686,,,,,0.15,0.035887028128263686,0.019694638556693237,,,,
mmlu.prompt2_mqa_single_correct_sk:marketing|0,mmlu,mmlu.mqa_single_correct_sk:marketing,2,0,234,em_relaxed,0.2094017094017094,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0811965811965812,0.20512820512820512,,,0.026453508054040356,,,,,0.2094017094017094,0.02665569965392276,0.017893784904018422,,,,
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0,mmlu,mmlu.mqa_single_correct_sk:medical_genetics,2,0,100,em_relaxed,0.18,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07,0.18,,,0.03861229196653694,,,,,0.18,0.03861229196653694,0.02564323999762429,,,,
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0,mmlu,mmlu.mqa_single_correct_sk:miscellaneous,2,0,779,em_relaxed,0.2092426187419769,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0693196405648267,0.2092426187419769,,,0.014583336673745648,,,,,0.2092426187419769,0.014583336673745648,0.009106230246473392,,,,
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0,mmlu,mmlu.mqa_single_correct_sk:moral_disputes,2,0,326,em_relaxed,0.25766871165644173,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.08895705521472393,0.25766871165644173,,,0.0242598463239315,,,,,0.25766871165644173,0.0242598463239315,0.015791302137788298,,,,
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0,mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,2,0,749,em_relaxed,0.0026702269692923898,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0026702269692923898,,,0.001886873052453708,,,,,0.0026702269692923898,0.001886873052453708,0.0,,,,
mmlu.prompt2_mqa_single_correct_sk:nutrition|0,mmlu,mmlu.mqa_single_correct_sk:nutrition,2,0,303,em_relaxed,0.19801980198019803,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0297029702970297,0.19801980198019803,,,0.02293151046482573,,,,,0.19801980198019803,0.02293151046482573,0.00976897099052845,,,,
mmlu.prompt2_mqa_single_correct_sk:philosophy|0,mmlu,mmlu.mqa_single_correct_sk:philosophy,2,0,308,em_relaxed,0.2922077922077922,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.10064935064935066,0.2922077922077922,,,0.02595551182455085,,,,,0.2922077922077922,0.02595551182455085,0.017171207138792156,,,,
mmlu.prompt2_mqa_single_correct_sk:prehistory|0,mmlu,mmlu.mqa_single_correct_sk:prehistory,2,0,319,em_relaxed,0.18808777429467086,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06583072100313479,0.18808777429467086,,,0.021913980163123535,,,,,0.18808777429467086,0.021913980163123535,0.013906362189890747,,,,
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0,mmlu,mmlu.mqa_single_correct_sk:professional_accounting,2,0,276,em_relaxed,0.15217391304347827,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.14130434782608695,,,0.02100540442245809,,,,,0.15217391304347827,0.02165993479655374,0.0,,,,
mmlu.prompt2_mqa_single_correct_sk:professional_law|0,mmlu,mmlu.mqa_single_correct_sk:professional_law,2,0,1171,em_relaxed,0.14175918018787362,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.004269854824935952,0.14175918018787362,,,0.01019734819812428,,,,,0.14175918018787362,0.01019734819812428,0.001906270167826036,,,,
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0,mmlu,mmlu.mqa_single_correct_sk:professional_medicine,2,0,264,em_relaxed,0.19696969696969696,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.003787878787878788,0.19696969696969696,,,0.024523802005628457,,,,,0.19696969696969696,0.024523802005628457,0.003787878787878786,,,,
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0,mmlu,mmlu.mqa_single_correct_sk:professional_psychology,2,0,596,em_relaxed,0.21308724832214765,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.050335570469798654,0.21308724832214765,,,0.01678740388759121,,,,,0.21308724832214765,0.01678740388759121,0.008963220911107225,,,,
mmlu.prompt2_mqa_single_correct_sk:public_relations|0,mmlu,mmlu.mqa_single_correct_sk:public_relations,2,0,109,em_relaxed,0.23853211009174313,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.09174311926605505,0.23853211009174313,,,0.041009771029340336,,,,,0.23853211009174313,0.041009771029340336,0.02777660875318803,,,,
mmlu.prompt2_mqa_single_correct_sk:security_studies|0,mmlu,mmlu.mqa_single_correct_sk:security_studies,2,0,222,em_relaxed,0.21621621621621623,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04504504504504504,0.21621621621621623,,,0.02769147769493752,,,,,0.21621621621621623,0.02769147769493752,0.013951433044338322,,,,
mmlu.prompt2_mqa_single_correct_sk:sociology|0,mmlu,mmlu.mqa_single_correct_sk:sociology,2,0,197,em_relaxed,0.3553299492385787,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07106598984771574,0.3553299492385787,,,0.034186655739878676,,,,,0.3553299492385787,0.034186655739878676,0.018352502878671184,,,,
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0,mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,2,0,99,em_relaxed,0.25252525252525254,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.24242424242424243,,,0.04329004329004326,,,,,0.25252525252525254,0.043887175543277195,0.01731601731601731,,,,
mmlu.prompt2_mqa_single_correct_sk:virology|0,mmlu,mmlu.mqa_single_correct_sk:virology,2,0,164,em_relaxed,0.16463414634146342,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03048780487804878,0.16463414634146342,,,0.029047216079384597,,,,,0.16463414634146342,0.029047216079384597,0.013466226472535378,,,,
mmlu.prompt2_mqa_single_correct_sk:world_religions|0,mmlu,mmlu.mqa_single_correct_sk:world_religions,2,0,171,em_relaxed,0.0935672514619883,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04678362573099415,0.0935672514619883,,,0.022335993231163274,,,,,0.0935672514619883,0.022335993231163274,0.016196389956235627,,,,
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0,mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,3,0,100,em_relaxed,0.06,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.06,,,0.023868325657594173,,,,,0.06,0.023868325657594173,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:anatomy|0,mmlu,mmlu.mqa_single_correct_sk:anatomy,3,0,134,em_relaxed,0.13432835820895522,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.029850746268656716,0.13432835820895522,,,0.029568863219678807,,,,,0.13432835820895522,0.029568863219678807,0.01475608176283378,,,,
mmlu.prompt3_mqa_single_correct_sk:astronomy|0,mmlu,mmlu.mqa_single_correct_sk:astronomy,3,0,152,em_relaxed,0.1513157894736842,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.019736842105263157,0.14473684210526316,,,0.028631951845930412,,,,,0.1513157894736842,0.029162631596843985,0.011319355505772411,,,,
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0,mmlu,mmlu.mqa_single_correct_sk:business_ethics,3,0,100,em_relaxed,0.13,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.13,,,0.03379976689896308,,,,,0.13,0.03379976689896308,0.009999999999999998,,,,
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0,mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,3,0,259,em_relaxed,0.1969111969111969,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03088803088803089,0.1969111969111969,,,0.024757519994692216,,,,,0.1969111969111969,0.024757519994692216,0.010771402065873048,,,,
mmlu.prompt3_mqa_single_correct_sk:college_biology|0,mmlu,mmlu.mqa_single_correct_sk:college_biology,3,0,142,em_relaxed,0.176056338028169,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.035211267605633804,0.176056338028169,,,0.03207487957537854,,,,,0.176056338028169,0.03207487957537854,0.015521989489917001,,,,
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:college_chemistry,3,0,100,em_relaxed,0.16,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.16,,,0.03684529491774709,,,,,0.16,0.03684529491774709,0.01969463855669323,,,,
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:college_computer_science,3,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03,0.11,,,0.031446603773522035,,,,,0.15,0.035887028128263714,0.01714466079977653,,,,
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:college_mathematics,3,0,99,em_relaxed,0.09090909090909091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.08080808080808081,,,0.027530712883605948,,,,,0.09090909090909091,0.02903984386363362,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0,mmlu,mmlu.mqa_single_correct_sk:college_medicine,3,0,172,em_relaxed,0.1569767441860465,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.040697674418604654,0.1569767441860465,,,0.027818863343493188,,,,,0.1569767441860465,0.027818863343493188,0.01511000089316197,,,,
mmlu.prompt3_mqa_single_correct_sk:college_physics|0,mmlu,mmlu.mqa_single_correct_sk:college_physics,3,0,102,em_relaxed,0.12745098039215685,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.12745098039215685,,,0.03318224921942076,,,,,0.12745098039215685,0.03318224921942076,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:computer_security|0,mmlu,mmlu.mqa_single_correct_sk:computer_security,3,0,100,em_relaxed,0.13,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.13,,,0.03379976689896309,,,,,0.13,0.03379976689896309,0.014070529413628963,,,,
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0,mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,3,0,235,em_relaxed,0.1276595744680851,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05106382978723404,0.1276595744680851,,,0.021815328994143183,,,,,0.1276595744680851,0.021815328994143183,0.014390214367756187,,,,
mmlu.prompt3_mqa_single_correct_sk:econometrics|0,mmlu,mmlu.mqa_single_correct_sk:econometrics,3,0,112,em_relaxed,0.10714285714285714,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.017857142857142856,0.09821428571428571,,,0.02824733740077989,,,,,0.10714285714285714,0.02935696201881377,0.012569900164490043,,,,
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0,mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,3,0,145,em_relaxed,0.12413793103448276,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.041379310344827586,0.12413793103448276,,,0.0274782369836366,,,,,0.12413793103448276,0.0274782369836366,0.016597159859992713,,,,
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,3,0,373,em_relaxed,0.08847184986595175,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.08579088471849866,,,0.01452017587959949,,,,,0.08847184986595175,0.014723671750091158,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0,mmlu,mmlu.mqa_single_correct_sk:formal_logic,3,0,115,em_relaxed,0.11304347826086956,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.06956521739130435,,,0.02382794649967283,,,,,0.11304347826086956,0.029656609316837397,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:global_facts|0,mmlu,mmlu.mqa_single_correct_sk:global_facts,3,0,100,em_relaxed,0.12,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.12,,,0.03265986323710906,,,,,0.12,0.03265986323710906,0.009999999999999997,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_biology,3,0,282,em_relaxed,0.1950354609929078,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02127659574468085,0.1879432624113475,,,0.023305230769714243,,,,,0.1950354609929078,0.02363698094391562,0.00860850820920177,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,3,0,198,em_relaxed,0.09090909090909091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005050505050505051,0.08585858585858586,,,0.01996022556317289,,,,,0.09090909090909091,0.020482086775424218,0.00505050505050505,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,3,0,99,em_relaxed,0.13131313131313133,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.010101010101010102,0.12121212121212122,,,0.03296871474399963,,,,,0.13131313131313133,0.034117144062155554,0.010101010101010109,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,3,0,135,em_relaxed,0.21481481481481482,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02962962962962963,0.2074074074074074,,,0.03502553170678318,,,,,0.21481481481481482,0.03547854198560824,0.014648038602753806,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0,mmlu,mmlu.mqa_single_correct_sk:high_school_geography,3,0,198,em_relaxed,0.20202020202020202,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07575757575757576,0.20202020202020202,,,0.028606204289229872,,,,,0.20202020202020202,0.028606204289229872,0.018852670234993107,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,3,0,193,em_relaxed,0.30569948186528495,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.046632124352331605,0.3005181347150259,,,0.0330881859441575,,,,,0.30569948186528495,0.033248379397581594,0.015216761819262604,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,3,0,387,em_relaxed,0.15762273901808785,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01808785529715762,0.14987080103359174,,,0.018168008667456638,,,,,0.15762273901808785,0.018546804758294254,0.006783224657656192,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,3,0,266,em_relaxed,0.05639097744360902,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.05639097744360902,,,0.014170263190664145,,,,,0.05639097744360902,0.014170263190664145,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,3,0,217,em_relaxed,0.17050691244239632,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.013824884792626729,0.16589861751152074,,,0.025310686717794278,,,,,0.17050691244239632,0.025588834483805927,0.007944762237164815,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_physics,3,0,150,em_relaxed,0.17333333333333334,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02666666666666667,0.17333333333333334,,,0.031010810941145068,,,,,0.17333333333333334,0.031010810941145068,0.013198422611557013,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,3,0,536,em_relaxed,0.18470149253731344,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03731343283582089,0.18470149253731344,,,0.016777090030491517,,,,,0.18470149253731344,0.016777090030491517,0.008194041665419557,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,3,0,214,em_relaxed,0.16355140186915887,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.1542056074766355,,,0.024745297661074854,,,,,0.16355140186915887,0.025342938086817392,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,3,0,57,em_relaxed,0.12280701754385964,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.12280701754385964,,,0.04385964912280699,,,,,0.12280701754385964,0.04385964912280699,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,3,0,180,em_relaxed,0.26666666666666666,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005555555555555556,0.26666666666666666,,,0.03305282343736876,,,,,0.26666666666666666,0.03305282343736876,0.005555555555555554,,,,
mmlu.prompt3_mqa_single_correct_sk:human_aging|0,mmlu,mmlu.mqa_single_correct_sk:human_aging,3,0,219,em_relaxed,0.1278538812785388,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0684931506849315,0.1278538812785388,,,0.02261639276396452,,,,,0.1278538812785388,0.02261639276396452,0.017107571020219335,,,,
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0,mmlu,mmlu.mqa_single_correct_sk:human_sexuality,3,0,109,em_relaxed,0.1559633027522936,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05504587155963303,0.1559633027522936,,,0.03491242362801975,,,,,0.1559633027522936,0.03491242362801975,0.021946024609185506,,,,
mmlu.prompt3_mqa_single_correct_sk:international_law|0,mmlu,mmlu.mqa_single_correct_sk:international_law,3,0,118,em_relaxed,0.15254237288135594,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0423728813559322,0.15254237288135594,,,0.03324002290477697,,,,,0.15254237288135594,0.03324002290477697,0.018622984668462274,,,,
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0,mmlu,mmlu.mqa_single_correct_sk:jurisprudence,3,0,107,em_relaxed,0.18691588785046728,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.037383177570093455,0.18691588785046728,,,0.03786499587281739,,,,,0.18691588785046728,0.03786499587281739,0.01842518672106723,,,,
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0,mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,3,0,158,em_relaxed,0.17721518987341772,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0379746835443038,0.17721518987341772,,,0.030475004517281346,,,,,0.17721518987341772,0.030475004517281346,0.015254237576434418,,,,
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0,mmlu,mmlu.mqa_single_correct_sk:machine_learning,3,0,112,em_relaxed,0.125,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.008928571428571428,0.11607142857142858,,,0.03040253249894973,,,,,0.125,0.03139045014587016,0.008928571428571423,,,,
mmlu.prompt3_mqa_single_correct_sk:management|0,mmlu,mmlu.mqa_single_correct_sk:management,3,0,100,em_relaxed,0.2,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06,0.2,,,0.040201512610368445,,,,,0.2,0.040201512610368445,0.023868325657594162,,,,
mmlu.prompt3_mqa_single_correct_sk:marketing|0,mmlu,mmlu.mqa_single_correct_sk:marketing,3,0,234,em_relaxed,0.1623931623931624,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05555555555555555,0.1623931623931624,,,0.02416161812798774,,,,,0.1623931623931624,0.02416161812798774,0.01500631280644691,,,,
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0,mmlu,mmlu.mqa_single_correct_sk:medical_genetics,3,0,100,em_relaxed,0.14,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.14,,,0.034873508801977704,,,,,0.14,0.034873508801977704,0.009999999999999997,,,,
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0,mmlu,mmlu.mqa_single_correct_sk:miscellaneous,3,0,779,em_relaxed,0.17843388960205392,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.055198973042362,0.17715019255455713,,,0.013688041138952063,,,,,0.17843388960205392,0.013726826073077934,0.00818739755268085,,,,
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0,mmlu,mmlu.mqa_single_correct_sk:moral_disputes,3,0,326,em_relaxed,0.2147239263803681,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06748466257668712,0.2116564417177914,,,0.02265853497858724,,,,,0.2147239263803681,0.022777692313730433,0.013915180741090785,,,,
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0,mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,3,0,749,em_relaxed,0.01335113484646195,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0053404539385847796,,,0.0026648668562206297,,,,,0.01335113484646195,0.004196522950335422,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:nutrition|0,mmlu,mmlu.mqa_single_correct_sk:nutrition,3,0,303,em_relaxed,0.18811881188118812,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.036303630363036306,0.18811881188118812,,,0.022488416775417838,,,,,0.18811881188118812,0.022488416775417838,0.010763206393659715,,,,
mmlu.prompt3_mqa_single_correct_sk:philosophy|0,mmlu,mmlu.mqa_single_correct_sk:philosophy,3,0,308,em_relaxed,0.237012987012987,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05194805194805195,0.237012987012987,,,0.024270317456654856,,,,,0.237012987012987,0.024270317456654856,0.01266576753511598,,,,
mmlu.prompt3_mqa_single_correct_sk:prehistory|0,mmlu,mmlu.mqa_single_correct_sk:prehistory,3,0,319,em_relaxed,0.15047021943573669,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04075235109717868,0.14733542319749215,,,0.019876006858615635,,,,,0.15047021943573669,0.020049383410796876,0.011087359931149772,,,,
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0,mmlu,mmlu.mqa_single_correct_sk:professional_accounting,3,0,276,em_relaxed,0.14492753623188406,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.13043478260869565,,,0.020308668108561446,,,,,0.14492753623188406,0.02122807279169736,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:professional_law|0,mmlu,mmlu.mqa_single_correct_sk:professional_law,3,0,1171,em_relaxed,0.1340734415029889,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0017079419299743809,0.12809564474807855,,,0.009770317447740377,,,,,0.1340734415029889,0.009961367847349073,0.0012071811002646103,,,,
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0,mmlu,mmlu.mqa_single_correct_sk:professional_medicine,3,0,264,em_relaxed,0.09090909090909091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.09090909090909091,,,0.01772676309015986,,,,,0.09090909090909091,0.01772676309015986,0.0,,,,
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0,mmlu,mmlu.mqa_single_correct_sk:professional_psychology,3,0,596,em_relaxed,0.19630872483221476,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.018456375838926176,0.19630872483221476,,,0.016283806947934177,,,,,0.19630872483221476,0.016283806947934177,0.005517845461546564,,,,
mmlu.prompt3_mqa_single_correct_sk:public_relations|0,mmlu,mmlu.mqa_single_correct_sk:public_relations,3,0,109,em_relaxed,0.21100917431192662,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07339449541284404,0.21100917431192662,,,0.03926221238036853,,,,,0.21100917431192662,0.03926221238036853,0.025093850407933094,,,,
mmlu.prompt3_mqa_single_correct_sk:security_studies|0,mmlu,mmlu.mqa_single_correct_sk:security_studies,3,0,222,em_relaxed,0.15765765765765766,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04954954954954955,0.15765765765765766,,,0.024513522045494097,,,,,0.15765765765765766,0.024513522045494097,0.014597835283794243,,,,
mmlu.prompt3_mqa_single_correct_sk:sociology|0,mmlu,mmlu.mqa_single_correct_sk:sociology,3,0,197,em_relaxed,0.3350253807106599,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05583756345177665,0.3350253807106599,,,0.03371423842235921,,,,,0.3350253807106599,0.03371423842235921,0.016400555297895067,,,,
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0,mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,3,0,99,em_relaxed,0.2727272727272727,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.050505050505050504,0.26262626262626265,,,0.044452876769839444,,,,,0.2727272727272727,0.044988332664126655,0.022120793242071984,,,,
mmlu.prompt3_mqa_single_correct_sk:virology|0,mmlu,mmlu.mqa_single_correct_sk:virology,3,0,164,em_relaxed,0.1402439024390244,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.024390243902439025,0.1402439024390244,,,0.027197928271392544,,,,,0.1402439024390244,0.027197928271392544,0.012082375722391943,,,,
mmlu.prompt3_mqa_single_correct_sk:world_religions|0,mmlu,mmlu.mqa_single_correct_sk:world_religions,3,0,171,em_relaxed,0.10526315789473684,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04678362573099415,0.10526315789473684,,,0.02353755765789256,,,,,0.10526315789473684,0.02353755765789256,0.01619638995623562,,,,
movie_review.prompt1_sentiment_binary_sk|0,movie_review,movie_review.sentiment_binary_sk,1,0,2008,em_norm_relaxed,0.6329681274900398,32,test,mteb/slovak_movie_review_sentiment,default,29a7405aabcfd4860a51ae6f65a5650d63108f26,,,0.6324701195219123,0.010761991548945318,,0.38745019920318724,0.6329681274900398,0.010758931054934645,0.010874391376667836,,,,,,,
movie_review.prompt2_sentiment_binary_sk|0,movie_review,movie_review.sentiment_binary_sk,2,0,2008,em_norm_relaxed,0.48306772908366535,32,test,mteb/slovak_movie_review_sentiment,default,29a7405aabcfd4860a51ae6f65a5650d63108f26,,,0.47758964143426297,0.011149609394535145,,0.01045816733067729,0.48306772908366535,0.011154424009707972,0.002270759742696006,,,,,,,
movie_review.prompt3_sentiment_binary_sk|0,movie_review,movie_review.sentiment_binary_sk,3,0,2008,em_norm_relaxed,0.11105577689243028,32,test,mteb/slovak_movie_review_sentiment,default,29a7405aabcfd4860a51ae6f65a5650d63108f26,,,0.09412350597609562,0.006517932474820501,,0.04731075697211155,0.11105577689243028,0.007013489433691037,0.004738947450976647,,,,,,,
multi_wiki_qa.prompt1_qa_sk|0,multi_wiki_qa,multi_wiki_qa.qa_sk,1,0,4997,exact_match_slk_prefix,0.36401841104662797,126,train,alexandrainst/multi-wiki-qa,sk,9bdb3350487e399bd1cd873e1395290d636db5ce,,,,,,,,,,,,,0.36401841104662797,0.00680726616133214,0.5156574509584114,0.0058319674487835125
multi_wiki_qa.prompt2_qa_sk|0,multi_wiki_qa,multi_wiki_qa.qa_sk,2,0,4997,exact_match_slk_prefix,0.36001600960576347,126,train,alexandrainst/multi-wiki-qa,sk,9bdb3350487e399bd1cd873e1395290d636db5ce,,,,,,,,,,,,,0.36001600960576347,0.006791008078277957,0.5090494926903776,0.005852156127470673
multi_wiki_qa.prompt3_qa_sk|0,multi_wiki_qa,multi_wiki_qa.qa_sk,3,0,4997,exact_match_slk_prefix,0.34000400240144085,126,train,alexandrainst/multi-wiki-qa,sk,9bdb3350487e399bd1cd873e1395290d636db5ce,,,,,,,,,,,,,0.34000400240144085,0.006701954125609285,0.5050071786470199,0.005797838571943781
senti_sk.prompt1_sentiment_3_class_sk|0,senti_sk,senti_sk.sentiment_3_class_sk,1,0,4677,em_norm_relaxed,0.3200769724182168,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.31922172332691895,0.006817290681927466,,0.22642719692110327,0.3200769724182168,0.0068221276139399335,0.006120369470744044,,,,,,,
senti_sk.prompt1_sentiment_binary_sk|0,senti_sk,senti_sk.sentiment_binary_sk,1,0,3728,em_norm_relaxed,0.31223175965665234,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.3111587982832618,0.007583521183648285,,0.1968884120171674,0.31223175965665234,0.0075906663188793026,0.006513558648330005,,,,,,,
senti_sk.prompt2_sentiment_3_class_sk|0,senti_sk,senti_sk.sentiment_3_class_sk,2,0,4677,em_norm_relaxed,0.383365405174257,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.37716484926234767,0.007087856422819165,,0.019029292281376953,0.383365405174257,0.007110221850748201,0.001998030377272694,,,,,,,
senti_sk.prompt2_sentiment_binary_sk|0,senti_sk,senti_sk.sentiment_binary_sk,2,0,3728,em_norm_relaxed,0.4965128755364807,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.49302575107296137,0.008189324033951761,,0.0037553648068669528,0.4965128755364807,0.008189921623320459,0.001001911513818721,,,,,,,
senti_sk.prompt3_sentiment_3_class_sk|0,senti_sk,senti_sk.sentiment_3_class_sk,3,0,4677,em_norm_relaxed,0.40880906564036773,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.26448578148385715,0.006449996068936491,,0.10540945050245884,0.40880906564036773,0.0071893047031815266,0.004490706548553942,,,,,,,
senti_sk.prompt3_sentiment_binary_sk|0,senti_sk,senti_sk.sentiment_binary_sk,3,0,3728,em_norm_relaxed,0.25160944206008584,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.1990343347639485,0.0065402033880141534,,0.03165236051502146,0.25160944206008584,0.0071080082101389235,0.0028677349189843607,,,,,,,
skj1.prompt1_skj1_sk|0,skj1,skj1.skj1_sk,1,0,100,em_relaxed,0.02,32,train,slovak-nlp/SKJ1,default,e615c60b415389b629703f82d3a79810dbde6734,0.0,0.02,,,0.014070529413628957,,,,,0.02,0.014070529413628957,0.0,,,,
skj1.prompt2_skj1_sk|0,skj1,skj1.skj1_sk,2,0,100,em_relaxed,0.06,32,train,slovak-nlp/SKJ1,default,e615c60b415389b629703f82d3a79810dbde6734,0.0,0.05,,,0.021904291355759067,,,,,0.06,0.023868325657594166,0.0,,,,
skj1.prompt3_skj1_sk|0,skj1,skj1.skj1_sk,3,0,100,em_relaxed,0.05,32,train,slovak-nlp/SKJ1,default,e615c60b415389b629703f82d3a79810dbde6734,0.0,0.03,,,0.017144660799776543,,,,,0.05,0.021904291355759067,0.0,,,,
sklep.prompt1_hate_speech_sk:hate-speech|0,sklep,sklep.hate_speech_sk:hate-speech,1,0,1319,em_norm_relaxed,0.6921910538286581,32,test,slovak-nlp/sklep,hate-speech,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.6921910538286581,0.012714401009923647,,0.0,0.6921910538286581,0.012714401009923647,0.0,,,,,,,
sklep.prompt1_nli_3_classes_sk:nli|0,sklep,sklep.nli_3_classes_sk:nli,1,0,5004,em_norm_relaxed,0.016786570743405275,32,test,slovak-nlp/sklep,nli,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.016386890487609912,0.0017949203375471519,,0.0003996802557953637,0.016786570743405275,0.0018163086111764766,0.00028258837305266234,,,,,,,
sklep.prompt1_nli_binary_sk:rte|0,sklep,sklep.nli_binary_sk:rte,1,0,1660,em_norm_relaxed,0.030120481927710843,32,test,slovak-nlp/sklep,rte,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.030120481927710843,0.004196301257202765,,0.0006024096385542169,0.030120481927710843,0.004196301257202765,0.0006024096385542043,,,,,,,
sklep.prompt1_qa_sk:question-answering|0,sklep,sklep.qa_sk:question-answering,1,0,9583,exact_match_slk_prefix,0.16456224564332672,126,test,slovak-nlp/sklep,question-answering,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,,,,,,,,,,,0.16456224564332672,0.003787861300566259,0.3730151221096053,0.003855179221962441
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0,sklep,sklep.sentiment_binary_sk:sentiment-analysis,1,0,1042,em_norm_relaxed,0.8809980806142035,32,test,slovak-nlp/sklep,sentiment-analysis,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.8800383877159309,0.010070396971355563,,0.44817658349328215,0.8809980806142035,0.010035501945449067,0.015413429449614385,,,,,,,
sklep.prompt2_hate_speech_sk:hate-speech|0,sklep,sklep.hate_speech_sk:hate-speech,2,0,1319,em_norm_relaxed,0.7247915087187263,32,test,slovak-nlp/sklep,hate-speech,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.7247915087187263,0.012302114305862654,,0.0,0.7247915087187263,0.012302114305862654,0.0,,,,,,,
sklep.prompt2_nli_3_classes_sk:nli|0,sklep,sklep.nli_3_classes_sk:nli,2,0,5004,em_norm_relaxed,0.09972022382094324,32,test,slovak-nlp/sklep,nli,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.09952038369304557,0.004232312358957502,,0.0,0.09972022382094324,0.004236089409386789,0.0,,,,,,,
sklep.prompt2_nli_binary_sk:rte|0,sklep,sklep.nli_binary_sk:rte,2,0,1660,em_norm_relaxed,0.3301204819277108,32,test,slovak-nlp/sklep,rte,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.3301204819277108,0.011545464009906971,,0.0018072289156626507,0.3301204819277108,0.011545464009906971,0.001042774975713993,,,,,,,
sklep.prompt2_qa_sk:question-answering|0,sklep,sklep.qa_sk:question-answering,2,0,9583,exact_match_slk_prefix,0.16518835437754356,126,test,slovak-nlp/sklep,question-answering,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,,,,,,,,,,,0.16518835437754356,0.003793637935655721,0.3751701402109962,0.0038611217329234445
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0,sklep,sklep.sentiment_binary_sk:sentiment-analysis,2,0,1042,em_norm_relaxed,0.3512476007677543,32,test,slovak-nlp/sklep,sentiment-analysis,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.34836852207293667,0.014767096602802959,,0.0009596928982725527,0.3512476007677543,0.014795198864354053,0.0009596928982725573,,,,,,,
sklep.prompt3_hate_speech_sk:hate-speech|0,sklep,sklep.hate_speech_sk:hate-speech,3,0,1319,em_norm_relaxed,0.6967399545109931,32,test,slovak-nlp/sklep,hate-speech,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.6967399545109931,0.012661502663418697,,0.0,0.6967399545109931,0.012661502663418697,0.0,,,,,,,
sklep.prompt3_nli_3_classes_sk:nli|0,sklep,sklep.nli_3_classes_sk:nli,3,0,5004,em_norm_relaxed,0.001398880895283773,32,test,slovak-nlp/sklep,nli,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.001398880895283773,0.0005284101391342323,,0.0,0.001398880895283773,0.0005284101391342323,0.0,,,,,,,
sklep.prompt3_nli_binary_sk:rte|0,sklep,sklep.nli_binary_sk:rte,3,0,1660,em_norm_relaxed,0.2319277108433735,32,test,slovak-nlp/sklep,rte,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.2319277108433735,0.010362252629508847,,0.0012048192771084338,0.2319277108433735,0.010362252629508847,0.0008516790803748009,,,,,,,
sklep.prompt3_qa_sk:question-answering|0,sklep,sklep.qa_sk:question-answering,3,0,9583,exact_match_slk_prefix,0.16143170197224252,126,test,slovak-nlp/sklep,question-answering,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,,,,,,,,,,,0.16143170197224252,0.0037586816998114083,0.3769392643579436,0.003773157275405368
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0,sklep,sklep.sentiment_binary_sk:sentiment-analysis,3,0,1042,em_norm_relaxed,0.15355086372360843,32,test,slovak-nlp/sklep,sentiment-analysis,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.118042226487524,0.010000396690181516,,0.015355086372360844,0.15355086372360843,0.011173807886451477,0.0038110143844746576,,,,,,,
slovakcopa.prompt1_copa_sk|0,slovakcopa,slovakcopa.copa_sk,1,0,500,em_relaxed,0.618,32,test,NaiveNeuron/SlovakCOPA,default,5c61e4cd9343f9b4fab623bdc50bb895cc170771,0.0,0.508,,,0.022380208834928028,,,,,0.618,0.021750820591250834,0.0,,,,
slovakcopa.prompt2_copa_sk|0,slovakcopa,slovakcopa.copa_sk,2,0,500,em_relaxed,0.51,32,test,NaiveNeuron/SlovakCOPA,default,5c61e4cd9343f9b4fab623bdc50bb895cc170771,0.0,0.326,,,0.020984009562393567,,,,,0.51,0.02237859698923078,0.0,,,,
slovakcopa.prompt3_copa_sk|0,slovakcopa,slovakcopa.copa_sk,3,0,500,em_relaxed,0.086,32,test,NaiveNeuron/SlovakCOPA,default,5c61e4cd9343f9b4fab623bdc50bb895cc170771,0.0,0.084,,,0.01241758401564368,,,,,0.086,0.01255081854271602,0.0,,,,
toxic_sk.prompt1_toxicity_sk|0,toxic_sk,toxic_sk.toxicity_sk,1,0,881,em_norm_relaxed,0.45629965947786605,32,test,TUKE-KEMT/toxic-sk,default,2fb416e18785c128e7ec934c07a14c803a88cdb3,,,0.45629965947786605,0.016790496508382532,,0.0,0.45629965947786605,0.016790496508382532,0.0,,,,,,,
toxic_sk.prompt2_toxicity_sk|0,toxic_sk,toxic_sk.toxicity_sk,2,0,881,em_norm_relaxed,0.47559591373439275,32,test,TUKE-KEMT/toxic-sk,default,2fb416e18785c128e7ec934c07a14c803a88cdb3,,,0.47559591373439275,0.016834908286515442,,0.0,0.47559591373439275,0.016834908286515442,0.0,,,,,,,
toxic_sk.prompt3_toxicity_sk|0,toxic_sk,toxic_sk.toxicity_sk,3,0,881,em_norm_relaxed,0.4960272417707151,32,test,TUKE-KEMT/toxic-sk,default,2fb416e18785c128e7ec934c07a14c803a88cdb3,,,0.4960272417707151,0.016854464514836336,,0.007945516458569807,0.4960272417707151,0.016854464514836336,0.00299286751118699,,,,,,,
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0,truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,1,0,778,em_relaxed,0.007712082262210797,32,val,alexandrainst/m_truthfulqa,sk,f0445d470f1925882b990f5f247fdcf288972f60,0.0,0.007712082262210797,,,0.003138297919968842,,,,,0.007712082262210797,0.003138297919968842,0.0,,,,
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0,truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,2,0,778,em_relaxed,0.016709511568123392,32,val,alexandrainst/m_truthfulqa,sk,f0445d470f1925882b990f5f247fdcf288972f60,0.0,0.016709511568123392,,,0.004598458668423289,,,,,0.016709511568123392,0.004598458668423289,0.0,,,,
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0,truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,3,0,778,em_relaxed,0.011568123393316195,32,val,alexandrainst/m_truthfulqa,sk,f0445d470f1925882b990f5f247fdcf288972f60,0.0,0.011568123393316195,,,0.00383613885044431,,,,,0.011568123393316195,0.00383613885044431,0.0,,,,
1 task family task_group_without_prompt prompt num_fewshots sample_count primary_metric primary_value generation_size evaluation_splits hf_repo hf_subset hf_revision em em_conservative em_conservative_norm em_conservative_norm_stderr em_conservative_stderr em_norm em_norm_relaxed em_norm_relaxed_stderr em_norm_stderr em_relaxed em_relaxed_stderr em_stderr exact_match_slk_prefix exact_match_slk_prefix_stderr f1_slk f1_slk_stderr
2 arc.prompt1_mqa_single_correct_sk|0 arc arc.mqa_single_correct_sk 1 0 1169 em_relaxed 0.1710863986313088 32 test jon-tow/okapi_arc_challenge sk 56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e 0.013686911890504704 0.1710863986313088 0.011018968811092407 0.1710863986313088 0.011018968811092407 0.0033996852610928244
3 arc.prompt2_mqa_single_correct_sk|0 arc arc.mqa_single_correct_sk 2 0 1169 em_relaxed 0.22412318220701455 32 test jon-tow/okapi_arc_challenge sk 56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e 0.03507271171941831 0.223267750213858 0.012185048029719049 0.22412318220701455 0.012201644195165715 0.005382828530594407
4 arc.prompt3_mqa_single_correct_sk|0 arc arc.mqa_single_correct_sk 3 0 1169 em_relaxed 0.1770744225834046 32 test jon-tow/okapi_arc_challenge sk 56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e 0.022241231822070145 0.17621899059024806 0.011148355810710043 0.1770744225834046 0.011169578281264629 0.004314931013145922
5 belebele.prompt1_mqa_single_correct_sk|0 belebele belebele.mqa_single_correct_sk 1 0 900 em_relaxed 0.21555555555555556 32 test facebook/belebele slk_Latn 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2 0.02 0.21555555555555556 0.01371452783236968 0.21555555555555556 0.01371452783236968 0.004669261421755523
6 belebele.prompt2_mqa_single_correct_sk|0 belebele belebele.mqa_single_correct_sk 2 0 900 em_relaxed 0.2111111111111111 32 test facebook/belebele slk_Latn 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2 0.016666666666666666 0.2111111111111111 0.013610798969328482 0.2111111111111111 0.013610798969328482 0.004269675894690937
7 belebele.prompt3_mqa_single_correct_sk|0 belebele belebele.mqa_single_correct_sk 3 0 900 em_relaxed 0.21333333333333335 32 test facebook/belebele slk_Latn 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2 0.03 0.2111111111111111 0.013610798969328486 0.21333333333333335 0.013662962863011163 0.005689402360648914
8 financial_exams.prompt1_completion_sk|0 financial_exams financial_exams.completion_sk 1 0 1334 em_relaxed 0.20089955022488756 32 test TUKE-KEMT/slovak-financial-exam default 7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c 0.009745127436281859 0.20089955022488756 0.010974256206370726 0.20089955022488756 0.010974256206370726 0.002690618851324012
9 financial_exams.prompt2_completion_sk|0 financial_exams financial_exams.completion_sk 2 0 1334 em_relaxed 0.2383808095952024 32 test TUKE-KEMT/slovak-financial-exam default 7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c 0.07571214392803598 0.20539730134932535 0.011065150105060274 0.2383808095952024 0.011670501255680532 0.007245549384888845
10 financial_exams.prompt3_completion_sk|0 financial_exams financial_exams.completion_sk 3 0 1334 em_relaxed 0.24887556221889057 32 test TUKE-KEMT/slovak-financial-exam default 7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c 0.0037481259370314842 0.24887556221889057 0.01184218927916494 0.24887556221889057 0.01184218927916494 0.0016736960391157028
11 globalmmlu-lite.prompt1_mqa_single_correct_sk|0 globalmmlu-lite globalmmlu-lite.mqa_single_correct_sk 1 0 400 em_relaxed 0.1825 32 test CohereLabs/Global-MMLU-Lite sk 600af1bff2d447c7fc3aee22f11f607560d5fcb7 0.025 0.1825 0.01933698975881026 0.1825 0.01933698975881026 0.007816023641463065
12 globalmmlu-lite.prompt2_mqa_single_correct_sk|0 globalmmlu-lite globalmmlu-lite.mqa_single_correct_sk 2 0 400 em_relaxed 0.21 32 test CohereLabs/Global-MMLU-Lite sk 600af1bff2d447c7fc3aee22f11f607560d5fcb7 0.035 0.21 0.020390916450326858 0.21 0.020390916450326858 0.009200495791370004
13 globalmmlu-lite.prompt3_mqa_single_correct_sk|0 globalmmlu-lite globalmmlu-lite.mqa_single_correct_sk 3 0 400 em_relaxed 0.2025 32 test CohereLabs/Global-MMLU-Lite sk 600af1bff2d447c7fc3aee22f11f607560d5fcb7 0.025 0.2025 0.02011830610666578 0.2025 0.02011830610666578 0.007816023641463063
14 mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0 mmlu mmlu.mqa_single_correct_sk:abstract_algebra 1 0 100 em_relaxed 0.01 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.01 0.009999999999999997 0.01 0.009999999999999997 0.0
15 mmlu.prompt1_mqa_single_correct_sk:anatomy|0 mmlu mmlu.mqa_single_correct_sk:anatomy 1 0 134 em_relaxed 0.12686567164179105 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.022388059701492536 0.12686567164179105 0.028859365115889928 0.12686567164179105 0.028859365115889928 0.012828198053329663
16 mmlu.prompt1_mqa_single_correct_sk:astronomy|0 mmlu mmlu.mqa_single_correct_sk:astronomy 1 0 152 em_relaxed 0.14473684210526316 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03289473684210526 0.13815789473684212 0.028081042939576552 0.14473684210526316 0.028631951845930412 0.014514818445457586
17 mmlu.prompt1_mqa_single_correct_sk:business_ethics|0 mmlu mmlu.mqa_single_correct_sk:business_ethics 1 0 100 em_relaxed 0.14 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02 0.14 0.03487350880197772 0.14 0.03487350880197772 0.014070529413628957
18 mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0 mmlu mmlu.mqa_single_correct_sk:clinical_knowledge 1 0 259 em_relaxed 0.15444015444015444 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.011583011583011582 0.15444015444015444 0.022497923516297567 0.15444015444015444 0.022497923516297567 0.006661484058899665
19 mmlu.prompt1_mqa_single_correct_sk:college_biology|0 mmlu mmlu.mqa_single_correct_sk:college_biology 1 0 142 em_relaxed 0.15492957746478872 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.028169014084507043 0.15492957746478872 0.030472217725828407 0.15492957746478872 0.030472217725828407 0.013933866272175799
20 mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0 mmlu mmlu.mqa_single_correct_sk:college_chemistry 1 0 100 em_relaxed 0.08 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04 0.08 0.027265992434429093 0.08 0.027265992434429093 0.019694638556693227
21 mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0 mmlu mmlu.mqa_single_correct_sk:college_computer_science 1 0 100 em_relaxed 0.15 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01 0.11 0.031446603773522035 0.15 0.035887028128263714 0.009999999999999998
22 mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0 mmlu mmlu.mqa_single_correct_sk:college_mathematics 1 0 99 em_relaxed 0.06060606060606061 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.050505050505050504 0.02212079324207198 0.06060606060606061 0.024102876029567216 0.0
23 mmlu.prompt1_mqa_single_correct_sk:college_medicine|0 mmlu mmlu.mqa_single_correct_sk:college_medicine 1 0 172 em_relaxed 0.1511627906976744 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.011627906976744186 0.1511627906976744 0.027392810919920886 0.1511627906976744 0.027392810919920886 0.008198095184581854
24 mmlu.prompt1_mqa_single_correct_sk:college_physics|0 mmlu mmlu.mqa_single_correct_sk:college_physics 1 0 102 em_relaxed 0.0784313725490196 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.00980392156862745 0.0784313725490196 0.026751495257385273 0.0784313725490196 0.026751495257385273 0.009803921568627454
25 mmlu.prompt1_mqa_single_correct_sk:computer_security|0 mmlu mmlu.mqa_single_correct_sk:computer_security 1 0 100 em_relaxed 0.08 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02 0.08 0.027265992434429082 0.08 0.027265992434429082 0.014070529413628966
26 mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0 mmlu mmlu.mqa_single_correct_sk:conceptual_physics 1 0 235 em_relaxed 0.09361702127659574 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03829787234042553 0.09361702127659574 0.01904256081095342 0.09361702127659574 0.01904256081095342 0.012545838230269625
27 mmlu.prompt1_mqa_single_correct_sk:econometrics|0 mmlu mmlu.mqa_single_correct_sk:econometrics 1 0 112 em_relaxed 0.125 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.026785714285714284 0.125 0.03139045014587016 0.125 0.03139045014587016 0.015324784082343043
28 mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0 mmlu mmlu.mqa_single_correct_sk:electrical_engineering 1 0 145 em_relaxed 0.0896551724137931 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.041379310344827586 0.0896551724137931 0.023807258824376967 0.0896551724137931 0.023807258824376967 0.016597159859992713
29 mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0 mmlu mmlu.mqa_single_correct_sk:elementary_mathematics 1 0 373 em_relaxed 0.06166219839142091 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.005361930294906166 0.058981233243967826 0.012214750400069162 0.06166219839142091 0.012471469772330128 0.003786357797685454
30 mmlu.prompt1_mqa_single_correct_sk:formal_logic|0 mmlu mmlu.mqa_single_correct_sk:formal_logic 1 0 115 em_relaxed 0.12173913043478261 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.0782608695652174 0.025154976812194616 0.12173913043478261 0.030624883036764184 0.0
31 mmlu.prompt1_mqa_single_correct_sk:global_facts|0 mmlu mmlu.mqa_single_correct_sk:global_facts 1 0 100 em_relaxed 0.11 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06 0.11 0.031446603773522035 0.11 0.031446603773522035 0.02386832565759416
32 mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0 mmlu mmlu.mqa_single_correct_sk:high_school_biology 1 0 282 em_relaxed 0.18439716312056736 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02127659574468085 0.1773049645390071 0.022783829728850895 0.18439716312056736 0.023134670670309505 0.00860850820920177
33 mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0 mmlu mmlu.mqa_single_correct_sk:high_school_chemistry 1 0 198 em_relaxed 0.08585858585858586 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.010101010101010102 0.08080808080808081 0.019417681889724533 0.08585858585858586 0.019960225563172885 0.007124341521250813
34 mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0 mmlu mmlu.mqa_single_correct_sk:high_school_computer_science 1 0 99 em_relaxed 0.10101010101010101 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.10101010101010101 0.030440148787487714 0.10101010101010101 0.030440148787487714 0.0
35 mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_european_history 1 0 135 em_relaxed 0.24444444444444444 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.014814814814814815 0.24444444444444444 0.03712537833614866 0.24444444444444444 0.03712537833614866 0.010436494549594362
36 mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0 mmlu mmlu.mqa_single_correct_sk:high_school_geography 1 0 198 em_relaxed 0.15656565656565657 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.030303030303030304 0.15656565656565657 0.025890520358141454 0.15656565656565657 0.025890520358141454 0.012213156893572809
37 mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0 mmlu mmlu.mqa_single_correct_sk:high_school_government_and_politics 1 0 193 em_relaxed 0.30569948186528495 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.025906735751295335 0.30569948186528495 0.03324837939758159 0.30569948186528495 0.03324837939758159 0.011464523356953171
38 mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0 mmlu mmlu.mqa_single_correct_sk:high_school_macroeconomics 1 0 387 em_relaxed 0.12919896640826872 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.012919896640826873 0.12661498708010335 0.016925900082204414 0.12919896640826872 0.017072429785340966 0.005747937884708704
39 mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0 mmlu mmlu.mqa_single_correct_sk:high_school_mathematics 1 0 266 em_relaxed 0.03759398496240601 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.03759398496240601 0.011684641953170118 0.03759398496240601 0.011684641953170118 0.0
40 mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0 mmlu mmlu.mqa_single_correct_sk:high_school_microeconomics 1 0 217 em_relaxed 0.14285714285714285 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.004608294930875576 0.14285714285714285 0.0238095238095238 0.14285714285714285 0.0238095238095238 0.004608294930875571
41 mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0 mmlu mmlu.mqa_single_correct_sk:high_school_physics 1 0 150 em_relaxed 0.13333333333333333 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02666666666666667 0.13333333333333333 0.027848530325897826 0.13333333333333333 0.027848530325897826 0.013198422611557032
42 mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0 mmlu mmlu.mqa_single_correct_sk:high_school_psychology 1 0 536 em_relaxed 0.16791044776119404 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.016791044776119403 0.16791044776119404 0.016160210122502162 0.16791044776119404 0.016160210122502162 0.005555010467354857
43 mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0 mmlu mmlu.mqa_single_correct_sk:high_school_statistics 1 0 214 em_relaxed 0.1822429906542056 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.17289719626168223 0.025910984835548666 0.1822429906542056 0.026451344562470138 0.0
44 mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_us_history 1 0 57 em_relaxed 0.19298245614035087 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.19298245614035087 0.05273590328947574 0.19298245614035087 0.05273590328947574 0.0
45 mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_world_history 1 0 180 em_relaxed 0.2777777777777778 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.005555555555555556 0.2777777777777778 0.033477857599386325 0.2777777777777778 0.033477857599386325 0.005555555555555546
46 mmlu.prompt1_mqa_single_correct_sk:human_aging|0 mmlu mmlu.mqa_single_correct_sk:human_aging 1 0 219 em_relaxed 0.1187214611872146 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0365296803652968 0.1187214611872146 0.021907507389103187 0.1187214611872146 0.021907507389103187 0.012706146532849703
47 mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0 mmlu mmlu.mqa_single_correct_sk:human_sexuality 1 0 109 em_relaxed 0.13761467889908258 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.045871559633027525 0.13761467889908258 0.03314905716923366 0.13761467889908258 0.03314905716923366 0.020130904850286974
48 mmlu.prompt1_mqa_single_correct_sk:international_law|0 mmlu mmlu.mqa_single_correct_sk:international_law 1 0 118 em_relaxed 0.13559322033898305 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03389830508474576 0.13559322033898305 0.03165083260501198 0.13559322033898305 0.03165083260501198 0.016730444637044904
49 mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0 mmlu mmlu.mqa_single_correct_sk:jurisprudence 1 0 107 em_relaxed 0.14018691588785046 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.056074766355140186 0.14018691588785046 0.03372118539369118 0.14018691588785046 0.03372118539369118 0.02234599008672909
50 mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0 mmlu mmlu.mqa_single_correct_sk:logical_fallacies 1 0 158 em_relaxed 0.13924050632911392 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02531645569620253 0.13924050632911392 0.027629554439252756 0.13924050632911392 0.027629554439252756 0.012536706047327604
51 mmlu.prompt1_mqa_single_correct_sk:machine_learning|0 mmlu mmlu.mqa_single_correct_sk:machine_learning 1 0 112 em_relaxed 0.11607142857142858 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.008928571428571428 0.10714285714285714 0.02935696201881377 0.11607142857142858 0.03040253249894973 0.008928571428571423
52 mmlu.prompt1_mqa_single_correct_sk:management|0 mmlu mmlu.mqa_single_correct_sk:management 1 0 100 em_relaxed 0.15 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02 0.15 0.0358870281282637 0.15 0.0358870281282637 0.01407052941362896
53 mmlu.prompt1_mqa_single_correct_sk:marketing|0 mmlu mmlu.mqa_single_correct_sk:marketing 1 0 234 em_relaxed 0.15384615384615385 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03418803418803419 0.15384615384615385 0.023636873317489277 0.15384615384615385 0.023636873317489277 0.011904341997629792
54 mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0 mmlu mmlu.mqa_single_correct_sk:medical_genetics 1 0 100 em_relaxed 0.18 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04 0.18 0.038612291966536955 0.18 0.038612291966536955 0.019694638556693223
55 mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0 mmlu mmlu.mqa_single_correct_sk:miscellaneous 1 0 779 em_relaxed 0.14505776636713735 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03209242618741977 0.14505776636713735 0.01262551435912769 0.14505776636713735 0.01262551435912769 0.006318709962464618
56 mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0 mmlu mmlu.mqa_single_correct_sk:moral_disputes 1 0 326 em_relaxed 0.20245398773006135 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06134969325153374 0.20245398773006135 0.022289450459613447 0.20245398773006135 0.022289450459613447 0.01331117614747688
57 mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0 mmlu mmlu.mqa_single_correct_sk:moral_scenarios 1 0 749 em_relaxed 0.006675567423230975 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.0053404539385847796 0.0026648668562206292 0.006675567423230975 0.0029774114441068625 0.0
58 mmlu.prompt1_mqa_single_correct_sk:nutrition|0 mmlu mmlu.mqa_single_correct_sk:nutrition 1 0 303 em_relaxed 0.1848184818481848 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.036303630363036306 0.1848184818481848 0.022335536574954167 0.1848184818481848 0.022335536574954167 0.010763206393659719
59 mmlu.prompt1_mqa_single_correct_sk:philosophy|0 mmlu mmlu.mqa_single_correct_sk:philosophy 1 0 308 em_relaxed 0.21103896103896103 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06818181818181818 0.21103896103896103 0.023288413887090093 0.21103896103896103 0.023288413887090093 0.014385689805382164
60 mmlu.prompt1_mqa_single_correct_sk:prehistory|0 mmlu mmlu.mqa_single_correct_sk:prehistory 1 0 319 em_relaxed 0.15987460815047022 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04075235109717868 0.15987460815047022 0.02055172319966096 0.15987460815047022 0.02055172319966096 0.011087359931149773
61 mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0 mmlu mmlu.mqa_single_correct_sk:professional_accounting 1 0 276 em_relaxed 0.15217391304347827 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0036231884057971015 0.14130434782608695 0.02100540442245809 0.15217391304347827 0.02165993479655374 0.0036231884057970937
62 mmlu.prompt1_mqa_single_correct_sk:professional_law|0 mmlu mmlu.mqa_single_correct_sk:professional_law 1 0 1171 em_relaxed 0.12724167378309137 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0017079419299743809 0.12724167378309137 0.009742462799429212 0.12724167378309137 0.009742462799429212 0.0012071811002646103
63 mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0 mmlu mmlu.mqa_single_correct_sk:professional_medicine 1 0 264 em_relaxed 0.13257575757575757 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.13257575757575757 0.020910766968363862 0.13257575757575757 0.020910766968363862 0.0
64 mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0 mmlu mmlu.mqa_single_correct_sk:professional_psychology 1 0 596 em_relaxed 0.20134228187919462 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03355704697986577 0.20134228187919462 0.016439528475054862 0.20134228187919462 0.016439528475054862 0.0073828066874755535
65 mmlu.prompt1_mqa_single_correct_sk:public_relations|0 mmlu mmlu.mqa_single_correct_sk:public_relations 1 0 109 em_relaxed 0.1743119266055046 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.027522935779816515 0.1743119266055046 0.03650561586028542 0.1743119266055046 0.03650561586028542 0.015742553732396008
66 mmlu.prompt1_mqa_single_correct_sk:security_studies|0 mmlu mmlu.mqa_single_correct_sk:security_studies 1 0 222 em_relaxed 0.1981981981981982 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04504504504504504 0.1981981981981982 0.026815582053277207 0.1981981981981982 0.026815582053277207 0.01395143304433832
67 mmlu.prompt1_mqa_single_correct_sk:sociology|0 mmlu mmlu.mqa_single_correct_sk:sociology 1 0 197 em_relaxed 0.3096446700507614 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.050761421319796954 0.3096446700507614 0.03302479343505637 0.3096446700507614 0.03302479343505637 0.015679295366526198
68 mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0 mmlu mmlu.mqa_single_correct_sk:us_foreign_policy 1 0 99 em_relaxed 0.23232323232323232 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.030303030303030304 0.2222222222222222 0.041996052556580815 0.23232323232323232 0.04266016017054687 0.01731601731601731
69 mmlu.prompt1_mqa_single_correct_sk:virology|0 mmlu mmlu.mqa_single_correct_sk:virology 1 0 164 em_relaxed 0.12195121951219512 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03048780487804878 0.12195121951219512 0.025630589418441155 0.12195121951219512 0.025630589418441155 0.013466226472535378
70 mmlu.prompt1_mqa_single_correct_sk:world_religions|0 mmlu mmlu.mqa_single_correct_sk:world_religions 1 0 171 em_relaxed 0.05263157894736842 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.023391812865497075 0.05263157894736842 0.017126088775157082 0.05263157894736842 0.017126088775157082 0.01159224790573493
71 mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0 mmlu mmlu.mqa_single_correct_sk:abstract_algebra 2 0 100 em_relaxed 0.05 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01 0.04 0.019694638556693237 0.05 0.021904291355759026 0.009999999999999997
72 mmlu.prompt2_mqa_single_correct_sk:anatomy|0 mmlu mmlu.mqa_single_correct_sk:anatomy 2 0 134 em_relaxed 0.1791044776119403 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05970149253731343 0.1791044776119403 0.03324844546213894 0.1791044776119403 0.03324844546213894 0.020544692585423446
73 mmlu.prompt2_mqa_single_correct_sk:astronomy|0 mmlu mmlu.mqa_single_correct_sk:astronomy 2 0 152 em_relaxed 0.17763157894736842 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05263157894736842 0.17105263157894737 0.03064360707167709 0.17763157894736842 0.031103182383123398 0.01817164224152954
74 mmlu.prompt2_mqa_single_correct_sk:business_ethics|0 mmlu mmlu.mqa_single_correct_sk:business_ethics 2 0 100 em_relaxed 0.22 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03 0.21 0.04093601807403326 0.22 0.0416333199893227 0.017144660799776522
75 mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0 mmlu mmlu.mqa_single_correct_sk:clinical_knowledge 2 0 259 em_relaxed 0.24324324324324326 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04633204633204633 0.23938223938223938 0.026565582359277095 0.24324324324324326 0.026710910573068247 0.013086679986931563
76 mmlu.prompt2_mqa_single_correct_sk:college_biology|0 mmlu mmlu.mqa_single_correct_sk:college_biology 2 0 142 em_relaxed 0.19014084507042253 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04225352112676056 0.19014084507042253 0.03304706552420638 0.19014084507042253 0.03304706552420638 0.01694131738100396
77 mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0 mmlu mmlu.mqa_single_correct_sk:college_chemistry 2 0 100 em_relaxed 0.14 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.09 0.14 0.03487350880197772 0.14 0.03487350880197772 0.028762349126466143
78 mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0 mmlu mmlu.mqa_single_correct_sk:college_computer_science 2 0 100 em_relaxed 0.17 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01 0.12 0.032659863237109066 0.17 0.03775251680686371 0.009999999999999995
79 mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0 mmlu mmlu.mqa_single_correct_sk:college_mathematics 2 0 99 em_relaxed 0.04040404040404041 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.030303030303030304 0.017316017316017295 0.04040404040404041 0.019890402239668436 0.0
80 mmlu.prompt2_mqa_single_correct_sk:college_medicine|0 mmlu mmlu.mqa_single_correct_sk:college_medicine 2 0 172 em_relaxed 0.20930232558139536 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06395348837209303 0.20930232558139536 0.031109583909764628 0.20930232558139536 0.031109583909764628 0.018710387348210598
81 mmlu.prompt2_mqa_single_correct_sk:college_physics|0 mmlu mmlu.mqa_single_correct_sk:college_physics 2 0 102 em_relaxed 0.12745098039215685 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0392156862745098 0.12745098039215685 0.03318224921942078 0.12745098039215685 0.03318224921942078 0.01931444240436096
82 mmlu.prompt2_mqa_single_correct_sk:computer_security|0 mmlu mmlu.mqa_single_correct_sk:computer_security 2 0 100 em_relaxed 0.11 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02 0.11 0.03144660377352203 0.11 0.03144660377352203 0.014070529413628963
83 mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0 mmlu mmlu.mqa_single_correct_sk:conceptual_physics 2 0 235 em_relaxed 0.14042553191489363 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0851063829787234 0.14042553191489363 0.022712077616627854 0.14042553191489363 0.022712077616627854 0.018241411344572286
84 mmlu.prompt2_mqa_single_correct_sk:econometrics|0 mmlu mmlu.mqa_single_correct_sk:econometrics 2 0 112 em_relaxed 0.11607142857142858 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.017857142857142856 0.11607142857142858 0.03040253249894973 0.11607142857142858 0.03040253249894973 0.012569900164490047
85 mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0 mmlu mmlu.mqa_single_correct_sk:electrical_engineering 2 0 145 em_relaxed 0.1310344827586207 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06896551724137931 0.1310344827586207 0.028119838672931184 0.1310344827586207 0.028119838672931184 0.021116290886061897
86 mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0 mmlu mmlu.mqa_single_correct_sk:elementary_mathematics 2 0 373 em_relaxed 0.10187667560321716 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0160857908847185 0.09651474530831099 0.015310377502350168 0.10187667560321716 0.015683171805464465 0.006522714286001283
87 mmlu.prompt2_mqa_single_correct_sk:formal_logic|0 mmlu mmlu.mqa_single_correct_sk:formal_logic 2 0 115 em_relaxed 0.13043478260869565 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.008695652173913044 0.08695652173913043 0.026390303420299282 0.13043478260869565 0.03154244565695702 0.008695652173913031
88 mmlu.prompt2_mqa_single_correct_sk:global_facts|0 mmlu mmlu.mqa_single_correct_sk:global_facts 2 0 100 em_relaxed 0.18 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.13 0.18 0.03861229196653694 0.18 0.03861229196653694 0.03379976689896309
89 mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0 mmlu mmlu.mqa_single_correct_sk:high_school_biology 2 0 282 em_relaxed 0.24822695035460993 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04964539007092199 0.24113475177304963 0.02551873104953778 0.24822695035460993 0.025770015644290406 0.012957736571882147
90 mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0 mmlu mmlu.mqa_single_correct_sk:high_school_chemistry 2 0 198 em_relaxed 0.11616161616161616 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.020202020202020204 0.1111111111111111 0.022390787638216763 0.11616161616161616 0.022828881775249377 0.01002380379631872
91 mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0 mmlu mmlu.mqa_single_correct_sk:high_school_computer_science 2 0 99 em_relaxed 0.12121212121212122 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.030303030303030304 0.12121212121212122 0.03296871474399962 0.12121212121212122 0.03296871474399962 0.01731601731601732
92 mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_european_history 2 0 135 em_relaxed 0.23703703703703705 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.022222222222222223 0.23703703703703705 0.03673731683969506 0.23703703703703705 0.03673731683969506 0.012733899715059683
93 mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0 mmlu mmlu.mqa_single_correct_sk:high_school_geography 2 0 198 em_relaxed 0.22727272727272727 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0707070707070707 0.22727272727272727 0.02985751567338641 0.22727272727272727 0.02985751567338641 0.01826310542019951
94 mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0 mmlu mmlu.mqa_single_correct_sk:high_school_government_and_politics 2 0 193 em_relaxed 0.32124352331606215 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.046632124352331605 0.32124352331606215 0.033699508685490674 0.32124352331606215 0.033699508685490674 0.015216761819262577
95 mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0 mmlu mmlu.mqa_single_correct_sk:high_school_macroeconomics 2 0 387 em_relaxed 0.1731266149870801 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.031007751937984496 0.17054263565891473 0.019143434938568855 0.1731266149870801 0.01925784882518206 0.00882270258926331
96 mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0 mmlu mmlu.mqa_single_correct_sk:high_school_mathematics 2 0 266 em_relaxed 0.06766917293233082 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.011278195488721804 0.06766917293233082 0.015429701322264242 0.06766917293233082 0.015429701322264242 0.0064868510822409155
97 mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0 mmlu mmlu.mqa_single_correct_sk:high_school_microeconomics 2 0 217 em_relaxed 0.17511520737327188 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02304147465437788 0.17511520737327188 0.025860189581375218 0.17511520737327188 0.025860189581375218 0.010208603193741356
98 mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0 mmlu mmlu.mqa_single_correct_sk:high_school_physics 2 0 150 em_relaxed 0.14666666666666667 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04 0.14666666666666667 0.02898223889753507 0.14666666666666667 0.02898223889753507 0.016053601490175616
99 mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0 mmlu mmlu.mqa_single_correct_sk:high_school_psychology 2 0 536 em_relaxed 0.22388059701492538 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04477611940298507 0.22388059701492538 0.01802169384796668 0.22388059701492538 0.01802169384796668 0.008941264053122127
100 mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0 mmlu mmlu.mqa_single_correct_sk:high_school_statistics 2 0 214 em_relaxed 0.18691588785046728 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.004672897196261682 0.17757009345794392 0.02618451638732211 0.18691588785046728 0.026711670245826564 0.004672897196261678
101 mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_us_history 2 0 57 em_relaxed 0.21052631578947367 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.21052631578947367 0.05447885994808057 0.21052631578947367 0.05447885994808057 0.0
102 mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_world_history 2 0 180 em_relaxed 0.2777777777777778 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.005555555555555556 0.2777777777777778 0.03347785759938633 0.2777777777777778 0.03347785759938633 0.005555555555555554
103 mmlu.prompt2_mqa_single_correct_sk:human_aging|0 mmlu mmlu.mqa_single_correct_sk:human_aging 2 0 219 em_relaxed 0.1735159817351598 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0867579908675799 0.1735159817351598 0.02564830822412722 0.1735159817351598 0.02564830822412722 0.01906423736413682
104 mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0 mmlu mmlu.mqa_single_correct_sk:human_sexuality 2 0 109 em_relaxed 0.21100917431192662 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05504587155963303 0.21100917431192662 0.039262212380368526 0.21100917431192662 0.039262212380368526 0.0219460246091855
105 mmlu.prompt2_mqa_single_correct_sk:international_law|0 mmlu mmlu.mqa_single_correct_sk:international_law 2 0 118 em_relaxed 0.16101694915254236 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05084745762711865 0.16101694915254236 0.03397969368949715 0.16101694915254236 0.03397969368949715 0.020309989475094187
106 mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0 mmlu mmlu.mqa_single_correct_sk:jurisprudence 2 0 107 em_relaxed 0.19626168224299065 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.056074766355140186 0.19626168224299065 0.038576441428227824 0.19626168224299065 0.038576441428227824 0.02234599008672909
107 mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0 mmlu mmlu.mqa_single_correct_sk:logical_fallacies 2 0 158 em_relaxed 0.25316455696202533 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05063291139240506 0.25316455696202533 0.0347027655481443 0.25316455696202533 0.0347027655481443 0.017497810541168043
108 mmlu.prompt2_mqa_single_correct_sk:machine_learning|0 mmlu mmlu.mqa_single_correct_sk:machine_learning 2 0 112 em_relaxed 0.14285714285714285 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.026785714285714284 0.125 0.03139045014587016 0.14285714285714285 0.033213611069662675 0.015324784082343059
109 mmlu.prompt2_mqa_single_correct_sk:management|0 mmlu mmlu.mqa_single_correct_sk:management 2 0 100 em_relaxed 0.15 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04 0.15 0.035887028128263686 0.15 0.035887028128263686 0.019694638556693237
110 mmlu.prompt2_mqa_single_correct_sk:marketing|0 mmlu mmlu.mqa_single_correct_sk:marketing 2 0 234 em_relaxed 0.2094017094017094 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0811965811965812 0.20512820512820512 0.026453508054040356 0.2094017094017094 0.02665569965392276 0.017893784904018422
111 mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0 mmlu mmlu.mqa_single_correct_sk:medical_genetics 2 0 100 em_relaxed 0.18 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.07 0.18 0.03861229196653694 0.18 0.03861229196653694 0.02564323999762429
112 mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0 mmlu mmlu.mqa_single_correct_sk:miscellaneous 2 0 779 em_relaxed 0.2092426187419769 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0693196405648267 0.2092426187419769 0.014583336673745648 0.2092426187419769 0.014583336673745648 0.009106230246473392
113 mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0 mmlu mmlu.mqa_single_correct_sk:moral_disputes 2 0 326 em_relaxed 0.25766871165644173 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.08895705521472393 0.25766871165644173 0.0242598463239315 0.25766871165644173 0.0242598463239315 0.015791302137788298
114 mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0 mmlu mmlu.mqa_single_correct_sk:moral_scenarios 2 0 749 em_relaxed 0.0026702269692923898 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.0026702269692923898 0.001886873052453708 0.0026702269692923898 0.001886873052453708 0.0
115 mmlu.prompt2_mqa_single_correct_sk:nutrition|0 mmlu mmlu.mqa_single_correct_sk:nutrition 2 0 303 em_relaxed 0.19801980198019803 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0297029702970297 0.19801980198019803 0.02293151046482573 0.19801980198019803 0.02293151046482573 0.00976897099052845
116 mmlu.prompt2_mqa_single_correct_sk:philosophy|0 mmlu mmlu.mqa_single_correct_sk:philosophy 2 0 308 em_relaxed 0.2922077922077922 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.10064935064935066 0.2922077922077922 0.02595551182455085 0.2922077922077922 0.02595551182455085 0.017171207138792156
117 mmlu.prompt2_mqa_single_correct_sk:prehistory|0 mmlu mmlu.mqa_single_correct_sk:prehistory 2 0 319 em_relaxed 0.18808777429467086 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06583072100313479 0.18808777429467086 0.021913980163123535 0.18808777429467086 0.021913980163123535 0.013906362189890747
118 mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0 mmlu mmlu.mqa_single_correct_sk:professional_accounting 2 0 276 em_relaxed 0.15217391304347827 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.14130434782608695 0.02100540442245809 0.15217391304347827 0.02165993479655374 0.0
119 mmlu.prompt2_mqa_single_correct_sk:professional_law|0 mmlu mmlu.mqa_single_correct_sk:professional_law 2 0 1171 em_relaxed 0.14175918018787362 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.004269854824935952 0.14175918018787362 0.01019734819812428 0.14175918018787362 0.01019734819812428 0.001906270167826036
120 mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0 mmlu mmlu.mqa_single_correct_sk:professional_medicine 2 0 264 em_relaxed 0.19696969696969696 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.003787878787878788 0.19696969696969696 0.024523802005628457 0.19696969696969696 0.024523802005628457 0.003787878787878786
121 mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0 mmlu mmlu.mqa_single_correct_sk:professional_psychology 2 0 596 em_relaxed 0.21308724832214765 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.050335570469798654 0.21308724832214765 0.01678740388759121 0.21308724832214765 0.01678740388759121 0.008963220911107225
122 mmlu.prompt2_mqa_single_correct_sk:public_relations|0 mmlu mmlu.mqa_single_correct_sk:public_relations 2 0 109 em_relaxed 0.23853211009174313 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.09174311926605505 0.23853211009174313 0.041009771029340336 0.23853211009174313 0.041009771029340336 0.02777660875318803
123 mmlu.prompt2_mqa_single_correct_sk:security_studies|0 mmlu mmlu.mqa_single_correct_sk:security_studies 2 0 222 em_relaxed 0.21621621621621623 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04504504504504504 0.21621621621621623 0.02769147769493752 0.21621621621621623 0.02769147769493752 0.013951433044338322
124 mmlu.prompt2_mqa_single_correct_sk:sociology|0 mmlu mmlu.mqa_single_correct_sk:sociology 2 0 197 em_relaxed 0.3553299492385787 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.07106598984771574 0.3553299492385787 0.034186655739878676 0.3553299492385787 0.034186655739878676 0.018352502878671184
125 mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0 mmlu mmlu.mqa_single_correct_sk:us_foreign_policy 2 0 99 em_relaxed 0.25252525252525254 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.030303030303030304 0.24242424242424243 0.04329004329004326 0.25252525252525254 0.043887175543277195 0.01731601731601731
126 mmlu.prompt2_mqa_single_correct_sk:virology|0 mmlu mmlu.mqa_single_correct_sk:virology 2 0 164 em_relaxed 0.16463414634146342 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03048780487804878 0.16463414634146342 0.029047216079384597 0.16463414634146342 0.029047216079384597 0.013466226472535378
127 mmlu.prompt2_mqa_single_correct_sk:world_religions|0 mmlu mmlu.mqa_single_correct_sk:world_religions 2 0 171 em_relaxed 0.0935672514619883 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04678362573099415 0.0935672514619883 0.022335993231163274 0.0935672514619883 0.022335993231163274 0.016196389956235627
128 mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0 mmlu mmlu.mqa_single_correct_sk:abstract_algebra 3 0 100 em_relaxed 0.06 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.06 0.023868325657594173 0.06 0.023868325657594173 0.0
129 mmlu.prompt3_mqa_single_correct_sk:anatomy|0 mmlu mmlu.mqa_single_correct_sk:anatomy 3 0 134 em_relaxed 0.13432835820895522 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.029850746268656716 0.13432835820895522 0.029568863219678807 0.13432835820895522 0.029568863219678807 0.01475608176283378
130 mmlu.prompt3_mqa_single_correct_sk:astronomy|0 mmlu mmlu.mqa_single_correct_sk:astronomy 3 0 152 em_relaxed 0.1513157894736842 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.019736842105263157 0.14473684210526316 0.028631951845930412 0.1513157894736842 0.029162631596843985 0.011319355505772411
131 mmlu.prompt3_mqa_single_correct_sk:business_ethics|0 mmlu mmlu.mqa_single_correct_sk:business_ethics 3 0 100 em_relaxed 0.13 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01 0.13 0.03379976689896308 0.13 0.03379976689896308 0.009999999999999998
132 mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0 mmlu mmlu.mqa_single_correct_sk:clinical_knowledge 3 0 259 em_relaxed 0.1969111969111969 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03088803088803089 0.1969111969111969 0.024757519994692216 0.1969111969111969 0.024757519994692216 0.010771402065873048
133 mmlu.prompt3_mqa_single_correct_sk:college_biology|0 mmlu mmlu.mqa_single_correct_sk:college_biology 3 0 142 em_relaxed 0.176056338028169 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.035211267605633804 0.176056338028169 0.03207487957537854 0.176056338028169 0.03207487957537854 0.015521989489917001
134 mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0 mmlu mmlu.mqa_single_correct_sk:college_chemistry 3 0 100 em_relaxed 0.16 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04 0.16 0.03684529491774709 0.16 0.03684529491774709 0.01969463855669323
135 mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0 mmlu mmlu.mqa_single_correct_sk:college_computer_science 3 0 100 em_relaxed 0.15 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03 0.11 0.031446603773522035 0.15 0.035887028128263714 0.01714466079977653
136 mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0 mmlu mmlu.mqa_single_correct_sk:college_mathematics 3 0 99 em_relaxed 0.09090909090909091 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.08080808080808081 0.027530712883605948 0.09090909090909091 0.02903984386363362 0.0
137 mmlu.prompt3_mqa_single_correct_sk:college_medicine|0 mmlu mmlu.mqa_single_correct_sk:college_medicine 3 0 172 em_relaxed 0.1569767441860465 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.040697674418604654 0.1569767441860465 0.027818863343493188 0.1569767441860465 0.027818863343493188 0.01511000089316197
138 mmlu.prompt3_mqa_single_correct_sk:college_physics|0 mmlu mmlu.mqa_single_correct_sk:college_physics 3 0 102 em_relaxed 0.12745098039215685 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.12745098039215685 0.03318224921942076 0.12745098039215685 0.03318224921942076 0.0
139 mmlu.prompt3_mqa_single_correct_sk:computer_security|0 mmlu mmlu.mqa_single_correct_sk:computer_security 3 0 100 em_relaxed 0.13 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02 0.13 0.03379976689896309 0.13 0.03379976689896309 0.014070529413628963
140 mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0 mmlu mmlu.mqa_single_correct_sk:conceptual_physics 3 0 235 em_relaxed 0.1276595744680851 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05106382978723404 0.1276595744680851 0.021815328994143183 0.1276595744680851 0.021815328994143183 0.014390214367756187
141 mmlu.prompt3_mqa_single_correct_sk:econometrics|0 mmlu mmlu.mqa_single_correct_sk:econometrics 3 0 112 em_relaxed 0.10714285714285714 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.017857142857142856 0.09821428571428571 0.02824733740077989 0.10714285714285714 0.02935696201881377 0.012569900164490043
142 mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0 mmlu mmlu.mqa_single_correct_sk:electrical_engineering 3 0 145 em_relaxed 0.12413793103448276 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.041379310344827586 0.12413793103448276 0.0274782369836366 0.12413793103448276 0.0274782369836366 0.016597159859992713
143 mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0 mmlu mmlu.mqa_single_correct_sk:elementary_mathematics 3 0 373 em_relaxed 0.08847184986595175 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.08579088471849866 0.01452017587959949 0.08847184986595175 0.014723671750091158 0.0
144 mmlu.prompt3_mqa_single_correct_sk:formal_logic|0 mmlu mmlu.mqa_single_correct_sk:formal_logic 3 0 115 em_relaxed 0.11304347826086956 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.06956521739130435 0.02382794649967283 0.11304347826086956 0.029656609316837397 0.0
145 mmlu.prompt3_mqa_single_correct_sk:global_facts|0 mmlu mmlu.mqa_single_correct_sk:global_facts 3 0 100 em_relaxed 0.12 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01 0.12 0.03265986323710906 0.12 0.03265986323710906 0.009999999999999997
146 mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0 mmlu mmlu.mqa_single_correct_sk:high_school_biology 3 0 282 em_relaxed 0.1950354609929078 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02127659574468085 0.1879432624113475 0.023305230769714243 0.1950354609929078 0.02363698094391562 0.00860850820920177
147 mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0 mmlu mmlu.mqa_single_correct_sk:high_school_chemistry 3 0 198 em_relaxed 0.09090909090909091 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.005050505050505051 0.08585858585858586 0.01996022556317289 0.09090909090909091 0.020482086775424218 0.00505050505050505
148 mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0 mmlu mmlu.mqa_single_correct_sk:high_school_computer_science 3 0 99 em_relaxed 0.13131313131313133 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.010101010101010102 0.12121212121212122 0.03296871474399963 0.13131313131313133 0.034117144062155554 0.010101010101010109
149 mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_european_history 3 0 135 em_relaxed 0.21481481481481482 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02962962962962963 0.2074074074074074 0.03502553170678318 0.21481481481481482 0.03547854198560824 0.014648038602753806
150 mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0 mmlu mmlu.mqa_single_correct_sk:high_school_geography 3 0 198 em_relaxed 0.20202020202020202 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.07575757575757576 0.20202020202020202 0.028606204289229872 0.20202020202020202 0.028606204289229872 0.018852670234993107
151 mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0 mmlu mmlu.mqa_single_correct_sk:high_school_government_and_politics 3 0 193 em_relaxed 0.30569948186528495 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.046632124352331605 0.3005181347150259 0.0330881859441575 0.30569948186528495 0.033248379397581594 0.015216761819262604
152 mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0 mmlu mmlu.mqa_single_correct_sk:high_school_macroeconomics 3 0 387 em_relaxed 0.15762273901808785 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01808785529715762 0.14987080103359174 0.018168008667456638 0.15762273901808785 0.018546804758294254 0.006783224657656192
153 mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0 mmlu mmlu.mqa_single_correct_sk:high_school_mathematics 3 0 266 em_relaxed 0.05639097744360902 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.05639097744360902 0.014170263190664145 0.05639097744360902 0.014170263190664145 0.0
154 mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0 mmlu mmlu.mqa_single_correct_sk:high_school_microeconomics 3 0 217 em_relaxed 0.17050691244239632 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.013824884792626729 0.16589861751152074 0.025310686717794278 0.17050691244239632 0.025588834483805927 0.007944762237164815
155 mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0 mmlu mmlu.mqa_single_correct_sk:high_school_physics 3 0 150 em_relaxed 0.17333333333333334 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.02666666666666667 0.17333333333333334 0.031010810941145068 0.17333333333333334 0.031010810941145068 0.013198422611557013
156 mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0 mmlu mmlu.mqa_single_correct_sk:high_school_psychology 3 0 536 em_relaxed 0.18470149253731344 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.03731343283582089 0.18470149253731344 0.016777090030491517 0.18470149253731344 0.016777090030491517 0.008194041665419557
157 mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0 mmlu mmlu.mqa_single_correct_sk:high_school_statistics 3 0 214 em_relaxed 0.16355140186915887 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.1542056074766355 0.024745297661074854 0.16355140186915887 0.025342938086817392 0.0
158 mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_us_history 3 0 57 em_relaxed 0.12280701754385964 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.12280701754385964 0.04385964912280699 0.12280701754385964 0.04385964912280699 0.0
159 mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0 mmlu mmlu.mqa_single_correct_sk:high_school_world_history 3 0 180 em_relaxed 0.26666666666666666 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.005555555555555556 0.26666666666666666 0.03305282343736876 0.26666666666666666 0.03305282343736876 0.005555555555555554
160 mmlu.prompt3_mqa_single_correct_sk:human_aging|0 mmlu mmlu.mqa_single_correct_sk:human_aging 3 0 219 em_relaxed 0.1278538812785388 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0684931506849315 0.1278538812785388 0.02261639276396452 0.1278538812785388 0.02261639276396452 0.017107571020219335
161 mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0 mmlu mmlu.mqa_single_correct_sk:human_sexuality 3 0 109 em_relaxed 0.1559633027522936 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05504587155963303 0.1559633027522936 0.03491242362801975 0.1559633027522936 0.03491242362801975 0.021946024609185506
162 mmlu.prompt3_mqa_single_correct_sk:international_law|0 mmlu mmlu.mqa_single_correct_sk:international_law 3 0 118 em_relaxed 0.15254237288135594 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0423728813559322 0.15254237288135594 0.03324002290477697 0.15254237288135594 0.03324002290477697 0.018622984668462274
163 mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0 mmlu mmlu.mqa_single_correct_sk:jurisprudence 3 0 107 em_relaxed 0.18691588785046728 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.037383177570093455 0.18691588785046728 0.03786499587281739 0.18691588785046728 0.03786499587281739 0.01842518672106723
164 mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0 mmlu mmlu.mqa_single_correct_sk:logical_fallacies 3 0 158 em_relaxed 0.17721518987341772 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0379746835443038 0.17721518987341772 0.030475004517281346 0.17721518987341772 0.030475004517281346 0.015254237576434418
165 mmlu.prompt3_mqa_single_correct_sk:machine_learning|0 mmlu mmlu.mqa_single_correct_sk:machine_learning 3 0 112 em_relaxed 0.125 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.008928571428571428 0.11607142857142858 0.03040253249894973 0.125 0.03139045014587016 0.008928571428571423
166 mmlu.prompt3_mqa_single_correct_sk:management|0 mmlu mmlu.mqa_single_correct_sk:management 3 0 100 em_relaxed 0.2 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06 0.2 0.040201512610368445 0.2 0.040201512610368445 0.023868325657594162
167 mmlu.prompt3_mqa_single_correct_sk:marketing|0 mmlu mmlu.mqa_single_correct_sk:marketing 3 0 234 em_relaxed 0.1623931623931624 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05555555555555555 0.1623931623931624 0.02416161812798774 0.1623931623931624 0.02416161812798774 0.01500631280644691
168 mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0 mmlu mmlu.mqa_single_correct_sk:medical_genetics 3 0 100 em_relaxed 0.14 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.01 0.14 0.034873508801977704 0.14 0.034873508801977704 0.009999999999999997
169 mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0 mmlu mmlu.mqa_single_correct_sk:miscellaneous 3 0 779 em_relaxed 0.17843388960205392 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.055198973042362 0.17715019255455713 0.013688041138952063 0.17843388960205392 0.013726826073077934 0.00818739755268085
170 mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0 mmlu mmlu.mqa_single_correct_sk:moral_disputes 3 0 326 em_relaxed 0.2147239263803681 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.06748466257668712 0.2116564417177914 0.02265853497858724 0.2147239263803681 0.022777692313730433 0.013915180741090785
171 mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0 mmlu mmlu.mqa_single_correct_sk:moral_scenarios 3 0 749 em_relaxed 0.01335113484646195 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.0053404539385847796 0.0026648668562206297 0.01335113484646195 0.004196522950335422 0.0
172 mmlu.prompt3_mqa_single_correct_sk:nutrition|0 mmlu mmlu.mqa_single_correct_sk:nutrition 3 0 303 em_relaxed 0.18811881188118812 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.036303630363036306 0.18811881188118812 0.022488416775417838 0.18811881188118812 0.022488416775417838 0.010763206393659715
173 mmlu.prompt3_mqa_single_correct_sk:philosophy|0 mmlu mmlu.mqa_single_correct_sk:philosophy 3 0 308 em_relaxed 0.237012987012987 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05194805194805195 0.237012987012987 0.024270317456654856 0.237012987012987 0.024270317456654856 0.01266576753511598
174 mmlu.prompt3_mqa_single_correct_sk:prehistory|0 mmlu mmlu.mqa_single_correct_sk:prehistory 3 0 319 em_relaxed 0.15047021943573669 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04075235109717868 0.14733542319749215 0.019876006858615635 0.15047021943573669 0.020049383410796876 0.011087359931149772
175 mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0 mmlu mmlu.mqa_single_correct_sk:professional_accounting 3 0 276 em_relaxed 0.14492753623188406 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.13043478260869565 0.020308668108561446 0.14492753623188406 0.02122807279169736 0.0
176 mmlu.prompt3_mqa_single_correct_sk:professional_law|0 mmlu mmlu.mqa_single_correct_sk:professional_law 3 0 1171 em_relaxed 0.1340734415029889 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0017079419299743809 0.12809564474807855 0.009770317447740377 0.1340734415029889 0.009961367847349073 0.0012071811002646103
177 mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0 mmlu mmlu.mqa_single_correct_sk:professional_medicine 3 0 264 em_relaxed 0.09090909090909091 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.0 0.09090909090909091 0.01772676309015986 0.09090909090909091 0.01772676309015986 0.0
178 mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0 mmlu mmlu.mqa_single_correct_sk:professional_psychology 3 0 596 em_relaxed 0.19630872483221476 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.018456375838926176 0.19630872483221476 0.016283806947934177 0.19630872483221476 0.016283806947934177 0.005517845461546564
179 mmlu.prompt3_mqa_single_correct_sk:public_relations|0 mmlu mmlu.mqa_single_correct_sk:public_relations 3 0 109 em_relaxed 0.21100917431192662 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.07339449541284404 0.21100917431192662 0.03926221238036853 0.21100917431192662 0.03926221238036853 0.025093850407933094
180 mmlu.prompt3_mqa_single_correct_sk:security_studies|0 mmlu mmlu.mqa_single_correct_sk:security_studies 3 0 222 em_relaxed 0.15765765765765766 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04954954954954955 0.15765765765765766 0.024513522045494097 0.15765765765765766 0.024513522045494097 0.014597835283794243
181 mmlu.prompt3_mqa_single_correct_sk:sociology|0 mmlu mmlu.mqa_single_correct_sk:sociology 3 0 197 em_relaxed 0.3350253807106599 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.05583756345177665 0.3350253807106599 0.03371423842235921 0.3350253807106599 0.03371423842235921 0.016400555297895067
182 mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0 mmlu mmlu.mqa_single_correct_sk:us_foreign_policy 3 0 99 em_relaxed 0.2727272727272727 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.050505050505050504 0.26262626262626265 0.044452876769839444 0.2727272727272727 0.044988332664126655 0.022120793242071984
183 mmlu.prompt3_mqa_single_correct_sk:virology|0 mmlu mmlu.mqa_single_correct_sk:virology 3 0 164 em_relaxed 0.1402439024390244 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.024390243902439025 0.1402439024390244 0.027197928271392544 0.1402439024390244 0.027197928271392544 0.012082375722391943
184 mmlu.prompt3_mqa_single_correct_sk:world_religions|0 mmlu mmlu.mqa_single_correct_sk:world_religions 3 0 171 em_relaxed 0.10526315789473684 32 test lighteval/okapi_mmlu sk b3e738ae24d5d96c4b425d4893f377451196cd2e 0.04678362573099415 0.10526315789473684 0.02353755765789256 0.10526315789473684 0.02353755765789256 0.01619638995623562
185 movie_review.prompt1_sentiment_binary_sk|0 movie_review movie_review.sentiment_binary_sk 1 0 2008 em_norm_relaxed 0.6329681274900398 32 test mteb/slovak_movie_review_sentiment default 29a7405aabcfd4860a51ae6f65a5650d63108f26 0.6324701195219123 0.010761991548945318 0.38745019920318724 0.6329681274900398 0.010758931054934645 0.010874391376667836
186 movie_review.prompt2_sentiment_binary_sk|0 movie_review movie_review.sentiment_binary_sk 2 0 2008 em_norm_relaxed 0.48306772908366535 32 test mteb/slovak_movie_review_sentiment default 29a7405aabcfd4860a51ae6f65a5650d63108f26 0.47758964143426297 0.011149609394535145 0.01045816733067729 0.48306772908366535 0.011154424009707972 0.002270759742696006
187 movie_review.prompt3_sentiment_binary_sk|0 movie_review movie_review.sentiment_binary_sk 3 0 2008 em_norm_relaxed 0.11105577689243028 32 test mteb/slovak_movie_review_sentiment default 29a7405aabcfd4860a51ae6f65a5650d63108f26 0.09412350597609562 0.006517932474820501 0.04731075697211155 0.11105577689243028 0.007013489433691037 0.004738947450976647
188 multi_wiki_qa.prompt1_qa_sk|0 multi_wiki_qa multi_wiki_qa.qa_sk 1 0 4997 exact_match_slk_prefix 0.36401841104662797 126 train alexandrainst/multi-wiki-qa sk 9bdb3350487e399bd1cd873e1395290d636db5ce 0.36401841104662797 0.00680726616133214 0.5156574509584114 0.0058319674487835125
189 multi_wiki_qa.prompt2_qa_sk|0 multi_wiki_qa multi_wiki_qa.qa_sk 2 0 4997 exact_match_slk_prefix 0.36001600960576347 126 train alexandrainst/multi-wiki-qa sk 9bdb3350487e399bd1cd873e1395290d636db5ce 0.36001600960576347 0.006791008078277957 0.5090494926903776 0.005852156127470673
190 multi_wiki_qa.prompt3_qa_sk|0 multi_wiki_qa multi_wiki_qa.qa_sk 3 0 4997 exact_match_slk_prefix 0.34000400240144085 126 train alexandrainst/multi-wiki-qa sk 9bdb3350487e399bd1cd873e1395290d636db5ce 0.34000400240144085 0.006701954125609285 0.5050071786470199 0.005797838571943781
191 senti_sk.prompt1_sentiment_3_class_sk|0 senti_sk senti_sk.sentiment_3_class_sk 1 0 4677 em_norm_relaxed 0.3200769724182168 32 test TUKE-KEMT/senti-sk default 3c8a92168ff2b692c653de65c97bda573127d15d 0.31922172332691895 0.006817290681927466 0.22642719692110327 0.3200769724182168 0.0068221276139399335 0.006120369470744044
192 senti_sk.prompt1_sentiment_binary_sk|0 senti_sk senti_sk.sentiment_binary_sk 1 0 3728 em_norm_relaxed 0.31223175965665234 32 test TUKE-KEMT/senti-sk default 3c8a92168ff2b692c653de65c97bda573127d15d 0.3111587982832618 0.007583521183648285 0.1968884120171674 0.31223175965665234 0.0075906663188793026 0.006513558648330005
193 senti_sk.prompt2_sentiment_3_class_sk|0 senti_sk senti_sk.sentiment_3_class_sk 2 0 4677 em_norm_relaxed 0.383365405174257 32 test TUKE-KEMT/senti-sk default 3c8a92168ff2b692c653de65c97bda573127d15d 0.37716484926234767 0.007087856422819165 0.019029292281376953 0.383365405174257 0.007110221850748201 0.001998030377272694
194 senti_sk.prompt2_sentiment_binary_sk|0 senti_sk senti_sk.sentiment_binary_sk 2 0 3728 em_norm_relaxed 0.4965128755364807 32 test TUKE-KEMT/senti-sk default 3c8a92168ff2b692c653de65c97bda573127d15d 0.49302575107296137 0.008189324033951761 0.0037553648068669528 0.4965128755364807 0.008189921623320459 0.001001911513818721
195 senti_sk.prompt3_sentiment_3_class_sk|0 senti_sk senti_sk.sentiment_3_class_sk 3 0 4677 em_norm_relaxed 0.40880906564036773 32 test TUKE-KEMT/senti-sk default 3c8a92168ff2b692c653de65c97bda573127d15d 0.26448578148385715 0.006449996068936491 0.10540945050245884 0.40880906564036773 0.0071893047031815266 0.004490706548553942
196 senti_sk.prompt3_sentiment_binary_sk|0 senti_sk senti_sk.sentiment_binary_sk 3 0 3728 em_norm_relaxed 0.25160944206008584 32 test TUKE-KEMT/senti-sk default 3c8a92168ff2b692c653de65c97bda573127d15d 0.1990343347639485 0.0065402033880141534 0.03165236051502146 0.25160944206008584 0.0071080082101389235 0.0028677349189843607
197 skj1.prompt1_skj1_sk|0 skj1 skj1.skj1_sk 1 0 100 em_relaxed 0.02 32 train slovak-nlp/SKJ1 default e615c60b415389b629703f82d3a79810dbde6734 0.0 0.02 0.014070529413628957 0.02 0.014070529413628957 0.0
198 skj1.prompt2_skj1_sk|0 skj1 skj1.skj1_sk 2 0 100 em_relaxed 0.06 32 train slovak-nlp/SKJ1 default e615c60b415389b629703f82d3a79810dbde6734 0.0 0.05 0.021904291355759067 0.06 0.023868325657594166 0.0
199 skj1.prompt3_skj1_sk|0 skj1 skj1.skj1_sk 3 0 100 em_relaxed 0.05 32 train slovak-nlp/SKJ1 default e615c60b415389b629703f82d3a79810dbde6734 0.0 0.03 0.017144660799776543 0.05 0.021904291355759067 0.0
200 sklep.prompt1_hate_speech_sk:hate-speech|0 sklep sklep.hate_speech_sk:hate-speech 1 0 1319 em_norm_relaxed 0.6921910538286581 32 test slovak-nlp/sklep hate-speech 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.6921910538286581 0.012714401009923647 0.0 0.6921910538286581 0.012714401009923647 0.0
201 sklep.prompt1_nli_3_classes_sk:nli|0 sklep sklep.nli_3_classes_sk:nli 1 0 5004 em_norm_relaxed 0.016786570743405275 32 test slovak-nlp/sklep nli 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.016386890487609912 0.0017949203375471519 0.0003996802557953637 0.016786570743405275 0.0018163086111764766 0.00028258837305266234
202 sklep.prompt1_nli_binary_sk:rte|0 sklep sklep.nli_binary_sk:rte 1 0 1660 em_norm_relaxed 0.030120481927710843 32 test slovak-nlp/sklep rte 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.030120481927710843 0.004196301257202765 0.0006024096385542169 0.030120481927710843 0.004196301257202765 0.0006024096385542043
203 sklep.prompt1_qa_sk:question-answering|0 sklep sklep.qa_sk:question-answering 1 0 9583 exact_match_slk_prefix 0.16456224564332672 126 test slovak-nlp/sklep question-answering 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.16456224564332672 0.003787861300566259 0.3730151221096053 0.003855179221962441
204 sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0 sklep sklep.sentiment_binary_sk:sentiment-analysis 1 0 1042 em_norm_relaxed 0.8809980806142035 32 test slovak-nlp/sklep sentiment-analysis 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.8800383877159309 0.010070396971355563 0.44817658349328215 0.8809980806142035 0.010035501945449067 0.015413429449614385
205 sklep.prompt2_hate_speech_sk:hate-speech|0 sklep sklep.hate_speech_sk:hate-speech 2 0 1319 em_norm_relaxed 0.7247915087187263 32 test slovak-nlp/sklep hate-speech 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.7247915087187263 0.012302114305862654 0.0 0.7247915087187263 0.012302114305862654 0.0
206 sklep.prompt2_nli_3_classes_sk:nli|0 sklep sklep.nli_3_classes_sk:nli 2 0 5004 em_norm_relaxed 0.09972022382094324 32 test slovak-nlp/sklep nli 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.09952038369304557 0.004232312358957502 0.0 0.09972022382094324 0.004236089409386789 0.0
207 sklep.prompt2_nli_binary_sk:rte|0 sklep sklep.nli_binary_sk:rte 2 0 1660 em_norm_relaxed 0.3301204819277108 32 test slovak-nlp/sklep rte 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.3301204819277108 0.011545464009906971 0.0018072289156626507 0.3301204819277108 0.011545464009906971 0.001042774975713993
208 sklep.prompt2_qa_sk:question-answering|0 sklep sklep.qa_sk:question-answering 2 0 9583 exact_match_slk_prefix 0.16518835437754356 126 test slovak-nlp/sklep question-answering 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.16518835437754356 0.003793637935655721 0.3751701402109962 0.0038611217329234445
209 sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0 sklep sklep.sentiment_binary_sk:sentiment-analysis 2 0 1042 em_norm_relaxed 0.3512476007677543 32 test slovak-nlp/sklep sentiment-analysis 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.34836852207293667 0.014767096602802959 0.0009596928982725527 0.3512476007677543 0.014795198864354053 0.0009596928982725573
210 sklep.prompt3_hate_speech_sk:hate-speech|0 sklep sklep.hate_speech_sk:hate-speech 3 0 1319 em_norm_relaxed 0.6967399545109931 32 test slovak-nlp/sklep hate-speech 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.6967399545109931 0.012661502663418697 0.0 0.6967399545109931 0.012661502663418697 0.0
211 sklep.prompt3_nli_3_classes_sk:nli|0 sklep sklep.nli_3_classes_sk:nli 3 0 5004 em_norm_relaxed 0.001398880895283773 32 test slovak-nlp/sklep nli 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.001398880895283773 0.0005284101391342323 0.0 0.001398880895283773 0.0005284101391342323 0.0
212 sklep.prompt3_nli_binary_sk:rte|0 sklep sklep.nli_binary_sk:rte 3 0 1660 em_norm_relaxed 0.2319277108433735 32 test slovak-nlp/sklep rte 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.2319277108433735 0.010362252629508847 0.0012048192771084338 0.2319277108433735 0.010362252629508847 0.0008516790803748009
213 sklep.prompt3_qa_sk:question-answering|0 sklep sklep.qa_sk:question-answering 3 0 9583 exact_match_slk_prefix 0.16143170197224252 126 test slovak-nlp/sklep question-answering 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.16143170197224252 0.0037586816998114083 0.3769392643579436 0.003773157275405368
214 sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0 sklep sklep.sentiment_binary_sk:sentiment-analysis 3 0 1042 em_norm_relaxed 0.15355086372360843 32 test slovak-nlp/sklep sentiment-analysis 10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9 0.118042226487524 0.010000396690181516 0.015355086372360844 0.15355086372360843 0.011173807886451477 0.0038110143844746576
215 slovakcopa.prompt1_copa_sk|0 slovakcopa slovakcopa.copa_sk 1 0 500 em_relaxed 0.618 32 test NaiveNeuron/SlovakCOPA default 5c61e4cd9343f9b4fab623bdc50bb895cc170771 0.0 0.508 0.022380208834928028 0.618 0.021750820591250834 0.0
216 slovakcopa.prompt2_copa_sk|0 slovakcopa slovakcopa.copa_sk 2 0 500 em_relaxed 0.51 32 test NaiveNeuron/SlovakCOPA default 5c61e4cd9343f9b4fab623bdc50bb895cc170771 0.0 0.326 0.020984009562393567 0.51 0.02237859698923078 0.0
217 slovakcopa.prompt3_copa_sk|0 slovakcopa slovakcopa.copa_sk 3 0 500 em_relaxed 0.086 32 test NaiveNeuron/SlovakCOPA default 5c61e4cd9343f9b4fab623bdc50bb895cc170771 0.0 0.084 0.01241758401564368 0.086 0.01255081854271602 0.0
218 toxic_sk.prompt1_toxicity_sk|0 toxic_sk toxic_sk.toxicity_sk 1 0 881 em_norm_relaxed 0.45629965947786605 32 test TUKE-KEMT/toxic-sk default 2fb416e18785c128e7ec934c07a14c803a88cdb3 0.45629965947786605 0.016790496508382532 0.0 0.45629965947786605 0.016790496508382532 0.0
219 toxic_sk.prompt2_toxicity_sk|0 toxic_sk toxic_sk.toxicity_sk 2 0 881 em_norm_relaxed 0.47559591373439275 32 test TUKE-KEMT/toxic-sk default 2fb416e18785c128e7ec934c07a14c803a88cdb3 0.47559591373439275 0.016834908286515442 0.0 0.47559591373439275 0.016834908286515442 0.0
220 toxic_sk.prompt3_toxicity_sk|0 toxic_sk toxic_sk.toxicity_sk 3 0 881 em_norm_relaxed 0.4960272417707151 32 test TUKE-KEMT/toxic-sk default 2fb416e18785c128e7ec934c07a14c803a88cdb3 0.4960272417707151 0.016854464514836336 0.007945516458569807 0.4960272417707151 0.016854464514836336 0.00299286751118699
221 truthfulqa.prompt1_mqa_single_correct_sk:mc1|0 truthfulqa truthfulqa.mqa_single_correct_sk:mc1 1 0 778 em_relaxed 0.007712082262210797 32 val alexandrainst/m_truthfulqa sk f0445d470f1925882b990f5f247fdcf288972f60 0.0 0.007712082262210797 0.003138297919968842 0.007712082262210797 0.003138297919968842 0.0
222 truthfulqa.prompt2_mqa_single_correct_sk:mc1|0 truthfulqa truthfulqa.mqa_single_correct_sk:mc1 2 0 778 em_relaxed 0.016709511568123392 32 val alexandrainst/m_truthfulqa sk f0445d470f1925882b990f5f247fdcf288972f60 0.0 0.016709511568123392 0.004598458668423289 0.016709511568123392 0.004598458668423289 0.0
223 truthfulqa.prompt3_mqa_single_correct_sk:mc1|0 truthfulqa truthfulqa.mqa_single_correct_sk:mc1 3 0 778 em_relaxed 0.011568123393316195 32 val alexandrainst/m_truthfulqa sk f0445d470f1925882b990f5f247fdcf288972f60 0.0 0.011568123393316195 0.00383613885044431 0.011568123393316195 0.00383613885044431 0.0

18023
aya_300k_lr_experiments/benchmarks/skbench/results/results_2026-10-22T21-55-04.268099.json Normal file
View File

File diff suppressed because it is too large Load Diff

221
aya_300k_lr_experiments/benchmarks/skbench/results/three_prompt_summary.csv Normal file
View File

@ -0,0 +1,221 @@
family,task_group_without_prompt,metric,task_variants,prompt_count,evaluated_instances,task_macro_mean,sample_weighted_mean,min_task_value,max_task_value
arc,arc.mqa_single_correct_sk,em,3,3,3507,0.023666951810664385,0.023666951810664385,0.013686911890504704,0.03507271171941831
arc,arc.mqa_single_correct_sk,em_conservative,3,3,3507,0.19019104647847163,0.19019104647847163,0.1710863986313088,0.223267750213858
arc,arc.mqa_single_correct_sk,em_relaxed,3,3,3507,0.19076133447390933,0.19076133447390933,0.1710863986313088,0.22412318220701455
belebele,belebele.mqa_single_correct_sk,em,3,3,2700,0.022222222222222223,0.022222222222222223,0.016666666666666666,0.03
belebele,belebele.mqa_single_correct_sk,em_conservative,3,3,2700,0.21259259259259258,0.21259259259259258,0.2111111111111111,0.21555555555555556
belebele,belebele.mqa_single_correct_sk,em_relaxed,3,3,2700,0.21333333333333335,0.21333333333333335,0.2111111111111111,0.21555555555555556
financial_exams,financial_exams.completion_sk,em,3,3,4002,0.029735132433783108,0.029735132433783108,0.0037481259370314842,0.07571214392803598
financial_exams,financial_exams.completion_sk,em_conservative,3,3,4002,0.21839080459770116,0.21839080459770116,0.20089955022488756,0.24887556221889057
financial_exams,financial_exams.completion_sk,em_relaxed,3,3,4002,0.22938530734632684,0.22938530734632684,0.20089955022488756,0.24887556221889057
globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,em,3,3,1200,0.028333333333333335,0.028333333333333332,0.025,0.035
globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,em_conservative,3,3,1200,0.19833333333333333,0.19833333333333333,0.1825,0.21
globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,em_relaxed,3,3,1200,0.19833333333333333,0.19833333333333333,0.1825,0.21
mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,em,3,3,300,0.0033333333333333335,0.0033333333333333335,0.0,0.01
mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,em_conservative,3,3,300,0.03666666666666667,0.03666666666666667,0.01,0.06
mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,em_relaxed,3,3,300,0.04,0.04,0.01,0.06
mmlu,mmlu.mqa_single_correct_sk:anatomy,em,3,3,402,0.03731343283582089,0.03731343283582089,0.022388059701492536,0.05970149253731343
mmlu,mmlu.mqa_single_correct_sk:anatomy,em_conservative,3,3,402,0.14676616915422885,0.14676616915422885,0.12686567164179105,0.1791044776119403
mmlu,mmlu.mqa_single_correct_sk:anatomy,em_relaxed,3,3,402,0.14676616915422885,0.14676616915422885,0.12686567164179105,0.1791044776119403
mmlu,mmlu.mqa_single_correct_sk:astronomy,em,3,3,456,0.03508771929824561,0.03508771929824561,0.019736842105263157,0.05263157894736842
mmlu,mmlu.mqa_single_correct_sk:astronomy,em_conservative,3,3,456,0.1513157894736842,0.1513157894736842,0.13815789473684212,0.17105263157894737
mmlu,mmlu.mqa_single_correct_sk:astronomy,em_relaxed,3,3,456,0.15789473684210525,0.15789473684210525,0.14473684210526316,0.17763157894736842
mmlu,mmlu.mqa_single_correct_sk:business_ethics,em,3,3,300,0.02,0.02,0.01,0.03
mmlu,mmlu.mqa_single_correct_sk:business_ethics,em_conservative,3,3,300,0.16,0.16,0.13,0.21
mmlu,mmlu.mqa_single_correct_sk:business_ethics,em_relaxed,3,3,300,0.16333333333333333,0.16333333333333333,0.13,0.22
mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,em,3,3,777,0.029601029601029602,0.029601029601029602,0.011583011583011582,0.04633204633204633
mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,em_conservative,3,3,777,0.1969111969111969,0.1969111969111969,0.15444015444015444,0.23938223938223938
mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,em_relaxed,3,3,777,0.1981981981981982,0.1981981981981982,0.15444015444015444,0.24324324324324326
mmlu,mmlu.mqa_single_correct_sk:college_biology,em,3,3,426,0.035211267605633804,0.035211267605633804,0.028169014084507043,0.04225352112676056
mmlu,mmlu.mqa_single_correct_sk:college_biology,em_conservative,3,3,426,0.1737089201877934,0.17370892018779344,0.15492957746478872,0.19014084507042253
mmlu,mmlu.mqa_single_correct_sk:college_biology,em_relaxed,3,3,426,0.1737089201877934,0.17370892018779344,0.15492957746478872,0.19014084507042253
mmlu,mmlu.mqa_single_correct_sk:college_chemistry,em,3,3,300,0.056666666666666664,0.056666666666666664,0.04,0.09
mmlu,mmlu.mqa_single_correct_sk:college_chemistry,em_conservative,3,3,300,0.12666666666666668,0.12666666666666668,0.08,0.16
mmlu,mmlu.mqa_single_correct_sk:college_chemistry,em_relaxed,3,3,300,0.12666666666666668,0.12666666666666668,0.08,0.16
mmlu,mmlu.mqa_single_correct_sk:college_computer_science,em,3,3,300,0.016666666666666666,0.016666666666666666,0.01,0.03
mmlu,mmlu.mqa_single_correct_sk:college_computer_science,em_conservative,3,3,300,0.11333333333333333,0.11333333333333333,0.11,0.12
mmlu,mmlu.mqa_single_correct_sk:college_computer_science,em_relaxed,3,3,300,0.15666666666666668,0.15666666666666668,0.15,0.17
mmlu,mmlu.mqa_single_correct_sk:college_mathematics,em,3,3,297,0.0,0.0,0.0,0.0
mmlu,mmlu.mqa_single_correct_sk:college_mathematics,em_conservative,3,3,297,0.05387205387205388,0.05387205387205387,0.030303030303030304,0.08080808080808081
mmlu,mmlu.mqa_single_correct_sk:college_mathematics,em_relaxed,3,3,297,0.06397306397306397,0.06397306397306397,0.04040404040404041,0.09090909090909091
mmlu,mmlu.mqa_single_correct_sk:college_medicine,em,3,3,516,0.03875968992248062,0.03875968992248062,0.011627906976744186,0.06395348837209303
mmlu,mmlu.mqa_single_correct_sk:college_medicine,em_conservative,3,3,516,0.17248062015503876,0.17248062015503876,0.1511627906976744,0.20930232558139536
mmlu,mmlu.mqa_single_correct_sk:college_medicine,em_relaxed,3,3,516,0.17248062015503876,0.17248062015503876,0.1511627906976744,0.20930232558139536
mmlu,mmlu.mqa_single_correct_sk:college_physics,em,3,3,306,0.016339869281045753,0.016339869281045753,0.0,0.0392156862745098
mmlu,mmlu.mqa_single_correct_sk:college_physics,em_conservative,3,3,306,0.1111111111111111,0.1111111111111111,0.0784313725490196,0.12745098039215685
mmlu,mmlu.mqa_single_correct_sk:college_physics,em_relaxed,3,3,306,0.1111111111111111,0.1111111111111111,0.0784313725490196,0.12745098039215685
mmlu,mmlu.mqa_single_correct_sk:computer_security,em,3,3,300,0.02,0.02,0.02,0.02
mmlu,mmlu.mqa_single_correct_sk:computer_security,em_conservative,3,3,300,0.10666666666666667,0.10666666666666667,0.08,0.13
mmlu,mmlu.mqa_single_correct_sk:computer_security,em_relaxed,3,3,300,0.10666666666666667,0.10666666666666667,0.08,0.13
mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,em,3,3,705,0.05815602836879433,0.05815602836879433,0.03829787234042553,0.0851063829787234
mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,em_conservative,3,3,705,0.12056737588652482,0.12056737588652482,0.09361702127659574,0.14042553191489363
mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,em_relaxed,3,3,705,0.12056737588652482,0.12056737588652482,0.09361702127659574,0.14042553191489363
mmlu,mmlu.mqa_single_correct_sk:econometrics,em,3,3,336,0.020833333333333332,0.020833333333333332,0.017857142857142856,0.026785714285714284
mmlu,mmlu.mqa_single_correct_sk:econometrics,em_conservative,3,3,336,0.1130952380952381,0.1130952380952381,0.09821428571428571,0.125
mmlu,mmlu.mqa_single_correct_sk:econometrics,em_relaxed,3,3,336,0.11607142857142858,0.11607142857142858,0.10714285714285714,0.125
mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,em,3,3,435,0.05057471264367816,0.05057471264367816,0.041379310344827586,0.06896551724137931
mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,em_conservative,3,3,435,0.11494252873563218,0.11494252873563218,0.0896551724137931,0.1310344827586207
mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,em_relaxed,3,3,435,0.11494252873563218,0.11494252873563218,0.0896551724137931,0.1310344827586207
mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,em,3,3,1119,0.0071492403932082215,0.0071492403932082215,0.0,0.0160857908847185
mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,em_conservative,3,3,1119,0.08042895442359249,0.08042895442359249,0.058981233243967826,0.09651474530831099
mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,em_relaxed,3,3,1119,0.0840035746201966,0.0840035746201966,0.06166219839142091,0.10187667560321716
mmlu,mmlu.mqa_single_correct_sk:formal_logic,em,3,3,345,0.002898550724637681,0.002898550724637681,0.0,0.008695652173913044
mmlu,mmlu.mqa_single_correct_sk:formal_logic,em_conservative,3,3,345,0.0782608695652174,0.0782608695652174,0.06956521739130435,0.08695652173913043
mmlu,mmlu.mqa_single_correct_sk:formal_logic,em_relaxed,3,3,345,0.12173913043478261,0.12173913043478261,0.11304347826086956,0.13043478260869565
mmlu,mmlu.mqa_single_correct_sk:global_facts,em,3,3,300,0.06666666666666667,0.06666666666666667,0.01,0.13
mmlu,mmlu.mqa_single_correct_sk:global_facts,em_conservative,3,3,300,0.13666666666666666,0.13666666666666666,0.11,0.18
mmlu,mmlu.mqa_single_correct_sk:global_facts,em_relaxed,3,3,300,0.13666666666666666,0.13666666666666666,0.11,0.18
mmlu,mmlu.mqa_single_correct_sk:high_school_biology,em,3,3,846,0.030732860520094562,0.030732860520094562,0.02127659574468085,0.04964539007092199
mmlu,mmlu.mqa_single_correct_sk:high_school_biology,em_conservative,3,3,846,0.20212765957446807,0.20212765957446807,0.1773049645390071,0.24113475177304963
mmlu,mmlu.mqa_single_correct_sk:high_school_biology,em_relaxed,3,3,846,0.20921985815602837,0.20921985815602837,0.18439716312056736,0.24822695035460993
mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,em,3,3,594,0.011784511784511786,0.011784511784511785,0.005050505050505051,0.020202020202020204
mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,em_conservative,3,3,594,0.09259259259259259,0.09259259259259259,0.08080808080808081,0.1111111111111111
mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,em_relaxed,3,3,594,0.09764309764309764,0.09764309764309764,0.08585858585858586,0.11616161616161616
mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,em,3,3,297,0.01346801346801347,0.013468013468013467,0.0,0.030303030303030304
mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,em_conservative,3,3,297,0.11447811447811448,0.11447811447811448,0.10101010101010101,0.12121212121212122
mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,em_relaxed,3,3,297,0.11784511784511785,0.11784511784511785,0.10101010101010101,0.13131313131313133
mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,em,3,3,405,0.022222222222222223,0.022222222222222223,0.014814814814814815,0.02962962962962963
mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,em_conservative,3,3,405,0.22962962962962963,0.22962962962962963,0.2074074074074074,0.24444444444444444
mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,em_relaxed,3,3,405,0.23209876543209876,0.23209876543209876,0.21481481481481482,0.24444444444444444
mmlu,mmlu.mqa_single_correct_sk:high_school_geography,em,3,3,594,0.058922558922558925,0.058922558922558925,0.030303030303030304,0.07575757575757576
mmlu,mmlu.mqa_single_correct_sk:high_school_geography,em_conservative,3,3,594,0.19528619528619529,0.19528619528619529,0.15656565656565657,0.22727272727272727
mmlu,mmlu.mqa_single_correct_sk:high_school_geography,em_relaxed,3,3,594,0.19528619528619529,0.19528619528619529,0.15656565656565657,0.22727272727272727
mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,em,3,3,579,0.039723661485319514,0.039723661485319514,0.025906735751295335,0.046632124352331605
mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,em_conservative,3,3,579,0.309153713298791,0.30915371329879104,0.3005181347150259,0.32124352331606215
mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,em_relaxed,3,3,579,0.31088082901554404,0.310880829015544,0.30569948186528495,0.32124352331606215
mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,em,3,3,1161,0.020671834625322995,0.020671834625323,0.012919896640826873,0.031007751937984496
mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,em_conservative,3,3,1161,0.14900947459086994,0.14900947459086994,0.12661498708010335,0.17054263565891473
mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,em_relaxed,3,3,1161,0.15331610680447888,0.1533161068044789,0.12919896640826872,0.1731266149870801
mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,em,3,3,798,0.0037593984962406013,0.0037593984962406013,0.0,0.011278195488721804
mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,em_conservative,3,3,798,0.05388471177944862,0.05388471177944862,0.03759398496240601,0.06766917293233082
mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,em_relaxed,3,3,798,0.05388471177944862,0.05388471177944862,0.03759398496240601,0.06766917293233082
mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,em,3,3,651,0.013824884792626729,0.013824884792626729,0.004608294930875576,0.02304147465437788
mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,em_conservative,3,3,651,0.16129032258064516,0.16129032258064516,0.14285714285714285,0.17511520737327188
mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,em_relaxed,3,3,651,0.16282642089093702,0.16282642089093702,0.14285714285714285,0.17511520737327188
mmlu,mmlu.mqa_single_correct_sk:high_school_physics,em,3,3,450,0.031111111111111114,0.03111111111111111,0.02666666666666667,0.04
mmlu,mmlu.mqa_single_correct_sk:high_school_physics,em_conservative,3,3,450,0.1511111111111111,0.1511111111111111,0.13333333333333333,0.17333333333333334
mmlu,mmlu.mqa_single_correct_sk:high_school_physics,em_relaxed,3,3,450,0.1511111111111111,0.1511111111111111,0.13333333333333333,0.17333333333333334
mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,em,3,3,1608,0.03296019900497512,0.03296019900497513,0.016791044776119403,0.04477611940298507
mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,em_conservative,3,3,1608,0.1921641791044776,0.1921641791044776,0.16791044776119404,0.22388059701492538
mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,em_relaxed,3,3,1608,0.1921641791044776,0.1921641791044776,0.16791044776119404,0.22388059701492538
mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,em,3,3,642,0.001557632398753894,0.0015576323987538938,0.0,0.004672897196261682
mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,em_conservative,3,3,642,0.16822429906542055,0.16822429906542055,0.1542056074766355,0.17757009345794392
mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,em_relaxed,3,3,642,0.17757009345794392,0.17757009345794392,0.16355140186915887,0.18691588785046728
mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,em,3,3,171,0.0,0.0,0.0,0.0
mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,em_conservative,3,3,171,0.17543859649122806,0.17543859649122806,0.12280701754385964,0.21052631578947367
mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,em_relaxed,3,3,171,0.17543859649122806,0.17543859649122806,0.12280701754385964,0.21052631578947367
mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,em,3,3,540,0.005555555555555556,0.005555555555555556,0.005555555555555556,0.005555555555555556
mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,em_conservative,3,3,540,0.2740740740740741,0.2740740740740741,0.26666666666666666,0.2777777777777778
mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,em_relaxed,3,3,540,0.2740740740740741,0.2740740740740741,0.26666666666666666,0.2777777777777778
mmlu,mmlu.mqa_single_correct_sk:human_aging,em,3,3,657,0.0639269406392694,0.0639269406392694,0.0365296803652968,0.0867579908675799
mmlu,mmlu.mqa_single_correct_sk:human_aging,em_conservative,3,3,657,0.1400304414003044,0.1400304414003044,0.1187214611872146,0.1735159817351598
mmlu,mmlu.mqa_single_correct_sk:human_aging,em_relaxed,3,3,657,0.1400304414003044,0.1400304414003044,0.1187214611872146,0.1735159817351598
mmlu,mmlu.mqa_single_correct_sk:human_sexuality,em,3,3,327,0.051987767584097865,0.05198776758409786,0.045871559633027525,0.05504587155963303
mmlu,mmlu.mqa_single_correct_sk:human_sexuality,em_conservative,3,3,327,0.16819571865443425,0.16819571865443425,0.13761467889908258,0.21100917431192662
mmlu,mmlu.mqa_single_correct_sk:human_sexuality,em_relaxed,3,3,327,0.16819571865443425,0.16819571865443425,0.13761467889908258,0.21100917431192662
mmlu,mmlu.mqa_single_correct_sk:international_law,em,3,3,354,0.0423728813559322,0.0423728813559322,0.03389830508474576,0.05084745762711865
mmlu,mmlu.mqa_single_correct_sk:international_law,em_conservative,3,3,354,0.1497175141242938,0.1497175141242938,0.13559322033898305,0.16101694915254236
mmlu,mmlu.mqa_single_correct_sk:international_law,em_relaxed,3,3,354,0.1497175141242938,0.1497175141242938,0.13559322033898305,0.16101694915254236
mmlu,mmlu.mqa_single_correct_sk:jurisprudence,em,3,3,321,0.04984423676012461,0.04984423676012461,0.037383177570093455,0.056074766355140186
mmlu,mmlu.mqa_single_correct_sk:jurisprudence,em_conservative,3,3,321,0.17445482866043613,0.17445482866043613,0.14018691588785046,0.19626168224299065
mmlu,mmlu.mqa_single_correct_sk:jurisprudence,em_relaxed,3,3,321,0.17445482866043613,0.17445482866043613,0.14018691588785046,0.19626168224299065
mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,em,3,3,474,0.0379746835443038,0.0379746835443038,0.02531645569620253,0.05063291139240506
mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,em_conservative,3,3,474,0.189873417721519,0.189873417721519,0.13924050632911392,0.25316455696202533
mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,em_relaxed,3,3,474,0.189873417721519,0.189873417721519,0.13924050632911392,0.25316455696202533
mmlu,mmlu.mqa_single_correct_sk:machine_learning,em,3,3,336,0.01488095238095238,0.01488095238095238,0.008928571428571428,0.026785714285714284
mmlu,mmlu.mqa_single_correct_sk:machine_learning,em_conservative,3,3,336,0.11607142857142858,0.11607142857142858,0.10714285714285714,0.125
mmlu,mmlu.mqa_single_correct_sk:machine_learning,em_relaxed,3,3,336,0.12797619047619047,0.12797619047619047,0.11607142857142858,0.14285714285714285
mmlu,mmlu.mqa_single_correct_sk:management,em,3,3,300,0.04,0.04,0.02,0.06
mmlu,mmlu.mqa_single_correct_sk:management,em_conservative,3,3,300,0.16666666666666666,0.16666666666666666,0.15,0.2
mmlu,mmlu.mqa_single_correct_sk:management,em_relaxed,3,3,300,0.16666666666666666,0.16666666666666666,0.15,0.2
mmlu,mmlu.mqa_single_correct_sk:marketing,em,3,3,702,0.05698005698005698,0.05698005698005698,0.03418803418803419,0.0811965811965812
mmlu,mmlu.mqa_single_correct_sk:marketing,em_conservative,3,3,702,0.1737891737891738,0.1737891737891738,0.15384615384615385,0.20512820512820512
mmlu,mmlu.mqa_single_correct_sk:marketing,em_relaxed,3,3,702,0.17521367521367523,0.1752136752136752,0.15384615384615385,0.2094017094017094
mmlu,mmlu.mqa_single_correct_sk:medical_genetics,em,3,3,300,0.04,0.04,0.01,0.07
mmlu,mmlu.mqa_single_correct_sk:medical_genetics,em_conservative,3,3,300,0.16666666666666666,0.16666666666666666,0.14,0.18
mmlu,mmlu.mqa_single_correct_sk:medical_genetics,em_relaxed,3,3,300,0.16666666666666666,0.16666666666666666,0.14,0.18
mmlu,mmlu.mqa_single_correct_sk:miscellaneous,em,3,3,2337,0.05220367993153616,0.05220367993153616,0.03209242618741977,0.0693196405648267
mmlu,mmlu.mqa_single_correct_sk:miscellaneous,em_conservative,3,3,2337,0.17715019255455713,0.17715019255455713,0.14505776636713735,0.2092426187419769
mmlu,mmlu.mqa_single_correct_sk:miscellaneous,em_relaxed,3,3,2337,0.1775780915703894,0.1775780915703894,0.14505776636713735,0.2092426187419769
mmlu,mmlu.mqa_single_correct_sk:moral_disputes,em,3,3,978,0.07259713701431493,0.07259713701431493,0.06134969325153374,0.08895705521472393
mmlu,mmlu.mqa_single_correct_sk:moral_disputes,em_conservative,3,3,978,0.22392638036809817,0.22392638036809817,0.20245398773006135,0.25766871165644173
mmlu,mmlu.mqa_single_correct_sk:moral_disputes,em_relaxed,3,3,978,0.22494887525562374,0.2249488752556237,0.20245398773006135,0.25766871165644173
mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,em,3,3,2247,0.0,0.0,0.0,0.0
mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,em_conservative,3,3,2247,0.004450378282153983,0.004450378282153983,0.0026702269692923898,0.0053404539385847796
mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,em_relaxed,3,3,2247,0.0075656430796617715,0.0075656430796617715,0.0026702269692923898,0.01335113484646195
mmlu,mmlu.mqa_single_correct_sk:nutrition,em,3,3,909,0.034103410341034104,0.034103410341034104,0.0297029702970297,0.036303630363036306
mmlu,mmlu.mqa_single_correct_sk:nutrition,em_conservative,3,3,909,0.19031903190319033,0.19031903190319033,0.1848184818481848,0.19801980198019803
mmlu,mmlu.mqa_single_correct_sk:nutrition,em_relaxed,3,3,909,0.19031903190319033,0.19031903190319033,0.1848184818481848,0.19801980198019803
mmlu,mmlu.mqa_single_correct_sk:philosophy,em,3,3,924,0.0735930735930736,0.0735930735930736,0.05194805194805195,0.10064935064935066
mmlu,mmlu.mqa_single_correct_sk:philosophy,em_conservative,3,3,924,0.24675324675324675,0.24675324675324675,0.21103896103896103,0.2922077922077922
mmlu,mmlu.mqa_single_correct_sk:philosophy,em_relaxed,3,3,924,0.24675324675324675,0.24675324675324675,0.21103896103896103,0.2922077922077922
mmlu,mmlu.mqa_single_correct_sk:prehistory,em,3,3,957,0.04911180773249738,0.04911180773249739,0.04075235109717868,0.06583072100313479
mmlu,mmlu.mqa_single_correct_sk:prehistory,em_conservative,3,3,957,0.16509926854754442,0.16509926854754442,0.14733542319749215,0.18808777429467086
mmlu,mmlu.mqa_single_correct_sk:prehistory,em_relaxed,3,3,957,0.16614420062695925,0.16614420062695925,0.15047021943573669,0.18808777429467086
mmlu,mmlu.mqa_single_correct_sk:professional_accounting,em,3,3,828,0.0012077294685990338,0.0012077294685990338,0.0,0.0036231884057971015
mmlu,mmlu.mqa_single_correct_sk:professional_accounting,em_conservative,3,3,828,0.13768115942028986,0.13768115942028986,0.13043478260869565,0.14130434782608695
mmlu,mmlu.mqa_single_correct_sk:professional_accounting,em_relaxed,3,3,828,0.1497584541062802,0.1497584541062802,0.14492753623188406,0.15217391304347827
mmlu,mmlu.mqa_single_correct_sk:professional_law,em,3,3,3513,0.0025619128949615714,0.0025619128949615714,0.0017079419299743809,0.004269854824935952
mmlu,mmlu.mqa_single_correct_sk:professional_law,em_conservative,3,3,3513,0.13236549957301452,0.13236549957301452,0.12724167378309137,0.14175918018787362
mmlu,mmlu.mqa_single_correct_sk:professional_law,em_relaxed,3,3,3513,0.13435809849131797,0.13435809849131797,0.12724167378309137,0.14175918018787362
mmlu,mmlu.mqa_single_correct_sk:professional_medicine,em,3,3,792,0.0012626262626262627,0.0012626262626262627,0.0,0.003787878787878788
mmlu,mmlu.mqa_single_correct_sk:professional_medicine,em_conservative,3,3,792,0.14015151515151514,0.14015151515151514,0.09090909090909091,0.19696969696969696
mmlu,mmlu.mqa_single_correct_sk:professional_medicine,em_relaxed,3,3,792,0.14015151515151514,0.14015151515151514,0.09090909090909091,0.19696969696969696
mmlu,mmlu.mqa_single_correct_sk:professional_psychology,em,3,3,1788,0.03411633109619687,0.03411633109619687,0.018456375838926176,0.050335570469798654
mmlu,mmlu.mqa_single_correct_sk:professional_psychology,em_conservative,3,3,1788,0.203579418344519,0.203579418344519,0.19630872483221476,0.21308724832214765
mmlu,mmlu.mqa_single_correct_sk:professional_psychology,em_relaxed,3,3,1788,0.203579418344519,0.203579418344519,0.19630872483221476,0.21308724832214765
mmlu,mmlu.mqa_single_correct_sk:public_relations,em,3,3,327,0.06422018348623854,0.06422018348623854,0.027522935779816515,0.09174311926605505
mmlu,mmlu.mqa_single_correct_sk:public_relations,em_conservative,3,3,327,0.20795107033639146,0.20795107033639143,0.1743119266055046,0.23853211009174313
mmlu,mmlu.mqa_single_correct_sk:public_relations,em_relaxed,3,3,327,0.20795107033639146,0.20795107033639143,0.1743119266055046,0.23853211009174313
mmlu,mmlu.mqa_single_correct_sk:security_studies,em,3,3,666,0.046546546546546545,0.046546546546546545,0.04504504504504504,0.04954954954954955
mmlu,mmlu.mqa_single_correct_sk:security_studies,em_conservative,3,3,666,0.1906906906906907,0.1906906906906907,0.15765765765765766,0.21621621621621623
mmlu,mmlu.mqa_single_correct_sk:security_studies,em_relaxed,3,3,666,0.1906906906906907,0.1906906906906907,0.15765765765765766,0.21621621621621623
mmlu,mmlu.mqa_single_correct_sk:sociology,em,3,3,591,0.05922165820642978,0.05922165820642978,0.050761421319796954,0.07106598984771574
mmlu,mmlu.mqa_single_correct_sk:sociology,em_conservative,3,3,591,0.3333333333333333,0.3333333333333333,0.3096446700507614,0.3553299492385787
mmlu,mmlu.mqa_single_correct_sk:sociology,em_relaxed,3,3,591,0.3333333333333333,0.3333333333333333,0.3096446700507614,0.3553299492385787
mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,em,3,3,297,0.037037037037037035,0.037037037037037035,0.030303030303030304,0.050505050505050504
mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,em_conservative,3,3,297,0.24242424242424243,0.24242424242424243,0.2222222222222222,0.26262626262626265
mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,em_relaxed,3,3,297,0.25252525252525254,0.25252525252525254,0.23232323232323232,0.2727272727272727
mmlu,mmlu.mqa_single_correct_sk:virology,em,3,3,492,0.028455284552845527,0.028455284552845527,0.024390243902439025,0.03048780487804878
mmlu,mmlu.mqa_single_correct_sk:virology,em_conservative,3,3,492,0.14227642276422764,0.14227642276422764,0.12195121951219512,0.16463414634146342
mmlu,mmlu.mqa_single_correct_sk:virology,em_relaxed,3,3,492,0.14227642276422764,0.14227642276422764,0.12195121951219512,0.16463414634146342
mmlu,mmlu.mqa_single_correct_sk:world_religions,em,3,3,513,0.03898635477582846,0.03898635477582846,0.023391812865497075,0.04678362573099415
mmlu,mmlu.mqa_single_correct_sk:world_religions,em_conservative,3,3,513,0.08382066276803118,0.08382066276803118,0.05263157894736842,0.10526315789473684
mmlu,mmlu.mqa_single_correct_sk:world_religions,em_relaxed,3,3,513,0.08382066276803118,0.08382066276803118,0.05263157894736842,0.10526315789473684
movie_review,movie_review.sentiment_binary_sk,em_conservative_norm,3,3,6024,0.40139442231075695,0.40139442231075695,0.09412350597609562,0.6324701195219123
movie_review,movie_review.sentiment_binary_sk,em_norm,3,3,6024,0.14840637450199204,0.14840637450199204,0.01045816733067729,0.38745019920318724
movie_review,movie_review.sentiment_binary_sk,em_norm_relaxed,3,3,6024,0.40903054448871184,0.40903054448871184,0.11105577689243028,0.6329681274900398
multi_wiki_qa,multi_wiki_qa.qa_sk,exact_match_slk_prefix,3,3,14991,0.35467947435127745,0.35467947435127745,0.34000400240144085,0.36401841104662797
multi_wiki_qa,multi_wiki_qa.qa_sk,f1_slk,3,3,14991,0.5099047074319363,0.5099047074319363,0.5050071786470199,0.5156574509584114
senti_sk,senti_sk.sentiment_3_class_sk,em_conservative_norm,3,3,14031,0.3202907846910413,0.3202907846910413,0.26448578148385715,0.37716484926234767
senti_sk,senti_sk.sentiment_3_class_sk,em_norm,3,3,14031,0.1169553132349797,0.1169553132349797,0.019029292281376953,0.22642719692110327
senti_sk,senti_sk.sentiment_3_class_sk,em_norm_relaxed,3,3,14031,0.3707504810776138,0.3707504810776139,0.3200769724182168,0.40880906564036773
senti_sk,senti_sk.sentiment_binary_sk,em_conservative_norm,3,3,11184,0.3344062947067239,0.3344062947067239,0.1990343347639485,0.49302575107296137
senti_sk,senti_sk.sentiment_binary_sk,em_norm,3,3,11184,0.07743204577968527,0.07743204577968527,0.0037553648068669528,0.1968884120171674
senti_sk,senti_sk.sentiment_binary_sk,em_norm_relaxed,3,3,11184,0.3534513590844063,0.3534513590844063,0.25160944206008584,0.4965128755364807
skj1,skj1.skj1_sk,em,3,3,300,0.0,0.0,0.0,0.0
skj1,skj1.skj1_sk,em_conservative,3,3,300,0.03333333333333333,0.03333333333333333,0.02,0.05
skj1,skj1.skj1_sk,em_relaxed,3,3,300,0.043333333333333335,0.043333333333333335,0.02,0.06
sklep,sklep.hate_speech_sk:hate-speech,em_conservative_norm,3,3,3957,0.7045741723527925,0.7045741723527925,0.6921910538286581,0.7247915087187263
sklep,sklep.hate_speech_sk:hate-speech,em_norm,3,3,3957,0.0,0.0,0.0,0.0
sklep,sklep.hate_speech_sk:hate-speech,em_norm_relaxed,3,3,3957,0.7045741723527925,0.7045741723527925,0.6921910538286581,0.7247915087187263
sklep,sklep.nli_3_classes_sk:nli,em_conservative_norm,3,3,15012,0.03910205169197975,0.03910205169197975,0.001398880895283773,0.09952038369304557
sklep,sklep.nli_3_classes_sk:nli,em_norm,3,3,15012,0.0001332267519317879,0.0001332267519317879,0.0,0.0003996802557953637
sklep,sklep.nli_3_classes_sk:nli,em_norm_relaxed,3,3,15012,0.03930189181987743,0.03930189181987743,0.001398880895283773,0.09972022382094324
sklep,sklep.nli_binary_sk:rte,em_conservative_norm,3,3,4980,0.19738955823293172,0.19738955823293172,0.030120481927710843,0.3301204819277108
sklep,sklep.nli_binary_sk:rte,em_norm,3,3,4980,0.0012048192771084338,0.0012048192771084338,0.0006024096385542169,0.0018072289156626507
sklep,sklep.nli_binary_sk:rte,em_norm_relaxed,3,3,4980,0.19738955823293172,0.19738955823293172,0.030120481927710843,0.3301204819277108
sklep,sklep.qa_sk:question-answering,exact_match_slk_prefix,3,3,28749,0.16372743399770426,0.16372743399770426,0.16143170197224252,0.16518835437754356
sklep,sklep.qa_sk:question-answering,f1_slk,3,3,28749,0.3750415088928484,0.3750415088928484,0.3730151221096053,0.3769392643579436
sklep,sklep.sentiment_binary_sk:sentiment-analysis,em_conservative_norm,3,3,3126,0.4488163787587972,0.4488163787587972,0.118042226487524,0.8800383877159309
sklep,sklep.sentiment_binary_sk:sentiment-analysis,em_norm,3,3,3126,0.1548304542546385,0.1548304542546385,0.0009596928982725527,0.44817658349328215
sklep,sklep.sentiment_binary_sk:sentiment-analysis,em_norm_relaxed,3,3,3126,0.4619321817018554,0.4619321817018554,0.15355086372360843,0.8809980806142035
slovakcopa,slovakcopa.copa_sk,em,3,3,1500,0.0,0.0,0.0,0.0
slovakcopa,slovakcopa.copa_sk,em_conservative,3,3,1500,0.306,0.306,0.084,0.508
slovakcopa,slovakcopa.copa_sk,em_relaxed,3,3,1500,0.4046666666666667,0.4046666666666667,0.086,0.618
toxic_sk,toxic_sk.toxicity_sk,em_conservative_norm,3,3,2643,0.4759742716609913,0.4759742716609913,0.45629965947786605,0.4960272417707151
toxic_sk,toxic_sk.toxicity_sk,em_norm,3,3,2643,0.0026485054861899358,0.0026485054861899358,0.0,0.007945516458569807
toxic_sk,toxic_sk.toxicity_sk,em_norm_relaxed,3,3,2643,0.4759742716609913,0.4759742716609913,0.45629965947786605,0.4960272417707151
truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,em,3,3,2334,0.0,0.0,0.0,0.0
truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,em_conservative,3,3,2334,0.011996572407883462,0.011996572407883462,0.007712082262210797,0.016709511568123392
truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,em_relaxed,3,3,2334,0.011996572407883462,0.011996572407883462,0.007712082262210797,0.016709511568123392
1 family task_group_without_prompt metric task_variants prompt_count evaluated_instances task_macro_mean sample_weighted_mean min_task_value max_task_value
2 arc arc.mqa_single_correct_sk em 3 3 3507 0.023666951810664385 0.023666951810664385 0.013686911890504704 0.03507271171941831
3 arc arc.mqa_single_correct_sk em_conservative 3 3 3507 0.19019104647847163 0.19019104647847163 0.1710863986313088 0.223267750213858
4 arc arc.mqa_single_correct_sk em_relaxed 3 3 3507 0.19076133447390933 0.19076133447390933 0.1710863986313088 0.22412318220701455
5 belebele belebele.mqa_single_correct_sk em 3 3 2700 0.022222222222222223 0.022222222222222223 0.016666666666666666 0.03
6 belebele belebele.mqa_single_correct_sk em_conservative 3 3 2700 0.21259259259259258 0.21259259259259258 0.2111111111111111 0.21555555555555556
7 belebele belebele.mqa_single_correct_sk em_relaxed 3 3 2700 0.21333333333333335 0.21333333333333335 0.2111111111111111 0.21555555555555556
8 financial_exams financial_exams.completion_sk em 3 3 4002 0.029735132433783108 0.029735132433783108 0.0037481259370314842 0.07571214392803598
9 financial_exams financial_exams.completion_sk em_conservative 3 3 4002 0.21839080459770116 0.21839080459770116 0.20089955022488756 0.24887556221889057
10 financial_exams financial_exams.completion_sk em_relaxed 3 3 4002 0.22938530734632684 0.22938530734632684 0.20089955022488756 0.24887556221889057
11 globalmmlu-lite globalmmlu-lite.mqa_single_correct_sk em 3 3 1200 0.028333333333333335 0.028333333333333332 0.025 0.035
12 globalmmlu-lite globalmmlu-lite.mqa_single_correct_sk em_conservative 3 3 1200 0.19833333333333333 0.19833333333333333 0.1825 0.21
13 globalmmlu-lite globalmmlu-lite.mqa_single_correct_sk em_relaxed 3 3 1200 0.19833333333333333 0.19833333333333333 0.1825 0.21
14 mmlu mmlu.mqa_single_correct_sk:abstract_algebra em 3 3 300 0.0033333333333333335 0.0033333333333333335 0.0 0.01
15 mmlu mmlu.mqa_single_correct_sk:abstract_algebra em_conservative 3 3 300 0.03666666666666667 0.03666666666666667 0.01 0.06
16 mmlu mmlu.mqa_single_correct_sk:abstract_algebra em_relaxed 3 3 300 0.04 0.04 0.01 0.06
17 mmlu mmlu.mqa_single_correct_sk:anatomy em 3 3 402 0.03731343283582089 0.03731343283582089 0.022388059701492536 0.05970149253731343
18 mmlu mmlu.mqa_single_correct_sk:anatomy em_conservative 3 3 402 0.14676616915422885 0.14676616915422885 0.12686567164179105 0.1791044776119403
19 mmlu mmlu.mqa_single_correct_sk:anatomy em_relaxed 3 3 402 0.14676616915422885 0.14676616915422885 0.12686567164179105 0.1791044776119403
20 mmlu mmlu.mqa_single_correct_sk:astronomy em 3 3 456 0.03508771929824561 0.03508771929824561 0.019736842105263157 0.05263157894736842
21 mmlu mmlu.mqa_single_correct_sk:astronomy em_conservative 3 3 456 0.1513157894736842 0.1513157894736842 0.13815789473684212 0.17105263157894737
22 mmlu mmlu.mqa_single_correct_sk:astronomy em_relaxed 3 3 456 0.15789473684210525 0.15789473684210525 0.14473684210526316 0.17763157894736842
23 mmlu mmlu.mqa_single_correct_sk:business_ethics em 3 3 300 0.02 0.02 0.01 0.03
24 mmlu mmlu.mqa_single_correct_sk:business_ethics em_conservative 3 3 300 0.16 0.16 0.13 0.21
25 mmlu mmlu.mqa_single_correct_sk:business_ethics em_relaxed 3 3 300 0.16333333333333333 0.16333333333333333 0.13 0.22
26 mmlu mmlu.mqa_single_correct_sk:clinical_knowledge em 3 3 777 0.029601029601029602 0.029601029601029602 0.011583011583011582 0.04633204633204633
27 mmlu mmlu.mqa_single_correct_sk:clinical_knowledge em_conservative 3 3 777 0.1969111969111969 0.1969111969111969 0.15444015444015444 0.23938223938223938
28 mmlu mmlu.mqa_single_correct_sk:clinical_knowledge em_relaxed 3 3 777 0.1981981981981982 0.1981981981981982 0.15444015444015444 0.24324324324324326
29 mmlu mmlu.mqa_single_correct_sk:college_biology em 3 3 426 0.035211267605633804 0.035211267605633804 0.028169014084507043 0.04225352112676056
30 mmlu mmlu.mqa_single_correct_sk:college_biology em_conservative 3 3 426 0.1737089201877934 0.17370892018779344 0.15492957746478872 0.19014084507042253
31 mmlu mmlu.mqa_single_correct_sk:college_biology em_relaxed 3 3 426 0.1737089201877934 0.17370892018779344 0.15492957746478872 0.19014084507042253
32 mmlu mmlu.mqa_single_correct_sk:college_chemistry em 3 3 300 0.056666666666666664 0.056666666666666664 0.04 0.09
33 mmlu mmlu.mqa_single_correct_sk:college_chemistry em_conservative 3 3 300 0.12666666666666668 0.12666666666666668 0.08 0.16
34 mmlu mmlu.mqa_single_correct_sk:college_chemistry em_relaxed 3 3 300 0.12666666666666668 0.12666666666666668 0.08 0.16
35 mmlu mmlu.mqa_single_correct_sk:college_computer_science em 3 3 300 0.016666666666666666 0.016666666666666666 0.01 0.03
36 mmlu mmlu.mqa_single_correct_sk:college_computer_science em_conservative 3 3 300 0.11333333333333333 0.11333333333333333 0.11 0.12
37 mmlu mmlu.mqa_single_correct_sk:college_computer_science em_relaxed 3 3 300 0.15666666666666668 0.15666666666666668 0.15 0.17
38 mmlu mmlu.mqa_single_correct_sk:college_mathematics em 3 3 297 0.0 0.0 0.0 0.0
39 mmlu mmlu.mqa_single_correct_sk:college_mathematics em_conservative 3 3 297 0.05387205387205388 0.05387205387205387 0.030303030303030304 0.08080808080808081
40 mmlu mmlu.mqa_single_correct_sk:college_mathematics em_relaxed 3 3 297 0.06397306397306397 0.06397306397306397 0.04040404040404041 0.09090909090909091
41 mmlu mmlu.mqa_single_correct_sk:college_medicine em 3 3 516 0.03875968992248062 0.03875968992248062 0.011627906976744186 0.06395348837209303
42 mmlu mmlu.mqa_single_correct_sk:college_medicine em_conservative 3 3 516 0.17248062015503876 0.17248062015503876 0.1511627906976744 0.20930232558139536
43 mmlu mmlu.mqa_single_correct_sk:college_medicine em_relaxed 3 3 516 0.17248062015503876 0.17248062015503876 0.1511627906976744 0.20930232558139536
44 mmlu mmlu.mqa_single_correct_sk:college_physics em 3 3 306 0.016339869281045753 0.016339869281045753 0.0 0.0392156862745098
45 mmlu mmlu.mqa_single_correct_sk:college_physics em_conservative 3 3 306 0.1111111111111111 0.1111111111111111 0.0784313725490196 0.12745098039215685
46 mmlu mmlu.mqa_single_correct_sk:college_physics em_relaxed 3 3 306 0.1111111111111111 0.1111111111111111 0.0784313725490196 0.12745098039215685
47 mmlu mmlu.mqa_single_correct_sk:computer_security em 3 3 300 0.02 0.02 0.02 0.02
48 mmlu mmlu.mqa_single_correct_sk:computer_security em_conservative 3 3 300 0.10666666666666667 0.10666666666666667 0.08 0.13
49 mmlu mmlu.mqa_single_correct_sk:computer_security em_relaxed 3 3 300 0.10666666666666667 0.10666666666666667 0.08 0.13
50 mmlu mmlu.mqa_single_correct_sk:conceptual_physics em 3 3 705 0.05815602836879433 0.05815602836879433 0.03829787234042553 0.0851063829787234
51 mmlu mmlu.mqa_single_correct_sk:conceptual_physics em_conservative 3 3 705 0.12056737588652482 0.12056737588652482 0.09361702127659574 0.14042553191489363
52 mmlu mmlu.mqa_single_correct_sk:conceptual_physics em_relaxed 3 3 705 0.12056737588652482 0.12056737588652482 0.09361702127659574 0.14042553191489363
53 mmlu mmlu.mqa_single_correct_sk:econometrics em 3 3 336 0.020833333333333332 0.020833333333333332 0.017857142857142856 0.026785714285714284
54 mmlu mmlu.mqa_single_correct_sk:econometrics em_conservative 3 3 336 0.1130952380952381 0.1130952380952381 0.09821428571428571 0.125
55 mmlu mmlu.mqa_single_correct_sk:econometrics em_relaxed 3 3 336 0.11607142857142858 0.11607142857142858 0.10714285714285714 0.125
56 mmlu mmlu.mqa_single_correct_sk:electrical_engineering em 3 3 435 0.05057471264367816 0.05057471264367816 0.041379310344827586 0.06896551724137931
57 mmlu mmlu.mqa_single_correct_sk:electrical_engineering em_conservative 3 3 435 0.11494252873563218 0.11494252873563218 0.0896551724137931 0.1310344827586207
58 mmlu mmlu.mqa_single_correct_sk:electrical_engineering em_relaxed 3 3 435 0.11494252873563218 0.11494252873563218 0.0896551724137931 0.1310344827586207
59 mmlu mmlu.mqa_single_correct_sk:elementary_mathematics em 3 3 1119 0.0071492403932082215 0.0071492403932082215 0.0 0.0160857908847185
60 mmlu mmlu.mqa_single_correct_sk:elementary_mathematics em_conservative 3 3 1119 0.08042895442359249 0.08042895442359249 0.058981233243967826 0.09651474530831099
61 mmlu mmlu.mqa_single_correct_sk:elementary_mathematics em_relaxed 3 3 1119 0.0840035746201966 0.0840035746201966 0.06166219839142091 0.10187667560321716
62 mmlu mmlu.mqa_single_correct_sk:formal_logic em 3 3 345 0.002898550724637681 0.002898550724637681 0.0 0.008695652173913044
63 mmlu mmlu.mqa_single_correct_sk:formal_logic em_conservative 3 3 345 0.0782608695652174 0.0782608695652174 0.06956521739130435 0.08695652173913043
64 mmlu mmlu.mqa_single_correct_sk:formal_logic em_relaxed 3 3 345 0.12173913043478261 0.12173913043478261 0.11304347826086956 0.13043478260869565
65 mmlu mmlu.mqa_single_correct_sk:global_facts em 3 3 300 0.06666666666666667 0.06666666666666667 0.01 0.13
66 mmlu mmlu.mqa_single_correct_sk:global_facts em_conservative 3 3 300 0.13666666666666666 0.13666666666666666 0.11 0.18
67 mmlu mmlu.mqa_single_correct_sk:global_facts em_relaxed 3 3 300 0.13666666666666666 0.13666666666666666 0.11 0.18
68 mmlu mmlu.mqa_single_correct_sk:high_school_biology em 3 3 846 0.030732860520094562 0.030732860520094562 0.02127659574468085 0.04964539007092199
69 mmlu mmlu.mqa_single_correct_sk:high_school_biology em_conservative 3 3 846 0.20212765957446807 0.20212765957446807 0.1773049645390071 0.24113475177304963
70 mmlu mmlu.mqa_single_correct_sk:high_school_biology em_relaxed 3 3 846 0.20921985815602837 0.20921985815602837 0.18439716312056736 0.24822695035460993
71 mmlu mmlu.mqa_single_correct_sk:high_school_chemistry em 3 3 594 0.011784511784511786 0.011784511784511785 0.005050505050505051 0.020202020202020204
72 mmlu mmlu.mqa_single_correct_sk:high_school_chemistry em_conservative 3 3 594 0.09259259259259259 0.09259259259259259 0.08080808080808081 0.1111111111111111
73 mmlu mmlu.mqa_single_correct_sk:high_school_chemistry em_relaxed 3 3 594 0.09764309764309764 0.09764309764309764 0.08585858585858586 0.11616161616161616
74 mmlu mmlu.mqa_single_correct_sk:high_school_computer_science em 3 3 297 0.01346801346801347 0.013468013468013467 0.0 0.030303030303030304
75 mmlu mmlu.mqa_single_correct_sk:high_school_computer_science em_conservative 3 3 297 0.11447811447811448 0.11447811447811448 0.10101010101010101 0.12121212121212122
76 mmlu mmlu.mqa_single_correct_sk:high_school_computer_science em_relaxed 3 3 297 0.11784511784511785 0.11784511784511785 0.10101010101010101 0.13131313131313133
77 mmlu mmlu.mqa_single_correct_sk:high_school_european_history em 3 3 405 0.022222222222222223 0.022222222222222223 0.014814814814814815 0.02962962962962963
78 mmlu mmlu.mqa_single_correct_sk:high_school_european_history em_conservative 3 3 405 0.22962962962962963 0.22962962962962963 0.2074074074074074 0.24444444444444444
79 mmlu mmlu.mqa_single_correct_sk:high_school_european_history em_relaxed 3 3 405 0.23209876543209876 0.23209876543209876 0.21481481481481482 0.24444444444444444
80 mmlu mmlu.mqa_single_correct_sk:high_school_geography em 3 3 594 0.058922558922558925 0.058922558922558925 0.030303030303030304 0.07575757575757576
81 mmlu mmlu.mqa_single_correct_sk:high_school_geography em_conservative 3 3 594 0.19528619528619529 0.19528619528619529 0.15656565656565657 0.22727272727272727
82 mmlu mmlu.mqa_single_correct_sk:high_school_geography em_relaxed 3 3 594 0.19528619528619529 0.19528619528619529 0.15656565656565657 0.22727272727272727
83 mmlu mmlu.mqa_single_correct_sk:high_school_government_and_politics em 3 3 579 0.039723661485319514 0.039723661485319514 0.025906735751295335 0.046632124352331605
84 mmlu mmlu.mqa_single_correct_sk:high_school_government_and_politics em_conservative 3 3 579 0.309153713298791 0.30915371329879104 0.3005181347150259 0.32124352331606215
85 mmlu mmlu.mqa_single_correct_sk:high_school_government_and_politics em_relaxed 3 3 579 0.31088082901554404 0.310880829015544 0.30569948186528495 0.32124352331606215
86 mmlu mmlu.mqa_single_correct_sk:high_school_macroeconomics em 3 3 1161 0.020671834625322995 0.020671834625323 0.012919896640826873 0.031007751937984496
87 mmlu mmlu.mqa_single_correct_sk:high_school_macroeconomics em_conservative 3 3 1161 0.14900947459086994 0.14900947459086994 0.12661498708010335 0.17054263565891473
88 mmlu mmlu.mqa_single_correct_sk:high_school_macroeconomics em_relaxed 3 3 1161 0.15331610680447888 0.1533161068044789 0.12919896640826872 0.1731266149870801
89 mmlu mmlu.mqa_single_correct_sk:high_school_mathematics em 3 3 798 0.0037593984962406013 0.0037593984962406013 0.0 0.011278195488721804
90 mmlu mmlu.mqa_single_correct_sk:high_school_mathematics em_conservative 3 3 798 0.05388471177944862 0.05388471177944862 0.03759398496240601 0.06766917293233082
91 mmlu mmlu.mqa_single_correct_sk:high_school_mathematics em_relaxed 3 3 798 0.05388471177944862 0.05388471177944862 0.03759398496240601 0.06766917293233082
92 mmlu mmlu.mqa_single_correct_sk:high_school_microeconomics em 3 3 651 0.013824884792626729 0.013824884792626729 0.004608294930875576 0.02304147465437788
93 mmlu mmlu.mqa_single_correct_sk:high_school_microeconomics em_conservative 3 3 651 0.16129032258064516 0.16129032258064516 0.14285714285714285 0.17511520737327188
94 mmlu mmlu.mqa_single_correct_sk:high_school_microeconomics em_relaxed 3 3 651 0.16282642089093702 0.16282642089093702 0.14285714285714285 0.17511520737327188
95 mmlu mmlu.mqa_single_correct_sk:high_school_physics em 3 3 450 0.031111111111111114 0.03111111111111111 0.02666666666666667 0.04
96 mmlu mmlu.mqa_single_correct_sk:high_school_physics em_conservative 3 3 450 0.1511111111111111 0.1511111111111111 0.13333333333333333 0.17333333333333334
97 mmlu mmlu.mqa_single_correct_sk:high_school_physics em_relaxed 3 3 450 0.1511111111111111 0.1511111111111111 0.13333333333333333 0.17333333333333334
98 mmlu mmlu.mqa_single_correct_sk:high_school_psychology em 3 3 1608 0.03296019900497512 0.03296019900497513 0.016791044776119403 0.04477611940298507
99 mmlu mmlu.mqa_single_correct_sk:high_school_psychology em_conservative 3 3 1608 0.1921641791044776 0.1921641791044776 0.16791044776119404 0.22388059701492538
100 mmlu mmlu.mqa_single_correct_sk:high_school_psychology em_relaxed 3 3 1608 0.1921641791044776 0.1921641791044776 0.16791044776119404 0.22388059701492538
101 mmlu mmlu.mqa_single_correct_sk:high_school_statistics em 3 3 642 0.001557632398753894 0.0015576323987538938 0.0 0.004672897196261682
102 mmlu mmlu.mqa_single_correct_sk:high_school_statistics em_conservative 3 3 642 0.16822429906542055 0.16822429906542055 0.1542056074766355 0.17757009345794392
103 mmlu mmlu.mqa_single_correct_sk:high_school_statistics em_relaxed 3 3 642 0.17757009345794392 0.17757009345794392 0.16355140186915887 0.18691588785046728
104 mmlu mmlu.mqa_single_correct_sk:high_school_us_history em 3 3 171 0.0 0.0 0.0 0.0
105 mmlu mmlu.mqa_single_correct_sk:high_school_us_history em_conservative 3 3 171 0.17543859649122806 0.17543859649122806 0.12280701754385964 0.21052631578947367
106 mmlu mmlu.mqa_single_correct_sk:high_school_us_history em_relaxed 3 3 171 0.17543859649122806 0.17543859649122806 0.12280701754385964 0.21052631578947367
107 mmlu mmlu.mqa_single_correct_sk:high_school_world_history em 3 3 540 0.005555555555555556 0.005555555555555556 0.005555555555555556 0.005555555555555556
108 mmlu mmlu.mqa_single_correct_sk:high_school_world_history em_conservative 3 3 540 0.2740740740740741 0.2740740740740741 0.26666666666666666 0.2777777777777778
109 mmlu mmlu.mqa_single_correct_sk:high_school_world_history em_relaxed 3 3 540 0.2740740740740741 0.2740740740740741 0.26666666666666666 0.2777777777777778
110 mmlu mmlu.mqa_single_correct_sk:human_aging em 3 3 657 0.0639269406392694 0.0639269406392694 0.0365296803652968 0.0867579908675799
111 mmlu mmlu.mqa_single_correct_sk:human_aging em_conservative 3 3 657 0.1400304414003044 0.1400304414003044 0.1187214611872146 0.1735159817351598
112 mmlu mmlu.mqa_single_correct_sk:human_aging em_relaxed 3 3 657 0.1400304414003044 0.1400304414003044 0.1187214611872146 0.1735159817351598
113 mmlu mmlu.mqa_single_correct_sk:human_sexuality em 3 3 327 0.051987767584097865 0.05198776758409786 0.045871559633027525 0.05504587155963303
114 mmlu mmlu.mqa_single_correct_sk:human_sexuality em_conservative 3 3 327 0.16819571865443425 0.16819571865443425 0.13761467889908258 0.21100917431192662
115 mmlu mmlu.mqa_single_correct_sk:human_sexuality em_relaxed 3 3 327 0.16819571865443425 0.16819571865443425 0.13761467889908258 0.21100917431192662
116 mmlu mmlu.mqa_single_correct_sk:international_law em 3 3 354 0.0423728813559322 0.0423728813559322 0.03389830508474576 0.05084745762711865
117 mmlu mmlu.mqa_single_correct_sk:international_law em_conservative 3 3 354 0.1497175141242938 0.1497175141242938 0.13559322033898305 0.16101694915254236
118 mmlu mmlu.mqa_single_correct_sk:international_law em_relaxed 3 3 354 0.1497175141242938 0.1497175141242938 0.13559322033898305 0.16101694915254236
119 mmlu mmlu.mqa_single_correct_sk:jurisprudence em 3 3 321 0.04984423676012461 0.04984423676012461 0.037383177570093455 0.056074766355140186
120 mmlu mmlu.mqa_single_correct_sk:jurisprudence em_conservative 3 3 321 0.17445482866043613 0.17445482866043613 0.14018691588785046 0.19626168224299065
121 mmlu mmlu.mqa_single_correct_sk:jurisprudence em_relaxed 3 3 321 0.17445482866043613 0.17445482866043613 0.14018691588785046 0.19626168224299065
122 mmlu mmlu.mqa_single_correct_sk:logical_fallacies em 3 3 474 0.0379746835443038 0.0379746835443038 0.02531645569620253 0.05063291139240506
123 mmlu mmlu.mqa_single_correct_sk:logical_fallacies em_conservative 3 3 474 0.189873417721519 0.189873417721519 0.13924050632911392 0.25316455696202533
124 mmlu mmlu.mqa_single_correct_sk:logical_fallacies em_relaxed 3 3 474 0.189873417721519 0.189873417721519 0.13924050632911392 0.25316455696202533
125 mmlu mmlu.mqa_single_correct_sk:machine_learning em 3 3 336 0.01488095238095238 0.01488095238095238 0.008928571428571428 0.026785714285714284
126 mmlu mmlu.mqa_single_correct_sk:machine_learning em_conservative 3 3 336 0.11607142857142858 0.11607142857142858 0.10714285714285714 0.125
127 mmlu mmlu.mqa_single_correct_sk:machine_learning em_relaxed 3 3 336 0.12797619047619047 0.12797619047619047 0.11607142857142858 0.14285714285714285
128 mmlu mmlu.mqa_single_correct_sk:management em 3 3 300 0.04 0.04 0.02 0.06
129 mmlu mmlu.mqa_single_correct_sk:management em_conservative 3 3 300 0.16666666666666666 0.16666666666666666 0.15 0.2
130 mmlu mmlu.mqa_single_correct_sk:management em_relaxed 3 3 300 0.16666666666666666 0.16666666666666666 0.15 0.2
131 mmlu mmlu.mqa_single_correct_sk:marketing em 3 3 702 0.05698005698005698 0.05698005698005698 0.03418803418803419 0.0811965811965812
132 mmlu mmlu.mqa_single_correct_sk:marketing em_conservative 3 3 702 0.1737891737891738 0.1737891737891738 0.15384615384615385 0.20512820512820512
133 mmlu mmlu.mqa_single_correct_sk:marketing em_relaxed 3 3 702 0.17521367521367523 0.1752136752136752 0.15384615384615385 0.2094017094017094
134 mmlu mmlu.mqa_single_correct_sk:medical_genetics em 3 3 300 0.04 0.04 0.01 0.07
135 mmlu mmlu.mqa_single_correct_sk:medical_genetics em_conservative 3 3 300 0.16666666666666666 0.16666666666666666 0.14 0.18
136 mmlu mmlu.mqa_single_correct_sk:medical_genetics em_relaxed 3 3 300 0.16666666666666666 0.16666666666666666 0.14 0.18
137 mmlu mmlu.mqa_single_correct_sk:miscellaneous em 3 3 2337 0.05220367993153616 0.05220367993153616 0.03209242618741977 0.0693196405648267
138 mmlu mmlu.mqa_single_correct_sk:miscellaneous em_conservative 3 3 2337 0.17715019255455713 0.17715019255455713 0.14505776636713735 0.2092426187419769
139 mmlu mmlu.mqa_single_correct_sk:miscellaneous em_relaxed 3 3 2337 0.1775780915703894 0.1775780915703894 0.14505776636713735 0.2092426187419769
140 mmlu mmlu.mqa_single_correct_sk:moral_disputes em 3 3 978 0.07259713701431493 0.07259713701431493 0.06134969325153374 0.08895705521472393
141 mmlu mmlu.mqa_single_correct_sk:moral_disputes em_conservative 3 3 978 0.22392638036809817 0.22392638036809817 0.20245398773006135 0.25766871165644173
142 mmlu mmlu.mqa_single_correct_sk:moral_disputes em_relaxed 3 3 978 0.22494887525562374 0.2249488752556237 0.20245398773006135 0.25766871165644173
143 mmlu mmlu.mqa_single_correct_sk:moral_scenarios em 3 3 2247 0.0 0.0 0.0 0.0
144 mmlu mmlu.mqa_single_correct_sk:moral_scenarios em_conservative 3 3 2247 0.004450378282153983 0.004450378282153983 0.0026702269692923898 0.0053404539385847796
145 mmlu mmlu.mqa_single_correct_sk:moral_scenarios em_relaxed 3 3 2247 0.0075656430796617715 0.0075656430796617715 0.0026702269692923898 0.01335113484646195
146 mmlu mmlu.mqa_single_correct_sk:nutrition em 3 3 909 0.034103410341034104 0.034103410341034104 0.0297029702970297 0.036303630363036306
147 mmlu mmlu.mqa_single_correct_sk:nutrition em_conservative 3 3 909 0.19031903190319033 0.19031903190319033 0.1848184818481848 0.19801980198019803
148 mmlu mmlu.mqa_single_correct_sk:nutrition em_relaxed 3 3 909 0.19031903190319033 0.19031903190319033 0.1848184818481848 0.19801980198019803
149 mmlu mmlu.mqa_single_correct_sk:philosophy em 3 3 924 0.0735930735930736 0.0735930735930736 0.05194805194805195 0.10064935064935066
150 mmlu mmlu.mqa_single_correct_sk:philosophy em_conservative 3 3 924 0.24675324675324675 0.24675324675324675 0.21103896103896103 0.2922077922077922
151 mmlu mmlu.mqa_single_correct_sk:philosophy em_relaxed 3 3 924 0.24675324675324675 0.24675324675324675 0.21103896103896103 0.2922077922077922
152 mmlu mmlu.mqa_single_correct_sk:prehistory em 3 3 957 0.04911180773249738 0.04911180773249739 0.04075235109717868 0.06583072100313479
153 mmlu mmlu.mqa_single_correct_sk:prehistory em_conservative 3 3 957 0.16509926854754442 0.16509926854754442 0.14733542319749215 0.18808777429467086
154 mmlu mmlu.mqa_single_correct_sk:prehistory em_relaxed 3 3 957 0.16614420062695925 0.16614420062695925 0.15047021943573669 0.18808777429467086
155 mmlu mmlu.mqa_single_correct_sk:professional_accounting em 3 3 828 0.0012077294685990338 0.0012077294685990338 0.0 0.0036231884057971015
156 mmlu mmlu.mqa_single_correct_sk:professional_accounting em_conservative 3 3 828 0.13768115942028986 0.13768115942028986 0.13043478260869565 0.14130434782608695
157 mmlu mmlu.mqa_single_correct_sk:professional_accounting em_relaxed 3 3 828 0.1497584541062802 0.1497584541062802 0.14492753623188406 0.15217391304347827
158 mmlu mmlu.mqa_single_correct_sk:professional_law em 3 3 3513 0.0025619128949615714 0.0025619128949615714 0.0017079419299743809 0.004269854824935952
159 mmlu mmlu.mqa_single_correct_sk:professional_law em_conservative 3 3 3513 0.13236549957301452 0.13236549957301452 0.12724167378309137 0.14175918018787362
160 mmlu mmlu.mqa_single_correct_sk:professional_law em_relaxed 3 3 3513 0.13435809849131797 0.13435809849131797 0.12724167378309137 0.14175918018787362
161 mmlu mmlu.mqa_single_correct_sk:professional_medicine em 3 3 792 0.0012626262626262627 0.0012626262626262627 0.0 0.003787878787878788
162 mmlu mmlu.mqa_single_correct_sk:professional_medicine em_conservative 3 3 792 0.14015151515151514 0.14015151515151514 0.09090909090909091 0.19696969696969696
163 mmlu mmlu.mqa_single_correct_sk:professional_medicine em_relaxed 3 3 792 0.14015151515151514 0.14015151515151514 0.09090909090909091 0.19696969696969696
164 mmlu mmlu.mqa_single_correct_sk:professional_psychology em 3 3 1788 0.03411633109619687 0.03411633109619687 0.018456375838926176 0.050335570469798654
165 mmlu mmlu.mqa_single_correct_sk:professional_psychology em_conservative 3 3 1788 0.203579418344519 0.203579418344519 0.19630872483221476 0.21308724832214765
166 mmlu mmlu.mqa_single_correct_sk:professional_psychology em_relaxed 3 3 1788 0.203579418344519 0.203579418344519 0.19630872483221476 0.21308724832214765
167 mmlu mmlu.mqa_single_correct_sk:public_relations em 3 3 327 0.06422018348623854 0.06422018348623854 0.027522935779816515 0.09174311926605505
168 mmlu mmlu.mqa_single_correct_sk:public_relations em_conservative 3 3 327 0.20795107033639146 0.20795107033639143 0.1743119266055046 0.23853211009174313
169 mmlu mmlu.mqa_single_correct_sk:public_relations em_relaxed 3 3 327 0.20795107033639146 0.20795107033639143 0.1743119266055046 0.23853211009174313
170 mmlu mmlu.mqa_single_correct_sk:security_studies em 3 3 666 0.046546546546546545 0.046546546546546545 0.04504504504504504 0.04954954954954955
171 mmlu mmlu.mqa_single_correct_sk:security_studies em_conservative 3 3 666 0.1906906906906907 0.1906906906906907 0.15765765765765766 0.21621621621621623
172 mmlu mmlu.mqa_single_correct_sk:security_studies em_relaxed 3 3 666 0.1906906906906907 0.1906906906906907 0.15765765765765766 0.21621621621621623
173 mmlu mmlu.mqa_single_correct_sk:sociology em 3 3 591 0.05922165820642978 0.05922165820642978 0.050761421319796954 0.07106598984771574
174 mmlu mmlu.mqa_single_correct_sk:sociology em_conservative 3 3 591 0.3333333333333333 0.3333333333333333 0.3096446700507614 0.3553299492385787
175 mmlu mmlu.mqa_single_correct_sk:sociology em_relaxed 3 3 591 0.3333333333333333 0.3333333333333333 0.3096446700507614 0.3553299492385787
176 mmlu mmlu.mqa_single_correct_sk:us_foreign_policy em 3 3 297 0.037037037037037035 0.037037037037037035 0.030303030303030304 0.050505050505050504
177 mmlu mmlu.mqa_single_correct_sk:us_foreign_policy em_conservative 3 3 297 0.24242424242424243 0.24242424242424243 0.2222222222222222 0.26262626262626265
178 mmlu mmlu.mqa_single_correct_sk:us_foreign_policy em_relaxed 3 3 297 0.25252525252525254 0.25252525252525254 0.23232323232323232 0.2727272727272727
179 mmlu mmlu.mqa_single_correct_sk:virology em 3 3 492 0.028455284552845527 0.028455284552845527 0.024390243902439025 0.03048780487804878
180 mmlu mmlu.mqa_single_correct_sk:virology em_conservative 3 3 492 0.14227642276422764 0.14227642276422764 0.12195121951219512 0.16463414634146342
181 mmlu mmlu.mqa_single_correct_sk:virology em_relaxed 3 3 492 0.14227642276422764 0.14227642276422764 0.12195121951219512 0.16463414634146342
182 mmlu mmlu.mqa_single_correct_sk:world_religions em 3 3 513 0.03898635477582846 0.03898635477582846 0.023391812865497075 0.04678362573099415
183 mmlu mmlu.mqa_single_correct_sk:world_religions em_conservative 3 3 513 0.08382066276803118 0.08382066276803118 0.05263157894736842 0.10526315789473684
184 mmlu mmlu.mqa_single_correct_sk:world_religions em_relaxed 3 3 513 0.08382066276803118 0.08382066276803118 0.05263157894736842 0.10526315789473684
185 movie_review movie_review.sentiment_binary_sk em_conservative_norm 3 3 6024 0.40139442231075695 0.40139442231075695 0.09412350597609562 0.6324701195219123
186 movie_review movie_review.sentiment_binary_sk em_norm 3 3 6024 0.14840637450199204 0.14840637450199204 0.01045816733067729 0.38745019920318724
187 movie_review movie_review.sentiment_binary_sk em_norm_relaxed 3 3 6024 0.40903054448871184 0.40903054448871184 0.11105577689243028 0.6329681274900398
188 multi_wiki_qa multi_wiki_qa.qa_sk exact_match_slk_prefix 3 3 14991 0.35467947435127745 0.35467947435127745 0.34000400240144085 0.36401841104662797
189 multi_wiki_qa multi_wiki_qa.qa_sk f1_slk 3 3 14991 0.5099047074319363 0.5099047074319363 0.5050071786470199 0.5156574509584114
190 senti_sk senti_sk.sentiment_3_class_sk em_conservative_norm 3 3 14031 0.3202907846910413 0.3202907846910413 0.26448578148385715 0.37716484926234767
191 senti_sk senti_sk.sentiment_3_class_sk em_norm 3 3 14031 0.1169553132349797 0.1169553132349797 0.019029292281376953 0.22642719692110327
192 senti_sk senti_sk.sentiment_3_class_sk em_norm_relaxed 3 3 14031 0.3707504810776138 0.3707504810776139 0.3200769724182168 0.40880906564036773
193 senti_sk senti_sk.sentiment_binary_sk em_conservative_norm 3 3 11184 0.3344062947067239 0.3344062947067239 0.1990343347639485 0.49302575107296137
194 senti_sk senti_sk.sentiment_binary_sk em_norm 3 3 11184 0.07743204577968527 0.07743204577968527 0.0037553648068669528 0.1968884120171674
195 senti_sk senti_sk.sentiment_binary_sk em_norm_relaxed 3 3 11184 0.3534513590844063 0.3534513590844063 0.25160944206008584 0.4965128755364807
196 skj1 skj1.skj1_sk em 3 3 300 0.0 0.0 0.0 0.0
197 skj1 skj1.skj1_sk em_conservative 3 3 300 0.03333333333333333 0.03333333333333333 0.02 0.05
198 skj1 skj1.skj1_sk em_relaxed 3 3 300 0.043333333333333335 0.043333333333333335 0.02 0.06
199 sklep sklep.hate_speech_sk:hate-speech em_conservative_norm 3 3 3957 0.7045741723527925 0.7045741723527925 0.6921910538286581 0.7247915087187263
200 sklep sklep.hate_speech_sk:hate-speech em_norm 3 3 3957 0.0 0.0 0.0 0.0
201 sklep sklep.hate_speech_sk:hate-speech em_norm_relaxed 3 3 3957 0.7045741723527925 0.7045741723527925 0.6921910538286581 0.7247915087187263
202 sklep sklep.nli_3_classes_sk:nli em_conservative_norm 3 3 15012 0.03910205169197975 0.03910205169197975 0.001398880895283773 0.09952038369304557
203 sklep sklep.nli_3_classes_sk:nli em_norm 3 3 15012 0.0001332267519317879 0.0001332267519317879 0.0 0.0003996802557953637
204 sklep sklep.nli_3_classes_sk:nli em_norm_relaxed 3 3 15012 0.03930189181987743 0.03930189181987743 0.001398880895283773 0.09972022382094324
205 sklep sklep.nli_binary_sk:rte em_conservative_norm 3 3 4980 0.19738955823293172 0.19738955823293172 0.030120481927710843 0.3301204819277108
206 sklep sklep.nli_binary_sk:rte em_norm 3 3 4980 0.0012048192771084338 0.0012048192771084338 0.0006024096385542169 0.0018072289156626507
207 sklep sklep.nli_binary_sk:rte em_norm_relaxed 3 3 4980 0.19738955823293172 0.19738955823293172 0.030120481927710843 0.3301204819277108
208 sklep sklep.qa_sk:question-answering exact_match_slk_prefix 3 3 28749 0.16372743399770426 0.16372743399770426 0.16143170197224252 0.16518835437754356
209 sklep sklep.qa_sk:question-answering f1_slk 3 3 28749 0.3750415088928484 0.3750415088928484 0.3730151221096053 0.3769392643579436
210 sklep sklep.sentiment_binary_sk:sentiment-analysis em_conservative_norm 3 3 3126 0.4488163787587972 0.4488163787587972 0.118042226487524 0.8800383877159309
211 sklep sklep.sentiment_binary_sk:sentiment-analysis em_norm 3 3 3126 0.1548304542546385 0.1548304542546385 0.0009596928982725527 0.44817658349328215
212 sklep sklep.sentiment_binary_sk:sentiment-analysis em_norm_relaxed 3 3 3126 0.4619321817018554 0.4619321817018554 0.15355086372360843 0.8809980806142035
213 slovakcopa slovakcopa.copa_sk em 3 3 1500 0.0 0.0 0.0 0.0
214 slovakcopa slovakcopa.copa_sk em_conservative 3 3 1500 0.306 0.306 0.084 0.508
215 slovakcopa slovakcopa.copa_sk em_relaxed 3 3 1500 0.4046666666666667 0.4046666666666667 0.086 0.618
216 toxic_sk toxic_sk.toxicity_sk em_conservative_norm 3 3 2643 0.4759742716609913 0.4759742716609913 0.45629965947786605 0.4960272417707151
217 toxic_sk toxic_sk.toxicity_sk em_norm 3 3 2643 0.0026485054861899358 0.0026485054861899358 0.0 0.007945516458569807
218 toxic_sk toxic_sk.toxicity_sk em_norm_relaxed 3 3 2643 0.4759742716609913 0.4759742716609913 0.45629965947786605 0.4960272417707151
219 truthfulqa truthfulqa.mqa_single_correct_sk:mc1 em 3 3 2334 0.0 0.0 0.0 0.0
220 truthfulqa truthfulqa.mqa_single_correct_sk:mc1 em_conservative 3 3 2334 0.011996572407883462 0.011996572407883462 0.007712082262210797 0.016709511568123392
221 truthfulqa truthfulqa.mqa_single_correct_sk:mc1 em_relaxed 3 3 2334 0.011996572407883462 0.011996572407883462 0.007712082262210797 0.016709511568123392

223
aya_300k_lr_experiments/benchmarks/skbench/results/truncation_audit_per_task.csv Normal file
View File

@ -0,0 +1,223 @@
task,sample_count,truncation_values_present,truncation_null_values,samples_with_truncation,truncated_tokens_total,max_truncated_tokens_per_sample,parquet_bytes,source_archive_member,extracted_relative_path
arc.prompt1_mqa_single_correct_sk|0,1169,1169,0,0,0,0,3889368,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt1_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt1_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
arc.prompt2_mqa_single_correct_sk|0,1169,1169,0,0,0,0,3568249,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt2_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt2_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
arc.prompt3_mqa_single_correct_sk|0,1169,1169,0,0,0,0,3818394,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt3_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt3_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
belebele.prompt1_mqa_single_correct_sk|0,900,900,0,0,0,0,5313887,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt1_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt1_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
belebele.prompt2_mqa_single_correct_sk|0,900,900,0,0,0,0,5121317,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt2_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt2_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
belebele.prompt3_mqa_single_correct_sk|0,900,900,0,0,0,0,5297164,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt3_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt3_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
financial_exams.prompt1_completion_sk|0,1334,1334,0,0,0,0,5500287,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt1_completion_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt1_completion_sk%7C0_2026-10-22T21-55-04.268099.parquet
financial_exams.prompt2_completion_sk|0,1334,1334,0,0,0,0,5419376,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt2_completion_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt2_completion_sk%7C0_2026-10-22T21-55-04.268099.parquet
financial_exams.prompt3_completion_sk|0,1334,1334,0,0,0,0,5136557,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt3_completion_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt3_completion_sk%7C0_2026-10-22T21-55-04.268099.parquet
globalmmlu-lite.prompt1_mqa_single_correct_sk|0,400,400,0,0,0,0,1676181,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt1_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt1_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
globalmmlu-lite.prompt2_mqa_single_correct_sk|0,400,400,0,0,0,0,1575910,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt2_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt2_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
globalmmlu-lite.prompt3_mqa_single_correct_sk|0,400,400,0,0,0,0,1652549,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt3_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt3_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0,100,100,0,0,0,0,323573,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aabstract_algebra%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:anatomy|0,134,134,0,0,0,0,431104,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:anatomy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aanatomy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:astronomy|0,152,152,0,0,0,0,548664,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:astronomy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aastronomy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:business_ethics|0,100,100,0,0,0,0,386302,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:business_ethics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Abusiness_ethics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0,259,259,0,0,0,0,840437,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aclinical_knowledge%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:college_biology|0,142,142,0,0,0,0,529694,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_biology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_biology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0,100,100,0,0,0,0,341700,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0,100,100,0,0,0,0,445224,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0,99,99,0,0,0,0,348406,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:college_medicine|0,172,172,0,0,0,0,761319,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_medicine|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_medicine%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:college_physics|0,102,102,0,0,0,0,370418,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:computer_security|0,100,100,0,0,0,0,352239,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:computer_security|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acomputer_security%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0,235,235,0,0,0,0,689127,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aconceptual_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:econometrics|0,112,112,0,0,0,0,469002,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:econometrics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aeconometrics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0,145,145,0,0,0,0,430149,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aelectrical_engineering%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0,373,373,0,0,0,0,1145930,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aelementary_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:formal_logic|0,115,115,0,0,0,0,484074,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:formal_logic|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aformal_logic%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:global_facts|0,100,100,0,0,0,0,301929,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:global_facts|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aglobal_facts%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0,282,282,0,0,0,0,1065950,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_biology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0,198,198,0,0,0,0,695762,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0,99,99,0,0,0,0,446198,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0,135,135,0,0,0,0,1362655,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_european_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0,198,198,0,0,0,0,603781,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_geography%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0,193,193,0,0,0,0,712168,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_government_and_politics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0,387,387,0,0,0,0,1377210,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_macroeconomics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0,266,266,0,0,0,0,855453,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0,217,217,0,0,0,0,801665,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_microeconomics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0,150,150,0,0,0,0,610907,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0,536,536,0,0,0,0,1832499,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_psychology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0,214,214,0,0,0,0,1027287,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_statistics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0,57,57,0,0,0,0,474194,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_us_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0,180,180,0,0,0,0,1764506,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_world_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:human_aging|0,219,219,0,0,0,0,666620,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:human_aging|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahuman_aging%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0,109,109,0,0,0,0,346704,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahuman_sexuality%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:international_law|0,118,118,0,0,0,0,521710,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:international_law|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ainternational_law%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0,107,107,0,0,0,0,382984,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ajurisprudence%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0,158,158,0,0,0,0,572812,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Alogical_fallacies%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:machine_learning|0,112,112,0,0,0,0,422205,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:machine_learning|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amachine_learning%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:management|0,100,100,0,0,0,0,301034,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:management|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amanagement%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:marketing|0,234,234,0,0,0,0,784276,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:marketing|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amarketing%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0,100,100,0,0,0,0,312791,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amedical_genetics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0,779,779,0,0,0,0,2183580,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amiscellaneous%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0,326,326,0,0,0,0,1152897,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amoral_disputes%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0,749,749,0,0,0,0,3250913,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amoral_scenarios%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:nutrition|0,303,303,0,0,0,0,1065386,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:nutrition|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Anutrition%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:philosophy|0,308,308,0,0,0,0,993683,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:philosophy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aphilosophy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:prehistory|0,319,319,0,0,0,0,1058794,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:prehistory|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprehistory%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0,276,276,0,0,0,0,1219526,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_accounting%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:professional_law|0,1171,1171,0,0,0,0,9388720,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_law|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_law%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0,264,264,0,0,0,0,1651766,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_medicine%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0,596,596,0,0,0,0,2228496,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_psychology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:public_relations|0,109,109,0,0,0,0,362397,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:public_relations|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Apublic_relations%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:security_studies|0,222,222,0,0,0,0,1351252,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:security_studies|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Asecurity_studies%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:sociology|0,197,197,0,0,0,0,704760,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:sociology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Asociology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0,99,99,0,0,0,0,343296,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aus_foreign_policy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:virology|0,164,164,0,0,0,0,523727,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:virology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Avirology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt1_mqa_single_correct_sk:world_religions|0,171,171,0,0,0,0,445297,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:world_religions|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aworld_religions%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0,100,100,0,0,0,0,299125,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aabstract_algebra%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:anatomy|0,134,134,0,0,0,0,395509,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:anatomy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aanatomy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:astronomy|0,152,152,0,0,0,0,507008,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:astronomy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aastronomy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:business_ethics|0,100,100,0,0,0,0,360042,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:business_ethics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Abusiness_ethics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0,259,259,0,0,0,0,765932,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aclinical_knowledge%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:college_biology|0,142,142,0,0,0,0,492071,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_biology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_biology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0,100,100,0,0,0,0,315493,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0,100,100,0,0,0,0,419754,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0,99,99,0,0,0,0,324232,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:college_medicine|0,172,172,0,0,0,0,715268,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_medicine|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_medicine%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:college_physics|0,102,102,0,0,0,0,342396,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:computer_security|0,100,100,0,0,0,0,326674,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:computer_security|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acomputer_security%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0,235,235,0,0,0,0,616251,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aconceptual_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:econometrics|0,112,112,0,0,0,0,440285,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:econometrics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aeconometrics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0,145,145,0,0,0,0,388061,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aelectrical_engineering%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0,373,373,0,0,0,0,1056284,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aelementary_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:formal_logic|0,115,115,0,0,0,0,457354,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:formal_logic|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aformal_logic%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:global_facts|0,100,100,0,0,0,0,275533,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:global_facts|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aglobal_facts%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0,282,282,0,0,0,0,987587,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_biology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0,198,198,0,0,0,0,644161,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0,99,99,0,0,0,0,422516,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0,135,135,0,0,0,0,1333703,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_european_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0,198,198,0,0,0,0,552814,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_geography%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0,193,193,0,0,0,0,667220,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_government_and_politics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0,387,387,0,0,0,0,1275788,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_macroeconomics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0,266,266,0,0,0,0,793238,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0,217,217,0,0,0,0,744021,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_microeconomics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0,150,150,0,0,0,0,573327,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0,536,536,0,0,0,0,1680422,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_psychology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0,214,214,0,0,0,0,977504,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_statistics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0,57,57,0,0,0,0,460045,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_us_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0,180,180,0,0,0,0,1723713,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_world_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:human_aging|0,219,219,0,0,0,0,604480,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:human_aging|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahuman_aging%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0,109,109,0,0,0,0,314874,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahuman_sexuality%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:international_law|0,118,118,0,0,0,0,496660,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:international_law|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ainternational_law%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0,107,107,0,0,0,0,356545,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ajurisprudence%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0,158,158,0,0,0,0,528533,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Alogical_fallacies%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:machine_learning|0,112,112,0,0,0,0,393833,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:machine_learning|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amachine_learning%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:management|0,100,100,0,0,0,0,274996,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:management|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amanagement%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:marketing|0,234,234,0,0,0,0,718284,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:marketing|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amarketing%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0,100,100,0,0,0,0,286172,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amedical_genetics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0,779,779,0,0,0,0,1979677,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amiscellaneous%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0,326,326,0,0,0,0,1065802,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amoral_disputes%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0,749,749,0,0,0,0,3013671,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amoral_scenarios%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:nutrition|0,303,303,0,0,0,0,991571,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:nutrition|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Anutrition%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:philosophy|0,308,308,0,0,0,0,906013,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:philosophy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aphilosophy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:prehistory|0,319,319,0,0,0,0,978440,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:prehistory|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprehistory%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0,276,276,0,0,0,0,1155027,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_accounting%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:professional_law|0,1171,1171,0,0,0,0,9115776,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_law|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_law%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0,264,264,0,0,0,0,1578359,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_medicine%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0,596,596,0,0,0,0,2077306,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_psychology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:public_relations|0,109,109,0,0,0,0,333951,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:public_relations|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Apublic_relations%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:security_studies|0,222,222,0,0,0,0,1301843,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:security_studies|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Asecurity_studies%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:sociology|0,197,197,0,0,0,0,658665,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:sociology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Asociology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0,99,99,0,0,0,0,318054,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aus_foreign_policy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:virology|0,164,164,0,0,0,0,479394,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:virology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Avirology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt2_mqa_single_correct_sk:world_religions|0,171,171,0,0,0,0,403615,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:world_religions|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aworld_religions%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0,100,100,0,0,0,0,317226,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aabstract_algebra%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:anatomy|0,134,134,0,0,0,0,424850,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:anatomy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aanatomy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:astronomy|0,152,152,0,0,0,0,538528,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:astronomy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aastronomy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:business_ethics|0,100,100,0,0,0,0,382738,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:business_ethics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Abusiness_ethics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0,259,259,0,0,0,0,822154,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aclinical_knowledge%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:college_biology|0,142,142,0,0,0,0,519029,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_biology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_biology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0,100,100,0,0,0,0,335057,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0,100,100,0,0,0,0,439058,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0,99,99,0,0,0,0,341549,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:college_medicine|0,172,172,0,0,0,0,753681,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_medicine|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_medicine%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:college_physics|0,102,102,0,0,0,0,363201,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:computer_security|0,100,100,0,0,0,0,348446,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:computer_security|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acomputer_security%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0,235,235,0,0,0,0,665228,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aconceptual_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:econometrics|0,112,112,0,0,0,0,463058,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:econometrics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aeconometrics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0,145,145,0,0,0,0,419331,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aelectrical_engineering%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0,373,373,0,0,0,0,1124101,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aelementary_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:formal_logic|0,115,115,0,0,0,0,479084,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:formal_logic|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aformal_logic%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:global_facts|0,100,100,0,0,0,0,294579,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:global_facts|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aglobal_facts%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0,282,282,0,0,0,0,1045929,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_biology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0,198,198,0,0,0,0,685593,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0,99,99,0,0,0,0,443316,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0,135,135,0,0,0,0,1361812,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_european_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0,198,198,0,0,0,0,591628,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_geography%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0,193,193,0,0,0,0,704619,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_government_and_politics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0,387,387,0,0,0,0,1350736,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_macroeconomics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0,266,266,0,0,0,0,836445,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0,217,217,0,0,0,0,785746,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_microeconomics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0,150,150,0,0,0,0,603087,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_physics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0,536,536,0,0,0,0,1801692,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_psychology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0,214,214,0,0,0,0,1021159,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_statistics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0,57,57,0,0,0,0,474029,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_us_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0,180,180,0,0,0,0,1759984,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_world_history%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:human_aging|0,219,219,0,0,0,0,651878,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:human_aging|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahuman_aging%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0,109,109,0,0,0,0,337916,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahuman_sexuality%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:international_law|0,118,118,0,0,0,0,515975,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:international_law|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ainternational_law%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0,107,107,0,0,0,0,378748,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ajurisprudence%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0,158,158,0,0,0,0,564282,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Alogical_fallacies%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:machine_learning|0,112,112,0,0,0,0,416217,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:machine_learning|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amachine_learning%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:management|0,100,100,0,0,0,0,293612,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:management|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amanagement%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:marketing|0,234,234,0,0,0,0,771738,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:marketing|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amarketing%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0,100,100,0,0,0,0,307187,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amedical_genetics%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0,779,779,0,0,0,0,2137196,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amiscellaneous%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0,326,326,0,0,0,0,1133086,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amoral_disputes%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0,749,749,0,0,0,0,3272242,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amoral_scenarios%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:nutrition|0,303,303,0,0,0,0,1050303,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:nutrition|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Anutrition%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:philosophy|0,308,308,0,0,0,0,973131,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:philosophy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aphilosophy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:prehistory|0,319,319,0,0,0,0,1039922,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:prehistory|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprehistory%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0,276,276,0,0,0,0,1209237,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_accounting%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:professional_law|0,1171,1171,0,0,0,0,9378846,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_law|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_law%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0,264,264,0,0,0,0,1649655,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_medicine%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0,596,596,0,0,0,0,2205361,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_psychology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:public_relations|0,109,109,0,0,0,0,355136,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:public_relations|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Apublic_relations%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:security_studies|0,222,222,0,0,0,0,1347999,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:security_studies|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Asecurity_studies%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:sociology|0,197,197,0,0,0,0,692741,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:sociology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Asociology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0,99,99,0,0,0,0,336616,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aus_foreign_policy%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:virology|0,164,164,0,0,0,0,511413,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:virology|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Avirology%7C0_2026-10-22T21-55-04.268099.parquet
mmlu.prompt3_mqa_single_correct_sk:world_religions|0,171,171,0,0,0,0,434303,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:world_religions|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aworld_religions%7C0_2026-10-22T21-55-04.268099.parquet
movie_review.prompt1_sentiment_binary_sk|0,2008,2008,0,0,0,0,6995385,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt1_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt1_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
movie_review.prompt2_sentiment_binary_sk|0,2008,2008,0,0,0,0,7383490,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt2_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt2_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
movie_review.prompt3_sentiment_binary_sk|0,2008,2008,0,0,0,0,6729072,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt3_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt3_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
multi_wiki_qa.prompt1_qa_sk|0,4997,4997,0,0,0,0,127233608,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet
multi_wiki_qa.prompt2_qa_sk|0,4997,4997,0,0,0,0,127542209,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet
multi_wiki_qa.prompt3_qa_sk|0,4997,4997,0,0,0,0,125437876,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet
senti_sk.prompt1_sentiment_3_class_sk|0,4677,4677,0,0,0,0,8652017,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_3_class_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_3_class_sk%7C0_2026-10-22T21-55-04.268099.parquet
senti_sk.prompt1_sentiment_binary_sk|0,3728,3728,0,0,0,0,6544907,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
senti_sk.prompt2_sentiment_3_class_sk|0,4677,4677,0,0,0,0,9896541,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_3_class_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_3_class_sk%7C0_2026-10-22T21-55-04.268099.parquet
senti_sk.prompt2_sentiment_binary_sk|0,3728,3728,0,0,0,0,7247865,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
senti_sk.prompt3_sentiment_3_class_sk|0,4677,4677,0,0,0,0,7883745,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_3_class_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_3_class_sk%7C0_2026-10-22T21-55-04.268099.parquet
senti_sk.prompt3_sentiment_binary_sk|0,3728,3728,0,0,0,0,6040540,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
skj1.prompt1_skj1_sk|0,100,100,0,0,0,0,320443,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt1_skj1_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt1_skj1_sk%7C0_2026-10-22T21-55-04.268099.parquet
skj1.prompt2_skj1_sk|0,100,100,0,0,0,0,301884,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt2_skj1_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt2_skj1_sk%7C0_2026-10-22T21-55-04.268099.parquet
skj1.prompt3_skj1_sk|0,100,100,0,0,0,0,305004,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt3_skj1_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt3_skj1_sk%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt1_hate_speech_sk:hate-speech|0,1319,1319,0,0,0,0,2492816,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_hate_speech_sk:hate-speech|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_hate_speech_sk%3Ahate-speech%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt1_nli_3_classes_sk:nli|0,5004,5004,0,0,0,0,14496139,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_3_classes_sk:nli|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_3_classes_sk%3Anli%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt1_nli_binary_sk:rte|0,1660,1660,0,0,0,0,6481644,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_binary_sk:rte|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_binary_sk%3Arte%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt1_qa_sk:question-answering|0,9583,9583,0,0,0,0,66012897,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0,1042,1042,0,0,0,0,1993964,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_sentiment_binary_sk%3Asentiment-analysis%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt2_hate_speech_sk:hate-speech|0,1319,1319,0,0,0,0,2209601,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_hate_speech_sk:hate-speech|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_hate_speech_sk%3Ahate-speech%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt2_nli_3_classes_sk:nli|0,5004,5004,0,0,0,0,18103600,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_3_classes_sk:nli|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_3_classes_sk%3Anli%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt2_nli_binary_sk:rte|0,1660,1660,0,0,0,0,5814265,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_binary_sk:rte|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_binary_sk%3Arte%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt2_qa_sk:question-answering|0,9583,9583,0,0,0,0,66791884,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0,1042,1042,0,0,0,0,2210267,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_sentiment_binary_sk%3Asentiment-analysis%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt3_hate_speech_sk:hate-speech|0,1319,1319,0,0,0,0,2647881,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_hate_speech_sk:hate-speech|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_hate_speech_sk%3Ahate-speech%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt3_nli_3_classes_sk:nli|0,5004,5004,0,0,0,0,14240432,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_3_classes_sk:nli|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_3_classes_sk%3Anli%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt3_nli_binary_sk:rte|0,1660,1660,0,0,0,0,6948585,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_binary_sk:rte|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_binary_sk%3Arte%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt3_qa_sk:question-answering|0,9583,9583,0,0,0,0,64169583,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet
sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0,1042,1042,0,0,0,0,1868158,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_sentiment_binary_sk%3Asentiment-analysis%7C0_2026-10-22T21-55-04.268099.parquet
slovakcopa.prompt1_copa_sk|0,500,500,0,0,0,0,1286131,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt1_copa_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt1_copa_sk%7C0_2026-10-22T21-55-04.268099.parquet
slovakcopa.prompt2_copa_sk|0,500,500,0,0,0,0,1319142,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt2_copa_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt2_copa_sk%7C0_2026-10-22T21-55-04.268099.parquet
slovakcopa.prompt3_copa_sk|0,500,500,0,0,0,0,1277486,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt3_copa_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt3_copa_sk%7C0_2026-10-22T21-55-04.268099.parquet
toxic_sk.prompt1_toxicity_sk|0,881,881,0,0,0,0,1374600,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt1_toxicity_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt1_toxicity_sk%7C0_2026-10-22T21-55-04.268099.parquet
toxic_sk.prompt2_toxicity_sk|0,881,881,0,0,0,0,1269485,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt2_toxicity_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt2_toxicity_sk%7C0_2026-10-22T21-55-04.268099.parquet
toxic_sk.prompt3_toxicity_sk|0,881,881,0,0,0,0,1663351,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt3_toxicity_sk|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt3_toxicity_sk%7C0_2026-10-22T21-55-04.268099.parquet
truthfulqa.prompt1_mqa_single_correct_sk:mc1|0,778,778,0,0,0,0,2939787,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt1_mqa_single_correct_sk:mc1|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt1_mqa_single_correct_sk%3Amc1%7C0_2026-10-22T21-55-04.268099.parquet
truthfulqa.prompt2_mqa_single_correct_sk:mc1|0,778,778,0,0,0,0,2746370,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt2_mqa_single_correct_sk:mc1|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt2_mqa_single_correct_sk%3Amc1%7C0_2026-10-22T21-55-04.268099.parquet
truthfulqa.prompt3_mqa_single_correct_sk:mc1|0,778,778,0,0,0,0,2882952,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt3_mqa_single_correct_sk:mc1|0_2026-10-22T21-55-04.268099.parquet,skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt3_mqa_single_correct_sk%3Amc1%7C0_2026-10-22T21-55-04.268099.parquet
1 task sample_count truncation_values_present truncation_null_values samples_with_truncation truncated_tokens_total max_truncated_tokens_per_sample parquet_bytes source_archive_member extracted_relative_path
2 arc.prompt1_mqa_single_correct_sk|0 1169 1169 0 0 0 0 3889368 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt1_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt1_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
3 arc.prompt2_mqa_single_correct_sk|0 1169 1169 0 0 0 0 3568249 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt2_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt2_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
4 arc.prompt3_mqa_single_correct_sk|0 1169 1169 0 0 0 0 3818394 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt3_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_arc.prompt3_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
5 belebele.prompt1_mqa_single_correct_sk|0 900 900 0 0 0 0 5313887 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt1_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt1_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
6 belebele.prompt2_mqa_single_correct_sk|0 900 900 0 0 0 0 5121317 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt2_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt2_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
7 belebele.prompt3_mqa_single_correct_sk|0 900 900 0 0 0 0 5297164 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt3_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_belebele.prompt3_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
8 financial_exams.prompt1_completion_sk|0 1334 1334 0 0 0 0 5500287 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt1_completion_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt1_completion_sk%7C0_2026-10-22T21-55-04.268099.parquet
9 financial_exams.prompt2_completion_sk|0 1334 1334 0 0 0 0 5419376 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt2_completion_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt2_completion_sk%7C0_2026-10-22T21-55-04.268099.parquet
10 financial_exams.prompt3_completion_sk|0 1334 1334 0 0 0 0 5136557 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt3_completion_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_financial_exams.prompt3_completion_sk%7C0_2026-10-22T21-55-04.268099.parquet
11 globalmmlu-lite.prompt1_mqa_single_correct_sk|0 400 400 0 0 0 0 1676181 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt1_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt1_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
12 globalmmlu-lite.prompt2_mqa_single_correct_sk|0 400 400 0 0 0 0 1575910 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt2_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt2_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
13 globalmmlu-lite.prompt3_mqa_single_correct_sk|0 400 400 0 0 0 0 1652549 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt3_mqa_single_correct_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_globalmmlu-lite.prompt3_mqa_single_correct_sk%7C0_2026-10-22T21-55-04.268099.parquet
14 mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0 100 100 0 0 0 0 323573 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aabstract_algebra%7C0_2026-10-22T21-55-04.268099.parquet
15 mmlu.prompt1_mqa_single_correct_sk:anatomy|0 134 134 0 0 0 0 431104 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:anatomy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aanatomy%7C0_2026-10-22T21-55-04.268099.parquet
16 mmlu.prompt1_mqa_single_correct_sk:astronomy|0 152 152 0 0 0 0 548664 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:astronomy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aastronomy%7C0_2026-10-22T21-55-04.268099.parquet
17 mmlu.prompt1_mqa_single_correct_sk:business_ethics|0 100 100 0 0 0 0 386302 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:business_ethics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Abusiness_ethics%7C0_2026-10-22T21-55-04.268099.parquet
18 mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0 259 259 0 0 0 0 840437 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aclinical_knowledge%7C0_2026-10-22T21-55-04.268099.parquet
19 mmlu.prompt1_mqa_single_correct_sk:college_biology|0 142 142 0 0 0 0 529694 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_biology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_biology%7C0_2026-10-22T21-55-04.268099.parquet
20 mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0 100 100 0 0 0 0 341700 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
21 mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0 100 100 0 0 0 0 445224 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
22 mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0 99 99 0 0 0 0 348406 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
23 mmlu.prompt1_mqa_single_correct_sk:college_medicine|0 172 172 0 0 0 0 761319 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_medicine|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_medicine%7C0_2026-10-22T21-55-04.268099.parquet
24 mmlu.prompt1_mqa_single_correct_sk:college_physics|0 102 102 0 0 0 0 370418 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:college_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acollege_physics%7C0_2026-10-22T21-55-04.268099.parquet
25 mmlu.prompt1_mqa_single_correct_sk:computer_security|0 100 100 0 0 0 0 352239 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:computer_security|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Acomputer_security%7C0_2026-10-22T21-55-04.268099.parquet
26 mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0 235 235 0 0 0 0 689127 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aconceptual_physics%7C0_2026-10-22T21-55-04.268099.parquet
27 mmlu.prompt1_mqa_single_correct_sk:econometrics|0 112 112 0 0 0 0 469002 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:econometrics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aeconometrics%7C0_2026-10-22T21-55-04.268099.parquet
28 mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0 145 145 0 0 0 0 430149 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aelectrical_engineering%7C0_2026-10-22T21-55-04.268099.parquet
29 mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0 373 373 0 0 0 0 1145930 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aelementary_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
30 mmlu.prompt1_mqa_single_correct_sk:formal_logic|0 115 115 0 0 0 0 484074 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:formal_logic|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aformal_logic%7C0_2026-10-22T21-55-04.268099.parquet
31 mmlu.prompt1_mqa_single_correct_sk:global_facts|0 100 100 0 0 0 0 301929 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:global_facts|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aglobal_facts%7C0_2026-10-22T21-55-04.268099.parquet
32 mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0 282 282 0 0 0 0 1065950 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_biology%7C0_2026-10-22T21-55-04.268099.parquet
33 mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0 198 198 0 0 0 0 695762 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
34 mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0 99 99 0 0 0 0 446198 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
35 mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0 135 135 0 0 0 0 1362655 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_european_history%7C0_2026-10-22T21-55-04.268099.parquet
36 mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0 198 198 0 0 0 0 603781 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_geography%7C0_2026-10-22T21-55-04.268099.parquet
37 mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0 193 193 0 0 0 0 712168 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_government_and_politics%7C0_2026-10-22T21-55-04.268099.parquet
38 mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0 387 387 0 0 0 0 1377210 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_macroeconomics%7C0_2026-10-22T21-55-04.268099.parquet
39 mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0 266 266 0 0 0 0 855453 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
40 mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0 217 217 0 0 0 0 801665 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_microeconomics%7C0_2026-10-22T21-55-04.268099.parquet
41 mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0 150 150 0 0 0 0 610907 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_physics%7C0_2026-10-22T21-55-04.268099.parquet
42 mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0 536 536 0 0 0 0 1832499 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_psychology%7C0_2026-10-22T21-55-04.268099.parquet
43 mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0 214 214 0 0 0 0 1027287 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_statistics%7C0_2026-10-22T21-55-04.268099.parquet
44 mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0 57 57 0 0 0 0 474194 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_us_history%7C0_2026-10-22T21-55-04.268099.parquet
45 mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0 180 180 0 0 0 0 1764506 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahigh_school_world_history%7C0_2026-10-22T21-55-04.268099.parquet
46 mmlu.prompt1_mqa_single_correct_sk:human_aging|0 219 219 0 0 0 0 666620 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:human_aging|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahuman_aging%7C0_2026-10-22T21-55-04.268099.parquet
47 mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0 109 109 0 0 0 0 346704 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ahuman_sexuality%7C0_2026-10-22T21-55-04.268099.parquet
48 mmlu.prompt1_mqa_single_correct_sk:international_law|0 118 118 0 0 0 0 521710 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:international_law|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ainternational_law%7C0_2026-10-22T21-55-04.268099.parquet
49 mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0 107 107 0 0 0 0 382984 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Ajurisprudence%7C0_2026-10-22T21-55-04.268099.parquet
50 mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0 158 158 0 0 0 0 572812 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Alogical_fallacies%7C0_2026-10-22T21-55-04.268099.parquet
51 mmlu.prompt1_mqa_single_correct_sk:machine_learning|0 112 112 0 0 0 0 422205 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:machine_learning|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amachine_learning%7C0_2026-10-22T21-55-04.268099.parquet
52 mmlu.prompt1_mqa_single_correct_sk:management|0 100 100 0 0 0 0 301034 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:management|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amanagement%7C0_2026-10-22T21-55-04.268099.parquet
53 mmlu.prompt1_mqa_single_correct_sk:marketing|0 234 234 0 0 0 0 784276 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:marketing|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amarketing%7C0_2026-10-22T21-55-04.268099.parquet
54 mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0 100 100 0 0 0 0 312791 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amedical_genetics%7C0_2026-10-22T21-55-04.268099.parquet
55 mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0 779 779 0 0 0 0 2183580 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amiscellaneous%7C0_2026-10-22T21-55-04.268099.parquet
56 mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0 326 326 0 0 0 0 1152897 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amoral_disputes%7C0_2026-10-22T21-55-04.268099.parquet
57 mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0 749 749 0 0 0 0 3250913 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Amoral_scenarios%7C0_2026-10-22T21-55-04.268099.parquet
58 mmlu.prompt1_mqa_single_correct_sk:nutrition|0 303 303 0 0 0 0 1065386 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:nutrition|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Anutrition%7C0_2026-10-22T21-55-04.268099.parquet
59 mmlu.prompt1_mqa_single_correct_sk:philosophy|0 308 308 0 0 0 0 993683 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:philosophy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aphilosophy%7C0_2026-10-22T21-55-04.268099.parquet
60 mmlu.prompt1_mqa_single_correct_sk:prehistory|0 319 319 0 0 0 0 1058794 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:prehistory|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprehistory%7C0_2026-10-22T21-55-04.268099.parquet
61 mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0 276 276 0 0 0 0 1219526 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_accounting%7C0_2026-10-22T21-55-04.268099.parquet
62 mmlu.prompt1_mqa_single_correct_sk:professional_law|0 1171 1171 0 0 0 0 9388720 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_law|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_law%7C0_2026-10-22T21-55-04.268099.parquet
63 mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0 264 264 0 0 0 0 1651766 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_medicine%7C0_2026-10-22T21-55-04.268099.parquet
64 mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0 596 596 0 0 0 0 2228496 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aprofessional_psychology%7C0_2026-10-22T21-55-04.268099.parquet
65 mmlu.prompt1_mqa_single_correct_sk:public_relations|0 109 109 0 0 0 0 362397 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:public_relations|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Apublic_relations%7C0_2026-10-22T21-55-04.268099.parquet
66 mmlu.prompt1_mqa_single_correct_sk:security_studies|0 222 222 0 0 0 0 1351252 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:security_studies|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Asecurity_studies%7C0_2026-10-22T21-55-04.268099.parquet
67 mmlu.prompt1_mqa_single_correct_sk:sociology|0 197 197 0 0 0 0 704760 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:sociology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Asociology%7C0_2026-10-22T21-55-04.268099.parquet
68 mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0 99 99 0 0 0 0 343296 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aus_foreign_policy%7C0_2026-10-22T21-55-04.268099.parquet
69 mmlu.prompt1_mqa_single_correct_sk:virology|0 164 164 0 0 0 0 523727 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:virology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Avirology%7C0_2026-10-22T21-55-04.268099.parquet
70 mmlu.prompt1_mqa_single_correct_sk:world_religions|0 171 171 0 0 0 0 445297 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk:world_religions|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt1_mqa_single_correct_sk%3Aworld_religions%7C0_2026-10-22T21-55-04.268099.parquet
71 mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0 100 100 0 0 0 0 299125 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aabstract_algebra%7C0_2026-10-22T21-55-04.268099.parquet
72 mmlu.prompt2_mqa_single_correct_sk:anatomy|0 134 134 0 0 0 0 395509 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:anatomy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aanatomy%7C0_2026-10-22T21-55-04.268099.parquet
73 mmlu.prompt2_mqa_single_correct_sk:astronomy|0 152 152 0 0 0 0 507008 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:astronomy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aastronomy%7C0_2026-10-22T21-55-04.268099.parquet
74 mmlu.prompt2_mqa_single_correct_sk:business_ethics|0 100 100 0 0 0 0 360042 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:business_ethics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Abusiness_ethics%7C0_2026-10-22T21-55-04.268099.parquet
75 mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0 259 259 0 0 0 0 765932 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aclinical_knowledge%7C0_2026-10-22T21-55-04.268099.parquet
76 mmlu.prompt2_mqa_single_correct_sk:college_biology|0 142 142 0 0 0 0 492071 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_biology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_biology%7C0_2026-10-22T21-55-04.268099.parquet
77 mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0 100 100 0 0 0 0 315493 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
78 mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0 100 100 0 0 0 0 419754 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
79 mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0 99 99 0 0 0 0 324232 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
80 mmlu.prompt2_mqa_single_correct_sk:college_medicine|0 172 172 0 0 0 0 715268 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_medicine|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_medicine%7C0_2026-10-22T21-55-04.268099.parquet
81 mmlu.prompt2_mqa_single_correct_sk:college_physics|0 102 102 0 0 0 0 342396 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:college_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acollege_physics%7C0_2026-10-22T21-55-04.268099.parquet
82 mmlu.prompt2_mqa_single_correct_sk:computer_security|0 100 100 0 0 0 0 326674 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:computer_security|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Acomputer_security%7C0_2026-10-22T21-55-04.268099.parquet
83 mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0 235 235 0 0 0 0 616251 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aconceptual_physics%7C0_2026-10-22T21-55-04.268099.parquet
84 mmlu.prompt2_mqa_single_correct_sk:econometrics|0 112 112 0 0 0 0 440285 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:econometrics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aeconometrics%7C0_2026-10-22T21-55-04.268099.parquet
85 mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0 145 145 0 0 0 0 388061 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aelectrical_engineering%7C0_2026-10-22T21-55-04.268099.parquet
86 mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0 373 373 0 0 0 0 1056284 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aelementary_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
87 mmlu.prompt2_mqa_single_correct_sk:formal_logic|0 115 115 0 0 0 0 457354 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:formal_logic|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aformal_logic%7C0_2026-10-22T21-55-04.268099.parquet
88 mmlu.prompt2_mqa_single_correct_sk:global_facts|0 100 100 0 0 0 0 275533 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:global_facts|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aglobal_facts%7C0_2026-10-22T21-55-04.268099.parquet
89 mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0 282 282 0 0 0 0 987587 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_biology%7C0_2026-10-22T21-55-04.268099.parquet
90 mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0 198 198 0 0 0 0 644161 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
91 mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0 99 99 0 0 0 0 422516 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
92 mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0 135 135 0 0 0 0 1333703 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_european_history%7C0_2026-10-22T21-55-04.268099.parquet
93 mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0 198 198 0 0 0 0 552814 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_geography%7C0_2026-10-22T21-55-04.268099.parquet
94 mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0 193 193 0 0 0 0 667220 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_government_and_politics%7C0_2026-10-22T21-55-04.268099.parquet
95 mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0 387 387 0 0 0 0 1275788 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_macroeconomics%7C0_2026-10-22T21-55-04.268099.parquet
96 mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0 266 266 0 0 0 0 793238 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
97 mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0 217 217 0 0 0 0 744021 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_microeconomics%7C0_2026-10-22T21-55-04.268099.parquet
98 mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0 150 150 0 0 0 0 573327 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_physics%7C0_2026-10-22T21-55-04.268099.parquet
99 mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0 536 536 0 0 0 0 1680422 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_psychology%7C0_2026-10-22T21-55-04.268099.parquet
100 mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0 214 214 0 0 0 0 977504 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_statistics%7C0_2026-10-22T21-55-04.268099.parquet
101 mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0 57 57 0 0 0 0 460045 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_us_history%7C0_2026-10-22T21-55-04.268099.parquet
102 mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0 180 180 0 0 0 0 1723713 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahigh_school_world_history%7C0_2026-10-22T21-55-04.268099.parquet
103 mmlu.prompt2_mqa_single_correct_sk:human_aging|0 219 219 0 0 0 0 604480 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:human_aging|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahuman_aging%7C0_2026-10-22T21-55-04.268099.parquet
104 mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0 109 109 0 0 0 0 314874 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ahuman_sexuality%7C0_2026-10-22T21-55-04.268099.parquet
105 mmlu.prompt2_mqa_single_correct_sk:international_law|0 118 118 0 0 0 0 496660 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:international_law|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ainternational_law%7C0_2026-10-22T21-55-04.268099.parquet
106 mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0 107 107 0 0 0 0 356545 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Ajurisprudence%7C0_2026-10-22T21-55-04.268099.parquet
107 mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0 158 158 0 0 0 0 528533 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Alogical_fallacies%7C0_2026-10-22T21-55-04.268099.parquet
108 mmlu.prompt2_mqa_single_correct_sk:machine_learning|0 112 112 0 0 0 0 393833 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:machine_learning|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amachine_learning%7C0_2026-10-22T21-55-04.268099.parquet
109 mmlu.prompt2_mqa_single_correct_sk:management|0 100 100 0 0 0 0 274996 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:management|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amanagement%7C0_2026-10-22T21-55-04.268099.parquet
110 mmlu.prompt2_mqa_single_correct_sk:marketing|0 234 234 0 0 0 0 718284 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:marketing|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amarketing%7C0_2026-10-22T21-55-04.268099.parquet
111 mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0 100 100 0 0 0 0 286172 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amedical_genetics%7C0_2026-10-22T21-55-04.268099.parquet
112 mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0 779 779 0 0 0 0 1979677 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amiscellaneous%7C0_2026-10-22T21-55-04.268099.parquet
113 mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0 326 326 0 0 0 0 1065802 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amoral_disputes%7C0_2026-10-22T21-55-04.268099.parquet
114 mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0 749 749 0 0 0 0 3013671 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Amoral_scenarios%7C0_2026-10-22T21-55-04.268099.parquet
115 mmlu.prompt2_mqa_single_correct_sk:nutrition|0 303 303 0 0 0 0 991571 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:nutrition|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Anutrition%7C0_2026-10-22T21-55-04.268099.parquet
116 mmlu.prompt2_mqa_single_correct_sk:philosophy|0 308 308 0 0 0 0 906013 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:philosophy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aphilosophy%7C0_2026-10-22T21-55-04.268099.parquet
117 mmlu.prompt2_mqa_single_correct_sk:prehistory|0 319 319 0 0 0 0 978440 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:prehistory|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprehistory%7C0_2026-10-22T21-55-04.268099.parquet
118 mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0 276 276 0 0 0 0 1155027 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_accounting%7C0_2026-10-22T21-55-04.268099.parquet
119 mmlu.prompt2_mqa_single_correct_sk:professional_law|0 1171 1171 0 0 0 0 9115776 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_law|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_law%7C0_2026-10-22T21-55-04.268099.parquet
120 mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0 264 264 0 0 0 0 1578359 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_medicine%7C0_2026-10-22T21-55-04.268099.parquet
121 mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0 596 596 0 0 0 0 2077306 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aprofessional_psychology%7C0_2026-10-22T21-55-04.268099.parquet
122 mmlu.prompt2_mqa_single_correct_sk:public_relations|0 109 109 0 0 0 0 333951 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:public_relations|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Apublic_relations%7C0_2026-10-22T21-55-04.268099.parquet
123 mmlu.prompt2_mqa_single_correct_sk:security_studies|0 222 222 0 0 0 0 1301843 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:security_studies|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Asecurity_studies%7C0_2026-10-22T21-55-04.268099.parquet
124 mmlu.prompt2_mqa_single_correct_sk:sociology|0 197 197 0 0 0 0 658665 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:sociology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Asociology%7C0_2026-10-22T21-55-04.268099.parquet
125 mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0 99 99 0 0 0 0 318054 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aus_foreign_policy%7C0_2026-10-22T21-55-04.268099.parquet
126 mmlu.prompt2_mqa_single_correct_sk:virology|0 164 164 0 0 0 0 479394 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:virology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Avirology%7C0_2026-10-22T21-55-04.268099.parquet
127 mmlu.prompt2_mqa_single_correct_sk:world_religions|0 171 171 0 0 0 0 403615 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk:world_religions|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt2_mqa_single_correct_sk%3Aworld_religions%7C0_2026-10-22T21-55-04.268099.parquet
128 mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0 100 100 0 0 0 0 317226 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aabstract_algebra%7C0_2026-10-22T21-55-04.268099.parquet
129 mmlu.prompt3_mqa_single_correct_sk:anatomy|0 134 134 0 0 0 0 424850 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:anatomy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aanatomy%7C0_2026-10-22T21-55-04.268099.parquet
130 mmlu.prompt3_mqa_single_correct_sk:astronomy|0 152 152 0 0 0 0 538528 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:astronomy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aastronomy%7C0_2026-10-22T21-55-04.268099.parquet
131 mmlu.prompt3_mqa_single_correct_sk:business_ethics|0 100 100 0 0 0 0 382738 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:business_ethics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Abusiness_ethics%7C0_2026-10-22T21-55-04.268099.parquet
132 mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0 259 259 0 0 0 0 822154 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aclinical_knowledge%7C0_2026-10-22T21-55-04.268099.parquet
133 mmlu.prompt3_mqa_single_correct_sk:college_biology|0 142 142 0 0 0 0 519029 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_biology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_biology%7C0_2026-10-22T21-55-04.268099.parquet
134 mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0 100 100 0 0 0 0 335057 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
135 mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0 100 100 0 0 0 0 439058 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
136 mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0 99 99 0 0 0 0 341549 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
137 mmlu.prompt3_mqa_single_correct_sk:college_medicine|0 172 172 0 0 0 0 753681 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_medicine|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_medicine%7C0_2026-10-22T21-55-04.268099.parquet
138 mmlu.prompt3_mqa_single_correct_sk:college_physics|0 102 102 0 0 0 0 363201 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:college_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acollege_physics%7C0_2026-10-22T21-55-04.268099.parquet
139 mmlu.prompt3_mqa_single_correct_sk:computer_security|0 100 100 0 0 0 0 348446 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:computer_security|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Acomputer_security%7C0_2026-10-22T21-55-04.268099.parquet
140 mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0 235 235 0 0 0 0 665228 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aconceptual_physics%7C0_2026-10-22T21-55-04.268099.parquet
141 mmlu.prompt3_mqa_single_correct_sk:econometrics|0 112 112 0 0 0 0 463058 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:econometrics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aeconometrics%7C0_2026-10-22T21-55-04.268099.parquet
142 mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0 145 145 0 0 0 0 419331 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aelectrical_engineering%7C0_2026-10-22T21-55-04.268099.parquet
143 mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0 373 373 0 0 0 0 1124101 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aelementary_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
144 mmlu.prompt3_mqa_single_correct_sk:formal_logic|0 115 115 0 0 0 0 479084 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:formal_logic|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aformal_logic%7C0_2026-10-22T21-55-04.268099.parquet
145 mmlu.prompt3_mqa_single_correct_sk:global_facts|0 100 100 0 0 0 0 294579 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:global_facts|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aglobal_facts%7C0_2026-10-22T21-55-04.268099.parquet
146 mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0 282 282 0 0 0 0 1045929 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_biology%7C0_2026-10-22T21-55-04.268099.parquet
147 mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0 198 198 0 0 0 0 685593 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_chemistry%7C0_2026-10-22T21-55-04.268099.parquet
148 mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0 99 99 0 0 0 0 443316 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_computer_science%7C0_2026-10-22T21-55-04.268099.parquet
149 mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0 135 135 0 0 0 0 1361812 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_european_history%7C0_2026-10-22T21-55-04.268099.parquet
150 mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0 198 198 0 0 0 0 591628 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_geography%7C0_2026-10-22T21-55-04.268099.parquet
151 mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0 193 193 0 0 0 0 704619 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_government_and_politics%7C0_2026-10-22T21-55-04.268099.parquet
152 mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0 387 387 0 0 0 0 1350736 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_macroeconomics%7C0_2026-10-22T21-55-04.268099.parquet
153 mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0 266 266 0 0 0 0 836445 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_mathematics%7C0_2026-10-22T21-55-04.268099.parquet
154 mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0 217 217 0 0 0 0 785746 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_microeconomics%7C0_2026-10-22T21-55-04.268099.parquet
155 mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0 150 150 0 0 0 0 603087 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_physics%7C0_2026-10-22T21-55-04.268099.parquet
156 mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0 536 536 0 0 0 0 1801692 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_psychology%7C0_2026-10-22T21-55-04.268099.parquet
157 mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0 214 214 0 0 0 0 1021159 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_statistics%7C0_2026-10-22T21-55-04.268099.parquet
158 mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0 57 57 0 0 0 0 474029 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_us_history%7C0_2026-10-22T21-55-04.268099.parquet
159 mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0 180 180 0 0 0 0 1759984 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahigh_school_world_history%7C0_2026-10-22T21-55-04.268099.parquet
160 mmlu.prompt3_mqa_single_correct_sk:human_aging|0 219 219 0 0 0 0 651878 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:human_aging|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahuman_aging%7C0_2026-10-22T21-55-04.268099.parquet
161 mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0 109 109 0 0 0 0 337916 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ahuman_sexuality%7C0_2026-10-22T21-55-04.268099.parquet
162 mmlu.prompt3_mqa_single_correct_sk:international_law|0 118 118 0 0 0 0 515975 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:international_law|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ainternational_law%7C0_2026-10-22T21-55-04.268099.parquet
163 mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0 107 107 0 0 0 0 378748 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Ajurisprudence%7C0_2026-10-22T21-55-04.268099.parquet
164 mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0 158 158 0 0 0 0 564282 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Alogical_fallacies%7C0_2026-10-22T21-55-04.268099.parquet
165 mmlu.prompt3_mqa_single_correct_sk:machine_learning|0 112 112 0 0 0 0 416217 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:machine_learning|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amachine_learning%7C0_2026-10-22T21-55-04.268099.parquet
166 mmlu.prompt3_mqa_single_correct_sk:management|0 100 100 0 0 0 0 293612 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:management|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amanagement%7C0_2026-10-22T21-55-04.268099.parquet
167 mmlu.prompt3_mqa_single_correct_sk:marketing|0 234 234 0 0 0 0 771738 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:marketing|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amarketing%7C0_2026-10-22T21-55-04.268099.parquet
168 mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0 100 100 0 0 0 0 307187 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amedical_genetics%7C0_2026-10-22T21-55-04.268099.parquet
169 mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0 779 779 0 0 0 0 2137196 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amiscellaneous%7C0_2026-10-22T21-55-04.268099.parquet
170 mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0 326 326 0 0 0 0 1133086 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amoral_disputes%7C0_2026-10-22T21-55-04.268099.parquet
171 mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0 749 749 0 0 0 0 3272242 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Amoral_scenarios%7C0_2026-10-22T21-55-04.268099.parquet
172 mmlu.prompt3_mqa_single_correct_sk:nutrition|0 303 303 0 0 0 0 1050303 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:nutrition|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Anutrition%7C0_2026-10-22T21-55-04.268099.parquet
173 mmlu.prompt3_mqa_single_correct_sk:philosophy|0 308 308 0 0 0 0 973131 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:philosophy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aphilosophy%7C0_2026-10-22T21-55-04.268099.parquet
174 mmlu.prompt3_mqa_single_correct_sk:prehistory|0 319 319 0 0 0 0 1039922 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:prehistory|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprehistory%7C0_2026-10-22T21-55-04.268099.parquet
175 mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0 276 276 0 0 0 0 1209237 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_accounting%7C0_2026-10-22T21-55-04.268099.parquet
176 mmlu.prompt3_mqa_single_correct_sk:professional_law|0 1171 1171 0 0 0 0 9378846 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_law|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_law%7C0_2026-10-22T21-55-04.268099.parquet
177 mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0 264 264 0 0 0 0 1649655 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_medicine%7C0_2026-10-22T21-55-04.268099.parquet
178 mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0 596 596 0 0 0 0 2205361 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aprofessional_psychology%7C0_2026-10-22T21-55-04.268099.parquet
179 mmlu.prompt3_mqa_single_correct_sk:public_relations|0 109 109 0 0 0 0 355136 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:public_relations|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Apublic_relations%7C0_2026-10-22T21-55-04.268099.parquet
180 mmlu.prompt3_mqa_single_correct_sk:security_studies|0 222 222 0 0 0 0 1347999 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:security_studies|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Asecurity_studies%7C0_2026-10-22T21-55-04.268099.parquet
181 mmlu.prompt3_mqa_single_correct_sk:sociology|0 197 197 0 0 0 0 692741 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:sociology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Asociology%7C0_2026-10-22T21-55-04.268099.parquet
182 mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0 99 99 0 0 0 0 336616 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aus_foreign_policy%7C0_2026-10-22T21-55-04.268099.parquet
183 mmlu.prompt3_mqa_single_correct_sk:virology|0 164 164 0 0 0 0 511413 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:virology|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Avirology%7C0_2026-10-22T21-55-04.268099.parquet
184 mmlu.prompt3_mqa_single_correct_sk:world_religions|0 171 171 0 0 0 0 434303 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk:world_religions|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_mmlu.prompt3_mqa_single_correct_sk%3Aworld_religions%7C0_2026-10-22T21-55-04.268099.parquet
185 movie_review.prompt1_sentiment_binary_sk|0 2008 2008 0 0 0 0 6995385 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt1_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt1_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
186 movie_review.prompt2_sentiment_binary_sk|0 2008 2008 0 0 0 0 7383490 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt2_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt2_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
187 movie_review.prompt3_sentiment_binary_sk|0 2008 2008 0 0 0 0 6729072 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt3_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_movie_review.prompt3_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
188 multi_wiki_qa.prompt1_qa_sk|0 4997 4997 0 0 0 0 127233608 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt1_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet
189 multi_wiki_qa.prompt2_qa_sk|0 4997 4997 0 0 0 0 127542209 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt2_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet
190 multi_wiki_qa.prompt3_qa_sk|0 4997 4997 0 0 0 0 125437876 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_multi_wiki_qa.prompt3_qa_sk%7C0_2026-10-22T21-55-04.268099.parquet
191 senti_sk.prompt1_sentiment_3_class_sk|0 4677 4677 0 0 0 0 8652017 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_3_class_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_3_class_sk%7C0_2026-10-22T21-55-04.268099.parquet
192 senti_sk.prompt1_sentiment_binary_sk|0 3728 3728 0 0 0 0 6544907 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt1_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
193 senti_sk.prompt2_sentiment_3_class_sk|0 4677 4677 0 0 0 0 9896541 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_3_class_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_3_class_sk%7C0_2026-10-22T21-55-04.268099.parquet
194 senti_sk.prompt2_sentiment_binary_sk|0 3728 3728 0 0 0 0 7247865 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt2_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
195 senti_sk.prompt3_sentiment_3_class_sk|0 4677 4677 0 0 0 0 7883745 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_3_class_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_3_class_sk%7C0_2026-10-22T21-55-04.268099.parquet
196 senti_sk.prompt3_sentiment_binary_sk|0 3728 3728 0 0 0 0 6040540 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_binary_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_senti_sk.prompt3_sentiment_binary_sk%7C0_2026-10-22T21-55-04.268099.parquet
197 skj1.prompt1_skj1_sk|0 100 100 0 0 0 0 320443 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt1_skj1_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt1_skj1_sk%7C0_2026-10-22T21-55-04.268099.parquet
198 skj1.prompt2_skj1_sk|0 100 100 0 0 0 0 301884 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt2_skj1_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt2_skj1_sk%7C0_2026-10-22T21-55-04.268099.parquet
199 skj1.prompt3_skj1_sk|0 100 100 0 0 0 0 305004 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt3_skj1_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_skj1.prompt3_skj1_sk%7C0_2026-10-22T21-55-04.268099.parquet
200 sklep.prompt1_hate_speech_sk:hate-speech|0 1319 1319 0 0 0 0 2492816 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_hate_speech_sk:hate-speech|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_hate_speech_sk%3Ahate-speech%7C0_2026-10-22T21-55-04.268099.parquet
201 sklep.prompt1_nli_3_classes_sk:nli|0 5004 5004 0 0 0 0 14496139 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_3_classes_sk:nli|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_3_classes_sk%3Anli%7C0_2026-10-22T21-55-04.268099.parquet
202 sklep.prompt1_nli_binary_sk:rte|0 1660 1660 0 0 0 0 6481644 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_binary_sk:rte|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_nli_binary_sk%3Arte%7C0_2026-10-22T21-55-04.268099.parquet
203 sklep.prompt1_qa_sk:question-answering|0 9583 9583 0 0 0 0 66012897 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet
204 sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0 1042 1042 0 0 0 0 1993964 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt1_sentiment_binary_sk%3Asentiment-analysis%7C0_2026-10-22T21-55-04.268099.parquet
205 sklep.prompt2_hate_speech_sk:hate-speech|0 1319 1319 0 0 0 0 2209601 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_hate_speech_sk:hate-speech|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_hate_speech_sk%3Ahate-speech%7C0_2026-10-22T21-55-04.268099.parquet
206 sklep.prompt2_nli_3_classes_sk:nli|0 5004 5004 0 0 0 0 18103600 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_3_classes_sk:nli|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_3_classes_sk%3Anli%7C0_2026-10-22T21-55-04.268099.parquet
207 sklep.prompt2_nli_binary_sk:rte|0 1660 1660 0 0 0 0 5814265 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_binary_sk:rte|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_nli_binary_sk%3Arte%7C0_2026-10-22T21-55-04.268099.parquet
208 sklep.prompt2_qa_sk:question-answering|0 9583 9583 0 0 0 0 66791884 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet
209 sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0 1042 1042 0 0 0 0 2210267 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt2_sentiment_binary_sk%3Asentiment-analysis%7C0_2026-10-22T21-55-04.268099.parquet
210 sklep.prompt3_hate_speech_sk:hate-speech|0 1319 1319 0 0 0 0 2647881 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_hate_speech_sk:hate-speech|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_hate_speech_sk%3Ahate-speech%7C0_2026-10-22T21-55-04.268099.parquet
211 sklep.prompt3_nli_3_classes_sk:nli|0 5004 5004 0 0 0 0 14240432 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_3_classes_sk:nli|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_3_classes_sk%3Anli%7C0_2026-10-22T21-55-04.268099.parquet
212 sklep.prompt3_nli_binary_sk:rte|0 1660 1660 0 0 0 0 6948585 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_binary_sk:rte|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_nli_binary_sk%3Arte%7C0_2026-10-22T21-55-04.268099.parquet
213 sklep.prompt3_qa_sk:question-answering|0 9583 9583 0 0 0 0 64169583 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk:question-answering|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_qa_sk%3Aquestion-answering%7C0_2026-10-22T21-55-04.268099.parquet
214 sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0 1042 1042 0 0 0 0 1868158 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_sklep.prompt3_sentiment_binary_sk%3Asentiment-analysis%7C0_2026-10-22T21-55-04.268099.parquet
215 slovakcopa.prompt1_copa_sk|0 500 500 0 0 0 0 1286131 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt1_copa_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt1_copa_sk%7C0_2026-10-22T21-55-04.268099.parquet
216 slovakcopa.prompt2_copa_sk|0 500 500 0 0 0 0 1319142 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt2_copa_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt2_copa_sk%7C0_2026-10-22T21-55-04.268099.parquet
217 slovakcopa.prompt3_copa_sk|0 500 500 0 0 0 0 1277486 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt3_copa_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_slovakcopa.prompt3_copa_sk%7C0_2026-10-22T21-55-04.268099.parquet
218 toxic_sk.prompt1_toxicity_sk|0 881 881 0 0 0 0 1374600 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt1_toxicity_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt1_toxicity_sk%7C0_2026-10-22T21-55-04.268099.parquet
219 toxic_sk.prompt2_toxicity_sk|0 881 881 0 0 0 0 1269485 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt2_toxicity_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt2_toxicity_sk%7C0_2026-10-22T21-55-04.268099.parquet
220 toxic_sk.prompt3_toxicity_sk|0 881 881 0 0 0 0 1663351 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt3_toxicity_sk|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_toxic_sk.prompt3_toxicity_sk%7C0_2026-10-22T21-55-04.268099.parquet
221 truthfulqa.prompt1_mqa_single_correct_sk:mc1|0 778 778 0 0 0 0 2939787 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt1_mqa_single_correct_sk:mc1|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt1_mqa_single_correct_sk%3Amc1%7C0_2026-10-22T21-55-04.268099.parquet
222 truthfulqa.prompt2_mqa_single_correct_sk:mc1|0 778 778 0 0 0 0 2746370 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt2_mqa_single_correct_sk:mc1|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt2_mqa_single_correct_sk%3Amc1%7C0_2026-10-22T21-55-04.268099.parquet
223 truthfulqa.prompt3_mqa_single_correct_sk:mc1|0 778 778 0 0 0 0 2882952 skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt3_mqa_single_correct_sk:mc1|0_2026-10-22T21-55-04.268099.parquet skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/details/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/2026-10-22T21-55-04.268099/details_truthfulqa.prompt3_mqa_single_correct_sk%3Amc1%7C0_2026-10-22T21-55-04.268099.parquet

50
aya_300k_lr_experiments/configs/lr_0p00008.yaml Normal file
View File

@ -0,0 +1,50 @@
stage: sft
do_train: true
do_eval: true
model_name_or_path: slovak-nlp/mistral-sk-7b
trust_remote_code: true
dataset_dir: data
dataset: aya_slovak_300k_fixed
template: alpaca
cutoff_len: 1024
packing: false
train_on_prompt: false
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 8.0e-05
lr_scheduler_type: cosine
warmup_steps: 1500
max_grad_norm: 0.5
bf16: true
fp16: false
optim: adamw_torch
flash_attn: auto
plot_loss: true
ddp_timeout: 180000000
ddp_find_unused_parameters: false
include_num_input_tokens_seen: true
enable_thinking: false
use_unsloth: true
use_unsloth_gc: true
streaming: false
eval_strategy: steps
val_size: 1000
eval_steps: 1000
overwrite_cache: false
overwrite_output_dir: false
output_dir: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00008
logging_steps: 10
save_steps: 2000
save_total_limit: 2
report_to: none
seed: 42
accelerator_config:
dispatch_batches: false
split_batches: false
num_train_epochs: 1.0

50
aya_300k_lr_experiments/configs/lr_0p00010.yaml Normal file
View File

@ -0,0 +1,50 @@
stage: sft
do_train: true
do_eval: true
model_name_or_path: slovak-nlp/mistral-sk-7b
trust_remote_code: true
dataset_dir: data
dataset: aya_slovak_300k_fixed
template: alpaca
cutoff_len: 1024
packing: false
train_on_prompt: false
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 0.0001
lr_scheduler_type: cosine
warmup_steps: 1500
max_grad_norm: 0.5
bf16: true
fp16: false
optim: adamw_torch
flash_attn: auto
plot_loss: true
ddp_timeout: 180000000
ddp_find_unused_parameters: false
include_num_input_tokens_seen: true
enable_thinking: false
use_unsloth: true
use_unsloth_gc: true
streaming: false
eval_strategy: steps
val_size: 1000
eval_steps: 1000
overwrite_cache: false
overwrite_output_dir: false
output_dir: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010
logging_steps: 10
save_steps: 2000
save_total_limit: 2
report_to: none
seed: 42
accelerator_config:
dispatch_batches: false
split_batches: false
num_train_epochs: 1.0

50
aya_300k_lr_experiments/configs/lr_0p00015.yaml Normal file
View File

@ -0,0 +1,50 @@
stage: sft
do_train: true
do_eval: true
model_name_or_path: slovak-nlp/mistral-sk-7b
trust_remote_code: true
dataset_dir: data
dataset: aya_slovak_300k_fixed
template: alpaca
cutoff_len: 1024
packing: false
train_on_prompt: false
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 0.00015
lr_scheduler_type: cosine
warmup_steps: 1500
max_grad_norm: 0.5
bf16: true
fp16: false
optim: adamw_torch
flash_attn: auto
plot_loss: true
ddp_timeout: 180000000
ddp_find_unused_parameters: false
include_num_input_tokens_seen: true
enable_thinking: false
use_unsloth: true
use_unsloth_gc: true
streaming: false
eval_strategy: steps
val_size: 1000
eval_steps: 1000
overwrite_cache: false
overwrite_output_dir: false
output_dir: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00015
logging_steps: 10
save_steps: 2000
save_total_limit: 2
report_to: none
seed: 42
accelerator_config:
dispatch_batches: false
split_batches: false
num_train_epochs: 1.0

50
aya_300k_lr_experiments/configs/lr_0p00018.yaml Normal file
View File

@ -0,0 +1,50 @@
stage: sft
do_train: true
do_eval: true
model_name_or_path: slovak-nlp/mistral-sk-7b
trust_remote_code: true
dataset_dir: data
dataset: aya_slovak_300k_fixed
template: alpaca
cutoff_len: 1024
packing: false
train_on_prompt: false
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 0.00018
lr_scheduler_type: cosine
warmup_steps: 1500
max_grad_norm: 0.5
bf16: true
fp16: false
optim: adamw_torch
flash_attn: auto
plot_loss: true
ddp_timeout: 180000000
ddp_find_unused_parameters: false
include_num_input_tokens_seen: true
enable_thinking: false
use_unsloth: true
use_unsloth_gc: true
streaming: false
eval_strategy: steps
val_size: 1000
eval_steps: 1000
overwrite_cache: false
overwrite_output_dir: true
output_dir: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00018
logging_steps: 10
save_steps: 2000
save_total_limit: 2
report_to: none
seed: 42
accelerator_config:
dispatch_batches: false
split_batches: false
num_train_epochs: 1.0

50
aya_300k_lr_experiments/configs/lr_0p00020.yaml Normal file
View File

@ -0,0 +1,50 @@
stage: sft
do_train: true
do_eval: true
model_name_or_path: slovak-nlp/mistral-sk-7b
trust_remote_code: true
dataset_dir: data
dataset: aya_slovak_300k_fixed
template: alpaca
cutoff_len: 1024
packing: false
train_on_prompt: false
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 0.0002
lr_scheduler_type: cosine
warmup_steps: 1500
max_grad_norm: 0.5
bf16: true
fp16: false
optim: adamw_torch
flash_attn: auto
plot_loss: true
ddp_timeout: 180000000
ddp_find_unused_parameters: false
include_num_input_tokens_seen: true
enable_thinking: false
use_unsloth: true
use_unsloth_gc: true
streaming: false
eval_strategy: steps
val_size: 1000
eval_steps: 1000
overwrite_cache: false
overwrite_output_dir: true
output_dir: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00020
logging_steps: 10
save_steps: 2000
save_total_limit: 2
report_to: none
seed: 42
accelerator_config:
dispatch_batches: false
split_batches: false
num_train_epochs: 1.0

50
aya_300k_lr_experiments/configs/lr_0p00023.yaml Normal file
View File

@ -0,0 +1,50 @@
stage: sft
do_train: true
do_eval: true
model_name_or_path: slovak-nlp/mistral-sk-7b
trust_remote_code: true
dataset_dir: data
dataset: aya_slovak_300k_fixed
template: alpaca
cutoff_len: 1024
packing: false
train_on_prompt: false
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.0
lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 0.00023
lr_scheduler_type: cosine
warmup_steps: 1500
max_grad_norm: 0.5
bf16: true
fp16: false
optim: adamw_torch
flash_attn: auto
plot_loss: true
ddp_timeout: 180000000
ddp_find_unused_parameters: false
include_num_input_tokens_seen: true
enable_thinking: false
use_unsloth: true
use_unsloth_gc: true
streaming: false
eval_strategy: steps
val_size: 1000
eval_steps: 1000
overwrite_cache: false
overwrite_output_dir: true
output_dir: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00023
logging_steps: 10
save_steps: 2000
save_total_limit: 2
report_to: none
seed: 42
accelerator_config:
dispatch_batches: false
split_batches: false
num_train_epochs: 1.0

12451
aya_300k_lr_experiments/logs/lr_0p00008.log Normal file
View File

File diff suppressed because one or more lines are too long

12453
aya_300k_lr_experiments/logs/lr_0p00010.log Normal file
View File

File diff suppressed because one or more lines are too long

12448
aya_300k_lr_experiments/logs/lr_0p00015.log Normal file
View File

File diff suppressed because one or more lines are too long

12451
aya_300k_lr_experiments/logs/lr_0p00018.log Normal file
View File

File diff suppressed because one or more lines are too long

12447
aya_300k_lr_experiments/logs/lr_0p00020.log Normal file
View File

File diff suppressed because one or more lines are too long

12446
aya_300k_lr_experiments/logs/lr_0p00023.log Normal file
View File

File diff suppressed because one or more lines are too long

12
aya_300k_lr_experiments/provenance/source_archive.json Normal file
View File

@ -0,0 +1,12 @@
{
"source_archive": "aya_300k_benchmarky_20261024_215708.tar.gz",
"sha256": "b45e431cbc63bc73ef953f8aa3837695ccec43783e3da2dfccb0d99f7d1fe79c",
"compressed_bytes": 349555561,
"scope": "Hruska Aya 300k learning-rate selection and evaluation of selected LR 0.00010; Titan excluded",
"base_model": "slovak-nlp/mistral-sk-7b",
"base_model_revision": "089497ae0d72018e9591895c54f774bf0b4a83a4",
"selection_criterion": "lowest final validation loss among the six completed 300k runs",
"intended_next_stage": "planned training on the full Aya dataset on Perun; not executed in this experiment",
"data_reproducibility_limit": "No source dataset revision, data-file checksum, or held-out sample IDs were included in this archive",
"curation": "Model weights, checkpoints, trainer_state.json, duplicated inference caches and full parquet details excluded from the Git subset. Original archive retained separately."
}

95
aya_300k_lr_experiments/reproduction.md Normal file
View File

@ -0,0 +1,95 @@
# Reprodukcia experimentu a benchmarkov
Hlavným cieľom série bolo vybrať vhodný learning rate na 300k slovenskej Aya vzorke pred plánovaným tréningom na celej množine na Perune. Hodnotenie SK-bench a lm-eval následne prebehlo iba pre vybraný LR `0.00010`, ktorý mal najnižší finálny validačný loss zo šiestich behov.
## Čo je pôvodný záznam a čo je odvodené
- `configs/`, `logs/`, jednotlivé tréningové JSON súbory, pôvodné grafy, LR súhrn a výsledné benchmarkové JSON pochádzajú z archívu `aya_300k_benchmarky_20261024_215708.tar.gz`.
- `results/eval_loss_history.csv`, `results/gradient_history.csv`, rozšírené tabuľky a graf finálneho loss podľa LR sú odvodené z tréningových záznamov. Pôvodné merania sa neprepisujú.
- `scripts/setup_aya_300k_lr_validation.sh`, oba tréningové runner skripty, oba pôvodné analyzátory a `run_lmeval_aya_300k_lr_0p00010.sh` sú archivované serverové skripty. Zachovávajú pôvodné absolútne cesty na Hruške.
- `scripts/analyze_training_results.py` je lokálny offline analyzátor vytvorený pri spracovaní výsledkov. Nepotrebuje váhy ani `trainer_state.json`.
- `scripts/rerun_skbench_lr_0p00010.sh` bol zrekonštruovaný pri spracovaní podľa odovzdaného príkazu, uložených parametrov a presného zoznamu 222 vykonaných úloh. Nie je vydávaný za pôvodný archivovaný runner.
Zdrojový archív vrátane úplných per-sample detailov zostáva samostatným primárnym záznamom. Jeho SHA-256 a veľkosť sú v [provenance/source_archive.json](provenance/source_archive.json).
## Príprava dát a tréning
Pôvodný setup načítal `CohereLabs/aya_collection_language_split`, subset `slovak`, split `train`. Ponechal len záznamy s neprázdnymi reťazcami `inputs` a `targets`, vykonal shuffle so seed 42 a vybral prvých 300 000 záznamov. Pri zápise sa odstránili okrajové medzery a vytvorili polia `instruction`, `input` a `output`. LLaMA-Factory následne použila 1000 príkladov na validáciu a 299 000 na tréning.
Archív neobsahuje konkrétnu revíziu zdrojového Aya datasetu, checksum vytvoreného JSONL súboru ani identifikátory validačného splitu. Recept preto dokumentuje postup, ale pri inom stave zdrojového datasetu nezaručuje identickú 300k vzorku. Pred budúcim tréningom na Perune je vhodné uchovať zdrojovú revíziu, checksum dát a rozdelenie na train/validation.
Tréningové konfigurácie používajú `model_name_or_path: slovak-nlp/mistral-sk-7b`. Logy potvrdzujú snapshot:
```text
089497ae0d72018e9591895c54f774bf0b4a83a4
```
Pri opakovaní možno tento snapshot explicitne zafixovať cez `model_revision`. Pôvodné YAML súbory zostávajú v zaznamenanom stave.
Setup na pôvodnej Hruške pracoval s cestami:
```text
/home/schwarc/LLaMA-Factory/data/aya_slovak_300k_fixed.jsonl
/home/schwarc/LLaMA-Factory/data/dataset_info.json
/home/schwarc/diplomovka/aya_300k_lr_validation/
/home/schwarc/diplomovka/scripts/
```
Registráciu datasetu prebral z existujúcej položky `aya_slovak_100k_fixed`. V novom prostredí musí registrácia zodpovedať Alpaca poliam `instruction`, `input`, `output`. Archivovaný setup aj tréningové skripty sú určené na prečítanie a prispôsobenie týmto cestám a novému výstupnému priečinku, nie na automatické prepísanie existujúcich dokončených behov.
Pôvodné poradie behov:
1. `run_aya_300k_lr_validation.sh`: LR `0.00018`, `0.00020`, `0.00023`.
2. `run_aya_300k_lr_validation_lower.sh`: LR `0.00015`, `0.00010`, `0.00008`.
Prvý pôvodný analyzátor sumarizuje len prvé tri LR. Pre všetkých šesť serverových behov slúži `analyze_aya_300k_lr_all.py`. Oba pôvodné analyzátory vyžadujú serverové `trainer_state.json`; tieto veľké súbory sa do Git výberu nekopírujú.
Na obnovenie malých odvodených tabuliek a LR grafu zo súborov v tomto repozitári slúži:
```bash
python aya_300k_lr_experiments/scripts/analyze_training_results.py
```
Analyzátor používa Python a Matplotlib (overené s verziou 3.11.2). Tréning ani benchmark týmto príkazom neprebieha.
## Zlúčenie modelu s adaptérom
[benchmarks/export/export_lr_0p00010.yaml](benchmarks/export/export_lr_0p00010.yaml) opisuje zlúčenie pôvodného modelového snapshotu s adaptérom z `lr_0p00010`. Export prebehol cez LLaMA-Factory na CPU do `bfloat16`, s template `alpaca` a tromi safetensors shardmi. Zlúčenie nevykonáva ďalší tréning.
Exportovaný model bol na serveri uložený v:
```text
/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
```
Modelové váhy a adaptér nie sú súčasťou tohto Git výberu. Metadata exportovaného modelu a chat template sú zachované v `benchmarks/export/`.
## lm-evaluation-harness
Archivovaný runner používa Conda prostredie `lm_eval_hruska`, checkout `/home/schwarc/lm-evaluation-harness` a custom úlohy v `/home/schwarc/diplomovka/lmeval_slovak_tasks`. Ich presný obsah, commit a verzie knižníc sú uložené v benchmarkových metadátach.
Zlúčený model sa načíta ako `pretrained=...`, bez druhého pripájania adaptéra cez `peft=`. Exportovaný tokenizer mal označenie `TokenizersBackend`, ktoré Transformers 4.48.2 nepozná. Úspešný beh preto explicitne používa pôvodný tokenizer z cache presného snapshotu základného Mistralu. Oba majú rovnaký pôvodný slovník, benchmarkové prompty sa neobalili Alpaca chat template.
Nastavenia úspešného behu: HF backend, BF16, batch size 1, zero-shot, bez limitu vzoriek, seeds `0/1234/1234/1234`. `CUDA_VISIBLE_DEVICES=1` vybral fyzickú GPU 1; `--device cuda:0` označoval túto jedinú viditeľnú GPU.
## SK-bench
Úspešný beh vychádza zo slovenského zero-shot výberu oficiálneho `tasks_list_zero.txt`. Presný použitý zoznam 222 položiek aj 190 vynechaných položiek sa zachoval v `benchmarks/skbench/configs/`. Do opakovaného porovnania ďalšieho modelu treba prevziať tento presný zoznam, nie nanovo meniaci sa zoznam dostupných úloh.
V čase behu nebolo prístupných 12 repozitárov zo `slovak-nlp`; navyše nebol schválený gated prístup k `natalia-nk/NLI-SK-annotated`. Pôvodný metadata report rozlišuje existenciu metadát, nie oprávnenie čítať dátové súbory. Preto jeho `OK` pri NLI-SK-annotated nie je dôkazom schváleného prístupu. Pri novom behu treba skontrolovať aj oprávnenie cez `HfApi.auth_check` a prípadné podmienky na stránke datasetu.
Reprodukčný runner používa existujúci checkout SK-bench a jeho `uv.lock`. Pri opakovaní treba overiť zhodu commitu a lockfile s archivovanými benchmarkovými metadátami. Príklad spustenia na pôvodnej Hruške:
```bash
bash aya_300k_lr_experiments/scripts/rerun_skbench_lr_0p00010.sh
```
Cesty možno nastaviť cez `SKBENCH_DIR`, `MODEL_DIR`, `OUTPUT_ROOT` a fyzickú GPU cez `GPU_ID`. Runner vždy vytvára nový výsledkový priečinok.
Zachované nastavenia: 1 GPU, tensor parallel size 1, BF16, GPU memory utilization 0.85, kontext 32768, max batched tokens 2048, eager režim, `VLLM_USE_FLASHINFER_SAMPLER=0` a uloženie detailov. Nepoužíva sa `--max-samples` ani globálne skrátenie výstupného rozpočtu úloh cez `--max-tokens`.
Šablóna Alpaca sa pri SK-bench používa. Greedy dekódovanie a seed 1234 pochádzajú z zaznamenanej konfigurácie Lighteval/vLLM. Pri vyhodnotení treba popri skóre sledovať aj dodržanie formátu odpovede a orezanie vstupov. Čiastkový beh sa neprezentuje ako oficiálne celkové skóre plnej sady.
## Kontrola dlhých vstupov v dokončenom SK-bench behu
Log obsahuje jednu hlášku `context_size + max_new_tokens=53860 ... max_length=32768. Truncating context to 32642 tokens.` Počet logových hlášok nie je počtom ovplyvnených príkladov. V detailoch bolo 159 426 nulových počítadiel `truncated_tokens_count`, hoci log orezanie explicitne potvrdzuje. Dĺžková kontrola navyše našla 69 MultiWikiQA vstupov na hranici 32642 tokenov. Sú to hraničné kandidáty (23 pri každom prompte), nie dokázaný presný počet orezaní. Bez pôvodnej tokenizácie pred orezaním nemožno skutočný počet spätne spoľahlivo určiť. Tento nesúlad je zaznamenaný v benchmarkovom audite; nulové počítadlá sa nepoužívajú ako dôkaz vyhodnotenia bez orezania.

7
aya_300k_lr_experiments/results/aya_300k_lr_summary.csv Normal file
View File

@ -0,0 +1,7 @@
learning_rate,global_step,final_eval_step,final_eval_loss,min_eval_step,min_eval_loss,max_grad_norm,p95_grad_norm,trainer_state
0.00008,18688,18688,0.875216543674469,18688,0.875216543674469,9.149866104125977,1.5837211608886719,/home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00008/trainer_state.json
0.00010,18688,18688,0.87261962890625,18688,0.87261962890625,8.578277587890625,1.6475107669830322,/home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010/trainer_state.json
0.00015,18688,18688,0.8842697143554688,18688,0.8842697143554688,28.051427841186523,1.9590225219726562,/home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00015/trainer_state.json
0.00018,18688,18688,0.8914602994918823,18000,0.891387939453125,6.5584211349487305,2.163313627243042,/home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00018/trainer_state.json
0.00020,18688,18688,0.8976690173149109,18000,0.897506594657898,10.969125747680664,2.306173801422119,/home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00020/trainer_state.json
0.00023,18688,18688,0.9032362699508667,18000,0.9031931161880493,23.10581398010254,2.501060724258423,/home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00023/trainer_state.json
1 learning_rate global_step final_eval_step final_eval_loss min_eval_step min_eval_loss max_grad_norm p95_grad_norm trainer_state
2 0.00008 18688 18688 0.875216543674469 18688 0.875216543674469 9.149866104125977 1.5837211608886719 /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00008/trainer_state.json
3 0.00010 18688 18688 0.87261962890625 18688 0.87261962890625 8.578277587890625 1.6475107669830322 /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010/trainer_state.json
4 0.00015 18688 18688 0.8842697143554688 18688 0.8842697143554688 28.051427841186523 1.9590225219726562 /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00015/trainer_state.json
5 0.00018 18688 18688 0.8914602994918823 18000 0.891387939453125 6.5584211349487305 2.163313627243042 /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00018/trainer_state.json
6 0.00020 18688 18688 0.8976690173149109 18000 0.897506594657898 10.969125747680664 2.306173801422119 /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00020/trainer_state.json
7 0.00023 18688 18688 0.9032362699508667 18000 0.9031931161880493 23.10581398010254 2.501060724258423 /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00023/trainer_state.json

115
aya_300k_lr_experiments/results/eval_loss_history.csv Normal file
View File

@ -0,0 +1,115 @@
learning_rate,run,step,eval_loss
8e-05,lr_0p00008,1000,1.0020490884780884
8e-05,lr_0p00008,2000,0.9672194719314575
8e-05,lr_0p00008,3000,0.9505742192268372
8e-05,lr_0p00008,4000,0.9352602362632751
8e-05,lr_0p00008,5000,0.9294734597206116
8e-05,lr_0p00008,6000,0.9264867305755615
8e-05,lr_0p00008,7000,0.9184143543243408
8e-05,lr_0p00008,8000,0.9120934009552002
8e-05,lr_0p00008,9000,0.9065558910369873
8e-05,lr_0p00008,10000,0.9004696011543274
8e-05,lr_0p00008,11000,0.8929902911186218
8e-05,lr_0p00008,12000,0.8897570371627808
8e-05,lr_0p00008,13000,0.8844991326332092
8e-05,lr_0p00008,14000,0.8800909519195557
8e-05,lr_0p00008,15000,0.8777939677238464
8e-05,lr_0p00008,16000,0.8767354488372803
8e-05,lr_0p00008,17000,0.875752329826355
8e-05,lr_0p00008,18000,0.875270426273346
8e-05,lr_0p00008,18688,0.875216543674469
0.0001,lr_0p00010,1000,1.0004310607910156
0.0001,lr_0p00010,2000,0.9658941626548767
0.0001,lr_0p00010,3000,0.9526628255844116
0.0001,lr_0p00010,4000,0.9393344521522522
0.0001,lr_0p00010,5000,0.9336668848991394
0.0001,lr_0p00010,6000,0.931794285774231
0.0001,lr_0p00010,7000,0.9227241277694702
0.0001,lr_0p00010,8000,0.9158084392547607
0.0001,lr_0p00010,9000,0.9100468158721924
0.0001,lr_0p00010,10000,0.9019826650619507
0.0001,lr_0p00010,11000,0.8941853642463684
0.0001,lr_0p00010,12000,0.8896107077598572
0.0001,lr_0p00010,13000,0.8851001262664795
0.0001,lr_0p00010,14000,0.8795706033706665
0.0001,lr_0p00010,15000,0.8770407438278198
0.0001,lr_0p00010,16000,0.8748454451560974
0.0001,lr_0p00010,17000,0.8731834888458252
0.0001,lr_0p00010,18000,0.8727287650108337
0.0001,lr_0p00010,18688,0.87261962890625
0.00015,lr_0p00015,1000,1.0024861097335815
0.00015,lr_0p00015,2000,0.9763674139976501
0.00015,lr_0p00015,3000,0.97147536277771
0.00015,lr_0p00015,4000,0.9657827019691467
0.00015,lr_0p00015,5000,0.9659448266029358
0.00015,lr_0p00015,6000,0.9597828388214111
0.00015,lr_0p00015,7000,0.951335608959198
0.00015,lr_0p00015,8000,0.9469588994979858
0.00015,lr_0p00015,9000,0.9383289813995361
0.00015,lr_0p00015,10000,0.9309172630310059
0.00015,lr_0p00015,11000,0.9210461974143982
0.00015,lr_0p00015,12000,0.9106600284576416
0.00015,lr_0p00015,13000,0.9036438465118408
0.00015,lr_0p00015,14000,0.8952069878578186
0.00015,lr_0p00015,15000,0.8909939527511597
0.00015,lr_0p00015,16000,0.8871906995773315
0.00015,lr_0p00015,17000,0.8851967453956604
0.00015,lr_0p00015,18000,0.8842737674713135
0.00015,lr_0p00015,18688,0.8842697143554688
0.00018,lr_0p00018,1000,0.9958739280700684
0.00018,lr_0p00018,2000,0.9904810786247253
0.00018,lr_0p00018,3000,0.9867064356803894
0.00018,lr_0p00018,4000,0.9833324551582336
0.00018,lr_0p00018,5000,0.9868490099906921
0.00018,lr_0p00018,6000,0.9755926728248596
0.00018,lr_0p00018,7000,0.9768977761268616
0.00018,lr_0p00018,8000,0.9687609076499939
0.00018,lr_0p00018,9000,0.9562658071517944
0.00018,lr_0p00018,10000,0.9435049891471863
0.00018,lr_0p00018,11000,0.9342814683914185
0.00018,lr_0p00018,12000,0.922635018825531
0.00018,lr_0p00018,13000,0.9130367636680603
0.00018,lr_0p00018,14000,0.904951810836792
0.00018,lr_0p00018,15000,0.8994976878166199
0.00018,lr_0p00018,16000,0.8953551650047302
0.00018,lr_0p00018,17000,0.8925029039382935
0.00018,lr_0p00018,18000,0.891387939453125
0.00018,lr_0p00018,18688,0.8914602994918823
0.0002,lr_0p00020,1000,0.9974449276924133
0.0002,lr_0p00020,2000,0.9973769783973694
0.0002,lr_0p00020,3000,1.0056378841400146
0.0002,lr_0p00020,4000,0.9973607063293457
0.0002,lr_0p00020,5000,1.0010013580322266
0.0002,lr_0p00020,6000,1.000449776649475
0.0002,lr_0p00020,7000,0.9865806698799133
0.0002,lr_0p00020,8000,0.9824715852737427
0.0002,lr_0p00020,9000,0.9666008353233337
0.0002,lr_0p00020,10000,0.9573416709899902
0.0002,lr_0p00020,11000,0.9432744383811951
0.0002,lr_0p00020,12000,0.9326162338256836
0.0002,lr_0p00020,13000,0.9201604127883911
0.0002,lr_0p00020,14000,0.9126585721969604
0.0002,lr_0p00020,15000,0.907073438167572
0.0002,lr_0p00020,16000,0.9014313817024231
0.0002,lr_0p00020,17000,0.8986518979072571
0.0002,lr_0p00020,18000,0.897506594657898
0.0002,lr_0p00020,18688,0.8976690173149109
0.00023,lr_0p00023,1000,1.0002912282943726
0.00023,lr_0p00023,2000,1.0145149230957031
0.00023,lr_0p00023,3000,1.0228914022445679
0.00023,lr_0p00023,4000,1.0323290824890137
0.00023,lr_0p00023,5000,1.0345656871795654
0.00023,lr_0p00023,6000,1.0246530771255493
0.00023,lr_0p00023,7000,1.0160919427871704
0.00023,lr_0p00023,8000,1.0105783939361572
0.00023,lr_0p00023,9000,0.9934014678001404
0.00023,lr_0p00023,10000,0.980603039264679
0.00023,lr_0p00023,11000,0.963624119758606
0.00023,lr_0p00023,12000,0.951417088508606
0.00023,lr_0p00023,13000,0.9340505599975586
0.00023,lr_0p00023,14000,0.9227956533432007
0.00023,lr_0p00023,15000,0.9158667325973511
0.00023,lr_0p00023,16000,0.908878743648529
0.00023,lr_0p00023,17000,0.9046502709388733
0.00023,lr_0p00023,18000,0.9031931161880493
0.00023,lr_0p00023,18688,0.9032362699508667
1 learning_rate run step eval_loss
2 8e-05 lr_0p00008 1000 1.0020490884780884
3 8e-05 lr_0p00008 2000 0.9672194719314575
4 8e-05 lr_0p00008 3000 0.9505742192268372
5 8e-05 lr_0p00008 4000 0.9352602362632751
6 8e-05 lr_0p00008 5000 0.9294734597206116
7 8e-05 lr_0p00008 6000 0.9264867305755615
8 8e-05 lr_0p00008 7000 0.9184143543243408
9 8e-05 lr_0p00008 8000 0.9120934009552002
10 8e-05 lr_0p00008 9000 0.9065558910369873
11 8e-05 lr_0p00008 10000 0.9004696011543274
12 8e-05 lr_0p00008 11000 0.8929902911186218
13 8e-05 lr_0p00008 12000 0.8897570371627808
14 8e-05 lr_0p00008 13000 0.8844991326332092
15 8e-05 lr_0p00008 14000 0.8800909519195557
16 8e-05 lr_0p00008 15000 0.8777939677238464
17 8e-05 lr_0p00008 16000 0.8767354488372803
18 8e-05 lr_0p00008 17000 0.875752329826355
19 8e-05 lr_0p00008 18000 0.875270426273346
20 8e-05 lr_0p00008 18688 0.875216543674469
21 0.0001 lr_0p00010 1000 1.0004310607910156
22 0.0001 lr_0p00010 2000 0.9658941626548767
23 0.0001 lr_0p00010 3000 0.9526628255844116
24 0.0001 lr_0p00010 4000 0.9393344521522522
25 0.0001 lr_0p00010 5000 0.9336668848991394
26 0.0001 lr_0p00010 6000 0.931794285774231
27 0.0001 lr_0p00010 7000 0.9227241277694702
28 0.0001 lr_0p00010 8000 0.9158084392547607
29 0.0001 lr_0p00010 9000 0.9100468158721924
30 0.0001 lr_0p00010 10000 0.9019826650619507
31 0.0001 lr_0p00010 11000 0.8941853642463684
32 0.0001 lr_0p00010 12000 0.8896107077598572
33 0.0001 lr_0p00010 13000 0.8851001262664795
34 0.0001 lr_0p00010 14000 0.8795706033706665
35 0.0001 lr_0p00010 15000 0.8770407438278198
36 0.0001 lr_0p00010 16000 0.8748454451560974
37 0.0001 lr_0p00010 17000 0.8731834888458252
38 0.0001 lr_0p00010 18000 0.8727287650108337
39 0.0001 lr_0p00010 18688 0.87261962890625
40 0.00015 lr_0p00015 1000 1.0024861097335815
41 0.00015 lr_0p00015 2000 0.9763674139976501
42 0.00015 lr_0p00015 3000 0.97147536277771
43 0.00015 lr_0p00015 4000 0.9657827019691467
44 0.00015 lr_0p00015 5000 0.9659448266029358
45 0.00015 lr_0p00015 6000 0.9597828388214111
46 0.00015 lr_0p00015 7000 0.951335608959198
47 0.00015 lr_0p00015 8000 0.9469588994979858
48 0.00015 lr_0p00015 9000 0.9383289813995361
49 0.00015 lr_0p00015 10000 0.9309172630310059
50 0.00015 lr_0p00015 11000 0.9210461974143982
51 0.00015 lr_0p00015 12000 0.9106600284576416
52 0.00015 lr_0p00015 13000 0.9036438465118408
53 0.00015 lr_0p00015 14000 0.8952069878578186
54 0.00015 lr_0p00015 15000 0.8909939527511597
55 0.00015 lr_0p00015 16000 0.8871906995773315
56 0.00015 lr_0p00015 17000 0.8851967453956604
57 0.00015 lr_0p00015 18000 0.8842737674713135
58 0.00015 lr_0p00015 18688 0.8842697143554688
59 0.00018 lr_0p00018 1000 0.9958739280700684
60 0.00018 lr_0p00018 2000 0.9904810786247253
61 0.00018 lr_0p00018 3000 0.9867064356803894
62 0.00018 lr_0p00018 4000 0.9833324551582336
63 0.00018 lr_0p00018 5000 0.9868490099906921
64 0.00018 lr_0p00018 6000 0.9755926728248596
65 0.00018 lr_0p00018 7000 0.9768977761268616
66 0.00018 lr_0p00018 8000 0.9687609076499939
67 0.00018 lr_0p00018 9000 0.9562658071517944
68 0.00018 lr_0p00018 10000 0.9435049891471863
69 0.00018 lr_0p00018 11000 0.9342814683914185
70 0.00018 lr_0p00018 12000 0.922635018825531
71 0.00018 lr_0p00018 13000 0.9130367636680603
72 0.00018 lr_0p00018 14000 0.904951810836792
73 0.00018 lr_0p00018 15000 0.8994976878166199
74 0.00018 lr_0p00018 16000 0.8953551650047302
75 0.00018 lr_0p00018 17000 0.8925029039382935
76 0.00018 lr_0p00018 18000 0.891387939453125
77 0.00018 lr_0p00018 18688 0.8914602994918823
78 0.0002 lr_0p00020 1000 0.9974449276924133
79 0.0002 lr_0p00020 2000 0.9973769783973694
80 0.0002 lr_0p00020 3000 1.0056378841400146
81 0.0002 lr_0p00020 4000 0.9973607063293457
82 0.0002 lr_0p00020 5000 1.0010013580322266
83 0.0002 lr_0p00020 6000 1.000449776649475
84 0.0002 lr_0p00020 7000 0.9865806698799133
85 0.0002 lr_0p00020 8000 0.9824715852737427
86 0.0002 lr_0p00020 9000 0.9666008353233337
87 0.0002 lr_0p00020 10000 0.9573416709899902
88 0.0002 lr_0p00020 11000 0.9432744383811951
89 0.0002 lr_0p00020 12000 0.9326162338256836
90 0.0002 lr_0p00020 13000 0.9201604127883911
91 0.0002 lr_0p00020 14000 0.9126585721969604
92 0.0002 lr_0p00020 15000 0.907073438167572
93 0.0002 lr_0p00020 16000 0.9014313817024231
94 0.0002 lr_0p00020 17000 0.8986518979072571
95 0.0002 lr_0p00020 18000 0.897506594657898
96 0.0002 lr_0p00020 18688 0.8976690173149109
97 0.00023 lr_0p00023 1000 1.0002912282943726
98 0.00023 lr_0p00023 2000 1.0145149230957031
99 0.00023 lr_0p00023 3000 1.0228914022445679
100 0.00023 lr_0p00023 4000 1.0323290824890137
101 0.00023 lr_0p00023 5000 1.0345656871795654
102 0.00023 lr_0p00023 6000 1.0246530771255493
103 0.00023 lr_0p00023 7000 1.0160919427871704
104 0.00023 lr_0p00023 8000 1.0105783939361572
105 0.00023 lr_0p00023 9000 0.9934014678001404
106 0.00023 lr_0p00023 10000 0.980603039264679
107 0.00023 lr_0p00023 11000 0.963624119758606
108 0.00023 lr_0p00023 12000 0.951417088508606
109 0.00023 lr_0p00023 13000 0.9340505599975586
110 0.00023 lr_0p00023 14000 0.9227956533432007
111 0.00023 lr_0p00023 15000 0.9158667325973511
112 0.00023 lr_0p00023 16000 0.908878743648529
113 0.00023 lr_0p00023 17000 0.9046502709388733
114 0.00023 lr_0p00023 18000 0.9031931161880493
115 0.00023 lr_0p00023 18688 0.9032362699508667

BIN
aya_300k_lr_experiments/results/eval_loss_over_steps.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 263 KiB

BIN
aya_300k_lr_experiments/results/grad_norm_over_steps.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 391 KiB

11209
aya_300k_lr_experiments/results/gradient_history.csv Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
aya_300k_lr_experiments/results/lr_vs_final_eval_loss.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 90 KiB

7
aya_300k_lr_experiments/results/training_run_summary.csv Normal file
View File

@ -0,0 +1,7 @@
learning_rate,run,global_step,epoch,final_eval_step,final_eval_loss,min_eval_step,min_eval_loss,max_grad_norm,p95_grad_norm,train_runtime,eval_observations,gradient_observations
8e-05,lr_0p00008,18688,1.0,18688,0.875216543674469,18688,0.875216543674469,9.149866104125977,1.5837211608886719,79243.6449,19,1868
0.0001,lr_0p00010,18688,1.0,18688,0.87261962890625,18688,0.87261962890625,8.578277587890625,1.6475107669830322,79344.5851,19,1868
0.00015,lr_0p00015,18688,1.0,18688,0.8842697143554688,18688,0.8842697143554688,28.051427841186523,1.9590225219726562,80127.6428,19,1868
0.00018,lr_0p00018,18688,1.0,18688,0.8914602994918823,18000,0.891387939453125,6.5584211349487305,2.163313627243042,79892.0077,19,1868
0.0002,lr_0p00020,18688,1.0,18688,0.8976690173149109,18000,0.897506594657898,10.969125747680664,2.306173801422119,79710.3252,19,1868
0.00023,lr_0p00023,18688,1.0,18688,0.9032362699508667,18000,0.9031931161880493,23.10581398010254,2.501060724258423,79789.3562,19,1868
1 learning_rate run global_step epoch final_eval_step final_eval_loss min_eval_step min_eval_loss max_grad_norm p95_grad_norm train_runtime eval_observations gradient_observations
2 8e-05 lr_0p00008 18688 1.0 18688 0.875216543674469 18688 0.875216543674469 9.149866104125977 1.5837211608886719 79243.6449 19 1868
3 0.0001 lr_0p00010 18688 1.0 18688 0.87261962890625 18688 0.87261962890625 8.578277587890625 1.6475107669830322 79344.5851 19 1868
4 0.00015 lr_0p00015 18688 1.0 18688 0.8842697143554688 18688 0.8842697143554688 28.051427841186523 1.9590225219726562 80127.6428 19 1868
5 0.00018 lr_0p00018 18688 1.0 18688 0.8914602994918823 18000 0.891387939453125 6.5584211349487305 2.163313627243042 79892.0077 19 1868
6 0.0002 lr_0p00020 18688 1.0 18688 0.8976690173149109 18000 0.897506594657898 10.969125747680664 2.306173801422119 79710.3252 19 1868
7 0.00023 lr_0p00023 18688 1.0 18688 0.9032362699508667 18000 0.9031931161880493 23.10581398010254 2.501060724258423 79789.3562 19 1868

92
aya_300k_lr_experiments/results/training_run_summary.json Normal file
View File

@ -0,0 +1,92 @@
[
{
"learning_rate": 8e-05,
"run": "lr_0p00008",
"global_step": 18688,
"epoch": 1.0,
"final_eval_step": 18688,
"final_eval_loss": 0.875216543674469,
"min_eval_step": 18688,
"min_eval_loss": 0.875216543674469,
"max_grad_norm": 9.149866104125977,
"p95_grad_norm": 1.5837211608886719,
"train_runtime": 79243.6449,
"eval_observations": 19,
"gradient_observations": 1868
},
{
"learning_rate": 0.0001,
"run": "lr_0p00010",
"global_step": 18688,
"epoch": 1.0,
"final_eval_step": 18688,
"final_eval_loss": 0.87261962890625,
"min_eval_step": 18688,
"min_eval_loss": 0.87261962890625,
"max_grad_norm": 8.578277587890625,
"p95_grad_norm": 1.6475107669830322,
"train_runtime": 79344.5851,
"eval_observations": 19,
"gradient_observations": 1868
},
{
"learning_rate": 0.00015,
"run": "lr_0p00015",
"global_step": 18688,
"epoch": 1.0,
"final_eval_step": 18688,
"final_eval_loss": 0.8842697143554688,
"min_eval_step": 18688,
"min_eval_loss": 0.8842697143554688,
"max_grad_norm": 28.051427841186523,
"p95_grad_norm": 1.9590225219726562,
"train_runtime": 80127.6428,
"eval_observations": 19,
"gradient_observations": 1868
},
{
"learning_rate": 0.00018,
"run": "lr_0p00018",
"global_step": 18688,
"epoch": 1.0,
"final_eval_step": 18688,
"final_eval_loss": 0.8914602994918823,
"min_eval_step": 18000,
"min_eval_loss": 0.891387939453125,
"max_grad_norm": 6.5584211349487305,
"p95_grad_norm": 2.163313627243042,
"train_runtime": 79892.0077,
"eval_observations": 19,
"gradient_observations": 1868
},
{
"learning_rate": 0.0002,
"run": "lr_0p00020",
"global_step": 18688,
"epoch": 1.0,
"final_eval_step": 18688,
"final_eval_loss": 0.8976690173149109,
"min_eval_step": 18000,
"min_eval_loss": 0.897506594657898,
"max_grad_norm": 10.969125747680664,
"p95_grad_norm": 2.306173801422119,
"train_runtime": 79710.3252,
"eval_observations": 19,
"gradient_observations": 1868
},
{
"learning_rate": 0.00023,
"run": "lr_0p00023",
"global_step": 18688,
"epoch": 1.0,
"final_eval_step": 18688,
"final_eval_loss": 0.9032362699508667,
"min_eval_step": 18000,
"min_eval_loss": 0.9031931161880493,
"max_grad_norm": 23.10581398010254,
"p95_grad_norm": 2.501060724258423,
"train_runtime": 79789.3562,
"eval_observations": 19,
"gradient_observations": 1868
}
]

338
aya_300k_lr_experiments/scripts/analyze_aya_300k_lr.py Normal file
View File

@ -0,0 +1,338 @@
from pathlib import Path
import csv
import json
import math
ROOT = Path(
"/home/schwarc/diplomovka/"
"aya_300k_lr_validation"
)
RESULT_DIR = ROOT / "results"
RESULT_DIR.mkdir(
parents=True,
exist_ok=True
)
RUNS = [
("0.00018", ROOT / "lr_0p00018"),
("0.00020", ROOT / "lr_0p00020"),
("0.00023", ROOT / "lr_0p00023"),
]
def load_best_state(output):
files = list(
output.rglob(
"trainer_state.json"
)
)
if not files:
return None
candidates = []
for p in files:
try:
data = json.loads(
p.read_text(
encoding="utf-8"
)
)
candidates.append(
(
int(
data.get(
"global_step",
0
)
),
data,
p,
)
)
except Exception:
pass
if not candidates:
return None
return max(
candidates,
key=lambda x: x[0]
)
rows = []
eval_curves = {}
grad_curves = {}
for lr, output in RUNS:
result = load_best_state(
output
)
if result is None:
continue
global_step, state, state_file = result
history = state.get(
"log_history",
[]
)
evals = []
grads = []
for item in history:
step = int(
item.get("step", 0)
)
if "eval_loss" in item:
try:
value = float(
item["eval_loss"]
)
if math.isfinite(value):
evals.append(
(step, value)
)
except Exception:
pass
if "grad_norm" in item:
try:
value = float(
item["grad_norm"]
)
if math.isfinite(value):
grads.append(
(step, value)
)
except Exception:
pass
if evals:
final_eval_step, final_eval = (
evals[-1]
)
min_eval_step, min_eval = min(
evals,
key=lambda x: x[1]
)
else:
final_eval_step = ""
final_eval = ""
min_eval_step = ""
min_eval = ""
grad_values = [
value
for _, value in grads
]
if grad_values:
max_grad = max(
grad_values
)
ordered = sorted(
grad_values
)
p95_index = int(
0.95
* (len(ordered) - 1)
)
p95_grad = (
ordered[p95_index]
)
else:
max_grad = ""
p95_grad = ""
rows.append(
{
"learning_rate": lr,
"global_step": global_step,
"final_eval_step":
final_eval_step,
"final_eval_loss":
final_eval,
"min_eval_step":
min_eval_step,
"min_eval_loss":
min_eval,
"max_grad_norm":
max_grad,
"p95_grad_norm":
p95_grad,
"trainer_state":
str(state_file),
}
)
eval_curves[lr] = evals
grad_curves[lr] = grads
csv_path = (
RESULT_DIR
/ "aya_300k_lr_summary.csv"
)
with csv_path.open(
"w",
encoding="utf-8",
newline=""
) as f:
fields = [
"learning_rate",
"global_step",
"final_eval_step",
"final_eval_loss",
"min_eval_step",
"min_eval_loss",
"max_grad_norm",
"p95_grad_norm",
"trainer_state",
]
writer = csv.DictWriter(
f,
fieldnames=fields
)
writer.writeheader()
writer.writerows(rows)
print()
print("===== 300K LR SUMMARY =====")
for row in rows:
print(
f"LR={row['learning_rate']} | "
f"step={row['global_step']} | "
f"final_eval={row['final_eval_loss']} | "
f"min_eval={row['min_eval_loss']} "
f"@ {row['min_eval_step']} | "
f"max_grad={row['max_grad_norm']} | "
f"p95_grad={row['p95_grad_norm']}"
)
print()
print("CSV:", csv_path)
try:
import matplotlib.pyplot as plt
if eval_curves:
plt.figure(
figsize=(12, 7)
)
for lr, points in (
eval_curves.items()
):
if not points:
continue
plt.plot(
[x for x, _ in points],
[y for _, y in points],
marker="o",
label=f"LR {lr}",
)
plt.xlabel(
"optimizer step"
)
plt.ylabel(
"eval loss"
)
plt.title(
"Aya 300k - eval loss"
)
plt.legend()
plt.grid(alpha=0.25)
plt.tight_layout()
plt.savefig(
RESULT_DIR
/ "eval_loss_over_steps.png",
dpi=200,
bbox_inches="tight"
)
plt.close()
if grad_curves:
plt.figure(
figsize=(12, 7)
)
for lr, points in (
grad_curves.items()
):
if not points:
continue
plt.plot(
[x for x, _ in points],
[y for _, y in points],
label=f"LR {lr}",
)
plt.xlabel(
"optimizer step"
)
plt.ylabel(
"grad norm"
)
plt.title(
"Aya 300k - gradient norm"
)
plt.yscale("log")
plt.legend()
plt.grid(alpha=0.25)
plt.tight_layout()
plt.savefig(
RESULT_DIR
/ "grad_norm_over_steps.png",
dpi=200,
bbox_inches="tight"
)
plt.close()
except Exception as exc:
print(
"Grafy sa nepodarilo vytvorit:",
exc
)

341
aya_300k_lr_experiments/scripts/analyze_aya_300k_lr_all.py Normal file
View File

@ -0,0 +1,341 @@
from pathlib import Path
import csv
import json
import math
ROOT = Path(
"/home/schwarc/diplomovka/"
"aya_300k_lr_validation"
)
RESULT_DIR = ROOT / "results"
RESULT_DIR.mkdir(
parents=True,
exist_ok=True
)
RUNS = [
("0.00008", ROOT / "lr_0p00008"),
("0.00010", ROOT / "lr_0p00010"),
("0.00015", ROOT / "lr_0p00015"),
("0.00018", ROOT / "lr_0p00018"),
("0.00020", ROOT / "lr_0p00020"),
("0.00023", ROOT / "lr_0p00023"),
]
def load_best_state(output):
files = list(
output.rglob(
"trainer_state.json"
)
)
if not files:
return None
candidates = []
for p in files:
try:
data = json.loads(
p.read_text(
encoding="utf-8"
)
)
candidates.append(
(
int(
data.get(
"global_step",
0
)
),
data,
p,
)
)
except Exception:
pass
if not candidates:
return None
return max(
candidates,
key=lambda x: x[0]
)
rows = []
eval_curves = {}
grad_curves = {}
for lr, output in RUNS:
result = load_best_state(
output
)
if result is None:
continue
global_step, state, state_file = result
history = state.get(
"log_history",
[]
)
evals = []
grads = []
for item in history:
step = int(
item.get("step", 0)
)
if "eval_loss" in item:
try:
value = float(
item["eval_loss"]
)
if math.isfinite(value):
evals.append(
(step, value)
)
except Exception:
pass
if "grad_norm" in item:
try:
value = float(
item["grad_norm"]
)
if math.isfinite(value):
grads.append(
(step, value)
)
except Exception:
pass
if evals:
final_eval_step, final_eval = (
evals[-1]
)
min_eval_step, min_eval = min(
evals,
key=lambda x: x[1]
)
else:
final_eval_step = ""
final_eval = ""
min_eval_step = ""
min_eval = ""
grad_values = [
value
for _, value in grads
]
if grad_values:
max_grad = max(
grad_values
)
ordered = sorted(
grad_values
)
p95_index = int(
0.95
* (len(ordered) - 1)
)
p95_grad = (
ordered[p95_index]
)
else:
max_grad = ""
p95_grad = ""
rows.append(
{
"learning_rate": lr,
"global_step": global_step,
"final_eval_step":
final_eval_step,
"final_eval_loss":
final_eval,
"min_eval_step":
min_eval_step,
"min_eval_loss":
min_eval,
"max_grad_norm":
max_grad,
"p95_grad_norm":
p95_grad,
"trainer_state":
str(state_file),
}
)
eval_curves[lr] = evals
grad_curves[lr] = grads
csv_path = (
RESULT_DIR
/ "aya_300k_lr_summary.csv"
)
with csv_path.open(
"w",
encoding="utf-8",
newline=""
) as f:
fields = [
"learning_rate",
"global_step",
"final_eval_step",
"final_eval_loss",
"min_eval_step",
"min_eval_loss",
"max_grad_norm",
"p95_grad_norm",
"trainer_state",
]
writer = csv.DictWriter(
f,
fieldnames=fields
)
writer.writeheader()
writer.writerows(rows)
print()
print("===== 300K LR SUMMARY =====")
for row in rows:
print(
f"LR={row['learning_rate']} | "
f"step={row['global_step']} | "
f"final_eval={row['final_eval_loss']} | "
f"min_eval={row['min_eval_loss']} "
f"@ {row['min_eval_step']} | "
f"max_grad={row['max_grad_norm']} | "
f"p95_grad={row['p95_grad_norm']}"
)
print()
print("CSV:", csv_path)
try:
import matplotlib.pyplot as plt
if eval_curves:
plt.figure(
figsize=(12, 7)
)
for lr, points in (
eval_curves.items()
):
if not points:
continue
plt.plot(
[x for x, _ in points],
[y for _, y in points],
marker="o",
label=f"LR {lr}",
)
plt.xlabel(
"optimizer step"
)
plt.ylabel(
"eval loss"
)
plt.title(
"Aya 300k - eval loss"
)
plt.legend()
plt.grid(alpha=0.25)
plt.tight_layout()
plt.savefig(
RESULT_DIR
/ "eval_loss_over_steps.png",
dpi=200,
bbox_inches="tight"
)
plt.close()
if grad_curves:
plt.figure(
figsize=(12, 7)
)
for lr, points in (
grad_curves.items()
):
if not points:
continue
plt.plot(
[x for x, _ in points],
[y for _, y in points],
label=f"LR {lr}",
)
plt.xlabel(
"optimizer step"
)
plt.ylabel(
"grad norm"
)
plt.title(
"Aya 300k - gradient norm"
)
plt.yscale("log")
plt.legend()
plt.grid(alpha=0.25)
plt.tight_layout()
plt.savefig(
RESULT_DIR
/ "grad_norm_over_steps.png",
dpi=200,
bbox_inches="tight"
)
plt.close()
except Exception as exc:
print(
"Grafy sa nepodarilo vytvorit:",
exc
)

119
aya_300k_lr_experiments/scripts/analyze_training_results.py Normal file
View File

@ -0,0 +1,119 @@
#!/usr/bin/env python3
"""Recompute run summaries and the LR comparison from compact curated evidence.
Run with no arguments from any directory. The standard library produces the CSV
and JSON summaries; matplotlib is required only for the PNG chart. No trainer
state, checkpoints, model weights, or tokenizers are needed.
The gradient P95 matches the archived summary: sort N observed norms and choose
the zero-based index floor(0.95 * (N - 1)), without interpolation.
"""
import csv
import json
import math
from collections import defaultdict
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
RESULTS = ROOT / "results"
def read_rows(filename):
with (RESULTS / filename).open(newline="", encoding="utf-8") as handle:
return list(csv.DictReader(handle))
def summarize():
evaluations = defaultdict(list)
gradients = defaultdict(list)
for row in read_rows("eval_loss_history.csv"):
evaluations[row["run"]].append(row)
for row in read_rows("gradient_history.csv"):
gradients[row["run"]].append(float(row["grad_norm"]))
raw_summary = {float(row["learning_rate"]): row for row in read_rows("aya_300k_lr_summary.csv")}
summaries = []
for run, rows in sorted(evaluations.items()):
rows.sort(key=lambda row: int(row["step"]))
lr = float(rows[0]["learning_rate"])
if any(float(row["learning_rate"]) != lr for row in rows):
raise ValueError(f"Inconsistent learning rate in {run}")
trial = ROOT / "trials" / run
train = json.loads((trial / "train_results.json").read_text(encoding="utf-8"))
final_eval = json.loads((trial / "eval_results.json").read_text(encoding="utf-8"))
norms = sorted(gradients[run])
if not norms or not (trial / "TRAINING_COMPLETE").is_file():
raise ValueError(f"Incomplete evidence for {run}")
final = rows[-1]
best = min(rows, key=lambda row: float(row["eval_loss"]))
summary = {
"learning_rate": lr, "run": run,
"global_step": int(raw_summary[lr]["global_step"]),
"epoch": float(train["epoch"]),
"final_eval_step": int(final["step"]),
"final_eval_loss": float(final["eval_loss"]),
"min_eval_step": int(best["step"]),
"min_eval_loss": float(best["eval_loss"]),
"max_grad_norm": max(norms),
"p95_grad_norm": norms[math.floor(0.95 * (len(norms) - 1))],
"train_runtime": float(train["train_runtime"]),
"eval_observations": len(rows),
"gradient_observations": len(norms),
}
if summary["final_eval_loss"] != float(final_eval["eval_loss"]):
raise ValueError(f"Final evaluation disagrees with eval_results.json: {run}")
for field in ("global_step", "final_eval_step", "final_eval_loss", "min_eval_step",
"min_eval_loss", "max_grad_norm", "p95_grad_norm"):
if float(summary[field]) != float(raw_summary[lr][field]):
raise ValueError(f"Archived summary disagrees at {run}: {field}")
summaries.append(summary)
if len(summaries) != 6 or any(row["global_step"] != 18688 or row["epoch"] != 1.0
or row["eval_observations"] != 19 or row["gradient_observations"] != 1868
for row in summaries):
raise ValueError("Expected six completed one-epoch runs with 19 eval and 1,868 gradient observations each")
with (RESULTS / "training_run_summary.csv").open("w", newline="", encoding="utf-8") as handle:
writer = csv.DictWriter(handle, fieldnames=list(summaries[0]))
writer.writeheader()
writer.writerows(summaries)
(RESULTS / "training_run_summary.json").write_text(json.dumps(summaries, indent=2) + "\n", encoding="utf-8")
return summaries
def plot(summaries):
try:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
except ImportError as exc:
raise SystemExit("Summaries recomputed. Install matplotlib to regenerate the PNG chart.") from exc
plt.rcParams.update({"font.family": "DejaVu Sans", "font.size": 11})
fig, ax = plt.subplots(figsize=(8, 4.8), layout="constrained")
x = [row["learning_rate"] * 1e4 for row in summaries]
y = [row["final_eval_loss"] for row in summaries]
ax.plot(x, y, color="#335c81", marker="o", linewidth=1.6, markersize=6)
selected = min(summaries, key=lambda row: row["final_eval_loss"])
sx, sy = selected["learning_rate"] * 1e4, selected["final_eval_loss"]
ax.scatter([sx], [sy], color="#ba4a00", s=75, zorder=3)
for xi, yi in zip(x, y):
ax.annotate(f"{yi:.5f}", (xi, yi), xytext=(0, 10), textcoords="offset points", ha="center", fontsize=9)
ax.annotate("Selected LR = 0.00010", (sx, sy), xytext=(20, -24),
textcoords="offset points", color="#8a3700", fontsize=10)
ax.set(title="Aya 300k: final validation loss by learning rate",
xlabel="Learning rate (×10⁻⁴)", ylabel="Final validation loss (lower is better)")
ax.set_xticks(x, [f"{value:.1f}" for value in x])
ax.set_ylim(min(y) - 0.009, max(y) + 0.012)
ax.grid(axis="y", color="#d8dee5", linewidth=0.7)
ax.spines[["top", "right"]].set_visible(False)
fig.savefig(RESULTS / "lr_vs_final_eval_loss.png", dpi=200, metadata={"Description": "Six manually chosen learning rates; final evaluation at step 18688 after one epoch."})
plt.close(fig)
def main():
summaries = summarize()
plot(summaries)
best = min(summaries, key=lambda row: row["final_eval_loss"])
print(f"Verified {len(summaries)} runs. Selected {best['run']}: final eval loss {best['final_eval_loss']:.15g}")
print(f"Saved summaries and chart in {RESULTS}")
if __name__ == "__main__":
main()

37
aya_300k_lr_experiments/scripts/rerun_skbench_lr_0p00010.sh Normal file
View File

@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Reconstructed from the completed run's saved parameters and frozen task list.
set -euo pipefail
PROJECT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)
SKBENCH_DIR=${SKBENCH_DIR:-/home/schwarc/diplomovka/sk-bench}
MODEL_DIR=${MODEL_DIR:-/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010}
OUTPUT_ROOT=${OUTPUT_ROOT:-/home/schwarc/diplomovka/skbench_evaluation/results}
GPU_ID=${GPU_ID:-1}
TASKS="$PROJECT/benchmarks/skbench/configs/tasks.txt"
OMITTED="$PROJECT/benchmarks/skbench/configs/omitted_tasks.txt"
test -f "$MODEL_DIR/config.json"
test -f "$TASKS"
test -f "$OMITTED"
cd "$SKBENCH_DIR"
RUN="$OUTPUT_ROOT/sk_zero_available_without_nli_lr_0p00010_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$RUN"
cp "$TASKS" "$RUN/tasks.txt"
cp "$OMITTED" "$RUN/omitted_tasks.txt"
cp uv.lock "$RUN/uv.lock"
git rev-parse HEAD > "$RUN/skbench_commit.txt"
CUDA_VISIBLE_DEVICES="$GPU_ID" VLLM_USE_FLASHINFER_SAMPLER=0 \
uv run --locked python eval_vllm.py \
--model-name "$MODEL_DIR" \
--tasks-file "$RUN/tasks.txt" \
--output-dir "$RUN" \
--tensor-parallel-size 1 \
--dtype bfloat16 \
--gpu-memory-utilization 0.85 \
--max-model-length 32768 \
--max-batched-tokens 2048 \
--enforce-eager \
--save-details \
2>&1 | tee "$RUN/run.log"

76
aya_300k_lr_experiments/scripts/run_aya_300k_lr_validation.sh Normal file
View File

@ -0,0 +1,76 @@
#!/usr/bin/env bash
set -euo pipefail
source /home/schwarc/miniconda3/etc/profile.d/conda.sh
conda activate llama_factory_unsloth
ROOT="/home/schwarc/diplomovka"
EXP="$ROOT/aya_300k_lr_validation"
cd /home/schwarc/LLaMA-Factory
export CUDA_VISIBLE_DEVICES=0
unset FORCE_TORCHRUN || true
export PYTHONUNBUFFERED=1
RUNS=(
"lr_0p00018"
"lr_0p00020"
"lr_0p00023"
)
for NAME in "${RUNS[@]}"; do
CFG="$EXP/configs/$NAME.yaml"
OUT="$EXP/$NAME"
LOG="$EXP/logs/$NAME.log"
DONE="$OUT/TRAINING_COMPLETE"
echo
echo "========================================"
echo "RUN: $NAME"
echo "========================================"
if [ -f "$DONE" ]; then
echo "Uz dokoncene - preskakujem."
continue
fi
# Ak existuje checkpoint po predoslom pade servera,
# radsej automaticky nic neprepiseme.
if [ -d "$OUT" ] && \
find "$OUT" -maxdepth 1 -type d \
-name 'checkpoint-*' | grep -q .; then
echo
echo "POZOR:"
echo "Nasiel som nedokonceny run s checkpointom:"
echo "$OUT"
echo
echo "Nebudem ho automaticky prepisovat."
echo "Treba ho najprv resume-nut."
exit 2
fi
llamafactory-cli train "$CFG" \
2>&1 | tee "$LOG"
mkdir -p "$OUT"
touch "$DONE"
python \
"$ROOT/scripts/analyze_aya_300k_lr.py"
done
echo
echo "========================================"
echo "VSETKY 300K LR BEHY HOTOVE"
echo "========================================"
python \
"$ROOT/scripts/analyze_aya_300k_lr.py"

76
aya_300k_lr_experiments/scripts/run_aya_300k_lr_validation_lower.sh Normal file
View File

@ -0,0 +1,76 @@
#!/usr/bin/env bash
set -euo pipefail
source /home/schwarc/miniconda3/etc/profile.d/conda.sh
conda activate llama_factory_unsloth
ROOT="/home/schwarc/diplomovka"
EXP="$ROOT/aya_300k_lr_validation"
cd /home/schwarc/LLaMA-Factory
export CUDA_VISIBLE_DEVICES=0
unset FORCE_TORCHRUN || true
export PYTHONUNBUFFERED=1
RUNS=(
"lr_0p00015"
"lr_0p00010"
"lr_0p00008"
)
for NAME in "${RUNS[@]}"; do
CFG="$EXP/configs/$NAME.yaml"
OUT="$EXP/$NAME"
LOG="$EXP/logs/$NAME.log"
DONE="$OUT/TRAINING_COMPLETE"
echo
echo "========================================"
echo "RUN: $NAME"
echo "========================================"
if [ -f "$DONE" ]; then
echo "Uz dokoncene - preskakujem."
continue
fi
# Ak existuje checkpoint po predoslom pade servera,
# radsej automaticky nic neprepiseme.
if [ -d "$OUT" ] && \
find "$OUT" -maxdepth 1 -type d \
-name 'checkpoint-*' | grep -q .; then
echo
echo "POZOR:"
echo "Nasiel som nedokonceny run s checkpointom:"
echo "$OUT"
echo
echo "Nebudem ho automaticky prepisovat."
echo "Treba ho najprv resume-nut."
exit 2
fi
llamafactory-cli train "$CFG" \
2>&1 | tee "$LOG"
mkdir -p "$OUT"
touch "$DONE"
python \
"$ROOT/scripts/analyze_aya_300k_lr_all.py"
done
echo
echo "========================================"
echo "VSETKY 300K LR BEHY HOTOVE"
echo "========================================"
python \
"$ROOT/scripts/analyze_aya_300k_lr_all.py"

74
aya_300k_lr_experiments/scripts/run_lmeval_aya_300k_lr_0p00010.sh Normal file
View File

@ -0,0 +1,74 @@
#!/usr/bin/env bash
set -euo pipefail
eval "$(/home/schwarc/miniconda3/bin/conda shell.bash hook)"
conda activate lm_eval_hruska
cd /home/schwarc/lm-evaluation-harness
# This directory already contains the base model merged with the Aya adapter.
MODEL="/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010"
TASKS="arc_sk,hellaswag_sk,m_mmlu_sk,truthfulqa_sk_mc1,truthfulqa_sk_mc2,sklegal,skquad"
INCLUDE_PATH="/home/schwarc/diplomovka/lmeval_slovak_tasks"
ROOT="/home/schwarc/diplomovka/aya_300k_lr_validation/lmeval"
NAME="lr_0p00010_$(date +%Y%m%d_%H%M%S)"
OUT="$ROOT/results/$NAME"
LOG="$ROOT/logs/$NAME.log"
SUMMARY="$ROOT/summary.csv"
test -f "$MODEL/config.json"
test -f "$MODEL/model.safetensors.index.json"
test -d "$INCLUDE_PATH"
mkdir -p "$OUT" "$ROOT/logs"
cp -a "$INCLUDE_PATH" "$OUT/custom_tasks"
git rev-parse HEAD > "$OUT/lm_eval_commit.txt"
printf '%s\n' "$TASKS" > "$OUT/tasks.txt"
printf '%s\n' "model=hf" "pretrained=$MODEL" "tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4" "dtype=bfloat16" \
"num_fewshot=0" "batch_size=1" "apply_chat_template=false" \
"CUDA_VISIBLE_DEVICES=1" > "$OUT/run_settings.txt"
python - <<'PY' > "$OUT/package_versions.json"
import importlib.metadata as md
import json
versions = {}
for package in ("lm_eval", "transformers", "torch", "datasets", "peft", "evaluate"):
try:
versions[package] = md.version(package)
except md.PackageNotFoundError:
versions[package] = None
print(json.dumps(versions, indent=2))
PY
if [[ ! -f "$SUMMARY" ]]; then
printf '%s\n' 'model,exit_code,start_time,end_time,log_file' > "$SUMMARY"
fi
START_TIME=$(date '+%Y-%m-%d %H:%M:%S')
printf 'Model: %s\nResults: %s\nLog: %s\n' "$MODEL" "$OUT" "$LOG"
set +e
CUDA_VISIBLE_DEVICES=1 lm_eval \
--model hf \
--model_args "pretrained=$MODEL,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16" \
--tasks "$TASKS" \
--include_path "$INCLUDE_PATH" \
--device cuda:0 \
--batch_size 1 \
--num_fewshot 0 \
--output_path "$OUT" \
2>&1 | tee "$LOG"
EXIT_CODES=("${PIPESTATUS[@]}")
set -e
EXIT_CODE=${EXIT_CODES[0]}
if (( EXIT_CODE == 0 )); then
EXIT_CODE=${EXIT_CODES[1]}
fi
END_TIME=$(date '+%Y-%m-%d %H:%M:%S')
printf '%s,%s,%s,%s,%s\n' "$NAME" "$EXIT_CODE" "$START_TIME" "$END_TIME" "$LOG" >> "$SUMMARY"
if (( EXIT_CODE == 0 )); then
printf 'Evaluation completed: %s\n' "$OUT"
else
printf 'Evaluation failed (exit code %s). Log: %s\n' "$EXIT_CODE" "$LOG" >&2
fi
exit "$EXIT_CODE"

899
aya_300k_lr_experiments/scripts/setup_aya_300k_lr_validation.sh Normal file
View File

@ -0,0 +1,899 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="/home/schwarc/diplomovka"
LF="/home/schwarc/LLaMA-Factory"
EXP="$ROOT/aya_300k_lr_validation"
CONFIG_DIR="$EXP/configs"
LOG_DIR="$EXP/logs"
RESULT_DIR="$EXP/results"
DATA_FILE="$LF/data/aya_slovak_300k_fixed.jsonl"
DATASET_INFO="$LF/data/dataset_info.json"
BASE_CONFIG="$ROOT/optuna_lr_hpo_1gpu/base_hpo_config.yaml"
mkdir -p "$EXP" "$CONFIG_DIR" "$LOG_DIR" "$RESULT_DIR" "$ROOT/scripts"
source /home/schwarc/miniconda3/etc/profile.d/conda.sh
conda activate llama_factory_unsloth
echo
echo "========================================"
echo "1. KONTROLA PROSTREDIA"
echo "========================================"
python - <<'PY'
import torch
import datasets
import yaml
print("Torch:", torch.__version__)
print("CUDA:", torch.cuda.is_available())
print("Datasets:", datasets.__version__)
if torch.cuda.is_available():
print("GPU 0:", torch.cuda.get_device_name(0))
PY
# ============================================================
# 2. VYTVORENIE FIXNEHO 300K DATASETU
# ============================================================
echo
echo "========================================"
echo "2. AYA SLOVAK 300K"
echo "========================================"
python - <<'PY'
from pathlib import Path
from datasets import load_dataset
import json
TARGET = Path(
"/home/schwarc/LLaMA-Factory/data/"
"aya_slovak_300k_fixed.jsonl"
)
EXPECTED = 300_000
SEED = 42
def count_lines(path):
with path.open("r", encoding="utf-8") as f:
return sum(1 for _ in f)
if TARGET.exists():
count = count_lines(TARGET)
if count == EXPECTED:
print(
f"Dataset uz existuje a ma spravne "
f"{count} riadkov."
)
raise SystemExit(0)
else:
raise RuntimeError(
f"{TARGET} existuje, ale ma "
f"{count} riadkov namiesto {EXPECTED}. "
"Nebudem ho automaticky prepisovat."
)
print("Nacitavam Aya Collection - slovak/train...")
ds = load_dataset(
"CohereLabs/aya_collection_language_split",
"slovak",
split="train",
)
print("Povodny train split:", len(ds))
def valid_batch(batch):
result = []
for inp, target in zip(
batch["inputs"],
batch["targets"],
):
ok = (
isinstance(inp, str)
and isinstance(target, str)
and bool(inp.strip())
and bool(target.strip())
)
result.append(ok)
return result
print("Filtrujem prazdne inputs/targets...")
ds = ds.filter(
valid_batch,
batched=True,
batch_size=10000,
num_proc=4,
)
print("Platne zaznamy:", len(ds))
if len(ds) < EXPECTED:
raise RuntimeError(
f"Po filtrovani ostalo iba {len(ds)} zaznamov."
)
print("Shuffle seed =", SEED)
ds = ds.shuffle(seed=SEED)
ds = ds.select(
range(EXPECTED)
)
print("Zapisujem:", TARGET)
with TARGET.open(
"w",
encoding="utf-8"
) as f:
for row in ds:
item = {
"instruction": row["inputs"].strip(),
"input": "",
"output": row["targets"].strip(),
}
f.write(
json.dumps(
item,
ensure_ascii=False
)
+ "\n"
)
count = count_lines(TARGET)
if count != EXPECTED:
raise RuntimeError(
f"Finalny dataset ma {count} riadkov."
)
print("HOTOVO.")
print("Pocet:", count)
print("Dataset:", TARGET)
PY
# ============================================================
# 3. DATASET_INFO.JSON
# ============================================================
echo
echo "========================================"
echo "3. DATASET_INFO"
echo "========================================"
python - <<'PY'
from pathlib import Path
import json
import copy
import shutil
p = Path(
"/home/schwarc/LLaMA-Factory/data/"
"dataset_info.json"
)
backup = p.parent / (
"dataset_info_before_aya300k_backup.json"
)
if not backup.exists():
shutil.copy2(p, backup)
with p.open(
"r",
encoding="utf-8"
) as f:
info = json.load(f)
source_name = "aya_slovak_100k_fixed"
new_name = "aya_slovak_300k_fixed"
if source_name not in info:
raise RuntimeError(
f"V dataset_info chyba {source_name}"
)
entry = copy.deepcopy(
info[source_name]
)
entry["file_name"] = (
"aya_slovak_300k_fixed.jsonl"
)
info[new_name] = entry
with p.open(
"w",
encoding="utf-8"
) as f:
json.dump(
info,
f,
ensure_ascii=False,
indent=2
)
print(
"Registrovany dataset:",
new_name
)
print(
json.dumps(
info[new_name],
ensure_ascii=False,
indent=2
)
)
PY
# ============================================================
# 4. VYTVORENIE 3 CONFIGOV
# ============================================================
echo
echo "========================================"
echo "4. CONFIGY"
echo "========================================"
python - <<'PY'
from pathlib import Path
import yaml
ROOT = Path(
"/home/schwarc/diplomovka"
)
EXP = ROOT / "aya_300k_lr_validation"
BASE = (
ROOT
/ "optuna_lr_hpo_1gpu"
/ "base_hpo_config.yaml"
)
if not BASE.exists():
BASE = (
ROOT
/ "configs"
/ "hruska_compare9_100k_fixed"
/ "10_r16_a32_lr1e4.yaml"
)
if not BASE.exists():
raise RuntimeError(
"Nenasiel som base config."
)
with BASE.open(
"r",
encoding="utf-8"
) as f:
base = yaml.safe_load(f)
# ------------------------------------------------------------
# FIXNE PARAMETRE PRE VSETKY 3 BEHY
# ------------------------------------------------------------
base["dataset"] = "aya_slovak_300k_fixed"
base["streaming"] = False
# presne 1 cely epoch
base.pop("max_steps", None)
base["num_train_epochs"] = 1.0
# 300k -> 1000 validation -> cca 299k train
base["val_size"] = 1000
# 1 GPU:
# 1 * 16 * 1 = effective batch 16
base["per_device_train_batch_size"] = 1
base["per_device_eval_batch_size"] = 1
base["gradient_accumulation_steps"] = 16
# cca 18 688 optimizer krokov
# ~8 % warmup
base["warmup_steps"] = 1500
base.pop("warmup_ratio", None)
# chceme sledovat priebeh
base["eval_strategy"] = "steps"
base["eval_steps"] = 1000
base["logging_steps"] = 10
# checkpointy pre pripad restartu servera
base["save_steps"] = 2000
base["save_total_limit"] = 2
base["seed"] = 42
base["overwrite_output_dir"] = True
base["report_to"] = "none"
# explicitne zachovame Unsloth + bf16
base["use_unsloth"] = True
base["use_unsloth_gc"] = True
base["bf16"] = True
base["fp16"] = False
# chceme rovnaky optimizer/scheduler
base["optim"] = "adamw_torch"
base["lr_scheduler_type"] = "cosine"
# rovnaka gradient clipping hranica
base["max_grad_norm"] = 0.5
runs = [
("lr_0p00018", 0.00018),
("lr_0p00020", 0.00020),
("lr_0p00023", 0.00023),
]
for name, lr in runs:
cfg = dict(base)
cfg["learning_rate"] = lr
cfg["output_dir"] = str(
EXP / name
)
path = (
EXP
/ "configs"
/ f"{name}.yaml"
)
with path.open(
"w",
encoding="utf-8"
) as f:
yaml.safe_dump(
cfg,
f,
sort_keys=False,
allow_unicode=True
)
print()
print(path)
print(" LR:", lr)
print(
" output:",
cfg["output_dir"]
)
print()
print("Base kontrola:")
print("dataset:", base["dataset"])
print("epochs:", base["num_train_epochs"])
print(
"batch:",
base["per_device_train_batch_size"]
)
print(
"grad_accum:",
base["gradient_accumulation_steps"]
)
print(
"warmup:",
base["warmup_steps"]
)
print(
"eval_steps:",
base["eval_steps"]
)
print(
"save_steps:",
base["save_steps"]
)
print(
"LoRA rank:",
base.get("lora_rank")
)
print(
"LoRA alpha:",
base.get("lora_alpha")
)
print(
"LoRA dropout:",
base.get("lora_dropout")
)
print(
"LoRA target:",
base.get("lora_target")
)
PY
# ============================================================
# 5. ANALYZATOR
# ============================================================
cat > "$ROOT/scripts/analyze_aya_300k_lr.py" <<'PY'
from pathlib import Path
import csv
import json
import math
ROOT = Path(
"/home/schwarc/diplomovka/"
"aya_300k_lr_validation"
)
RESULT_DIR = ROOT / "results"
RESULT_DIR.mkdir(
parents=True,
exist_ok=True
)
RUNS = [
("0.00018", ROOT / "lr_0p00018"),
("0.00020", ROOT / "lr_0p00020"),
("0.00023", ROOT / "lr_0p00023"),
]
def load_best_state(output):
files = list(
output.rglob(
"trainer_state.json"
)
)
if not files:
return None
candidates = []
for p in files:
try:
data = json.loads(
p.read_text(
encoding="utf-8"
)
)
candidates.append(
(
int(
data.get(
"global_step",
0
)
),
data,
p,
)
)
except Exception:
pass
if not candidates:
return None
return max(
candidates,
key=lambda x: x[0]
)
rows = []
eval_curves = {}
grad_curves = {}
for lr, output in RUNS:
result = load_best_state(
output
)
if result is None:
continue
global_step, state, state_file = result
history = state.get(
"log_history",
[]
)
evals = []
grads = []
for item in history:
step = int(
item.get("step", 0)
)
if "eval_loss" in item:
try:
value = float(
item["eval_loss"]
)
if math.isfinite(value):
evals.append(
(step, value)
)
except Exception:
pass
if "grad_norm" in item:
try:
value = float(
item["grad_norm"]
)
if math.isfinite(value):
grads.append(
(step, value)
)
except Exception:
pass
if evals:
final_eval_step, final_eval = (
evals[-1]
)
min_eval_step, min_eval = min(
evals,
key=lambda x: x[1]
)
else:
final_eval_step = ""
final_eval = ""
min_eval_step = ""
min_eval = ""
grad_values = [
value
for _, value in grads
]
if grad_values:
max_grad = max(
grad_values
)
ordered = sorted(
grad_values
)
p95_index = int(
0.95
* (len(ordered) - 1)
)
p95_grad = (
ordered[p95_index]
)
else:
max_grad = ""
p95_grad = ""
rows.append(
{
"learning_rate": lr,
"global_step": global_step,
"final_eval_step":
final_eval_step,
"final_eval_loss":
final_eval,
"min_eval_step":
min_eval_step,
"min_eval_loss":
min_eval,
"max_grad_norm":
max_grad,
"p95_grad_norm":
p95_grad,
"trainer_state":
str(state_file),
}
)
eval_curves[lr] = evals
grad_curves[lr] = grads
csv_path = (
RESULT_DIR
/ "aya_300k_lr_summary.csv"
)
with csv_path.open(
"w",
encoding="utf-8",
newline=""
) as f:
fields = [
"learning_rate",
"global_step",
"final_eval_step",
"final_eval_loss",
"min_eval_step",
"min_eval_loss",
"max_grad_norm",
"p95_grad_norm",
"trainer_state",
]
writer = csv.DictWriter(
f,
fieldnames=fields
)
writer.writeheader()
writer.writerows(rows)
print()
print("===== 300K LR SUMMARY =====")
for row in rows:
print(
f"LR={row['learning_rate']} | "
f"step={row['global_step']} | "
f"final_eval={row['final_eval_loss']} | "
f"min_eval={row['min_eval_loss']} "
f"@ {row['min_eval_step']} | "
f"max_grad={row['max_grad_norm']} | "
f"p95_grad={row['p95_grad_norm']}"
)
print()
print("CSV:", csv_path)
try:
import matplotlib.pyplot as plt
if eval_curves:
plt.figure(
figsize=(12, 7)
)
for lr, points in (
eval_curves.items()
):
if not points:
continue
plt.plot(
[x for x, _ in points],
[y for _, y in points],
marker="o",
label=f"LR {lr}",
)
plt.xlabel(
"optimizer step"
)
plt.ylabel(
"eval loss"
)
plt.title(
"Aya 300k - eval loss"
)
plt.legend()
plt.grid(alpha=0.25)
plt.tight_layout()
plt.savefig(
RESULT_DIR
/ "eval_loss_over_steps.png",
dpi=200,
bbox_inches="tight"
)
plt.close()
if grad_curves:
plt.figure(
figsize=(12, 7)
)
for lr, points in (
grad_curves.items()
):
if not points:
continue
plt.plot(
[x for x, _ in points],
[y for _, y in points],
label=f"LR {lr}",
)
plt.xlabel(
"optimizer step"
)
plt.ylabel(
"grad norm"
)
plt.title(
"Aya 300k - gradient norm"
)
plt.yscale("log")
plt.legend()
plt.grid(alpha=0.25)
plt.tight_layout()
plt.savefig(
RESULT_DIR
/ "grad_norm_over_steps.png",
dpi=200,
bbox_inches="tight"
)
plt.close()
except Exception as exc:
print(
"Grafy sa nepodarilo vytvorit:",
exc
)
PY
# ============================================================
# 6. RUNNER
# ============================================================
cat > "$ROOT/scripts/run_aya_300k_lr_validation.sh" <<'BASH2'
#!/usr/bin/env bash
set -euo pipefail
source /home/schwarc/miniconda3/etc/profile.d/conda.sh
conda activate llama_factory_unsloth
ROOT="/home/schwarc/diplomovka"
EXP="$ROOT/aya_300k_lr_validation"
cd /home/schwarc/LLaMA-Factory
export CUDA_VISIBLE_DEVICES=0
unset FORCE_TORCHRUN || true
export PYTHONUNBUFFERED=1
RUNS=(
"lr_0p00018"
"lr_0p00020"
"lr_0p00023"
)
for NAME in "${RUNS[@]}"; do
CFG="$EXP/configs/$NAME.yaml"
OUT="$EXP/$NAME"
LOG="$EXP/logs/$NAME.log"
DONE="$OUT/TRAINING_COMPLETE"
echo
echo "========================================"
echo "RUN: $NAME"
echo "========================================"
if [ -f "$DONE" ]; then
echo "Uz dokoncene - preskakujem."
continue
fi
# Ak existuje checkpoint po predoslom pade servera,
# radsej automaticky nic neprepiseme.
if [ -d "$OUT" ] && \
find "$OUT" -maxdepth 1 -type d \
-name 'checkpoint-*' | grep -q .; then
echo
echo "POZOR:"
echo "Nasiel som nedokonceny run s checkpointom:"
echo "$OUT"
echo
echo "Nebudem ho automaticky prepisovat."
echo "Treba ho najprv resume-nut."
exit 2
fi
llamafactory-cli train "$CFG" \
2>&1 | tee "$LOG"
mkdir -p "$OUT"
touch "$DONE"
python \
"$ROOT/scripts/analyze_aya_300k_lr.py"
done
echo
echo "========================================"
echo "VSETKY 300K LR BEHY HOTOVE"
echo "========================================"
python \
"$ROOT/scripts/analyze_aya_300k_lr.py"
BASH2
chmod +x \
"$ROOT/scripts/run_aya_300k_lr_validation.sh"
echo
echo "========================================"
echo "SETUP HOTOVY"
echo "========================================"
echo
echo "Dataset:"
wc -l "$DATA_FILE"
echo
echo "Configy:"
ls -1 "$CONFIG_DIR"
echo
echo "Runner:"
echo "$ROOT/scripts/run_aya_300k_lr_validation.sh"

74
aya_300k_lr_experiments/scripts/unpack_results_archive.py Normal file
View File

@ -0,0 +1,74 @@
from pathlib import Path, PurePosixPath
import argparse
import collections
import hashlib
import json
import re
import tarfile
parser = argparse.ArgumentParser(description="Safely extract an experiment archive and create audit path manifests (Python 3.11+).")
parser.add_argument("--archive", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
SOURCE = args.archive.resolve()
DEST = args.output.resolve()
if DEST.exists() and any(DEST.iterdir()):
raise SystemExit(f"Output directory must be empty: {DEST}")
DEST.mkdir(parents=True, exist_ok=True)
start_size = SOURCE.stat().st_size
def safe_component(value):
value = value.replace('%', '%25')
value = re.sub(r'[<>:"\\|?*\x00-\x1f]', lambda m: f'%{ord(m.group()):02X}', value)
while value.endswith((' ', '.')):
value = value[:-1] + f'%{ord(value[-1]):02X}'
if value.split('.')[0].upper() in {'CON', 'PRN', 'AUX', 'NUL', *(f'COM{i}' for i in range(1, 10)), *(f'LPT{i}' for i in range(1, 10))}:
value = '%5F' + value
return value
manifest = []
seen = set()
with tarfile.open(SOURCE, 'r:gz') as archive:
for member in archive:
original = PurePosixPath(member.name)
if original.is_absolute() or '..' in original.parts or '\\' in member.name:
raise RuntimeError(f'Unsafe archive path: {member.name!r}')
relative = Path(*(safe_component(x) for x in original.parts if x not in ('', '.')))
target = DEST / relative
if not target.resolve().is_relative_to(DEST.resolve()):
raise RuntimeError(f'Path escapes extraction root: {member.name!r}')
row = {'original': member.name, 'local': relative.as_posix(), 'size': member.size, 'type': 'other'}
if member.isdir():
target.mkdir(parents=True, exist_ok=True)
row['type'] = 'directory'
elif member.isfile():
key = relative.as_posix().casefold()
if key in seen:
raise RuntimeError(f'Archive filename collision: {member.name!r}')
seen.add(key)
target.parent.mkdir(parents=True, exist_ok=True)
extracted = archive.extractfile(member)
assert extracted is not None
with extracted, target.open('wb') as handle:
while chunk := extracted.read(1024 * 1024):
handle.write(chunk)
if target.stat().st_size != member.size:
raise RuntimeError(f'Incomplete extraction: {member.name!r}')
row['type'] = 'file'
else:
row['type'] = 'skipped_link_or_special'
manifest.append(row)
if SOURCE.stat().st_size != start_size:
raise RuntimeError('Archive changed during extraction')
with SOURCE.open('rb') as handle:
digest = hashlib.file_digest(handle, 'sha256').hexdigest()
(DEST / 'archive_manifest.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding='utf-8')
files = [x for x in manifest if x['type'] == 'file']
summary = {'source': str(SOURCE), 'compressed_bytes': start_size, 'sha256': digest,
'files': len(files), 'uncompressed_bytes': sum(x['size'] for x in files),
'skipped_special': [x for x in manifest if x['type'] == 'skipped_link_or_special'],
'top_level_files': dict(collections.Counter(PurePosixPath(x['original']).parts[0] for x in files)),
'largest_files': sorted(files, key=lambda x: x['size'], reverse=True)[:12]}
(DEST / 'archive_summary.json').write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding='utf-8')
print(json.dumps(summary, ensure_ascii=False, indent=2))

0
aya_300k_lr_experiments/trials/lr_0p00008/TRAINING_COMPLETE Normal file
View File

50
aya_300k_lr_experiments/trials/lr_0p00008/adapter_config.json Normal file
View File

@ -0,0 +1,50 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": {
"base_model_class": "MistralForCausalLM",
"parent_library": "transformers.models.mistral.modeling_mistral",
"unsloth_fixed": true
},
"base_model_name_or_path": "slovak-nlp/mistral-sk-7b",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 32,
"lora_bias": false,
"lora_dropout": 0.0,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.18.1",
"qalora_group_size": 16,
"r": 16,
"rank_pattern": {},
"revision": null,
"target_modules": [
"q_proj",
"o_proj",
"gate_proj",
"k_proj",
"up_proj",
"v_proj",
"down_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

13
aya_300k_lr_experiments/trials/lr_0p00008/all_results.json Normal file
View File

@ -0,0 +1,13 @@
{
"epoch": 1.0,
"eval_loss": 0.875216543674469,
"eval_runtime": 84.3566,
"eval_samples_per_second": 11.854,
"eval_steps_per_second": 11.854,
"num_input_tokens_seen": 62891360,
"total_flos": 2.69902163257983e+18,
"train_loss": 0.9401782060884041,
"train_runtime": 79243.6449,
"train_samples_per_second": 3.773,
"train_steps_per_second": 0.236
}

8
aya_300k_lr_experiments/trials/lr_0p00008/eval_results.json Normal file
View File

@ -0,0 +1,8 @@
{
"epoch": 1.0,
"eval_loss": 0.875216543674469,
"eval_runtime": 84.3566,
"eval_samples_per_second": 11.854,
"eval_steps_per_second": 11.854,
"num_input_tokens_seen": 62891360
}

9
aya_300k_lr_experiments/trials/lr_0p00008/train_results.json Normal file
View File

@ -0,0 +1,9 @@
{
"epoch": 1.0,
"num_input_tokens_seen": 62891360,
"total_flos": 2.69902163257983e+18,
"train_loss": 0.9401782060884041,
"train_runtime": 79243.6449,
"train_samples_per_second": 3.773,
"train_steps_per_second": 0.236
}

BIN
aya_300k_lr_experiments/trials/lr_0p00008/training_eval_loss.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
aya_300k_lr_experiments/trials/lr_0p00008/training_loss.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

0
aya_300k_lr_experiments/trials/lr_0p00010/TRAINING_COMPLETE Normal file
View File

50
aya_300k_lr_experiments/trials/lr_0p00010/adapter_config.json Normal file
View File

@ -0,0 +1,50 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": {
"base_model_class": "MistralForCausalLM",
"parent_library": "transformers.models.mistral.modeling_mistral",
"unsloth_fixed": true
},
"base_model_name_or_path": "slovak-nlp/mistral-sk-7b",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 32,
"lora_bias": false,
"lora_dropout": 0.0,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.18.1",
"qalora_group_size": 16,
"r": 16,
"rank_pattern": {},
"revision": null,
"target_modules": [
"down_proj",
"o_proj",
"up_proj",
"q_proj",
"gate_proj",
"k_proj",
"v_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

13
aya_300k_lr_experiments/trials/lr_0p00010/all_results.json Normal file
View File

@ -0,0 +1,13 @@
{
"epoch": 1.0,
"eval_loss": 0.87261962890625,
"eval_runtime": 84.2749,
"eval_samples_per_second": 11.866,
"eval_steps_per_second": 11.866,
"num_input_tokens_seen": 62891360,
"total_flos": 2.69902163257983e+18,
"train_loss": 0.942430813171684,
"train_runtime": 79344.5851,
"train_samples_per_second": 3.768,
"train_steps_per_second": 0.236
}

8
aya_300k_lr_experiments/trials/lr_0p00010/eval_results.json Normal file
View File

@ -0,0 +1,8 @@
{
"epoch": 1.0,
"eval_loss": 0.87261962890625,
"eval_runtime": 84.2749,
"eval_samples_per_second": 11.866,
"eval_steps_per_second": 11.866,
"num_input_tokens_seen": 62891360
}

9
aya_300k_lr_experiments/trials/lr_0p00010/train_results.json Normal file
View File

@ -0,0 +1,9 @@
{
"epoch": 1.0,
"num_input_tokens_seen": 62891360,
"total_flos": 2.69902163257983e+18,
"train_loss": 0.942430813171684,
"train_runtime": 79344.5851,
"train_samples_per_second": 3.768,
"train_steps_per_second": 0.236
}

Some files were not shown because too many files have changed in this diff Show More