Document six completed LR runs as preparation for full Aya training on Perun. Record selection of LR 0.00010 by lowest final validation loss, selected-model lm-eval and partial SK-bench results, reproducible provenance, and dataset-access/context-truncation limitations. Preserve existing 30k and Titan material.
4.0 KiB
4.0 KiB
| 1 | family | metric | task_variants | prompt_count | evaluated_instances | task_macro_mean | sample_weighted_mean | min_task_value | max_task_value |
|---|---|---|---|---|---|---|---|---|---|
| 2 | arc | em | 3 | 3 | 3507 | 0.023666951810664385 | 0.023666951810664385 | 0.013686911890504704 | 0.03507271171941831 |
| 3 | arc | em_conservative | 3 | 3 | 3507 | 0.19019104647847163 | 0.19019104647847163 | 0.1710863986313088 | 0.223267750213858 |
| 4 | arc | em_relaxed | 3 | 3 | 3507 | 0.19076133447390933 | 0.19076133447390933 | 0.1710863986313088 | 0.22412318220701455 |
| 5 | belebele | em | 3 | 3 | 2700 | 0.022222222222222223 | 0.022222222222222223 | 0.016666666666666666 | 0.03 |
| 6 | belebele | em_conservative | 3 | 3 | 2700 | 0.21259259259259258 | 0.21259259259259258 | 0.2111111111111111 | 0.21555555555555556 |
| 7 | belebele | em_relaxed | 3 | 3 | 2700 | 0.21333333333333335 | 0.21333333333333335 | 0.2111111111111111 | 0.21555555555555556 |
| 8 | financial_exams | em | 3 | 3 | 4002 | 0.029735132433783108 | 0.029735132433783108 | 0.0037481259370314842 | 0.07571214392803598 |
| 9 | financial_exams | em_conservative | 3 | 3 | 4002 | 0.21839080459770116 | 0.21839080459770116 | 0.20089955022488756 | 0.24887556221889057 |
| 10 | financial_exams | em_relaxed | 3 | 3 | 4002 | 0.22938530734632684 | 0.22938530734632684 | 0.20089955022488756 | 0.24887556221889057 |
| 11 | globalmmlu-lite | em | 3 | 3 | 1200 | 0.028333333333333335 | 0.028333333333333332 | 0.025 | 0.035 |
| 12 | globalmmlu-lite | em_conservative | 3 | 3 | 1200 | 0.19833333333333333 | 0.19833333333333333 | 0.1825 | 0.21 |
| 13 | globalmmlu-lite | em_relaxed | 3 | 3 | 1200 | 0.19833333333333333 | 0.19833333333333333 | 0.1825 | 0.21 |
| 14 | mmlu | em | 171 | 3 | 39186 | 0.03148675340198335 | 0.029576889705507068 | 0.0 | 0.13 |
| 15 | mmlu | em_conservative | 171 | 3 | 39186 | 0.15665550650347979 | 0.15464706783034757 | 0.0026702269692923898 | 0.3553299492385787 |
| 16 | mmlu | em_relaxed | 171 | 3 | 39186 | 0.16009412530257372 | 0.15722451896085338 | 0.0026702269692923898 | 0.3553299492385787 |
| 17 | movie_review | em_conservative_norm | 3 | 3 | 6024 | 0.40139442231075695 | 0.40139442231075695 | 0.09412350597609562 | 0.6324701195219123 |
| 18 | movie_review | em_norm | 3 | 3 | 6024 | 0.14840637450199204 | 0.14840637450199204 | 0.01045816733067729 | 0.38745019920318724 |
| 19 | movie_review | em_norm_relaxed | 3 | 3 | 6024 | 0.40903054448871184 | 0.40903054448871184 | 0.11105577689243028 | 0.6329681274900398 |
| 20 | multi_wiki_qa | exact_match_slk_prefix | 3 | 3 | 14991 | 0.35467947435127745 | 0.35467947435127745 | 0.34000400240144085 | 0.36401841104662797 |
| 21 | multi_wiki_qa | f1_slk | 3 | 3 | 14991 | 0.5099047074319363 | 0.5099047074319363 | 0.5050071786470199 | 0.5156574509584114 |
| 22 | senti_sk | em_conservative_norm | 6 | 3 | 25215 | 0.32734853969888256 | 0.32655165576046 | 0.1990343347639485 | 0.49302575107296137 |
| 23 | senti_sk | em_norm | 6 | 3 | 25215 | 0.09719367950733247 | 0.09942494546896688 | 0.0037553648068669528 | 0.22642719692110327 |
| 24 | senti_sk | em_norm_relaxed | 6 | 3 | 25215 | 0.3621009200810101 | 0.36307753321435654 | 0.25160944206008584 | 0.4965128755364807 |
| 25 | skj1 | em | 3 | 3 | 300 | 0.0 | 0.0 | 0.0 | 0.0 |
| 26 | skj1 | em_conservative | 3 | 3 | 300 | 0.03333333333333333 | 0.03333333333333333 | 0.02 | 0.05 |
| 27 | skj1 | em_relaxed | 3 | 3 | 300 | 0.043333333333333335 | 0.043333333333333335 | 0.02 | 0.06 |
| 28 | sklep | em_conservative_norm | 12 | 3 | 27075 | 0.34747054025912527 | 0.21277931671283473 | 0.001398880895283773 | 0.8800383877159309 |
| 29 | sklep | em_norm | 12 | 3 | 27075 | 0.039042125070919684 | 0.01817174515235457 | 0.0 | 0.44817658349328215 |
| 30 | sklep | em_norm_relaxed | 12 | 3 | 27075 | 0.3507994510268643 | 0.21440443213296398 | 0.001398880895283773 | 0.8809980806142035 |
| 31 | sklep | exact_match_slk_prefix | 3 | 3 | 28749 | 0.16372743399770426 | 0.16372743399770426 | 0.16143170197224252 | 0.16518835437754356 |
| 32 | sklep | f1_slk | 3 | 3 | 28749 | 0.3750415088928484 | 0.3750415088928484 | 0.3730151221096053 | 0.3769392643579436 |
| 33 | slovakcopa | em | 3 | 3 | 1500 | 0.0 | 0.0 | 0.0 | 0.0 |
| 34 | slovakcopa | em_conservative | 3 | 3 | 1500 | 0.306 | 0.306 | 0.084 | 0.508 |
| 35 | slovakcopa | em_relaxed | 3 | 3 | 1500 | 0.4046666666666667 | 0.4046666666666667 | 0.086 | 0.618 |
| 36 | toxic_sk | em_conservative_norm | 3 | 3 | 2643 | 0.4759742716609913 | 0.4759742716609913 | 0.45629965947786605 | 0.4960272417707151 |
| 37 | toxic_sk | em_norm | 3 | 3 | 2643 | 0.0026485054861899358 | 0.0026485054861899358 | 0.0 | 0.007945516458569807 |
| 38 | toxic_sk | em_norm_relaxed | 3 | 3 | 2643 | 0.4759742716609913 | 0.4759742716609913 | 0.45629965947786605 | 0.4960272417707151 |
| 39 | truthfulqa | em | 3 | 3 | 2334 | 0.0 | 0.0 | 0.0 | 0.0 |
| 40 | truthfulqa | em_conservative | 3 | 3 | 2334 | 0.011996572407883462 | 0.011996572407883462 | 0.007712082262210797 | 0.016709511568123392 |
| 41 | truthfulqa | em_relaxed | 3 | 3 | 2334 | 0.011996572407883462 | 0.011996572407883462 | 0.007712082262210797 | 0.016709511568123392 |