diff --git a/README.md b/README.md
index 247fcc5..357f2d8 100644
--- a/README.md
+++ b/README.md
@@ -866,3 +866,129 @@ Presné hodnoty LR, finálne aj minimálne eval loss, kroky, training loss a run
- Na tomto 30k tréningu mali nižšie skúšané LR vyšší finálny eval loss.
- Najlepší skúšaný LR leží na hornej hranici rozsahu.
+
+---
+
+# Hruška – hľadanie learning rate na 300k Aya príkladoch
+
+Po skúšobnom hľadaní LR pomocou Optuny na 30k príkladoch nasledovalo **overenie šiestich ručne zvolených hodnôt LR na väčšej, fixnej 300k vzorke**. Cieľom bolo nájsť vhodné východiskové nastavenie pre plánovaný tréning na celej slovenskej Aya množine na **Perune**. Tento 300k experiment ešte nebol tréningom na celej množine.
+
+Podklady k tréningu a následným benchmarkom vybraného modelu sú v priečinku [aya_300k_lr_experiments/](aya_300k_lr_experiments/). Postup opakovania a technické podmienky sú v [reproduction.md](aya_300k_lr_experiments/reproduction.md).
+
+## Dáta a spoločná konfigurácia
+
+Zo slovenského `train` splitu `CohereLabs/aya_collection_language_split` sa vyfiltrovali prázdne alebo neplatné `inputs`/`targets`, údaje sa premiešali so seed 42 a vybralo sa 300 000 záznamov. Polia sa previedli na Alpaca formát: `inputs` na `instruction`, prázdny `input` a `targets` na `output`.
+
+| Parameter | Hodnota |
+|---|---|
+| Základný model | `slovak-nlp/mistral-sk-7b` |
+| Použitý modelový snapshot | `089497ae0d72018e9591895c54f774bf0b4a83a4` |
+| Dataset | fixný `aya_slovak_300k_fixed`, bez streamingu |
+| Rozdelenie | 299 000 tréningových + 1 000 validačných príkladov |
+| Hardvér | Hruška, 1× RTX A4500, GPU 0 |
+| Framework a presnosť | LLaMA-Factory + Unsloth, `bf16` |
+| Fine-tuning | LoRA, rank 16, alpha 32, dropout 0.0 |
+| LoRA moduly | `q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj` |
+| Template / cutoff | `alpaca` / 1024 tokenov |
+| Packing / train on prompt | `false` / `false` |
+| Batch / gradient accumulation | 1 / 16; effective batch 16 |
+| Tréning | 1 epocha, 18 688 optimizer krokov |
+| Optimizer / scheduler | `adamw_torch` / cosine |
+| Warmup / max grad norm | 1500 krokov / 0.5 |
+| Evaluácia / logovanie | každých 1000 / 10 krokov + finálna evaluácia |
+| Seed | 42 |
+
+Pôvodné [konfigurácie](aya_300k_lr_experiments/configs/) a [tréningové logy](aya_300k_lr_experiments/logs/) sú zachované. Konfigurácie sa líšia hodnotou LR, výstupnou cestou a prevádzkovým príznakom `overwrite_output_dir`. Archív neobsahuje checksum vstupného datasetu ani identifikátory validačných vzoriek; úplnú identitu dát preto nemožno nezávisle overiť iba z uložených konfigurácií.
+
+## Výsledky a výber LR pre ďalší tréning
+
+Prvá séria obsahovala LR `0.00018`, `0.00020` a `0.00023`. Po jej vyhodnotení sa rozsah rozšíril smerom nadol o `0.00015`, `0.00010` a `0.00008`. Všetkých šesť behov dokončilo jednu epochu.
+
+| Learning rate | Finálny eval loss | Minimálny eval loss | Krok minima |
+|---:|---:|---:|---:|
+| 0.00008 | 0.875217 | 0.875217 | 18 688 |
+| **0.00010** | **0.872620** | **0.872620** | **18 688** |
+| 0.00015 | 0.884270 | 0.884270 | 18 688 |
+| 0.00018 | 0.891460 | 0.891388 | 18 000 |
+| 0.00020 | 0.897669 | 0.897507 | 18 000 |
+| 0.00023 | 0.903236 | 0.903193 | 18 000 |
+
+**LR `0.00010` bol vybraný preto, že dosiahol najnižší finálny validačný loss zo všetkých šiestich 300k behov: `0.87261962890625`.** Rozdiel oproti LR `0.00008` je približne 0.002597. Ide o najlepší bodový výsledok v tejto sérii; neprebehli opakovania s viacerými seedmi ani overenie optimálneho LR na celej Aya množine. Pre Perun slúži ako východisko.
+
+Na 30k skúške bol najlepší skúšaný LR `0.0002`. Výsledky 300k série preto ukazujú význam overenia LR na väčšej vzorke. Absolútne hodnoty eval loss medzi 30k, 100k a 300k sériami nie sú priamym porovnaním kvality modelov, keďže nemožno predpokladať zhodné validačné vzorky.
+
+Presné údaje sú v [pôvodnom LR súhrne](aya_300k_lr_experiments/results/aya_300k_lr_summary.csv), [rozšírenom súhrne behov](aya_300k_lr_experiments/results/training_run_summary.csv) a [histórii validačného loss](aya_300k_lr_experiments/results/eval_loss_history.csv).
+
+
+ 
+
+
+
+ 
+
+
+
+ 
+
+
+## Export vybraného modelu a benchmarky
+
+Na následné benchmarky sa vybral model s LR `0.00010`. Jeho LoRA adaptér sa zlúčil s pôvodným základným modelom do samostatného modelu v `bfloat16`. Export nepredstavoval ďalší tréning. [Exportná konfigurácia](aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml) používa rovnaký snapshot základného modelu a template `alpaca`.
+
+Benchmarky skúmajú schopnosti tohto vybraného modelu. **Ostatných päť LR zatiaľ týmito benchmarkmi nebolo porovnaných**, takže z nich nemožno vyhlásiť LR `0.00010` za najlepší LR podľa downstream metrík.
+
+### lm-evaluation-harness
+
+Použilo sa pôvodné prostredie `lm_eval_hruska`, HF backend, `bf16`, batch size 1 a zero-shot nastavenie bez aplikovania chat template. Zlúčený model sa načítal samostatne; tokenizer pochádzal z pôvodného modelového snapshotu. Tento postup obchádza nekompatibilitu exportovaného označenia `TokenizersBackend` so starším Transformers 4.48.2.
+
+Vyhodnotenie dokončilo všetkých sedem pôvodných úloh a skončilo s návratovým kódom 0. Zachované sú [výsledky](aya_300k_lr_experiments/benchmarks/lmeval/results/), [log](aya_300k_lr_experiments/benchmarks/lmeval/logs/), custom úlohy a metadáta prostredia. V nasledujúcej tabuľke sú accuracy metriky prevedené na percentá; SKQuad F1 je už v zdroji v škále 0–100.
+
+| Model | ARC acc_norm | HellaSwag acc_norm | M-MMLU acc | TQA MC1 | TQA MC2 | SKLegal acc_norm | SKQuad F1 |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| 100k – experiment 01, baseline | 34.05 | 52.09 | 35.50 | 17.99 | 34.39 | 46.85 | 36.68 |
+| 100k – experiment 08, attention-only | 35.24 | 52.18 | 34.48 | 18.77 | 36.74 | 47.68 | 43.38 |
+| 100k – experiment 10, r16/a32/LR 0.0001 | 32.93 | 51.57 | 34.48 | 20.57 | 34.80 | 46.18 | 37.76 |
+| **300k – vybraný LR 0.00010** | 34.73 | 51.64 | 31.03 | 19.67 | 35.07 | 46.10 | 38.73 |
+
+Oproti 100k experimentu 10 dosiahol vybraný 300k model bodovo vyššie ARC acc_norm (+1.80 percentuálneho bodu) a SKQuad F1 (+0.97 bodu), ale nižšie M-MMLU acc (−3.45 percentuálneho bodu). Ostatné zmeny sú menšie. V žiadnej zo siedmich hlavných metrík neprekonal maximum z celej desiatky 100k behov; tieto maximá navyše pochádzajú z rôznych konfigurácií. Výber najnižšieho validačného loss teda neznamená automaticky najlepší výsledok v každej downstream úlohe. Nejde o vyhodnotenie všetkých šiestich 300k LR ani o test štatistickej významnosti rozdielov.
+
+Úplné porovnanie so všetkými desiatimi staršími Hruška experimentmi je v [comparison_with_100k.csv](aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv). Vo všetkých porovnávaných Hruška behoch sa zhodujú konfigurácie a verzie úloh, počty vzoriek, seeds, evaluator commit `6d2abda4`, Transformers 4.48.2 a nastavenie bez chat template. Pole `task_hashes` je však v zdrojových JSON súboroch prázdne, takže nezávislé overenie obsahu benchmarkových dát pomocou hashov nie je možné. Staršie výsledky používali samostatný PEFT adaptér, nový výsledok zlúčený model v BF16; spôsob načítania a zaokrúhlenie pri zlúčení preto nie sú úplne identické. Podrobnosti sú v [audite porovnateľnosti](aya_300k_lr_experiments/benchmarks/lmeval/provenance/comparability_audit.json).
+
+### SK-bench
+
+Dokončil sa **čiastkový slovenský zero-shot beh s 222 položkami**. Položka predstavuje kombináciu úlohy, podúlohy a variantu promptu, nie jeden hodnotený príklad. Zahrnuté boli tri slovenské formulácie promptov dostupných úloh. Použila sa Alpaca šablóna zlúčeného modelu, greedy dekódovanie, seed 1234, `bf16`, jedna GPU a maximálny kontext 32 768 tokenov.
+
+Z pôvodných 412 slovenských položiek sa vynechalo 187 položiek pre 12 nedostupných repozitárov: Chiby, Autoškola, e-testy, IFEval-SK, OneRuler-SK, Scio, Slohry, Slovak school exams, Slovak Trivia, SlovakSum, SME-Sum a slovné analógie. Ich kontrola vracala `RepositoryNotFoundError`; tento stav sám osebe nerozlišuje neexistujúci repozitár od súkromného repozitára bez oprávnenia. Ďalšie tri položky NLI-SK-annotated sa vynechali, pretože prístup vyžadoval schválenie autorkou a pri spúšťaní behu ešte nebol udelený. Ostatné NLI úlohy zo skLEP v behu zostali.
+
+Uložené sú [vykonané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt), [vynechané úlohy](aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt), [report dostupnosti](aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json), úplný výsledný JSON a úspešný log. Audit 222 detailných parquet výstupov overil **159 426 hodnotení** (53 142 pre každý z troch promptov). **Log zároveň potvrdzuje orezanie príliš dlhého vstupu.** Všetky uložené `model_response.truncated_tokens_count` sú síce nulové, ale tieto počítadlá nezachytili orezanie hlásené enginom. Pri MultiWikiQA má 69 vstupov dĺžku presne na hranici použitého rozpočtu (23 pri každom prompte); ide o kandidátov konzistentných s orezaním, nie o spoľahlivo dokázaný počet orezaných príkladov. Výsledky dlhých QA vstupov preto treba interpretovať s týmto obmedzením. Podrobnosti sú v [audite](aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json) a [počítadlách po úlohách](aya_300k_lr_experiments/benchmarks/skbench/results/truncation_audit_per_task.csv).
+
+Výber opisných výsledkov z dostupných úloh. Uvedený priemer je vážený počtom hodnotených príkladov v príslušných variantoch; všetky tri prompty majú v danej úlohe rovnaký počet príkladov. Pri MMLU zahŕňa všetkých 57 predmetov a tri prompty. Toto nie je oficiálna agregácia úplného SK-bench.
+
+| Úloha | Metrika | Počet variantov | Priemer (%) |
+|---|---|---:|---:|
+| ARC | `em_relaxed` | 3 | 19.08 |
+| Belebele | `em_relaxed` | 3 | 21.33 |
+| Financial Exams | `em_relaxed` | 3 | 22.94 |
+| Global-MMLU-Lite | `em_relaxed` | 3 | 19.83 |
+| MMLU (57 predmetov) | `em_relaxed` | 171 | 15.72 |
+| SlovakCOPA | `em_relaxed` | 3 | 40.47 |
+| SKJ1 | `em_relaxed` | 3 | 4.33 |
+| TruthfulQA MC1 | `em_relaxed` | 3 | 1.20 |
+| MovieReview | `em_norm_relaxed` | 3 | 40.90 |
+| ToxicSK | `em_norm_relaxed` | 3 | 47.60 |
+| MultiWikiQA | `f1_slk` | 3 | 50.99 |
+| skLEP-QA | `f1_slk` | 3 | 37.50 |
+
+Úplné metriky sú v [per_task_metrics.csv](aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv); rodinné a promptové súhrny vrátane nevážených priemerov sú v [family_summary.csv](aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv) a [family_prompt_summary.csv](aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv).
+
+Výsledky treba čítať spolu s definíciou metriky a promptu. Napríklad pri ARC je priemer presnej metriky `em` približne 2.37 %, zatiaľ čo `em_relaxed` dosahuje 19.08 %. Rozdiel ukazuje citlivosť generatívneho hodnotenia na spôsob zápisu odpovede. Úplný surový JSON zachováva aj agregát `all`, ktorý sa tu neinterpretuje ako celkové skóre: mieša rôzne metriky a pokrýva neúplnú množinu úloh. Samotné MMLU tvorí 171 z 222 položiek, preto jednoduché spriemerovanie všetkých riadkov nie je vyváženým súhrnom schopností modelu.
+
+**Tento výsledok nie je oficiálne celkové skóre úplného SK-bench.** Pri porovnávaní ďalších modelov treba použiť rovnaký zmrazený zoznam úloh a rovnaký protokol. SK-bench používa generovanie a následné porovnávanie textovej odpovede, zatiaľ čo väčšina uvedených lm-eval úloh vyhodnocuje pravdepodobnosti možností. Odlišné sú aj konkrétne datasety, prompty a metriky; hodnoty medzi týmito dvoma frameworkmi preto nie sú priamo zameniteľné.
+
+Pri príprave sa sampler FlashInfer pokúšal kompilovať cez starší systémový CUDA toolkit. Úspešný beh používal `VLLM_USE_FLASHINFER_SAMPLER=0` a `--enforce-eager`. Podrobné verzie a parametre sú súčasťou benchmarkových metadát.
+
+## Uložené podklady
+
+Priečinok [aya_300k_lr_experiments/](aya_300k_lr_experiments/) obsahuje pôvodné konfigurácie, tréningové a benchmarkové logy, výsledné JSON/CSV súbory, grafy, vybrané metadáta a skripty. Veľké váhy, checkpointy, `trainer_state.json`, kompletné per-sample parquet súbory a duplicitné inference cache zostávajú v pôvodnom archíve. Grafy a odvodené tabuľky tréningu možno obnoviť z malých uložených histórií bez prístupu k váham modelov.
+
+Zdrojový archív a jeho SHA-256 sú zaznamenané v [source_archive.json](aya_300k_lr_experiments/provenance/source_archive.json).
diff --git a/aya_300k_lr_experiments/.gitattributes b/aya_300k_lr_experiments/.gitattributes
new file mode 100644
index 0000000..dfdb8b7
--- /dev/null
+++ b/aya_300k_lr_experiments/.gitattributes
@@ -0,0 +1 @@
+*.sh text eol=lf
diff --git a/aya_300k_lr_experiments/.gitignore b/aya_300k_lr_experiments/.gitignore
new file mode 100644
index 0000000..10b8036
--- /dev/null
+++ b/aya_300k_lr_experiments/.gitignore
@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+*.safetensors
+*.bin
+*.pt
+*.pth
+*.parquet
+checkpoint-*/
+trainer_state.json
diff --git a/aya_300k_lr_experiments/benchmarks/export/chat_template.jinja b/aya_300k_lr_experiments/benchmarks/export/chat_template.jinja
new file mode 100644
index 0000000..8d7a085
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/export/chat_template.jinja
@@ -0,0 +1,9 @@
+{% set system_message = 'Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+' %}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in loop_messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '### Instruction:
+' + content + '
+
+### Response:
+' }}{% elif message['role'] == 'assistant' %}{{ content + '' + '
+
+' }}{% endif %}{% endfor %}
\ No newline at end of file
diff --git a/aya_300k_lr_experiments/benchmarks/export/config.json b/aya_300k_lr_experiments/benchmarks/export/config.json
new file mode 100644
index 0000000..13eab00
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/export/config.json
@@ -0,0 +1,30 @@
+{
+ "architectures": [
+ "MistralForCausalLM"
+ ],
+ "attention_dropout": 0.0,
+ "bos_token_id": 1,
+ "dtype": "bfloat16",
+ "eos_token_id": 2,
+ "head_dim": 128,
+ "hidden_act": "silu",
+ "hidden_size": 4096,
+ "initializer_range": 0.02,
+ "intermediate_size": 14336,
+ "max_position_embeddings": 32768,
+ "model_type": "mistral",
+ "num_attention_heads": 32,
+ "num_hidden_layers": 32,
+ "num_key_value_heads": 8,
+ "pad_token_id": 2,
+ "rms_norm_eps": 1e-05,
+ "rope_parameters": {
+ "rope_theta": 10000.0,
+ "rope_type": "default"
+ },
+ "sliding_window": 4096,
+ "tie_word_embeddings": false,
+ "transformers_version": "5.5.0",
+ "use_cache": true,
+ "vocab_size": 32000
+}
diff --git a/aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml b/aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml
new file mode 100644
index 0000000..fcbcfa7
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/export/export_lr_0p00010.yaml
@@ -0,0 +1,12 @@
+model_name_or_path: slovak-nlp/mistral-sk-7b
+model_revision: 089497ae0d72018e9591895c54f774bf0b4a83a4
+adapter_name_or_path: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010
+
+template: alpaca
+finetuning_type: lora
+infer_dtype: bfloat16
+
+export_dir: /home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
+export_size: 5
+export_device: cpu
+export_legacy_format: false
diff --git a/aya_300k_lr_experiments/benchmarks/export/generation_config.json b/aya_300k_lr_experiments/benchmarks/export/generation_config.json
new file mode 100644
index 0000000..b7715d4
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/export/generation_config.json
@@ -0,0 +1,6 @@
+{
+ "_from_model_config": true,
+ "bos_token_id": 1,
+ "eos_token_id": 2,
+ "transformers_version": "5.5.0"
+}
diff --git a/aya_300k_lr_experiments/benchmarks/export/tokenizer_config.json b/aya_300k_lr_experiments/benchmarks/export/tokenizer_config.json
new file mode 100644
index 0000000..6a1eb78
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/export/tokenizer_config.json
@@ -0,0 +1,19 @@
+{
+ "add_prefix_space": true,
+ "backend": "tokenizers",
+ "bos_token": "",
+ "clean_up_tokenization_spaces": false,
+ "eos_token": "",
+ "extra_special_tokens": [],
+ "is_local": false,
+ "legacy": true,
+ "model_max_length": 1000000000000000019884624838656,
+ "pad_token": "",
+ "padding_side": "left",
+ "sp_model_kwargs": {},
+ "spaces_between_special_tokens": false,
+ "split_special_tokens": false,
+ "tokenizer_class": "TokenizersBackend",
+ "unk_token": "",
+ "use_default_system_prompt": false
+}
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv b/aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv
new file mode 100644
index 0000000..3376c2b
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/comparison_with_100k.csv
@@ -0,0 +1,12 @@
+model_id,training_samples,model_name,source_result_json,arc_acc_norm,hellaswag_acc_norm,mmlu_acc,truthfulqa_mc1_acc,truthfulqa_mc2_acc,sklegal_acc_norm,skquad_f1
+01_baseline_r16_a16_lr5e5_adamw,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_01_baseline_r16_a16_lr5e5_adamw,lmeval/mistral_aya(hruska)/01_baseline_r16_a16_lr5e5_adamw/results_2026-09-30T21-23-33.435952.json,0.34046193327630453,0.5209277807063785,0.3549992344204563,0.17994858611825193,0.34391089322085056,0.46851574212893554,36.6797346829585
+02_optimizer_adamw8bit_r16_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_02_optimizer_adamw8bit_r16_a16_lr5e5,lmeval/mistral_aya(hruska)/02_optimizer_adamw8bit_r16_a16_lr5e5/results_2026-10-01T04-23-23.791749.json,0.3378956372968349,0.521244069583553,0.35438677078548464,0.1928020565552699,0.35373417730948353,0.4715142428785607,36.21782844597198
+03_rank8_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_03_rank8_a16_lr5e5,lmeval/mistral_aya(hruska)/03_rank8_a16_lr5e5/results_2026-10-01T10-25-51.531765.json,0.33105218135158254,0.5167105956773853,0.34550604807839536,0.18380462724935734,0.3424520772406517,0.4715142428785607,36.36378811056779
+04_rank32_a32_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_04_rank32_a32_lr5e5,lmeval/mistral_aya(hruska)/04_rank32_a32_lr5e5/results_2026-10-01T17-22-48.146390.json,0.33532934131736525,0.5200843437005799,0.3449701423977951,0.19023136246786632,0.3367013908863517,0.47301349325337333,36.02282635500946
+05_r16_alpha32_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_05_r16_alpha32_lr5e5,lmeval/mistral_aya(hruska)/05_r16_alpha32_lr5e5/results_2026-10-02T00-19-55.725179.json,0.33361847733105215,0.5190300474433316,0.35714285714285715,0.19665809768637532,0.3555125811399826,0.46701649175412296,37.873456685535025
+06_lr1e4_r16_a16,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_06_lr1e4_r16_a16,lmeval/mistral_aya(hruska)/06_lr1e4_r16_a16/results_2026-10-02T06-59-24.237777.json,0.33105218135158254,0.5207169214549289,0.3549992344204563,0.19023136246786632,0.34733281230911506,0.46551724137931033,37.57962398015711
+07_lr2e5_r16_a16,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_07_lr2e5_r16_a16,lmeval/mistral_aya(hruska)/07_lr2e5_r16_a16/results_2026-10-02T14-27-49.730611.json,0.32763045337895635,0.522509225092251,0.3379268105956209,0.16709511568123395,0.32463255972324384,0.47226386806596704,33.83807391933327
+08_attention_only_r16_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_08_attention_only_r16_a16_lr5e5,lmeval/mistral_aya(hruska)/08_attention_only_r16_a16_lr5e5/results_2026-10-02T19-20-02.720498.json,0.35243798118049613,0.5217712177121772,0.3448170264890522,0.18766066838046272,0.3673805917101876,0.4767616191904048,43.37517960476381
+09_batch2_accum4_r16_a16_lr5e5,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_09_batch2_accum4_r16_a16_lr5e5,lmeval/mistral_aya(hruska)/09_batch2_accum4_r16_a16_lr5e5/results_2026-10-03T01-58-48.046423.json,0.3378956372968349,0.5209277807063785,0.35140101056499773,0.18637532133676094,0.340221777957301,0.46926536731634183,35.3128443275733
+10_r16_a32_lr1e4,100000,/home/schwarc/diplomovka/hruska_compare9_100k_fixed_10_r16_a32_lr1e4,lmeval/mistral_aya(hruska)/10_r16_a32_lr1e4/results_2026-10-03T07-58-30.464881.json,0.32934131736526945,0.515656299420137,0.3448170264890522,0.20565552699228792,0.34795284436431645,0.4617691154422789,37.75612784500817
+aya_300k_lr_0p00010,300000,/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json,0.3473053892215569,0.5163943068002108,0.3102893890675241,0.19665809768637532,0.3506542164316134,0.46101949025487254,38.72920807013749
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/full_metrics.csv b/aya_300k_lr_experiments/benchmarks/lmeval/full_metrics.csv
new file mode 100644
index 0000000..e76867e
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/full_metrics.csv
@@ -0,0 +1,35 @@
+model_id,training_samples,task,metric,value,unit,status
+aya_300k_lr_0p00010,300000,arc_sk,"acc,none",0.30025662959794697,0..1,reported
+aya_300k_lr_0p00010,300000,arc_sk,"acc_stderr,none",0.013412022629366669,0..1,reported
+aya_300k_lr_0p00010,300000,arc_sk,"acc_norm,none",0.3473053892215569,0..1,reported
+aya_300k_lr_0p00010,300000,arc_sk,"acc_norm_stderr,none",0.013931226499492353,0..1,reported
+aya_300k_lr_0p00010,300000,hellaswag_sk,"acc,none",0.4083289404322615,0..1,reported
+aya_300k_lr_0p00010,300000,hellaswag_sk,"acc_stderr,none",0.005047187883274763,0..1,reported
+aya_300k_lr_0p00010,300000,hellaswag_sk,"acc_norm,none",0.5163943068002108,0..1,reported
+aya_300k_lr_0p00010,300000,hellaswag_sk,"acc_norm_stderr,none",0.005131456510575215,0..1,reported
+aya_300k_lr_0p00010,300000,m_mmlu_sk,"acc,none",0.3102893890675241,0..1,reported
+aya_300k_lr_0p00010,300000,m_mmlu_sk,"acc_stderr,none",0.0040478899905589515,0..1,reported
+aya_300k_lr_0p00010,300000,sklegal,"acc,none",0.2856071964017991,0..1,reported
+aya_300k_lr_0p00010,300000,sklegal,"acc_stderr,none",0.012371946515772182,0..1,reported
+aya_300k_lr_0p00010,300000,sklegal,"acc_norm,none",0.46101949025487254,0..1,reported
+aya_300k_lr_0p00010,300000,sklegal,"acc_norm_stderr,none",0.013653094581189941,0..1,reported
+aya_300k_lr_0p00010,300000,skquad,"exact,none",15.120525931336742,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"exact_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"f1,none",38.72920807013749,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"f1_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"HasAns_exact,none",18.540838569047313,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"HasAns_exact_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"HasAns_f1,none",47.549942420326644,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"HasAns_f1_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"NoAns_exact,none",0.1681614349775785,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"NoAns_exact_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"NoAns_f1,none",0.1681614349775785,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"NoAns_f1_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"best_exact,none",18.75195658979443,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"best_exact_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,skquad,"best_f1,none",38.7292080701372,0..100,reported
+aya_300k_lr_0p00010,300000,skquad,"best_f1_stderr,none",N/A,0..100,unavailable
+aya_300k_lr_0p00010,300000,truthfulqa_sk_mc1,"acc,none",0.19665809768637532,0..1,reported
+aya_300k_lr_0p00010,300000,truthfulqa_sk_mc1,"acc_stderr,none",0.014259215011433178,0..1,reported
+aya_300k_lr_0p00010,300000,truthfulqa_sk_mc2,"acc,none",0.3506542164316134,0..1,reported
+aya_300k_lr_0p00010,300000,truthfulqa_sk_mc2,"acc_stderr,none",0.014662940445997986,0..1,reported
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/logs/lr_0p00010_20261023_224332.log b/aya_300k_lr_experiments/benchmarks/lmeval/logs/lr_0p00010_20261023_224332.log
new file mode 100644
index 0000000..b7fde91
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/logs/lr_0p00010_20261023_224332.log
@@ -0,0 +1,58 @@
+2026-10-23:22:43:38,177 INFO [lm_eval.__main__:307] Including path: /home/schwarc/diplomovka/lmeval_slovak_tasks
+2026-10-23:22:43:47,185 INFO [lm_eval.__main__:379] Selected Tasks: ['arc_sk', 'hellaswag_sk', 'm_mmlu_sk', 'sklegal', 'skquad', 'truthfulqa_sk_mc1', 'truthfulqa_sk_mc2']
+2026-10-23:22:43:47,186 INFO [lm_eval.evaluator:169] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-10-23:22:43:47,186 INFO [lm_eval.evaluator:206] Initializing hf model, with arguments: {'pretrained': '/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010', 'tokenizer': '/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4', 'dtype': 'bfloat16'}
+2026-10-23:22:43:47,315 INFO [lm_eval.models.huggingface:136] Using device 'cuda:0'
+2026-10-23:22:43:47,566 INFO [lm_eval.models.huggingface:376] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+
Loading checkpoint shards: 0%| | 0/3 [00:00 datasets.Dataset:\n\
+ \ def _process_doc(doc):\n # breakpoint()\n out_doc = {\n \
+ \ \"id\": doc[\"id\"],\n \"query\": \"Question: \" + preprocess(doc[\"\
+ instruction\"]) + \"\\nAnswer:\",\n \"choices\": [\n preprocess(option)\n\
+ \ for option in [\n doc[\"option_a\"],\n \
+ \ doc[\"option_b\"],\n doc[\"option_c\"],\n \
+ \ doc[\"option_d\"],\n doc[\"option_e\"],\n \
+ \ ]\n if option\n ],\n \"gold\"\
+ : [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n }\n \
+ \ return out_doc\n\n return dataset.map(_process_doc)\n"
+doc_to_choice: choices
+doc_to_target: gold
+training_split: train
+target_delimiter: ' '
+validation_split: validation
+fewshot_delimiter: '
+
+
+ '
+should_decontaminate: true
+doc_to_decontamination_query: query
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/hellaswag_sk.yaml b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/hellaswag_sk.yaml
new file mode 100644
index 0000000..2c96e72
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/hellaswag_sk.yaml
@@ -0,0 +1,35 @@
+tag:
+- hellaswag_multilingual
+task: hellaswag_sk
+repeats: 1
+metadata:
+ version: 1
+description: ''
+doc_to_text: query
+metric_list:
+- metric: acc
+ aggregation: mean
+ higher_is_better: true
+- metric: acc_norm
+ aggregation: mean
+ higher_is_better: true
+num_fewshot: 0
+output_type: multiple_choice
+unsafe_code: false
+dataset_name: sk
+dataset_path: alexandrainst/m_hellaswag
+process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
+ \ def _process_doc(doc):\n ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"\
+ ].capitalize()\n out_doc = {\n \"query\": preprocess(doc[\"activity_label\"\
+ ] + \": \" + ctx),\n \"choices\": [preprocess(ending) for ending in doc[\"\
+ endings\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return\
+ \ out_doc\n\n return dataset.map(_process_doc)\n"
+doc_to_choice: choices
+doc_to_target: '{{label.lstrip()}}'
+target_delimiter: ' '
+validation_split: val
+fewshot_delimiter: '
+
+
+ '
+should_decontaminate: false
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/m_mmlu_sk.yaml b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/m_mmlu_sk.yaml
new file mode 100644
index 0000000..501237a
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/m_mmlu_sk.yaml
@@ -0,0 +1,43 @@
+tag:
+- m_mmlu
+task: m_mmlu_sk
+repeats: 1
+metadata:
+ version: 0
+test_split: test
+description: ''
+doc_to_text: '{{instruction.strip()}}
+
+ A. {{option_a}}
+
+ B. {{option_b}}
+
+ C. {{option_c}}
+
+ D. {{option_d}}
+
+ Answer:'
+metric_list:
+- metric: acc
+ aggregation: mean
+ higher_is_better: true
+num_fewshot: 0
+output_type: multiple_choice
+unsafe_code: false
+dataset_name: sk
+dataset_path: alexandrainst/m_mmlu
+doc_to_choice:
+- A
+- B
+- C
+- D
+doc_to_target: answer
+fewshot_split: train
+fewshot_config:
+ sampler: first_n
+target_delimiter: ' '
+fewshot_delimiter: '
+
+
+ '
+should_decontaminate: false
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/sklegal.yaml b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/sklegal.yaml
new file mode 100644
index 0000000..814685f
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/sklegal.yaml
@@ -0,0 +1,23 @@
+task: sklegal
+repeats: 1
+metadata:
+ version: 1
+test_split: test
+description: ''
+doc_to_text: '{{prompt}} '
+metric_list:
+ - metric: acc
+ aggregation: mean
+ higher_is_better: true
+ - metric: acc_norm
+ aggregation: mean
+ higher_is_better: true
+num_fewshot: 0
+output_type: multiple_choice
+unsafe_code: false
+dataset_path: TUKE-KEMT/slovak-legal-exam
+doc_to_choice: '{{answers}}'
+doc_to_target: label
+target_delimiter: ' '
+fewshot_delimiter: "\n\n"
+should_decontaminate: false
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/skquad.yaml b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/skquad.yaml
new file mode 100644
index 0000000..2546a99
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/skquad.yaml
@@ -0,0 +1,2 @@
+task: skquad
+class: !function task.SQuAD2
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task.py b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task.py
new file mode 100644
index 0000000..c9e55f7
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task.py
@@ -0,0 +1,245 @@
+"""
+Know What You Don’t Know: Unanswerable Questions for SQuAD
+https://arxiv.org/pdf/1806.03822.pdf
+
+Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset,
+consisting of questions posed by crowdworkers on a set of Wikipedia articles,
+where the answer to every question is a segment of text, or span, from the
+corresponding reading passage, or the question might be unanswerable.
+SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable
+questions written adversarially by crowdworkers to look similar to answerable ones.
+To do well on SQuAD2.0, systems must not only answer questions when possible, but
+also determine when no answer is supported by the paragraph and abstain from answering.
+
+Homepage: https://rajpurkar.github.io/SQuAD-explorer/
+"""
+
+from functools import partial
+from math import exp
+
+import datasets
+from packaging import version
+
+from lm_eval.api.instance import Instance
+from lm_eval.api.task import ConfigurableTask
+
+
+_CITATION = """
+@misc{rajpurkar2018know,
+ title={Know What You Don't Know: Unanswerable Questions for SQuAD},
+ author={Pranav Rajpurkar and Robin Jia and Percy Liang},
+ year={2018},
+ eprint={1806.03822},
+ archivePrefix={arXiv},
+ primaryClass={cs.CL}
+}
+"""
+
+
+def _squad_metric(predictions, references):
+ import evaluate
+
+ squad_metric = evaluate.load("squad_v2")
+ return squad_metric.compute(predictions=predictions, references=references)
+
+
+def _squad_agg(key, items):
+ predictions, references = zip(*items)
+
+ return _squad_metric(predictions=predictions, references=references).get(key, 0)
+
+
+class SQuAD2(ConfigurableTask):
+ VERSION = 3
+ DATASET_PATH = "TUKE-DeutscheTelekom/skquad"
+ DATASET_NAME = None
+
+ def __init__(self, config=None):
+ super().__init__(config={"metadata": {"version": self.VERSION}})
+
+ # HF changed squad on us so we have to make sure we aren't running the old one
+ assert version.parse(datasets.__version__) >= version.parse("1.11.0"), (
+ "datasets v1.11.0 or later required for SQuAD"
+ )
+
+ def has_training_docs(self):
+ return True
+
+ def has_validation_docs(self):
+ return True
+
+ def has_test_docs(self):
+ return False
+
+ def training_docs(self):
+ return self.dataset["train"]
+
+ def validation_docs(self):
+ return self.dataset["validation"]
+
+ def doc_to_text(self, doc):
+ return (
+ "Title: "
+ + doc["title"]
+ + "\n\n"
+ + "Background: "
+ + doc["context"]
+ + "\n\n"
+ + "Question: "
+ + doc["question"]
+ + "\n\n"
+ + "Answer:"
+ )
+
+ def should_decontaminate(self):
+ return True
+
+ def doc_to_decontamination_query(self, doc):
+ return doc["context"]
+
+ def doc_to_target(self, doc):
+ answer_list = doc["answers"]["text"]
+ if len(answer_list) > 0:
+ answer = answer_list[0]
+ else:
+ answer = "unanswerable"
+ return " " + answer
+
+ def construct_requests(
+ self, doc, ctx, chat_template=None, apply_chat_template=False, **kwargs
+ ):
+ """Uses RequestFactory to construct Requests and returns an iterable of
+ Requests which will be sent to the LM.
+
+ :param doc:
+ The document as returned from training_docs, validation_docs, or test_docs.
+ :param ctx: str
+ The context string, generated by fewshot_context. This includes the natural
+ language description, as well as the few shot examples, and the question
+ part of the document for `doc`.
+ """
+
+ return [
+ Instance(
+ request_type="generate_until",
+ doc=doc,
+ arguments=(ctx, {"until": ["\n"]}),
+ idx=0,
+ **kwargs,
+ ),
+ Instance(
+ request_type="loglikelihood",
+ doc=doc,
+ arguments=(ctx, " " + "unanswerable"),
+ idx=0,
+ **kwargs,
+ ),
+ ]
+
+ def process_results(self, doc, results):
+ """Take a single document and the LM results and evaluates, returning a
+ dict where keys are the names of submetrics and values are the values of
+ the metric for that one document
+
+ :param doc:
+ The document as returned from training_docs, validation_docs, or test_docs.
+ :param results:
+ The results of the requests created in construct_requests.
+ """
+
+ continuation, (logprob_unanswerable, _) = results
+
+ no_answer_probability = exp(logprob_unanswerable)
+
+ predictions = {
+ "id": doc["id"],
+ "prediction_text": continuation,
+ "no_answer_probability": no_answer_probability,
+ }
+
+ references = {
+ "id": doc["id"],
+ "answers": doc["answers"],
+ }
+
+ return {
+ "exact": (
+ predictions,
+ references,
+ ), # Exact match (the normalized answer exactly match the gold answer)
+ "f1": (
+ predictions,
+ references,
+ ), # The F-score of predicted tokens versus the gold answer
+ "HasAns_exact": (
+ predictions,
+ references,
+ ), # Exact match (the normalized answer exactly match the gold answer)
+ "HasAns_f1": (
+ predictions,
+ references,
+ ), # The F-score of predicted tokens versus the gold answer
+ "NoAns_exact": (
+ predictions,
+ references,
+ ), # Exact match (the normalized answer exactly match the gold answer)
+ "NoAns_f1": (
+ predictions,
+ references,
+ ), # The F-score of predicted tokens versus the gold answer
+ "best_exact": (
+ predictions,
+ references,
+ ), # Best exact match (with varying threshold)
+ "best_f1": (predictions, references), # Best F1 (with varying threshold)
+ }
+
+ def aggregation(self):
+ """
+ :returns: {str: [float] -> float}
+ A dictionary where keys are the names of submetrics and values are
+ functions that aggregate a list of metrics
+ """
+ return {
+ "exact": partial(
+ _squad_agg, "exact"
+ ), # Exact match (the normalized answer exactly match the gold answer)
+ "f1": partial(
+ _squad_agg, "f1"
+ ), # The F-score of predicted tokens versus the gold answer
+ "HasAns_exact": partial(
+ _squad_agg, "HasAns_exact"
+ ), # Exact match (the normalized answer exactly match the gold answer)
+ "HasAns_f1": partial(
+ _squad_agg, "HasAns_f1"
+ ), # The F-score of predicted tokens versus the gold answer
+ "NoAns_exact": partial(
+ _squad_agg, "NoAns_exact"
+ ), # Exact match (the normalized answer exactly match the gold answer)
+ "NoAns_f1": partial(
+ _squad_agg, "NoAns_f1"
+ ), # The F-score of predicted tokens versus the gold answer
+ "best_exact": partial(
+ _squad_agg, "best_exact"
+ ), # Best exact match (with varying threshold)
+ "best_f1": partial(
+ _squad_agg, "best_f1"
+ ), # Best F1 (with varying threshold)
+ }
+
+ def higher_is_better(self):
+ """
+ :returns: {str: bool}
+ A dictionary where keys are the names of submetrics and values are
+ whether a higher value of the submetric is better
+ """
+ return {
+ "exact": True, # Exact match (the normalized answer exactly match the gold answer)
+ "f1": True, # The F-score of predicted tokens versus the gold answer
+ "HasAns_exact": True, # Exact match (the normalized answer exactly match the gold answer)
+ "HasAns_f1": True, # The F-score of predicted tokens versus the gold answer
+ "NoAns_exact": True, # Exact match (the normalized answer exactly match the gold answer)
+ "NoAns_f1": True, # The F-score of predicted tokens versus the gold answer
+ "best_exact": True, # Best exact match (with varying threshold)
+ "best_f1": True, # Best F1 (with varying threshold)
+ }
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task_configs_from_wandb.json b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task_configs_from_wandb.json
new file mode 100644
index 0000000..99a3cb3
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/task_configs_from_wandb.json
@@ -0,0 +1,232 @@
+{
+ "arc_sk": {
+ "tag": [
+ "arc_multilingual"
+ ],
+ "task": "arc_sk",
+ "repeats": 1,
+ "metadata": {
+ "version": 2
+ },
+ "test_split": "test",
+ "description": "",
+ "doc_to_text": "query",
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ },
+ {
+ "metric": "acc_norm",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "num_fewshot": 0,
+ "output_type": "multiple_choice",
+ "unsafe_code": false,
+ "dataset_name": "sk",
+ "dataset_path": "alexandrainst/m_arc",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n # breakpoint()\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"Question: \" + preprocess(doc[\"instruction\"]) + \"\\nAnswer:\",\n \"choices\": [\n preprocess(option)\n for option in [\n doc[\"option_a\"],\n doc[\"option_b\"],\n doc[\"option_c\"],\n doc[\"option_d\"],\n doc[\"option_e\"],\n ]\n if option\n ],\n \"gold\": [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_choice": "choices",
+ "doc_to_target": "gold",
+ "training_split": "train",
+ "target_delimiter": " ",
+ "validation_split": "validation",
+ "fewshot_delimiter": "\n\n",
+ "should_decontaminate": true,
+ "doc_to_decontamination_query": "query"
+ },
+ "skquad": {
+ "task": "skquad",
+ "repeats": 1,
+ "metadata": {
+ "version": 3
+ },
+ "description": "",
+ "num_fewshot": 0,
+ "output_type": "generate_until",
+ "unsafe_code": false,
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "generation_kwargs": {
+ "until": [
+ "\n\n"
+ ],
+ "do_sample": false
+ },
+ "should_decontaminate": false
+ },
+ "sklegal": {
+ "task": "sklegal",
+ "repeats": 1,
+ "metadata": {
+ "version": 1
+ },
+ "test_split": "test",
+ "description": "",
+ "doc_to_text": "{{prompt}} ",
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ },
+ {
+ "metric": "acc_norm",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "num_fewshot": 0,
+ "output_type": "multiple_choice",
+ "unsafe_code": false,
+ "dataset_path": "TUKE-KEMT/slovak-legal-exam",
+ "doc_to_choice": "{{answers}}",
+ "doc_to_target": "label",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "should_decontaminate": false
+ },
+ "m_mmlu_sk": {
+ "tag": [
+ "m_mmlu"
+ ],
+ "task": "m_mmlu_sk",
+ "repeats": 1,
+ "metadata": {
+ "version": 0
+ },
+ "test_split": "test",
+ "description": "",
+ "doc_to_text": "{{instruction.strip()}}\nA. {{option_a}}\nB. {{option_b}}\nC. {{option_c}}\nD. {{option_d}}\nAnswer:",
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "num_fewshot": 0,
+ "output_type": "multiple_choice",
+ "unsafe_code": false,
+ "dataset_name": "sk",
+ "dataset_path": "alexandrainst/m_mmlu",
+ "doc_to_choice": [
+ "A",
+ "B",
+ "C",
+ "D"
+ ],
+ "doc_to_target": "answer",
+ "fewshot_split": "train",
+ "fewshot_config": {
+ "sampler": "first_n"
+ },
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "should_decontaminate": false
+ },
+ "hellaswag_sk": {
+ "tag": [
+ "hellaswag_multilingual"
+ ],
+ "task": "hellaswag_sk",
+ "repeats": 1,
+ "metadata": {
+ "version": 1
+ },
+ "description": "",
+ "doc_to_text": "query",
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ },
+ {
+ "metric": "acc_norm",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "num_fewshot": 0,
+ "output_type": "multiple_choice",
+ "unsafe_code": false,
+ "dataset_name": "sk",
+ "dataset_path": "alexandrainst/m_hellaswag",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n out_doc = {\n \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_choice": "choices",
+ "doc_to_target": "{{label.lstrip()}}",
+ "target_delimiter": " ",
+ "validation_split": "val",
+ "fewshot_delimiter": "\n\n",
+ "should_decontaminate": false
+ },
+ "truthfulqa_sk_mc1": {
+ "tag": [
+ "truthfulqa_multilingual"
+ ],
+ "task": "truthfulqa_sk_mc1",
+ "repeats": 1,
+ "metadata": {
+ "version": 1
+ },
+ "description": "",
+ "doc_to_text": "query",
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "num_fewshot": 0,
+ "output_type": "multiple_choice",
+ "unsafe_code": false,
+ "dataset_name": "sk",
+ "dataset_path": "alexandrainst/m_truthfulqa",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_choice": "mc1_choices",
+ "doc_to_target": 0,
+ "target_delimiter": " ",
+ "validation_split": "val",
+ "fewshot_delimiter": "\n\n",
+ "should_decontaminate": true,
+ "doc_to_decontamination_query": "question"
+ },
+ "truthfulqa_sk_mc2": {
+ "tag": [
+ "truthfulqa_multilingual"
+ ],
+ "task": "truthfulqa_sk_mc2",
+ "repeats": 1,
+ "metadata": {
+ "version": 1
+ },
+ "description": "",
+ "doc_to_text": "query",
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "num_fewshot": 0,
+ "output_type": "multiple_choice",
+ "unsafe_code": false,
+ "dataset_name": "sk",
+ "dataset_path": "alexandrainst/m_truthfulqa",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_choice": "mc2_choices",
+ "doc_to_target": 0,
+ "process_results": "def process_results_mc2(doc, results):\n lls, is_greedy = zip(*results)\n\n # Split on the first `0` as everything before it is true (`1`).\n split_idx = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n # Compute the normalized probability mass for the correct answer.\n ll_true, ll_false = lls[:split_idx], lls[split_idx:]\n p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n p_true = p_true / (sum(p_true) + sum(p_false))\n\n return {\"acc\": sum(p_true)}\n",
+ "target_delimiter": " ",
+ "validation_split": "val",
+ "fewshot_delimiter": "\n\n",
+ "should_decontaminate": true,
+ "doc_to_decontamination_query": "question"
+ }
+}
\ No newline at end of file
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc1.yaml b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc1.yaml
new file mode 100644
index 0000000..92b1a45
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc1.yaml
@@ -0,0 +1,34 @@
+tag:
+- truthfulqa_multilingual
+task: truthfulqa_sk_mc1
+repeats: 1
+metadata:
+ version: 1
+description: ''
+doc_to_text: query
+metric_list:
+- metric: acc
+ aggregation: mean
+ higher_is_better: true
+num_fewshot: 0
+output_type: multiple_choice
+unsafe_code: false
+dataset_name: sk
+dataset_path: alexandrainst/m_truthfulqa
+process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
+ \ def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"\
+ question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"\
+ question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"\
+ ],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\"\
+ : {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n \
+ \ }\n return out_doc\n\n return dataset.map(_process_doc)\n"
+doc_to_choice: mc1_choices
+doc_to_target: 0
+target_delimiter: ' '
+validation_split: val
+fewshot_delimiter: '
+
+
+ '
+should_decontaminate: true
+doc_to_decontamination_query: question
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc2.yaml b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc2.yaml
new file mode 100644
index 0000000..6684fa7
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/custom_tasks/truthfulqa_sk_mc2.yaml
@@ -0,0 +1,40 @@
+tag:
+- truthfulqa_multilingual
+task: truthfulqa_sk_mc2
+repeats: 1
+metadata:
+ version: 1
+description: ''
+doc_to_text: query
+metric_list:
+- metric: acc
+ aggregation: mean
+ higher_is_better: true
+num_fewshot: 0
+output_type: multiple_choice
+unsafe_code: false
+dataset_name: sk
+dataset_path: alexandrainst/m_truthfulqa
+process_docs: "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\
+ \ def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"\
+ question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"\
+ question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"\
+ ],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\"\
+ : {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n \
+ \ }\n return out_doc\n\n return dataset.map(_process_doc)\n"
+doc_to_choice: mc2_choices
+doc_to_target: 0
+process_results: "def process_results_mc2(doc, results):\n lls, is_greedy = zip(*results)\n\
+ \n # Split on the first `0` as everything before it is true (`1`).\n split_idx\
+ \ = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n # Compute the normalized\
+ \ probability mass for the correct answer.\n ll_true, ll_false = lls[:split_idx],\
+ \ lls[split_idx:]\n p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n\
+ \ p_true = p_true / (sum(p_true) + sum(p_false))\n\n return {\"acc\": sum(p_true)}\n"
+target_delimiter: ' '
+validation_split: val
+fewshot_delimiter: '
+
+
+ '
+should_decontaminate: true
+doc_to_decontamination_query: question
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/lm_eval_commit.txt b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/lm_eval_commit.txt
new file mode 100644
index 0000000..af55d49
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/lm_eval_commit.txt
@@ -0,0 +1 @@
+6d2abda4fd171e68a8789330c4149e37c1ca0bda
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/package_versions.json b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/package_versions.json
new file mode 100644
index 0000000..f351a5e
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/package_versions.json
@@ -0,0 +1,8 @@
+{
+ "lm_eval": "0.4.8",
+ "transformers": "4.48.2",
+ "torch": "2.6.0+cu124",
+ "datasets": "5.0.1",
+ "peft": "0.14.0",
+ "evaluate": "0.4.6"
+}
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/run_settings.txt b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/run_settings.txt
new file mode 100644
index 0000000..47fe7cf
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/run_settings.txt
@@ -0,0 +1,8 @@
+model=hf
+pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
+tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4
+dtype=bfloat16
+num_fewshot=0
+batch_size=1
+apply_chat_template=false
+CUDA_VISIBLE_DEVICES=1
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/provenance/tasks.txt b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/tasks.txt
new file mode 100644
index 0000000..2ce65ea
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/provenance/tasks.txt
@@ -0,0 +1 @@
+arc_sk,hellaswag_sk,m_mmlu_sk,truthfulqa_sk_mc1,truthfulqa_sk_mc2,sklegal,skquad
diff --git a/aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json b/aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json
new file mode 100644
index 0000000..bd77cec
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/lmeval/results/results_2026-10-24T02-28-32.987157.json
@@ -0,0 +1,431 @@
+{
+ "results": {
+ "arc_sk": {
+ "alias": "arc_sk",
+ "acc,none": 0.30025662959794697,
+ "acc_stderr,none": 0.013412022629366669,
+ "acc_norm,none": 0.3473053892215569,
+ "acc_norm_stderr,none": 0.013931226499492353
+ },
+ "hellaswag_sk": {
+ "alias": "hellaswag_sk",
+ "acc,none": 0.4083289404322615,
+ "acc_stderr,none": 0.005047187883274763,
+ "acc_norm,none": 0.5163943068002108,
+ "acc_norm_stderr,none": 0.005131456510575215
+ },
+ "m_mmlu_sk": {
+ "alias": "m_mmlu_sk",
+ "acc,none": 0.3102893890675241,
+ "acc_stderr,none": 0.0040478899905589515
+ },
+ "sklegal": {
+ "alias": "sklegal",
+ "acc,none": 0.2856071964017991,
+ "acc_stderr,none": 0.012371946515772182,
+ "acc_norm,none": 0.46101949025487254,
+ "acc_norm_stderr,none": 0.013653094581189941
+ },
+ "skquad": {
+ "alias": "skquad",
+ "exact,none": 15.120525931336742,
+ "exact_stderr,none": "N/A",
+ "f1,none": 38.72920807013749,
+ "f1_stderr,none": "N/A",
+ "HasAns_exact,none": 18.540838569047313,
+ "HasAns_exact_stderr,none": "N/A",
+ "HasAns_f1,none": 47.549942420326644,
+ "HasAns_f1_stderr,none": "N/A",
+ "NoAns_exact,none": 0.1681614349775785,
+ "NoAns_exact_stderr,none": "N/A",
+ "NoAns_f1,none": 0.1681614349775785,
+ "NoAns_f1_stderr,none": "N/A",
+ "best_exact,none": 18.75195658979443,
+ "best_exact_stderr,none": "N/A",
+ "best_f1,none": 38.7292080701372,
+ "best_f1_stderr,none": "N/A"
+ },
+ "truthfulqa_sk_mc1": {
+ "alias": "truthfulqa_sk_mc1",
+ "acc,none": 0.19665809768637532,
+ "acc_stderr,none": 0.014259215011433178
+ },
+ "truthfulqa_sk_mc2": {
+ "alias": "truthfulqa_sk_mc2",
+ "acc,none": 0.3506542164316134,
+ "acc_stderr,none": 0.014662940445997986
+ }
+ },
+ "group_subtasks": {
+ "arc_sk": [],
+ "hellaswag_sk": [],
+ "m_mmlu_sk": [],
+ "sklegal": [],
+ "skquad": [],
+ "truthfulqa_sk_mc1": [],
+ "truthfulqa_sk_mc2": []
+ },
+ "configs": {
+ "arc_sk": {
+ "task": "arc_sk",
+ "tag": [
+ "arc_multilingual"
+ ],
+ "dataset_path": "alexandrainst/m_arc",
+ "dataset_name": "sk",
+ "training_split": "train",
+ "validation_split": "validation",
+ "test_split": "test",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n # breakpoint()\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"Question: \" + preprocess(doc[\"instruction\"]) + \"\\nAnswer:\",\n \"choices\": [\n preprocess(option)\n for option in [\n doc[\"option_a\"],\n doc[\"option_b\"],\n doc[\"option_c\"],\n doc[\"option_d\"],\n doc[\"option_e\"],\n ]\n if option\n ],\n \"gold\": [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answer\"]),\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_text": "query",
+ "doc_to_target": "gold",
+ "unsafe_code": false,
+ "doc_to_choice": "choices",
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "num_fewshot": 0,
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ },
+ {
+ "metric": "acc_norm",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "output_type": "multiple_choice",
+ "repeats": 1,
+ "should_decontaminate": true,
+ "doc_to_decontamination_query": "query",
+ "metadata": {
+ "version": 2.0
+ }
+ },
+ "hellaswag_sk": {
+ "task": "hellaswag_sk",
+ "tag": [
+ "hellaswag_multilingual"
+ ],
+ "dataset_path": "alexandrainst/m_hellaswag",
+ "dataset_name": "sk",
+ "validation_split": "val",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n out_doc = {\n \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_text": "query",
+ "doc_to_target": "{{label.lstrip()}}",
+ "unsafe_code": false,
+ "doc_to_choice": "choices",
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "num_fewshot": 0,
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ },
+ {
+ "metric": "acc_norm",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "output_type": "multiple_choice",
+ "repeats": 1,
+ "should_decontaminate": false,
+ "metadata": {
+ "version": 1.0
+ }
+ },
+ "m_mmlu_sk": {
+ "task": "m_mmlu_sk",
+ "tag": [
+ "m_mmlu"
+ ],
+ "dataset_path": "alexandrainst/m_mmlu",
+ "dataset_name": "sk",
+ "test_split": "test",
+ "fewshot_split": "train",
+ "doc_to_text": "{{instruction.strip()}}\nA. {{option_a}}\nB. {{option_b}}\nC. {{option_c}}\nD. {{option_d}}\nAnswer:",
+ "doc_to_target": "answer",
+ "unsafe_code": false,
+ "doc_to_choice": [
+ "A",
+ "B",
+ "C",
+ "D"
+ ],
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "fewshot_config": {
+ "sampler": "first_n"
+ },
+ "num_fewshot": 0,
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "output_type": "multiple_choice",
+ "repeats": 1,
+ "should_decontaminate": false,
+ "metadata": {
+ "version": 0.0
+ }
+ },
+ "sklegal": {
+ "task": "sklegal",
+ "dataset_path": "TUKE-KEMT/slovak-legal-exam",
+ "test_split": "test",
+ "doc_to_text": "{{prompt}} ",
+ "doc_to_target": "label",
+ "unsafe_code": false,
+ "doc_to_choice": "{{answers}}",
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "num_fewshot": 0,
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ },
+ {
+ "metric": "acc_norm",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "output_type": "multiple_choice",
+ "repeats": 1,
+ "should_decontaminate": false,
+ "metadata": {
+ "version": 1
+ }
+ },
+ "skquad": {
+ "task": "skquad",
+ "unsafe_code": false,
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "num_fewshot": 0,
+ "output_type": "generate_until",
+ "generation_kwargs": {
+ "until": [
+ "\n\n"
+ ],
+ "do_sample": false
+ },
+ "repeats": 1,
+ "should_decontaminate": false,
+ "metadata": {
+ "version": 3
+ }
+ },
+ "truthfulqa_sk_mc1": {
+ "task": "truthfulqa_sk_mc1",
+ "tag": [
+ "truthfulqa_multilingual"
+ ],
+ "dataset_path": "alexandrainst/m_truthfulqa",
+ "dataset_name": "sk",
+ "validation_split": "val",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_text": "query",
+ "doc_to_target": 0,
+ "unsafe_code": false,
+ "doc_to_choice": "mc1_choices",
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "num_fewshot": 0,
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "output_type": "multiple_choice",
+ "repeats": 1,
+ "should_decontaminate": true,
+ "doc_to_decontamination_query": "question",
+ "metadata": {
+ "version": 1.0
+ }
+ },
+ "truthfulqa_sk_mc2": {
+ "task": "truthfulqa_sk_mc2",
+ "tag": [
+ "truthfulqa_multilingual"
+ ],
+ "dataset_path": "alexandrainst/m_truthfulqa",
+ "dataset_name": "sk",
+ "validation_split": "val",
+ "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_doc(doc):\n out_doc = {\n \"question\": preprocess(doc[\"question\"]),\n \"query\": QA_PROMPT + \"\\n\\nQ: \" + preprocess(doc[\"question\"]) + \"\\nA:\",\n \"mc1_choices\": doc[\"mc1_targets_choices\"],\n \"mc2_choices\": doc[\"mc2_targets_choices\"],\n \"mc2_targets\": {\"labels\": doc[\"mc2_targets_labels\"]},\n \"gold\": \" \",\n }\n return out_doc\n\n return dataset.map(_process_doc)\n",
+ "doc_to_text": "query",
+ "doc_to_target": 0,
+ "unsafe_code": false,
+ "doc_to_choice": "mc2_choices",
+ "process_results": "def process_results_mc2(doc, results):\n lls, is_greedy = zip(*results)\n\n # Split on the first `0` as everything before it is true (`1`).\n split_idx = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n # Compute the normalized probability mass for the correct answer.\n ll_true, ll_false = lls[:split_idx], lls[split_idx:]\n p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n p_true = p_true / (sum(p_true) + sum(p_false))\n\n return {\"acc\": sum(p_true)}\n",
+ "description": "",
+ "target_delimiter": " ",
+ "fewshot_delimiter": "\n\n",
+ "num_fewshot": 0,
+ "metric_list": [
+ {
+ "metric": "acc",
+ "aggregation": "mean",
+ "higher_is_better": true
+ }
+ ],
+ "output_type": "multiple_choice",
+ "repeats": 1,
+ "should_decontaminate": true,
+ "doc_to_decontamination_query": "question",
+ "metadata": {
+ "version": 1.0
+ }
+ }
+ },
+ "versions": {
+ "arc_sk": 2.0,
+ "hellaswag_sk": 1.0,
+ "m_mmlu_sk": 0.0,
+ "sklegal": 1,
+ "skquad": 3,
+ "truthfulqa_sk_mc1": 1.0,
+ "truthfulqa_sk_mc2": 1.0
+ },
+ "n-shot": {
+ "arc_sk": 0,
+ "hellaswag_sk": 0,
+ "m_mmlu_sk": 0,
+ "sklegal": 0,
+ "skquad": 0,
+ "truthfulqa_sk_mc1": 0,
+ "truthfulqa_sk_mc2": 0
+ },
+ "higher_is_better": {
+ "arc_sk": {
+ "acc": true,
+ "acc_norm": true
+ },
+ "hellaswag_sk": {
+ "acc": true,
+ "acc_norm": true
+ },
+ "m_mmlu_sk": {
+ "acc": true
+ },
+ "sklegal": {
+ "acc": true,
+ "acc_norm": true
+ },
+ "skquad": {
+ "exact": true,
+ "f1": true,
+ "HasAns_exact": true,
+ "HasAns_f1": true,
+ "NoAns_exact": true,
+ "NoAns_f1": true,
+ "best_exact": true,
+ "best_f1": true
+ },
+ "truthfulqa_sk_mc1": {
+ "acc": true
+ },
+ "truthfulqa_sk_mc2": {
+ "acc": true
+ }
+ },
+ "n-samples": {
+ "truthfulqa_sk_mc2": {
+ "original": 778,
+ "effective": 778
+ },
+ "truthfulqa_sk_mc1": {
+ "original": 778,
+ "effective": 778
+ },
+ "skquad": {
+ "original": 9583,
+ "effective": 9583
+ },
+ "sklegal": {
+ "original": 1334,
+ "effective": 1334
+ },
+ "m_mmlu_sk": {
+ "original": 13062,
+ "effective": 13062
+ },
+ "hellaswag_sk": {
+ "original": 9485,
+ "effective": 9485
+ },
+ "arc_sk": {
+ "original": 1169,
+ "effective": 1169
+ }
+ },
+ "config": {
+ "model": "hf",
+ "model_args": "pretrained=/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010,tokenizer=/home/schwarc/.cache/huggingface/hub/models--slovak-nlp--mistral-sk-7b/snapshots/089497ae0d72018e9591895c54f774bf0b4a83a4,dtype=bfloat16",
+ "model_num_parameters": 7241732096,
+ "model_dtype": "torch.bfloat16",
+ "model_revision": "main",
+ "model_sha": "",
+ "batch_size": "1",
+ "batch_sizes": [],
+ "device": "cuda:0",
+ "use_cache": null,
+ "limit": null,
+ "bootstrap_iters": 100000,
+ "gen_kwargs": null,
+ "random_seed": 0,
+ "numpy_seed": 1234,
+ "torch_seed": 1234,
+ "fewshot_seed": 1234
+ },
+ "git_hash": "6d2abda4",
+ "date": 1792795427.1860082,
+ "pretty_env_info": "PyTorch version: 2.6.0+cu124\nIs debug build: False\nCUDA used to build PyTorch: 12.4\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 24.04.5 LTS (x86_64)\nGCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0\nClang version: Could not collect\nCMake version: version 3.28.3\nLibc version: glibc-2.39\n\nPython version: 3.11.16 (main, Aug 27 2026, 14:44:21) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.8.0-147-generic-x86_64-with-glibc2.39\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA RTX A4500\nGPU 1: NVIDIA RTX A4500\n\nNvidia driver version: 580.178.04\ncuDNN version: Could not collect\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 46 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 16\nOn-line CPU(s) list: 0-15\nVendor ID: GenuineIntel\nModel name: Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz\nCPU family: 6\nModel: 85\nThread(s) per core: 2\nCore(s) per socket: 8\nSocket(s): 1\nStepping: 7\nCPU(s) scaling MHz: 49%\nCPU max MHz: 4700.0000\nCPU min MHz: 1200.0000\nBogoMIPS: 7799.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi avx512_vnni md_clear flush_l1d arch_capabilities ibpb_exit_to_user\nVirtualization: VT-x\nL1d cache: 256 KiB (8 instances)\nL1i cache: 256 KiB (8 instances)\nL2 cache: 8 MiB (8 instances)\nL3 cache: 16.5 MiB (1 instance)\nNUMA node(s): 1\nNUMA node0 CPU(s): 0-15\nVulnerability Gather data sampling: Mitigation; Microcode\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit: KVM: Mitigation: VMX disabled\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable\nVulnerability Reg file data sampling: Not affected\nVulnerability Retbleed: Mitigation; Enhanced IBRS\nVulnerability Spec rstack overflow: Not affected\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop\nVulnerability Srbds: Not affected\nVulnerability Tsa: Not affected\nVulnerability Tsx async abort: Mitigation; TSX disabled\nVulnerability Vmscape: Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.4.6\n[pip3] nvidia-cublas-cu12==12.4.5.8\n[pip3] nvidia-cuda-cupti-cu12==12.4.127\n[pip3] nvidia-cuda-nvrtc-cu12==12.4.127\n[pip3] nvidia-cuda-runtime-cu12==12.4.127\n[pip3] nvidia-cudnn-cu12==9.1.0.70\n[pip3] nvidia-cufft-cu12==11.2.1.3\n[pip3] nvidia-curand-cu12==10.3.5.147\n[pip3] nvidia-cusolver-cu12==11.6.1.9\n[pip3] nvidia-cusparse-cu12==12.3.1.170\n[pip3] nvidia-cusparselt-cu12==0.6.2\n[pip3] nvidia-nccl-cu12==2.21.5\n[pip3] nvidia-nvjitlink-cu12==12.4.127\n[pip3] nvidia-nvtx-cu12==12.4.127\n[pip3] torch==2.6.0+cu124\n[pip3] triton==3.2.0\n[conda] numpy 2.4.6 pypi_0 pypi\n[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi\n[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi\n[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi\n[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi\n[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi\n[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi\n[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi\n[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi\n[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi\n[conda] nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi\n[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi\n[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi\n[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi\n[conda] torch 2.6.0+cu124 pypi_0 pypi\n[conda] triton 3.2.0 pypi_0 pypi",
+ "transformers_version": "4.48.2",
+ "upper_git_hash": null,
+ "tokenizer_pad_token": [
+ "",
+ "2"
+ ],
+ "tokenizer_eos_token": [
+ "",
+ "2"
+ ],
+ "tokenizer_bos_token": [
+ "",
+ "1"
+ ],
+ "eot_token_id": 2,
+ "max_length": 32768,
+ "task_hashes": {},
+ "model_source": "hf",
+ "model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010",
+ "model_name_sanitized": "__home__schwarc__diplomovka__skbench_evaluation__models__aya_300k_lr_0p00010",
+ "system_instruction": null,
+ "system_instruction_sha": null,
+ "fewshot_as_multiturn": false,
+ "chat_template": null,
+ "chat_template_sha": null,
+ "start_time": 896902.690480037,
+ "end_time": 910397.499522964,
+ "total_evaluation_time_seconds": "13494.809042926994"
+}
\ No newline at end of file
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/audit.md b/aya_300k_lr_experiments/benchmarks/skbench/audit.md
new file mode 100644
index 0000000..4e3b0e9
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/audit.md
@@ -0,0 +1,82 @@
+# Audit úspešného čiastočného SK-bench behu
+
+Vybraný model po tréningu na 300k vzorke Aya s LR `0.00010` dokončil beh `sk_zero_available_without_nli_lr_0p00010_20261022_185255`. Log končí hláškou `Benchmarking completed.` a obsahuje výsledné metriky. Benchmarkovalo sa iba toto LR, vybrané podľa najnižšieho finálneho validačného loss zo šiestich tréningových behov; výsledky preto neporovnávajú benchmarkovú kvalitu šiestich LR ani budúceho tréningu na celej Aya množine.
+
+## Pokrytie a počty
+
+- Zoznam `configs/tasks.txt`, 222 konfigurácií úloh vo výslednom JSON a 222 súborov DETAILS sa zhodujú. Názvy sa pri porovnaní dekódujú z Windows `%3A`/`%7C` a normalizuje sa koncové označenie zero-shot variantu.
+- Vyhodnotených bolo **222 zo 412 pôvodných task variantov**: 74 kombinácií úlohy/subtasku, každá s tromi promptmi. Ide o 13 rodín úloh. MMLU tvorí 171 variantov (57 predmetov × 3 prompty).
+- DETAILS obsahujú **159 426 vyhodnotených inštancií**, po **53 142 na každý prompt**. Tri prompty opakovane hodnotia tie isté základné testové otázky; 159 426 nie je počet nezávislých otázok ani dôkaz unikátnosti medzi rôznymi datasetmi.
+- `configs/omitted_tasks.txt` obsahuje 190 vynechaných variantov. Report dostupnosti eviduje 12 nedostupných repozitárov so 187 variantmi; tri ďalšie vynechané varianty patria samostatnému NLI-SK. Historický access report uvádza NLI-SK ako `OK`, ale jeho tri varianty v konečnom behu nie sú. NLI úlohy z rodiny SKLEP zostali zahrnuté.
+- `max_samples: null` znamená, že beh nemal globálny limit vzoriek. Počty sa získali z metadát jednotlivých DETAILS súborov, nie zo zaokrúhlených metrík.
+
+Tento dostupný subset sa nesmie označovať za oficiálne kompletné SK-bench skóre. Chýbajúce úlohy ani chýbajúce rodiny sa neimputovali.
+
+## Zaznamenané nastavenie
+
+Výsledný JSON potvrdzuje `bfloat16`, kontext `32768`, tensor/data/pipeline parallelism `1/1/1`, využitie GPU pamäte `0.85`, seed modelu `1234`, teplotu `0` a `enforce_eager: true`. Pole `generation_parameters.seed` je `null`; konkrétny seed je uložený v `model_config.seed`. Všetky úlohy majú `num_fewshots: 0`.
+
+216 task variantov používa najviac 32 generovaných tokenov a šesť QA variantov 126 tokenov; stop sekvencia je nový riadok. Tieto limity výstupu treba pri porovnaní výsledkov zachovať. Záznam neobsahuje spoľahlivý počet odpovedí ukončených výstupným limitom.
+
+Zmrazený commit SK-bench je `15823bbcc2d995e69441fcb7d34abb61bd8180c5`. Pôvodné `lighteval_sha` vo výslednom JSON je `?`; tento údaj sa dodatočne nevymýšľa. `uv.lock`, commit, modelová konfigurácia, per-task konfigurácie a revízie datasetov sú uchované v `provenance/` a `configs/`. Log zaznamenáva aj runtime patchovanie; samotný commit a lock preto bez rovnakého postupu nie sú úplným dôkazom identického prostredia.
+
+## Orezanie dlhých vstupov
+
+Kontrola projektovala `model_response.truncated_tokens_count` zo **všetkých 222 DETAILS súborov**, bez čítania duplicitného generation cache. Všetkých 159 426 hodnôt existuje, žiadna nie je `null` a všetky sú nula. Aj pôvodné súhrnné počítadlá sú nulové vrátane `non_truncated`, takže samy osebe nepotvrdzujú neprítomnosť orezania.
+
+**Log priamo potvrdzuje orezanie kontextu.** Vyhľadanie `truncat` bez rozlíšenia veľkosti písmen našlo presne jednu hlášku, na LF riadku 16033:
+
+```text
+context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens.
+```
+
+Následná diagnostika dĺžok zo všetkých DETAILS našla **69 tokenových polí presne na vstupnom limite 32 642 tokenov**, po 23 v každom z troch variantov `multi_wiki_qa`. Vo všetkých troch sa opakujú tie isté indexy riadkov. Žiadne uložené pole neprekročilo príslušný limit `32768 − generation_size`; ostatné úlohy mali vstupy pod limitom. Súvisiace uložené reťazce vstupu mali pri hraničných riadkoch až 121 852 znakov.
+
+Týchto 69 riadkov sú **hraniční kandidáti zodpovedajúci orezaniu**, nie spoľahlivo dokázaný počet orezaných vstupov. Log neuvádza identifikátor konkrétneho tasku ani počet dotknutých vzoriek. Bez opätovnej tokenizácie pôvodných vstupov nemožno z dĺžky uloženého reťazca určiť pôvodnú dĺžku v tokenoch alebo počet odstránených tokenov. Overený záver je: **orezanie nastalo aspoň raz; jeho presný rozsah nie je z dostupných počítadiel známy**.
+
+Strojový súhrn preto obsahuje `context_truncation_observed_in_log: true`, `actual_truncated_sample_count: null` a nulové počítadlá označuje prefixom `details_reported_`. Podrobnosti sú v `results/truncation_audit_per_task.csv`, `results/log_truncation_evidence.json`, `results/input_length_diagnostics_per_task.csv` a `results/input_length_budget_boundary_rows.csv`.
+
+## Metriky a agregácia
+
+`results/per_task_metrics.csv` obsahuje iba 222 reálne vykonaných task variantov, ich skutočné metriky a štandardné chyby priamo z výsledného JSON. Prioritná metrika na čítanie je `em_relaxed`, prípadne `em_norm_relaxed`, ak ju daná úloha skutočne používa. QA má vlastné metriky `exact_match_slk_prefix` a `f1_slk`; nepremenúvajú sa na všeobecnú accuracy.
+
+Odvodené súhrny nikdy nemiešajú odlišné metriky do jedného skóre:
+
+- `family_summary.csv`: priemer v rodine, vždy osobitne pre konkrétnu metriku, cez všetky dostupné subtasky a tri prompty.
+- `family_prompt_summary.csv`: tá istá agregácia osobitne pre prompt 1, 2 a 3.
+- `three_prompt_summary.csv`: priemer troch promptov pre každú zo 74 konkrétnych kombinácií úlohy/subtasku, opäť pre každú metriku osobitne.
+- `task_macro_mean` je aritmetický priemer task metrík s rovnakou váhou každého variantu. `sample_weighted_mean` váži tieto metriky počtom riadkov v DETAILS. Prompty sa v oboch prípadoch považujú za opakované vyhodnotenia, nie za nové nezávislé otázky. Súhrnné intervaly spoľahlivosti sa z per-task chýb nevymýšľajú.
+
+Pôvodný JSON má 226 kľúčov výsledkov: 222 úloh, tri vstavané MMLU promptové priemery a `all`. Štyri agregované riadky sú zachované v pôvodnom JSON, ale v odvodených súhrnoch sa znovu nepripočítavajú. `all` obsahuje viac metrík s rôznymi sadami úloh a netvorí oficiálne celkové skóre.
+
+Napríklad MMLU `em_relaxed` má task macro priemer `0.1600941253` a sample-weighted priemer `0.1572245190`. Rozdiel vzniká rôznou veľkosťou 57 predmetov. SKLEP klasifikačné úlohy s `em_norm_relaxed` majú macro `0.3507994510`, ale sample-weighted `0.2144044321`; výber váh tu zásadne mení opisný výsledok.
+
+## Obnovenie auditu
+
+Audit používa Python a `pyarrow==26.0.0`. Knižnica sa pri spracovaní nainštalovala len do pomocného priečinka pracovného projektu; nie je súčasťou tohto stromu. Pôvodné parquety, cache ani váhy sa do Git podkladov nekopírujú. Na opätovný audit treba samostatne rozbalený zdrojový archív a jeho `archive_manifest.json`.
+
+Archív treba najprv rozbaliť pomocou uloženého pomocného skriptu (Python 3.11+), ktorý vytvorí aj `archive_manifest.json` a `archive_summary.json`. Názvy s `:` a `|` prevedie na prenosné percent-encoded názvy a zachová mapovanie na originál. Priečinok určený cez `--output` musí byť nový alebo prázdny:
+
+```bash
+python scripts/unpack_results_archive.py \
+ --archive /path/to/aya_300k_benchmarky_20261024_215708.tar.gz \
+ --output /path/to/extracted/archive
+```
+
+Z koreňa experimentu možno potom spustiť:
+
+```bash
+python -m pip install --target .audit-deps -r benchmarks/skbench/provenance/requirements-audit.txt
+python benchmarks/skbench/provenance/audit_skbench.py \
+ --source /path/to/extracted/archive \
+ --output benchmarks/skbench \
+ --pyarrow-path .audit-deps
+python benchmarks/skbench/provenance/input_length_diagnostics.py \
+ --source /path/to/extracted/archive \
+ --output benchmarks/skbench/results \
+ --pyarrow-path .audit-deps
+```
+
+Prvý skript kontroluje pokrytie, kopíruje malé pôvodné artefakty, počíta metrikové súhrny a overuje truncation counter projekciou jediného stĺpca. Druhý skript projektuje uložené tokenové polia a vstupné reťazce, ale exportuje iba ich dĺžky a indexy hraničných riadkov. Texty otázok a odpovedí v týchto odvodených diagnostikách nie sú.
+
+SHA-256 presných kópií pôvodných súborov je v `provenance/copied_files_sha256.csv`; zdrojový archív identifikuje `provenance/archive_summary.json`.
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json b/aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json
new file mode 100644
index 0000000..cc43959
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/dataset_access_report.json
@@ -0,0 +1,622 @@
+[
+ {
+ "repo": "CohereLabs/Global-MMLU-Lite",
+ "revision": "600af1bff2d447c7fc3aee22f11f607560d5fcb7",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "globalmmlu-lite.prompt3_mqa_single_correct_sk|0",
+ "globalmmlu-lite.prompt1_mqa_single_correct_sk|0",
+ "globalmmlu-lite.prompt2_mqa_single_correct_sk|0"
+ ]
+ },
+ {
+ "repo": "NaiveNeuron/SlovakCOPA",
+ "revision": "5c61e4cd9343f9b4fab623bdc50bb895cc170771",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "slovakcopa.prompt3_copa_sk|0",
+ "slovakcopa.prompt1_copa_sk|0",
+ "slovakcopa.prompt2_copa_sk|0"
+ ]
+ },
+ {
+ "repo": "TUKE-KEMT/senti-sk",
+ "revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "senti_sk.prompt2_sentiment_binary_sk|0",
+ "senti_sk.prompt3_sentiment_binary_sk|0",
+ "senti_sk.prompt1_sentiment_binary_sk|0",
+ "senti_sk.prompt3_sentiment_3_class_sk|0",
+ "senti_sk.prompt1_sentiment_3_class_sk|0",
+ "senti_sk.prompt2_sentiment_3_class_sk|0"
+ ]
+ },
+ {
+ "repo": "TUKE-KEMT/slovak-financial-exam",
+ "revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "financial_exams.prompt1_completion_sk|0",
+ "financial_exams.prompt3_completion_sk|0",
+ "financial_exams.prompt2_completion_sk|0"
+ ]
+ },
+ {
+ "repo": "TUKE-KEMT/toxic-sk",
+ "revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "toxic_sk.prompt2_toxicity_sk|0",
+ "toxic_sk.prompt3_toxicity_sk|0",
+ "toxic_sk.prompt1_toxicity_sk|0"
+ ]
+ },
+ {
+ "repo": "alexandrainst/m_truthfulqa",
+ "revision": "f0445d470f1925882b990f5f247fdcf288972f60",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "truthfulqa.prompt3_mqa_single_correct_sk:mc1|0",
+ "truthfulqa.prompt1_mqa_single_correct_sk:mc1|0",
+ "truthfulqa.prompt2_mqa_single_correct_sk:mc1|0"
+ ]
+ },
+ {
+ "repo": "alexandrainst/multi-wiki-qa",
+ "revision": "9bdb3350487e399bd1cd873e1395290d636db5ce",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "multi_wiki_qa.prompt3_qa_sk|0",
+ "multi_wiki_qa.prompt2_qa_sk|0",
+ "multi_wiki_qa.prompt1_qa_sk|0"
+ ]
+ },
+ {
+ "repo": "facebook/belebele",
+ "revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "belebele.prompt3_mqa_single_correct_sk|0",
+ "belebele.prompt1_mqa_single_correct_sk|0",
+ "belebele.prompt2_mqa_single_correct_sk|0"
+ ]
+ },
+ {
+ "repo": "jon-tow/okapi_arc_challenge",
+ "revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "arc.prompt3_mqa_single_correct_sk|0",
+ "arc.prompt1_mqa_single_correct_sk|0",
+ "arc.prompt2_mqa_single_correct_sk|0"
+ ]
+ },
+ {
+ "repo": "lighteval/okapi_mmlu",
+ "revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0",
+ "mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0",
+ "mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0",
+ "mmlu.prompt3_mqa_single_correct_sk:anatomy|0",
+ "mmlu.prompt1_mqa_single_correct_sk:anatomy|0",
+ "mmlu.prompt2_mqa_single_correct_sk:anatomy|0",
+ "mmlu.prompt3_mqa_single_correct_sk:astronomy|0",
+ "mmlu.prompt1_mqa_single_correct_sk:astronomy|0",
+ "mmlu.prompt2_mqa_single_correct_sk:astronomy|0",
+ "mmlu.prompt3_mqa_single_correct_sk:business_ethics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:business_ethics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:business_ethics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0",
+ "mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0",
+ "mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0",
+ "mmlu.prompt3_mqa_single_correct_sk:college_biology|0",
+ "mmlu.prompt1_mqa_single_correct_sk:college_biology|0",
+ "mmlu.prompt2_mqa_single_correct_sk:college_biology|0",
+ "mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0",
+ "mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0",
+ "mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0",
+ "mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0",
+ "mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0",
+ "mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0",
+ "mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:college_medicine|0",
+ "mmlu.prompt1_mqa_single_correct_sk:college_medicine|0",
+ "mmlu.prompt2_mqa_single_correct_sk:college_medicine|0",
+ "mmlu.prompt3_mqa_single_correct_sk:college_physics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:college_physics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:college_physics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:computer_security|0",
+ "mmlu.prompt1_mqa_single_correct_sk:computer_security|0",
+ "mmlu.prompt2_mqa_single_correct_sk:computer_security|0",
+ "mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:econometrics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:econometrics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:econometrics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0",
+ "mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0",
+ "mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0",
+ "mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:formal_logic|0",
+ "mmlu.prompt1_mqa_single_correct_sk:formal_logic|0",
+ "mmlu.prompt2_mqa_single_correct_sk:formal_logic|0",
+ "mmlu.prompt3_mqa_single_correct_sk:global_facts|0",
+ "mmlu.prompt1_mqa_single_correct_sk:global_facts|0",
+ "mmlu.prompt2_mqa_single_correct_sk:global_facts|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0",
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0",
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0",
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0",
+ "mmlu.prompt3_mqa_single_correct_sk:human_aging|0",
+ "mmlu.prompt1_mqa_single_correct_sk:human_aging|0",
+ "mmlu.prompt2_mqa_single_correct_sk:human_aging|0",
+ "mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0",
+ "mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0",
+ "mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0",
+ "mmlu.prompt3_mqa_single_correct_sk:international_law|0",
+ "mmlu.prompt1_mqa_single_correct_sk:international_law|0",
+ "mmlu.prompt2_mqa_single_correct_sk:international_law|0",
+ "mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0",
+ "mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0",
+ "mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0",
+ "mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0",
+ "mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0",
+ "mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0",
+ "mmlu.prompt3_mqa_single_correct_sk:machine_learning|0",
+ "mmlu.prompt1_mqa_single_correct_sk:machine_learning|0",
+ "mmlu.prompt2_mqa_single_correct_sk:machine_learning|0",
+ "mmlu.prompt3_mqa_single_correct_sk:management|0",
+ "mmlu.prompt1_mqa_single_correct_sk:management|0",
+ "mmlu.prompt2_mqa_single_correct_sk:management|0",
+ "mmlu.prompt3_mqa_single_correct_sk:marketing|0",
+ "mmlu.prompt1_mqa_single_correct_sk:marketing|0",
+ "mmlu.prompt2_mqa_single_correct_sk:marketing|0",
+ "mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0",
+ "mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0",
+ "mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0",
+ "mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0",
+ "mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0",
+ "mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0",
+ "mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0",
+ "mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0",
+ "mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0",
+ "mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0",
+ "mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0",
+ "mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0",
+ "mmlu.prompt3_mqa_single_correct_sk:nutrition|0",
+ "mmlu.prompt1_mqa_single_correct_sk:nutrition|0",
+ "mmlu.prompt2_mqa_single_correct_sk:nutrition|0",
+ "mmlu.prompt3_mqa_single_correct_sk:philosophy|0",
+ "mmlu.prompt1_mqa_single_correct_sk:philosophy|0",
+ "mmlu.prompt2_mqa_single_correct_sk:philosophy|0",
+ "mmlu.prompt3_mqa_single_correct_sk:prehistory|0",
+ "mmlu.prompt1_mqa_single_correct_sk:prehistory|0",
+ "mmlu.prompt2_mqa_single_correct_sk:prehistory|0",
+ "mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0",
+ "mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0",
+ "mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0",
+ "mmlu.prompt3_mqa_single_correct_sk:professional_law|0",
+ "mmlu.prompt1_mqa_single_correct_sk:professional_law|0",
+ "mmlu.prompt2_mqa_single_correct_sk:professional_law|0",
+ "mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0",
+ "mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0",
+ "mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0",
+ "mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0",
+ "mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0",
+ "mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0",
+ "mmlu.prompt3_mqa_single_correct_sk:public_relations|0",
+ "mmlu.prompt1_mqa_single_correct_sk:public_relations|0",
+ "mmlu.prompt2_mqa_single_correct_sk:public_relations|0",
+ "mmlu.prompt3_mqa_single_correct_sk:security_studies|0",
+ "mmlu.prompt1_mqa_single_correct_sk:security_studies|0",
+ "mmlu.prompt2_mqa_single_correct_sk:security_studies|0",
+ "mmlu.prompt3_mqa_single_correct_sk:sociology|0",
+ "mmlu.prompt1_mqa_single_correct_sk:sociology|0",
+ "mmlu.prompt2_mqa_single_correct_sk:sociology|0",
+ "mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0",
+ "mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0",
+ "mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0",
+ "mmlu.prompt3_mqa_single_correct_sk:virology|0",
+ "mmlu.prompt1_mqa_single_correct_sk:virology|0",
+ "mmlu.prompt2_mqa_single_correct_sk:virology|0",
+ "mmlu.prompt3_mqa_single_correct_sk:world_religions|0",
+ "mmlu.prompt1_mqa_single_correct_sk:world_religions|0",
+ "mmlu.prompt2_mqa_single_correct_sk:world_religions|0"
+ ]
+ },
+ {
+ "repo": "mteb/slovak_movie_review_sentiment",
+ "revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "movie_review.prompt3_sentiment_binary_sk|0",
+ "movie_review.prompt2_sentiment_binary_sk|0",
+ "movie_review.prompt1_sentiment_binary_sk|0"
+ ]
+ },
+ {
+ "repo": "natalia-nk/NLI-SK-annotated",
+ "revision": "79914f425b59d8b9fabb6d38c37f6d81f9723f46",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "nli_sk.prompt1_nli_3_classes_sk|0",
+ "nli_sk.prompt2_nli_3_classes_sk|0",
+ "nli_sk.prompt3_nli_3_classes_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/SKJ1",
+ "revision": "e615c60b415389b629703f82d3a79810dbde6734",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "skj1.prompt3_skj1_sk|0",
+ "skj1.prompt1_skj1_sk|0",
+ "skj1.prompt2_skj1_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/chiby-classification",
+ "revision": "ce8533398057ace6da817758335633067648ca93",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-1e7a56917b7b19192d552e13;119d40c0-7b4d-43f3-8cd4-82370a8ee77e)",
+ "tasks": [
+ "chiby.prompt2_chiby_sk|0",
+ "chiby.prompt1_chiby_sk|0",
+ "chiby.prompt3_chiby_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/driving-test-dataset-slovak",
+ "revision": "37064890fbc00bfca59c035d38f52d265949361e",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-3129bdcb539f88e7672e2027;105ec634-cecc-4558-b3c9-e83508219914)",
+ "tasks": [
+ "driving_test.prompt3_mqa_single_correct_sk|0",
+ "driving_test.prompt1_mqa_single_correct_sk|0",
+ "driving_test.prompt2_mqa_single_correct_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/e-testy-sk",
+ "revision": "1a5e798e0473241097c4a26c511426167522adf5",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-4f270b5d6ba17a181d7897ee;9fecd2d5-dc7c-4a4c-8755-61fd6bcda935)",
+ "tasks": [
+ "e_tests.prompt3_mqa_single_correct_sk:biologia|0",
+ "e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0",
+ "e_tests.prompt3_mqa_single_correct_sk:informatika|0",
+ "e_tests.prompt3_mqa_single_correct_sk:geografia|0",
+ "e_tests.prompt3_mqa_single_correct_sk:matematika|0",
+ "e_tests.prompt3_mqa_single_correct_sk:chemia|0",
+ "e_tests.prompt3_mqa_single_correct_sk:dejepis|0",
+ "e_tests.prompt3_mqa_single_correct_sk:fyzika|0",
+ "e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0",
+ "e_tests.prompt1_mqa_single_correct_sk:biologia|0",
+ "e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0",
+ "e_tests.prompt1_mqa_single_correct_sk:informatika|0",
+ "e_tests.prompt1_mqa_single_correct_sk:geografia|0",
+ "e_tests.prompt1_mqa_single_correct_sk:matematika|0",
+ "e_tests.prompt1_mqa_single_correct_sk:chemia|0",
+ "e_tests.prompt1_mqa_single_correct_sk:dejepis|0",
+ "e_tests.prompt1_mqa_single_correct_sk:fyzika|0",
+ "e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0",
+ "e_tests.prompt2_mqa_single_correct_sk:biologia|0",
+ "e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0",
+ "e_tests.prompt2_mqa_single_correct_sk:informatika|0",
+ "e_tests.prompt2_mqa_single_correct_sk:geografia|0",
+ "e_tests.prompt2_mqa_single_correct_sk:matematika|0",
+ "e_tests.prompt2_mqa_single_correct_sk:chemia|0",
+ "e_tests.prompt2_mqa_single_correct_sk:dejepis|0",
+ "e_tests.prompt2_mqa_single_correct_sk:fyzika|0",
+ "e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:biologia|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:informatika|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:geografia|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:matematika|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:chemia|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0",
+ "e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:biologia|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:informatika|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:geografia|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:matematika|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:chemia|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0",
+ "e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:biologia|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:informatika|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:geografia|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:matematika|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:chemia|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0",
+ "e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0",
+ "e_tests.prompt1_qa_sk:biologia|0",
+ "e_tests.prompt1_qa_sk:obcianska-nauka|0",
+ "e_tests.prompt1_qa_sk:informatika|0",
+ "e_tests.prompt1_qa_sk:geografia|0",
+ "e_tests.prompt1_qa_sk:chemia|0",
+ "e_tests.prompt1_qa_sk:dejepis|0",
+ "e_tests.prompt1_qa_sk:fyzika|0",
+ "e_tests.prompt1_qa_sk:slovensky-jazyk|0",
+ "e_tests.prompt3_qa_sk:biologia|0",
+ "e_tests.prompt3_qa_sk:obcianska-nauka|0",
+ "e_tests.prompt3_qa_sk:informatika|0",
+ "e_tests.prompt3_qa_sk:geografia|0",
+ "e_tests.prompt3_qa_sk:chemia|0",
+ "e_tests.prompt3_qa_sk:dejepis|0",
+ "e_tests.prompt3_qa_sk:fyzika|0",
+ "e_tests.prompt3_qa_sk:slovensky-jazyk|0",
+ "e_tests.prompt2_qa_sk:biologia|0",
+ "e_tests.prompt2_qa_sk:obcianska-nauka|0",
+ "e_tests.prompt2_qa_sk:informatika|0",
+ "e_tests.prompt2_qa_sk:geografia|0",
+ "e_tests.prompt2_qa_sk:chemia|0",
+ "e_tests.prompt2_qa_sk:dejepis|0",
+ "e_tests.prompt2_qa_sk:fyzika|0",
+ "e_tests.prompt2_qa_sk:slovensky-jazyk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/ifeval-sk",
+ "revision": "2114c70fdcd22e8e3e5ea11f6b77067740959bf4",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-5666942d3a4121b676a928ff;cbd91d44-6ee2-44b0-b74b-3aca15012a31)",
+ "tasks": [
+ "ifeval_sk.ifeval|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/oneruler-sk-processed",
+ "revision": "c3a160bc7ce2dada86803f26ee699ca75ee26e41",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-342e828a639d5d19427fae4f;91605400-c335-4c30-8e40-78bdf0e40b3c)",
+ "tasks": [
+ "oneruler_sk.oneruler_slovak:cwe_4096|0",
+ "oneruler_sk.oneruler_slovak:cwe_8192|0",
+ "oneruler_sk.oneruler_slovak:cwe_16384|0",
+ "oneruler_sk.oneruler_slovak:niah_multikey_4096|0",
+ "oneruler_sk.oneruler_slovak:niah_multikey_8192|0",
+ "oneruler_sk.oneruler_slovak:niah_multikey_16384|0",
+ "oneruler_sk.oneruler_slovak:niah_multiquery_4096|0",
+ "oneruler_sk.oneruler_slovak:niah_multiquery_8192|0",
+ "oneruler_sk.oneruler_slovak:niah_multiquery_16384|0",
+ "oneruler_sk.oneruler_slovak:niah_multivalue_4096|0",
+ "oneruler_sk.oneruler_slovak:niah_multivalue_8192|0",
+ "oneruler_sk.oneruler_slovak:niah_multivalue_16384|0",
+ "oneruler_sk.oneruler_slovak:niah_none_4096|0",
+ "oneruler_sk.oneruler_slovak:niah_none_8192|0",
+ "oneruler_sk.oneruler_slovak:niah_none_16384|0",
+ "oneruler_sk.oneruler_slovak:niah_single_4096|0",
+ "oneruler_sk.oneruler_slovak:niah_single_8192|0",
+ "oneruler_sk.oneruler_slovak:niah_single_16384|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/scio-testy",
+ "revision": "78c67eeae5d48b92590945a1c589b7730bfce8df",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-203c6cfb4d094ab1703aa786;3f87f5ab-83e3-4b52-b012-e6e26eed3d43)",
+ "tasks": [
+ "scio.prompt2_mqa_single_correct_sk:aj|0",
+ "scio.prompt2_mqa_single_correct_sk:mat|0",
+ "scio.prompt2_mqa_single_correct_sk:bio|0",
+ "scio.prompt2_mqa_single_correct_sk:che|0",
+ "scio.prompt2_mqa_single_correct_sk:vsp|0",
+ "scio.prompt3_mqa_single_correct_sk:aj|0",
+ "scio.prompt3_mqa_single_correct_sk:mat|0",
+ "scio.prompt3_mqa_single_correct_sk:bio|0",
+ "scio.prompt3_mqa_single_correct_sk:che|0",
+ "scio.prompt3_mqa_single_correct_sk:vsp|0",
+ "scio.prompt1_mqa_single_correct_sk:aj|0",
+ "scio.prompt1_mqa_single_correct_sk:mat|0",
+ "scio.prompt1_mqa_single_correct_sk:bio|0",
+ "scio.prompt1_mqa_single_correct_sk:che|0",
+ "scio.prompt1_mqa_single_correct_sk:vsp|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/sklep",
+ "revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "status": "OK",
+ "error": "",
+ "tasks": [
+ "sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0",
+ "sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0",
+ "sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0",
+ "sklep.prompt1_hate_speech_sk:hate-speech|0",
+ "sklep.prompt3_hate_speech_sk:hate-speech|0",
+ "sklep.prompt2_hate_speech_sk:hate-speech|0",
+ "sklep.prompt1_nli_3_classes_sk:nli|0",
+ "sklep.prompt2_nli_3_classes_sk:nli|0",
+ "sklep.prompt3_nli_3_classes_sk:nli|0",
+ "sklep.prompt1_nli_binary_sk:rte|0",
+ "sklep.prompt3_nli_binary_sk:rte|0",
+ "sklep.prompt2_nli_binary_sk:rte|0",
+ "sklep.prompt3_qa_sk:question-answering|0",
+ "sklep.prompt2_qa_sk:question-answering|0",
+ "sklep.prompt1_qa_sk:question-answering|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/slohry",
+ "revision": "3b47acd8a549c65ec6ab34aacd538f3ebcbfa32f",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d7-18026919100903a2056ca192;a073f7f3-9b00-49ca-982c-c03741ca2364)",
+ "tasks": [
+ "slohry.prompt2_declension_sk:declension_high|0",
+ "slohry.prompt2_declension_sk:declension_mid|0",
+ "slohry.prompt2_declension_sk:declension_low|0",
+ "slohry.prompt1_declension_sk:declension_high|0",
+ "slohry.prompt1_declension_sk:declension_mid|0",
+ "slohry.prompt1_declension_sk:declension_low|0",
+ "slohry.prompt3_declension_sk:declension_high|0",
+ "slohry.prompt3_declension_sk:declension_mid|0",
+ "slohry.prompt3_declension_sk:declension_low|0",
+ "slohry.prompt3_gradation_sk:gradation_high|0",
+ "slohry.prompt3_gradation_sk:gradation_mid|0",
+ "slohry.prompt3_gradation_sk:gradation_low|0",
+ "slohry.prompt1_gradation_sk:gradation_high|0",
+ "slohry.prompt1_gradation_sk:gradation_mid|0",
+ "slohry.prompt1_gradation_sk:gradation_low|0",
+ "slohry.prompt2_gradation_sk:gradation_high|0",
+ "slohry.prompt2_gradation_sk:gradation_mid|0",
+ "slohry.prompt2_gradation_sk:gradation_low|0",
+ "slohry.prompt1_identity_form_sk:identity_form_high|0",
+ "slohry.prompt1_identity_form_sk:identity_form_mid|0",
+ "slohry.prompt1_identity_form_sk:identity_form_low|0",
+ "slohry.prompt2_identity_form_sk:identity_form_high|0",
+ "slohry.prompt2_identity_form_sk:identity_form_mid|0",
+ "slohry.prompt2_identity_form_sk:identity_form_low|0",
+ "slohry.prompt3_identity_form_sk:identity_form_high|0",
+ "slohry.prompt3_identity_form_sk:identity_form_mid|0",
+ "slohry.prompt3_identity_form_sk:identity_form_low|0",
+ "slohry.prompt2_inflection_sk:inflection_high|0",
+ "slohry.prompt2_inflection_sk:inflection_mid|0",
+ "slohry.prompt2_inflection_sk:inflection_low|0",
+ "slohry.prompt1_inflection_sk:inflection_high|0",
+ "slohry.prompt1_inflection_sk:inflection_mid|0",
+ "slohry.prompt1_inflection_sk:inflection_low|0",
+ "slohry.prompt3_inflection_sk:inflection_high|0",
+ "slohry.prompt3_inflection_sk:inflection_mid|0",
+ "slohry.prompt3_inflection_sk:inflection_low|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/slovak-school-exams-test-preprocessed",
+ "revision": "1d51f8205ee2341d11f05273d9c6d4ba3532a1df",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-7d40d0951ba637dd705fbdd4;c5ed7cb3-8e89-455e-af3d-699ae41c323d)",
+ "tasks": [
+ "slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0",
+ "slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0",
+ "slovak_tests.prompt3_mqa_single_correct_sk:maturita|0",
+ "slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0",
+ "slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0",
+ "slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0",
+ "slovak_tests.prompt1_mqa_single_correct_sk:maturita|0",
+ "slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0",
+ "slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0",
+ "slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0",
+ "slovak_tests.prompt2_mqa_single_correct_sk:maturita|0",
+ "slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0",
+ "slovak_tests.prompt1_qa_sk:monitor9|0",
+ "slovak_tests.prompt1_qa_sk:maturita|0",
+ "slovak_tests.prompt1_qa_sk:prijmacie_skusky|0",
+ "slovak_tests.prompt3_qa_sk:monitor9|0",
+ "slovak_tests.prompt3_qa_sk:maturita|0",
+ "slovak_tests.prompt3_qa_sk:prijmacie_skusky|0",
+ "slovak_tests.prompt2_qa_sk:monitor9|0",
+ "slovak_tests.prompt2_qa_sk:maturita|0",
+ "slovak_tests.prompt2_qa_sk:prijmacie_skusky|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/slovak_trivia",
+ "revision": "2f27ff5a3c266a379c164d3cb43fc9cf8e45b37a",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-60bbc6ed1f90512f2a40bf37;5a9d9ee9-412d-4cdd-a8a3-66accb5e4d3e)",
+ "tasks": [
+ "slovak_trivia.prompt2_mqa_single_correct_sk|0",
+ "slovak_trivia.prompt3_mqa_single_correct_sk|0",
+ "slovak_trivia.prompt1_mqa_single_correct_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/slovaksum-downsampled",
+ "revision": "2e4cdabe972ac040114701ea9e885e0a648a33fb",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-3266710308ee05503220c4de;bd8592d6-e022-40ef-a2dd-c9bd490e957f)",
+ "tasks": [
+ "slovaksum.prompt2_summarization_sk|0",
+ "slovaksum.prompt3_summarization_sk|0",
+ "slovaksum.prompt1_summarization_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/smesum-downsampled",
+ "revision": "cb427bdc96e444637508fb982915f706c19d81d4",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-4c1e6f5602eda70b00239325;126975ea-e381-456f-8fa6-0ffff90f3938)",
+ "tasks": [
+ "smesum.prompt2_summarization_sk|0",
+ "smesum.prompt3_summarization_sk|0",
+ "smesum.prompt1_summarization_sk|0"
+ ]
+ },
+ {
+ "repo": "slovak-nlp/word-analogies-sk",
+ "revision": "d6564d49c07b29ea93028f55a57c25eb231762f9",
+ "status": "NEDOSTUPNÉ",
+ "error": "RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6ada58d8-18fc3a1223dc59e9238fa860;c8c94f13-fd96-4630-9510-c1ab0db7a366)",
+ "tasks": [
+ "analogies_sk.prompt1_analogy_sk|0",
+ "analogies_sk.prompt2_analogy_sk|0",
+ "analogies_sk.prompt3_analogy_sk|0"
+ ]
+ }
+]
\ No newline at end of file
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/export_lr_0p00010.yaml b/aya_300k_lr_experiments/benchmarks/skbench/configs/export_lr_0p00010.yaml
new file mode 100644
index 0000000..fcbcfa7
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/export_lr_0p00010.yaml
@@ -0,0 +1,12 @@
+model_name_or_path: slovak-nlp/mistral-sk-7b
+model_revision: 089497ae0d72018e9591895c54f774bf0b4a83a4
+adapter_name_or_path: /home/schwarc/diplomovka/aya_300k_lr_validation/lr_0p00010
+
+template: alpaca
+finetuning_type: lora
+infer_dtype: bfloat16
+
+export_dir: /home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010
+export_size: 5
+export_device: cpu
+export_legacy_format: false
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/model_config.json b/aya_300k_lr_experiments/benchmarks/skbench/configs/model_config.json
new file mode 100644
index 0000000..ebbb6ba
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/model_config.json
@@ -0,0 +1,59 @@
+{
+ "lighteval_sha": "?",
+ "num_fewshot_seeds": 1,
+ "max_samples": null,
+ "job_id": "0",
+ "start_time": 796668.90423246,
+ "end_time": 807588.555648873,
+ "total_evaluation_time_secondes": "10919.65141641302",
+ "model_config": {
+ "model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010",
+ "generation_parameters": {
+ "num_blocks": null,
+ "block_size": null,
+ "early_stopping": null,
+ "repetition_penalty": null,
+ "frequency_penalty": null,
+ "length_penalty": null,
+ "presence_penalty": null,
+ "max_new_tokens": null,
+ "min_new_tokens": null,
+ "seed": null,
+ "stop_tokens": null,
+ "temperature": 0,
+ "top_k": null,
+ "min_p": null,
+ "top_p": null,
+ "truncate_prompt": null,
+ "cache_implementation": null,
+ "response_format": null,
+ "skip_special_tokens": true
+ },
+ "system_prompt": null,
+ "cache_dir": "~/.cache/huggingface/lighteval",
+ "tokenizer": null,
+ "revision": "main",
+ "dtype": "bfloat16",
+ "tensor_parallel_size": 1,
+ "data_parallel_size": 1,
+ "pipeline_parallel_size": 1,
+ "gpu_memory_utilization": 0.85,
+ "enable_prefix_caching": null,
+ "max_model_length": 32768,
+ "quantization": null,
+ "load_format": null,
+ "swap_space": 4,
+ "seed": 1234,
+ "trust_remote_code": false,
+ "add_special_tokens": true,
+ "multichoice_continuations_start_space": true,
+ "pairwise_tokenization": false,
+ "max_num_seqs": 128,
+ "max_num_batched_tokens": 2048,
+ "subfolder": null,
+ "is_async": false,
+ "override_chat_template": null,
+ "enforce_eager": true
+ },
+ "model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010"
+}
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt b/aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt
new file mode 100644
index 0000000..b7b2fdb
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/omitted_tasks.txt
@@ -0,0 +1,190 @@
+chiby.prompt2_chiby_sk|0
+chiby.prompt1_chiby_sk|0
+chiby.prompt3_chiby_sk|0
+ifeval_sk.ifeval|0
+oneruler_sk.oneruler_slovak:cwe_4096|0
+oneruler_sk.oneruler_slovak:cwe_8192|0
+oneruler_sk.oneruler_slovak:cwe_16384|0
+oneruler_sk.oneruler_slovak:niah_multikey_4096|0
+oneruler_sk.oneruler_slovak:niah_multikey_8192|0
+oneruler_sk.oneruler_slovak:niah_multikey_16384|0
+oneruler_sk.oneruler_slovak:niah_multiquery_4096|0
+oneruler_sk.oneruler_slovak:niah_multiquery_8192|0
+oneruler_sk.oneruler_slovak:niah_multiquery_16384|0
+oneruler_sk.oneruler_slovak:niah_multivalue_4096|0
+oneruler_sk.oneruler_slovak:niah_multivalue_8192|0
+oneruler_sk.oneruler_slovak:niah_multivalue_16384|0
+oneruler_sk.oneruler_slovak:niah_none_4096|0
+oneruler_sk.oneruler_slovak:niah_none_8192|0
+oneruler_sk.oneruler_slovak:niah_none_16384|0
+oneruler_sk.oneruler_slovak:niah_single_4096|0
+oneruler_sk.oneruler_slovak:niah_single_8192|0
+oneruler_sk.oneruler_slovak:niah_single_16384|0
+slovaksum.prompt2_summarization_sk|0
+slovaksum.prompt3_summarization_sk|0
+slovaksum.prompt1_summarization_sk|0
+smesum.prompt2_summarization_sk|0
+smesum.prompt3_summarization_sk|0
+smesum.prompt1_summarization_sk|0
+slovak_trivia.prompt2_mqa_single_correct_sk|0
+slovak_trivia.prompt3_mqa_single_correct_sk|0
+slovak_trivia.prompt1_mqa_single_correct_sk|0
+analogies_sk.prompt1_analogy_sk|0
+analogies_sk.prompt2_analogy_sk|0
+analogies_sk.prompt3_analogy_sk|0
+driving_test.prompt3_mqa_single_correct_sk|0
+driving_test.prompt1_mqa_single_correct_sk|0
+driving_test.prompt2_mqa_single_correct_sk|0
+e_tests.prompt3_mqa_single_correct_sk:biologia|0
+e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt3_mqa_single_correct_sk:informatika|0
+e_tests.prompt3_mqa_single_correct_sk:geografia|0
+e_tests.prompt3_mqa_single_correct_sk:matematika|0
+e_tests.prompt3_mqa_single_correct_sk:chemia|0
+e_tests.prompt3_mqa_single_correct_sk:dejepis|0
+e_tests.prompt3_mqa_single_correct_sk:fyzika|0
+e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt1_mqa_single_correct_sk:biologia|0
+e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt1_mqa_single_correct_sk:informatika|0
+e_tests.prompt1_mqa_single_correct_sk:geografia|0
+e_tests.prompt1_mqa_single_correct_sk:matematika|0
+e_tests.prompt1_mqa_single_correct_sk:chemia|0
+e_tests.prompt1_mqa_single_correct_sk:dejepis|0
+e_tests.prompt1_mqa_single_correct_sk:fyzika|0
+e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt2_mqa_single_correct_sk:biologia|0
+e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt2_mqa_single_correct_sk:informatika|0
+e_tests.prompt2_mqa_single_correct_sk:geografia|0
+e_tests.prompt2_mqa_single_correct_sk:matematika|0
+e_tests.prompt2_mqa_single_correct_sk:chemia|0
+e_tests.prompt2_mqa_single_correct_sk:dejepis|0
+e_tests.prompt2_mqa_single_correct_sk:fyzika|0
+e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
+e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
+e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
+slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
+slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
+slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
+e_tests.prompt1_qa_sk:biologia|0
+e_tests.prompt1_qa_sk:obcianska-nauka|0
+e_tests.prompt1_qa_sk:informatika|0
+e_tests.prompt1_qa_sk:geografia|0
+e_tests.prompt1_qa_sk:chemia|0
+e_tests.prompt1_qa_sk:dejepis|0
+e_tests.prompt1_qa_sk:fyzika|0
+e_tests.prompt1_qa_sk:slovensky-jazyk|0
+e_tests.prompt3_qa_sk:biologia|0
+e_tests.prompt3_qa_sk:obcianska-nauka|0
+e_tests.prompt3_qa_sk:informatika|0
+e_tests.prompt3_qa_sk:geografia|0
+e_tests.prompt3_qa_sk:chemia|0
+e_tests.prompt3_qa_sk:dejepis|0
+e_tests.prompt3_qa_sk:fyzika|0
+e_tests.prompt3_qa_sk:slovensky-jazyk|0
+e_tests.prompt2_qa_sk:biologia|0
+e_tests.prompt2_qa_sk:obcianska-nauka|0
+e_tests.prompt2_qa_sk:informatika|0
+e_tests.prompt2_qa_sk:geografia|0
+e_tests.prompt2_qa_sk:chemia|0
+e_tests.prompt2_qa_sk:dejepis|0
+e_tests.prompt2_qa_sk:fyzika|0
+e_tests.prompt2_qa_sk:slovensky-jazyk|0
+slovak_tests.prompt1_qa_sk:monitor9|0
+slovak_tests.prompt1_qa_sk:maturita|0
+slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
+slovak_tests.prompt3_qa_sk:monitor9|0
+slovak_tests.prompt3_qa_sk:maturita|0
+slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
+slovak_tests.prompt2_qa_sk:monitor9|0
+slovak_tests.prompt2_qa_sk:maturita|0
+slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
+slohry.prompt2_declension_sk:declension_high|0
+slohry.prompt2_declension_sk:declension_mid|0
+slohry.prompt2_declension_sk:declension_low|0
+slohry.prompt1_declension_sk:declension_high|0
+slohry.prompt1_declension_sk:declension_mid|0
+slohry.prompt1_declension_sk:declension_low|0
+slohry.prompt3_declension_sk:declension_high|0
+slohry.prompt3_declension_sk:declension_mid|0
+slohry.prompt3_declension_sk:declension_low|0
+slohry.prompt3_gradation_sk:gradation_high|0
+slohry.prompt3_gradation_sk:gradation_mid|0
+slohry.prompt3_gradation_sk:gradation_low|0
+slohry.prompt1_gradation_sk:gradation_high|0
+slohry.prompt1_gradation_sk:gradation_mid|0
+slohry.prompt1_gradation_sk:gradation_low|0
+slohry.prompt2_gradation_sk:gradation_high|0
+slohry.prompt2_gradation_sk:gradation_mid|0
+slohry.prompt2_gradation_sk:gradation_low|0
+slohry.prompt1_identity_form_sk:identity_form_high|0
+slohry.prompt1_identity_form_sk:identity_form_mid|0
+slohry.prompt1_identity_form_sk:identity_form_low|0
+slohry.prompt2_identity_form_sk:identity_form_high|0
+slohry.prompt2_identity_form_sk:identity_form_mid|0
+slohry.prompt2_identity_form_sk:identity_form_low|0
+slohry.prompt3_identity_form_sk:identity_form_high|0
+slohry.prompt3_identity_form_sk:identity_form_mid|0
+slohry.prompt3_identity_form_sk:identity_form_low|0
+slohry.prompt2_inflection_sk:inflection_high|0
+slohry.prompt2_inflection_sk:inflection_mid|0
+slohry.prompt2_inflection_sk:inflection_low|0
+slohry.prompt1_inflection_sk:inflection_high|0
+slohry.prompt1_inflection_sk:inflection_mid|0
+slohry.prompt1_inflection_sk:inflection_low|0
+slohry.prompt3_inflection_sk:inflection_high|0
+slohry.prompt3_inflection_sk:inflection_mid|0
+slohry.prompt3_inflection_sk:inflection_low|0
+scio.prompt2_mqa_single_correct_sk:aj|0
+scio.prompt2_mqa_single_correct_sk:mat|0
+scio.prompt2_mqa_single_correct_sk:bio|0
+scio.prompt2_mqa_single_correct_sk:che|0
+scio.prompt2_mqa_single_correct_sk:vsp|0
+scio.prompt3_mqa_single_correct_sk:aj|0
+scio.prompt3_mqa_single_correct_sk:mat|0
+scio.prompt3_mqa_single_correct_sk:bio|0
+scio.prompt3_mqa_single_correct_sk:che|0
+scio.prompt3_mqa_single_correct_sk:vsp|0
+scio.prompt1_mqa_single_correct_sk:aj|0
+scio.prompt1_mqa_single_correct_sk:mat|0
+scio.prompt1_mqa_single_correct_sk:bio|0
+scio.prompt1_mqa_single_correct_sk:che|0
+scio.prompt1_mqa_single_correct_sk:vsp|0
+nli_sk.prompt1_nli_3_classes_sk|0
+nli_sk.prompt2_nli_3_classes_sk|0
+nli_sk.prompt3_nli_3_classes_sk|0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/task_configs.json b/aya_300k_lr_experiments/benchmarks/skbench/configs/task_configs.json
new file mode 100644
index 0000000..a7c580e
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/task_configs.json
@@ -0,0 +1,13475 @@
+{
+ "mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:miscellaneous",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "financial_exams.prompt2_completion_sk|0": {
+ "name": "financial_exams.prompt2_completion_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/slovak-financial-exam",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:machine_learning|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:machine_learning",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "belebele.prompt3_mqa_single_correct_sk|0": {
+ "name": "belebele.prompt3_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "facebook/belebele",
+ "hf_subset": "slk_Latn",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_law|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:professional_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:professional_accounting",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_us_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_us_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_law|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:professional_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_qa_sk:question-answering|0": {
+ "name": "sklep.prompt1_qa_sk:question-answering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "question-answering",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_statistics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:professional_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "slovakcopa.prompt3_copa_sk|0": {
+ "name": "slovakcopa.prompt3_copa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "NaiveNeuron/SlovakCOPA",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "5c61e4cd9343f9b4fab623bdc50bb895cc170771",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "validation",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "movie_review.prompt2_sentiment_binary_sk|0": {
+ "name": "movie_review.prompt2_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "mteb/slovak_movie_review_sentiment",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_qa_sk:question-answering|0": {
+ "name": "sklep.prompt2_qa_sk:question-answering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "question-answering",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_biology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:nutrition|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:nutrition",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "arc.prompt3_mqa_single_correct_sk|0": {
+ "name": "arc.prompt3_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "jon-tow/okapi_arc_challenge",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": [
+ "solve"
+ ],
+ "scorer": "score",
+ "sample_fields": "record_to_sample",
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:econometrics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:econometrics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:prehistory|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:prehistory",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "toxic_sk.prompt3_toxicity_sk|0": {
+ "name": "toxic_sk.prompt3_toxicity_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/toxic-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "belebele.prompt1_mqa_single_correct_sk|0": {
+ "name": "belebele.prompt1_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "facebook/belebele",
+ "hf_subset": "slk_Latn",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "arc.prompt2_mqa_single_correct_sk|0": {
+ "name": "arc.prompt2_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "jon-tow/okapi_arc_challenge",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": [
+ "solve"
+ ],
+ "scorer": "score",
+ "sample_fields": "record_to_sample",
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "multi_wiki_qa.prompt2_qa_sk|0": {
+ "name": "multi_wiki_qa.prompt2_qa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/multi-wiki-qa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "9bdb3350487e399bd1cd873e1395290d636db5ce",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:management|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:management",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:abstract_algebra",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_nli_3_classes_sk:nli|0": {
+ "name": "sklep.prompt3_nli_3_classes_sk:nli",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "nli",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_medicine|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:business_ethics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:business_ethics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:international_law|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:international_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:international_law|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:international_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "truthfulqa.prompt3_mqa_single_correct_sk:mc1|0": {
+ "name": "truthfulqa.prompt3_mqa_single_correct_sk:mc1",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/m_truthfulqa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "f0445d470f1925882b990f5f247fdcf288972f60",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "val"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:prehistory|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:prehistory",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:nutrition|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:nutrition",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:conceptual_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_medicine|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_medicine|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_nli_binary_sk:rte|0": {
+ "name": "sklep.prompt2_nli_binary_sk:rte",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "rte",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_hate_speech_sk:hate-speech|0": {
+ "name": "sklep.prompt2_hate_speech_sk:hate-speech",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "hate-speech",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:human_sexuality",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_us_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:marketing|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:marketing",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:logical_fallacies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:conceptual_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:virology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:virology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:medical_genetics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:moral_scenarios",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:world_religions|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:world_religions",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt3_sentiment_binary_sk|0": {
+ "name": "senti_sk.prompt3_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_nli_3_classes_sk:nli|0": {
+ "name": "sklep.prompt1_nli_3_classes_sk:nli",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "nli",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:moral_scenarios",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:business_ethics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:business_ethics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:public_relations|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:public_relations",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_statistics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt2_sentiment_binary_sk|0": {
+ "name": "senti_sk.prompt2_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:medical_genetics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "belebele.prompt2_mqa_single_correct_sk|0": {
+ "name": "belebele.prompt2_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "facebook/belebele",
+ "hf_subset": "slk_Latn",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:virology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:virology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_nli_3_classes_sk:nli|0": {
+ "name": "sklep.prompt2_nli_3_classes_sk:nli",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "nli",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_european_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0": {
+ "name": "sklep.prompt3_sentiment_binary_sk:sentiment-analysis",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "sentiment-analysis",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:conceptual_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:moral_disputes",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:security_studies|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:security_studies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "skj1.prompt1_skj1_sk|0": {
+ "name": "skj1.prompt1_skj1_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/SKJ1",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "e615c60b415389b629703f82d3a79810dbde6734",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_physics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_hate_speech_sk:hate-speech|0": {
+ "name": "sklep.prompt3_hate_speech_sk:hate-speech",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "hate-speech",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:virology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:virology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:security_studies|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:security_studies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "financial_exams.prompt3_completion_sk|0": {
+ "name": "financial_exams.prompt3_completion_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/slovak-financial-exam",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:jurisprudence",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt3_sentiment_3_class_sk|0": {
+ "name": "senti_sk.prompt3_sentiment_3_class_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:electrical_engineering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_qa_sk:question-answering|0": {
+ "name": "sklep.prompt3_qa_sk:question-answering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "question-answering",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:professional_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:global_facts|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:global_facts",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:computer_security|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:computer_security",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:computer_security|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:computer_security",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_physics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "globalmmlu-lite.prompt2_mqa_single_correct_sk|0": {
+ "name": "globalmmlu-lite.prompt2_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "CohereLabs/Global-MMLU-Lite",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "600af1bff2d447c7fc3aee22f11f607560d5fcb7",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": "1"
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_world_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:anatomy|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:anatomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:security_studies|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:security_studies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "movie_review.prompt3_sentiment_binary_sk|0": {
+ "name": "movie_review.prompt3_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "mteb/slovak_movie_review_sentiment",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:astronomy|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:astronomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:medical_genetics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:abstract_algebra",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:electrical_engineering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_world_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:philosophy|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:philosophy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:human_sexuality",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:abstract_algebra",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:world_religions|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:world_religions",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "toxic_sk.prompt1_toxicity_sk|0": {
+ "name": "toxic_sk.prompt1_toxicity_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/toxic-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt1_sentiment_3_class_sk|0": {
+ "name": "senti_sk.prompt1_sentiment_3_class_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_biology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:public_relations|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:public_relations",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:formal_logic|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:formal_logic",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_geography",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:prehistory|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:prehistory",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:econometrics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:econometrics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "arc.prompt1_mqa_single_correct_sk|0": {
+ "name": "arc.prompt1_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "jon-tow/okapi_arc_challenge",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": [
+ "solve"
+ ],
+ "scorer": "score",
+ "sample_fields": "record_to_sample",
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:business_ethics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:business_ethics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:jurisprudence",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "truthfulqa.prompt2_mqa_single_correct_sk:mc1|0": {
+ "name": "truthfulqa.prompt2_mqa_single_correct_sk:mc1",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/m_truthfulqa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "f0445d470f1925882b990f5f247fdcf288972f60",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "val"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:miscellaneous",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:astronomy|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:astronomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_european_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt2_sentiment_3_class_sk|0": {
+ "name": "senti_sk.prompt2_sentiment_3_class_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_european_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:formal_logic|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:formal_logic",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0": {
+ "name": "sklep.prompt2_sentiment_binary_sk:sentiment-analysis",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "sentiment-analysis",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0": {
+ "name": "sklep.prompt1_sentiment_binary_sk:sentiment-analysis",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "sentiment-analysis",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "skj1.prompt2_skj1_sk|0": {
+ "name": "skj1.prompt2_skj1_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/SKJ1",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "e615c60b415389b629703f82d3a79810dbde6734",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:international_law|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:international_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_nli_binary_sk:rte|0": {
+ "name": "sklep.prompt1_nli_binary_sk:rte",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "rte",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:human_aging|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:human_aging",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:management|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:management",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:professional_accounting",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "truthfulqa.prompt1_mqa_single_correct_sk:mc1|0": {
+ "name": "truthfulqa.prompt1_mqa_single_correct_sk:mc1",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/m_truthfulqa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "f0445d470f1925882b990f5f247fdcf288972f60",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "val"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt1_sentiment_binary_sk|0": {
+ "name": "senti_sk.prompt1_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:astronomy|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:astronomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:human_aging|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:human_aging",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:formal_logic|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:formal_logic",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:sociology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:sociology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:econometrics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:econometrics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:professional_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "globalmmlu-lite.prompt3_mqa_single_correct_sk|0": {
+ "name": "globalmmlu-lite.prompt3_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "CohereLabs/Global-MMLU-Lite",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "600af1bff2d447c7fc3aee22f11f607560d5fcb7",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": "1"
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:human_sexuality",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:computer_security|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:computer_security",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "toxic_sk.prompt2_toxicity_sk|0": {
+ "name": "toxic_sk.prompt2_toxicity_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/toxic-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "multi_wiki_qa.prompt3_qa_sk|0": {
+ "name": "multi_wiki_qa.prompt3_qa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/multi-wiki-qa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "9bdb3350487e399bd1cd873e1395290d636db5ce",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "financial_exams.prompt1_completion_sk|0": {
+ "name": "financial_exams.prompt1_completion_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/slovak-financial-exam",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:philosophy|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:philosophy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:moral_disputes",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:marketing|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:marketing",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "globalmmlu-lite.prompt1_mqa_single_correct_sk|0": {
+ "name": "globalmmlu-lite.prompt1_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "CohereLabs/Global-MMLU-Lite",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "600af1bff2d447c7fc3aee22f11f607560d5fcb7",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": "1"
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_law|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:professional_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:machine_learning|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:machine_learning",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:management|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:management",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:sociology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:sociology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "skj1.prompt3_skj1_sk|0": {
+ "name": "skj1.prompt3_skj1_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/SKJ1",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "e615c60b415389b629703f82d3a79810dbde6734",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "movie_review.prompt1_sentiment_binary_sk|0": {
+ "name": "movie_review.prompt1_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "mteb/slovak_movie_review_sentiment",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:anatomy|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:anatomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:philosophy|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:philosophy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:human_aging|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:human_aging",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:professional_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:professional_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:electrical_engineering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:logical_fallacies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_hate_speech_sk:hate-speech|0": {
+ "name": "sklep.prompt1_hate_speech_sk:hate-speech",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "hate-speech",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:miscellaneous",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:world_religions|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:world_religions",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_world_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_biology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:global_facts|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:global_facts",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:moral_disputes",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_geography",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:moral_scenarios",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_physics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:professional_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:logical_fallacies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_nli_binary_sk:rte|0": {
+ "name": "sklep.prompt3_nli_binary_sk:rte",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "rte",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "slovakcopa.prompt2_copa_sk|0": {
+ "name": "slovakcopa.prompt2_copa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "NaiveNeuron/SlovakCOPA",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "5c61e4cd9343f9b4fab623bdc50bb895cc170771",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "validation",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:jurisprudence",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "multi_wiki_qa.prompt1_qa_sk|0": {
+ "name": "multi_wiki_qa.prompt1_qa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/multi-wiki-qa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "9bdb3350487e399bd1cd873e1395290d636db5ce",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:global_facts|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:global_facts",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:sociology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:sociology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:public_relations|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:public_relations",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_statistics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:anatomy|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:anatomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:marketing|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:marketing",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:professional_accounting",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_geography",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "slovakcopa.prompt1_copa_sk|0": {
+ "name": "slovakcopa.prompt1_copa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "NaiveNeuron/SlovakCOPA",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "5c61e4cd9343f9b4fab623bdc50bb895cc170771",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "validation",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:nutrition|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:nutrition",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:machine_learning|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:machine_learning",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ }
+}
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt
new file mode 100644
index 0000000..45b85f6
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks.txt
@@ -0,0 +1,222 @@
+arc.prompt3_mqa_single_correct_sk|0
+arc.prompt1_mqa_single_correct_sk|0
+arc.prompt2_mqa_single_correct_sk|0
+belebele.prompt3_mqa_single_correct_sk|0
+belebele.prompt1_mqa_single_correct_sk|0
+belebele.prompt2_mqa_single_correct_sk|0
+financial_exams.prompt1_completion_sk|0
+financial_exams.prompt3_completion_sk|0
+financial_exams.prompt2_completion_sk|0
+mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt3_mqa_single_correct_sk:anatomy|0
+mmlu.prompt1_mqa_single_correct_sk:anatomy|0
+mmlu.prompt2_mqa_single_correct_sk:anatomy|0
+mmlu.prompt3_mqa_single_correct_sk:astronomy|0
+mmlu.prompt1_mqa_single_correct_sk:astronomy|0
+mmlu.prompt2_mqa_single_correct_sk:astronomy|0
+mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt3_mqa_single_correct_sk:college_biology|0
+mmlu.prompt1_mqa_single_correct_sk:college_biology|0
+mmlu.prompt2_mqa_single_correct_sk:college_biology|0
+mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:college_physics|0
+mmlu.prompt1_mqa_single_correct_sk:college_physics|0
+mmlu.prompt2_mqa_single_correct_sk:college_physics|0
+mmlu.prompt3_mqa_single_correct_sk:computer_security|0
+mmlu.prompt1_mqa_single_correct_sk:computer_security|0
+mmlu.prompt2_mqa_single_correct_sk:computer_security|0
+mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt3_mqa_single_correct_sk:econometrics|0
+mmlu.prompt1_mqa_single_correct_sk:econometrics|0
+mmlu.prompt2_mqa_single_correct_sk:econometrics|0
+mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt3_mqa_single_correct_sk:global_facts|0
+mmlu.prompt1_mqa_single_correct_sk:global_facts|0
+mmlu.prompt2_mqa_single_correct_sk:global_facts|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt3_mqa_single_correct_sk:human_aging|0
+mmlu.prompt1_mqa_single_correct_sk:human_aging|0
+mmlu.prompt2_mqa_single_correct_sk:human_aging|0
+mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt3_mqa_single_correct_sk:international_law|0
+mmlu.prompt1_mqa_single_correct_sk:international_law|0
+mmlu.prompt2_mqa_single_correct_sk:international_law|0
+mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt3_mqa_single_correct_sk:management|0
+mmlu.prompt1_mqa_single_correct_sk:management|0
+mmlu.prompt2_mqa_single_correct_sk:management|0
+mmlu.prompt3_mqa_single_correct_sk:marketing|0
+mmlu.prompt1_mqa_single_correct_sk:marketing|0
+mmlu.prompt2_mqa_single_correct_sk:marketing|0
+mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt3_mqa_single_correct_sk:nutrition|0
+mmlu.prompt1_mqa_single_correct_sk:nutrition|0
+mmlu.prompt2_mqa_single_correct_sk:nutrition|0
+mmlu.prompt3_mqa_single_correct_sk:philosophy|0
+mmlu.prompt1_mqa_single_correct_sk:philosophy|0
+mmlu.prompt2_mqa_single_correct_sk:philosophy|0
+mmlu.prompt3_mqa_single_correct_sk:prehistory|0
+mmlu.prompt1_mqa_single_correct_sk:prehistory|0
+mmlu.prompt2_mqa_single_correct_sk:prehistory|0
+mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt3_mqa_single_correct_sk:professional_law|0
+mmlu.prompt1_mqa_single_correct_sk:professional_law|0
+mmlu.prompt2_mqa_single_correct_sk:professional_law|0
+mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:public_relations|0
+mmlu.prompt1_mqa_single_correct_sk:public_relations|0
+mmlu.prompt2_mqa_single_correct_sk:public_relations|0
+mmlu.prompt3_mqa_single_correct_sk:security_studies|0
+mmlu.prompt1_mqa_single_correct_sk:security_studies|0
+mmlu.prompt2_mqa_single_correct_sk:security_studies|0
+mmlu.prompt3_mqa_single_correct_sk:sociology|0
+mmlu.prompt1_mqa_single_correct_sk:sociology|0
+mmlu.prompt2_mqa_single_correct_sk:sociology|0
+mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt3_mqa_single_correct_sk:virology|0
+mmlu.prompt1_mqa_single_correct_sk:virology|0
+mmlu.prompt2_mqa_single_correct_sk:virology|0
+mmlu.prompt3_mqa_single_correct_sk:world_religions|0
+mmlu.prompt1_mqa_single_correct_sk:world_religions|0
+mmlu.prompt2_mqa_single_correct_sk:world_religions|0
+multi_wiki_qa.prompt3_qa_sk|0
+multi_wiki_qa.prompt2_qa_sk|0
+multi_wiki_qa.prompt1_qa_sk|0
+senti_sk.prompt2_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_binary_sk|0
+senti_sk.prompt1_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_3_class_sk|0
+senti_sk.prompt1_sentiment_3_class_sk|0
+senti_sk.prompt2_sentiment_3_class_sk|0
+skj1.prompt3_skj1_sk|0
+skj1.prompt1_skj1_sk|0
+skj1.prompt2_skj1_sk|0
+sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_hate_speech_sk:hate-speech|0
+sklep.prompt3_hate_speech_sk:hate-speech|0
+sklep.prompt2_hate_speech_sk:hate-speech|0
+sklep.prompt1_nli_3_classes_sk:nli|0
+sklep.prompt2_nli_3_classes_sk:nli|0
+sklep.prompt3_nli_3_classes_sk:nli|0
+sklep.prompt1_nli_binary_sk:rte|0
+sklep.prompt3_nli_binary_sk:rte|0
+sklep.prompt2_nli_binary_sk:rte|0
+sklep.prompt3_qa_sk:question-answering|0
+sklep.prompt2_qa_sk:question-answering|0
+sklep.prompt1_qa_sk:question-answering|0
+toxic_sk.prompt2_toxicity_sk|0
+toxic_sk.prompt3_toxicity_sk|0
+toxic_sk.prompt1_toxicity_sk|0
+truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
+slovakcopa.prompt3_copa_sk|0
+slovakcopa.prompt1_copa_sk|0
+slovakcopa.prompt2_copa_sk|0
+movie_review.prompt3_sentiment_binary_sk|0
+movie_review.prompt2_sentiment_binary_sk|0
+movie_review.prompt1_sentiment_binary_sk|0
+globalmmlu-lite.prompt3_mqa_single_correct_sk|0
+globalmmlu-lite.prompt1_mqa_single_correct_sk|0
+globalmmlu-lite.prompt2_mqa_single_correct_sk|0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero.txt b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero.txt
new file mode 100644
index 0000000..154f511
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero.txt
@@ -0,0 +1,412 @@
+arc.prompt3_mqa_single_correct_sk|0
+arc.prompt1_mqa_single_correct_sk|0
+arc.prompt2_mqa_single_correct_sk|0
+belebele.prompt3_mqa_single_correct_sk|0
+belebele.prompt1_mqa_single_correct_sk|0
+belebele.prompt2_mqa_single_correct_sk|0
+chiby.prompt2_chiby_sk|0
+chiby.prompt1_chiby_sk|0
+chiby.prompt3_chiby_sk|0
+financial_exams.prompt1_completion_sk|0
+financial_exams.prompt3_completion_sk|0
+financial_exams.prompt2_completion_sk|0
+ifeval_sk.ifeval|0
+mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt3_mqa_single_correct_sk:anatomy|0
+mmlu.prompt1_mqa_single_correct_sk:anatomy|0
+mmlu.prompt2_mqa_single_correct_sk:anatomy|0
+mmlu.prompt3_mqa_single_correct_sk:astronomy|0
+mmlu.prompt1_mqa_single_correct_sk:astronomy|0
+mmlu.prompt2_mqa_single_correct_sk:astronomy|0
+mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt3_mqa_single_correct_sk:college_biology|0
+mmlu.prompt1_mqa_single_correct_sk:college_biology|0
+mmlu.prompt2_mqa_single_correct_sk:college_biology|0
+mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:college_physics|0
+mmlu.prompt1_mqa_single_correct_sk:college_physics|0
+mmlu.prompt2_mqa_single_correct_sk:college_physics|0
+mmlu.prompt3_mqa_single_correct_sk:computer_security|0
+mmlu.prompt1_mqa_single_correct_sk:computer_security|0
+mmlu.prompt2_mqa_single_correct_sk:computer_security|0
+mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt3_mqa_single_correct_sk:econometrics|0
+mmlu.prompt1_mqa_single_correct_sk:econometrics|0
+mmlu.prompt2_mqa_single_correct_sk:econometrics|0
+mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt3_mqa_single_correct_sk:global_facts|0
+mmlu.prompt1_mqa_single_correct_sk:global_facts|0
+mmlu.prompt2_mqa_single_correct_sk:global_facts|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt3_mqa_single_correct_sk:human_aging|0
+mmlu.prompt1_mqa_single_correct_sk:human_aging|0
+mmlu.prompt2_mqa_single_correct_sk:human_aging|0
+mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt3_mqa_single_correct_sk:international_law|0
+mmlu.prompt1_mqa_single_correct_sk:international_law|0
+mmlu.prompt2_mqa_single_correct_sk:international_law|0
+mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt3_mqa_single_correct_sk:management|0
+mmlu.prompt1_mqa_single_correct_sk:management|0
+mmlu.prompt2_mqa_single_correct_sk:management|0
+mmlu.prompt3_mqa_single_correct_sk:marketing|0
+mmlu.prompt1_mqa_single_correct_sk:marketing|0
+mmlu.prompt2_mqa_single_correct_sk:marketing|0
+mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt3_mqa_single_correct_sk:nutrition|0
+mmlu.prompt1_mqa_single_correct_sk:nutrition|0
+mmlu.prompt2_mqa_single_correct_sk:nutrition|0
+mmlu.prompt3_mqa_single_correct_sk:philosophy|0
+mmlu.prompt1_mqa_single_correct_sk:philosophy|0
+mmlu.prompt2_mqa_single_correct_sk:philosophy|0
+mmlu.prompt3_mqa_single_correct_sk:prehistory|0
+mmlu.prompt1_mqa_single_correct_sk:prehistory|0
+mmlu.prompt2_mqa_single_correct_sk:prehistory|0
+mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt3_mqa_single_correct_sk:professional_law|0
+mmlu.prompt1_mqa_single_correct_sk:professional_law|0
+mmlu.prompt2_mqa_single_correct_sk:professional_law|0
+mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:public_relations|0
+mmlu.prompt1_mqa_single_correct_sk:public_relations|0
+mmlu.prompt2_mqa_single_correct_sk:public_relations|0
+mmlu.prompt3_mqa_single_correct_sk:security_studies|0
+mmlu.prompt1_mqa_single_correct_sk:security_studies|0
+mmlu.prompt2_mqa_single_correct_sk:security_studies|0
+mmlu.prompt3_mqa_single_correct_sk:sociology|0
+mmlu.prompt1_mqa_single_correct_sk:sociology|0
+mmlu.prompt2_mqa_single_correct_sk:sociology|0
+mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt3_mqa_single_correct_sk:virology|0
+mmlu.prompt1_mqa_single_correct_sk:virology|0
+mmlu.prompt2_mqa_single_correct_sk:virology|0
+mmlu.prompt3_mqa_single_correct_sk:world_religions|0
+mmlu.prompt1_mqa_single_correct_sk:world_religions|0
+mmlu.prompt2_mqa_single_correct_sk:world_religions|0
+multi_wiki_qa.prompt3_qa_sk|0
+multi_wiki_qa.prompt2_qa_sk|0
+multi_wiki_qa.prompt1_qa_sk|0
+nli_sk.prompt1_nli_3_classes_sk|0
+nli_sk.prompt2_nli_3_classes_sk|0
+nli_sk.prompt3_nli_3_classes_sk|0
+oneruler_sk.oneruler_slovak:cwe_4096|0
+oneruler_sk.oneruler_slovak:cwe_8192|0
+oneruler_sk.oneruler_slovak:cwe_16384|0
+oneruler_sk.oneruler_slovak:niah_multikey_4096|0
+oneruler_sk.oneruler_slovak:niah_multikey_8192|0
+oneruler_sk.oneruler_slovak:niah_multikey_16384|0
+oneruler_sk.oneruler_slovak:niah_multiquery_4096|0
+oneruler_sk.oneruler_slovak:niah_multiquery_8192|0
+oneruler_sk.oneruler_slovak:niah_multiquery_16384|0
+oneruler_sk.oneruler_slovak:niah_multivalue_4096|0
+oneruler_sk.oneruler_slovak:niah_multivalue_8192|0
+oneruler_sk.oneruler_slovak:niah_multivalue_16384|0
+oneruler_sk.oneruler_slovak:niah_none_4096|0
+oneruler_sk.oneruler_slovak:niah_none_8192|0
+oneruler_sk.oneruler_slovak:niah_none_16384|0
+oneruler_sk.oneruler_slovak:niah_single_4096|0
+oneruler_sk.oneruler_slovak:niah_single_8192|0
+oneruler_sk.oneruler_slovak:niah_single_16384|0
+senti_sk.prompt2_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_binary_sk|0
+senti_sk.prompt1_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_3_class_sk|0
+senti_sk.prompt1_sentiment_3_class_sk|0
+senti_sk.prompt2_sentiment_3_class_sk|0
+skj1.prompt3_skj1_sk|0
+skj1.prompt1_skj1_sk|0
+skj1.prompt2_skj1_sk|0
+sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_hate_speech_sk:hate-speech|0
+sklep.prompt3_hate_speech_sk:hate-speech|0
+sklep.prompt2_hate_speech_sk:hate-speech|0
+sklep.prompt1_nli_3_classes_sk:nli|0
+sklep.prompt2_nli_3_classes_sk:nli|0
+sklep.prompt3_nli_3_classes_sk:nli|0
+sklep.prompt1_nli_binary_sk:rte|0
+sklep.prompt3_nli_binary_sk:rte|0
+sklep.prompt2_nli_binary_sk:rte|0
+sklep.prompt3_qa_sk:question-answering|0
+sklep.prompt2_qa_sk:question-answering|0
+sklep.prompt1_qa_sk:question-answering|0
+toxic_sk.prompt2_toxicity_sk|0
+toxic_sk.prompt3_toxicity_sk|0
+toxic_sk.prompt1_toxicity_sk|0
+truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
+slovakcopa.prompt3_copa_sk|0
+slovakcopa.prompt1_copa_sk|0
+slovakcopa.prompt2_copa_sk|0
+slovaksum.prompt2_summarization_sk|0
+slovaksum.prompt3_summarization_sk|0
+slovaksum.prompt1_summarization_sk|0
+smesum.prompt2_summarization_sk|0
+smesum.prompt3_summarization_sk|0
+smesum.prompt1_summarization_sk|0
+movie_review.prompt3_sentiment_binary_sk|0
+movie_review.prompt2_sentiment_binary_sk|0
+movie_review.prompt1_sentiment_binary_sk|0
+slovak_trivia.prompt2_mqa_single_correct_sk|0
+slovak_trivia.prompt3_mqa_single_correct_sk|0
+slovak_trivia.prompt1_mqa_single_correct_sk|0
+analogies_sk.prompt1_analogy_sk|0
+analogies_sk.prompt2_analogy_sk|0
+analogies_sk.prompt3_analogy_sk|0
+driving_test.prompt3_mqa_single_correct_sk|0
+driving_test.prompt1_mqa_single_correct_sk|0
+driving_test.prompt2_mqa_single_correct_sk|0
+e_tests.prompt3_mqa_single_correct_sk:biologia|0
+e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt3_mqa_single_correct_sk:informatika|0
+e_tests.prompt3_mqa_single_correct_sk:geografia|0
+e_tests.prompt3_mqa_single_correct_sk:matematika|0
+e_tests.prompt3_mqa_single_correct_sk:chemia|0
+e_tests.prompt3_mqa_single_correct_sk:dejepis|0
+e_tests.prompt3_mqa_single_correct_sk:fyzika|0
+e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt1_mqa_single_correct_sk:biologia|0
+e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt1_mqa_single_correct_sk:informatika|0
+e_tests.prompt1_mqa_single_correct_sk:geografia|0
+e_tests.prompt1_mqa_single_correct_sk:matematika|0
+e_tests.prompt1_mqa_single_correct_sk:chemia|0
+e_tests.prompt1_mqa_single_correct_sk:dejepis|0
+e_tests.prompt1_mqa_single_correct_sk:fyzika|0
+e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt2_mqa_single_correct_sk:biologia|0
+e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt2_mqa_single_correct_sk:informatika|0
+e_tests.prompt2_mqa_single_correct_sk:geografia|0
+e_tests.prompt2_mqa_single_correct_sk:matematika|0
+e_tests.prompt2_mqa_single_correct_sk:chemia|0
+e_tests.prompt2_mqa_single_correct_sk:dejepis|0
+e_tests.prompt2_mqa_single_correct_sk:fyzika|0
+e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
+e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
+e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
+slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
+slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
+slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
+e_tests.prompt1_qa_sk:biologia|0
+e_tests.prompt1_qa_sk:obcianska-nauka|0
+e_tests.prompt1_qa_sk:informatika|0
+e_tests.prompt1_qa_sk:geografia|0
+e_tests.prompt1_qa_sk:chemia|0
+e_tests.prompt1_qa_sk:dejepis|0
+e_tests.prompt1_qa_sk:fyzika|0
+e_tests.prompt1_qa_sk:slovensky-jazyk|0
+e_tests.prompt3_qa_sk:biologia|0
+e_tests.prompt3_qa_sk:obcianska-nauka|0
+e_tests.prompt3_qa_sk:informatika|0
+e_tests.prompt3_qa_sk:geografia|0
+e_tests.prompt3_qa_sk:chemia|0
+e_tests.prompt3_qa_sk:dejepis|0
+e_tests.prompt3_qa_sk:fyzika|0
+e_tests.prompt3_qa_sk:slovensky-jazyk|0
+e_tests.prompt2_qa_sk:biologia|0
+e_tests.prompt2_qa_sk:obcianska-nauka|0
+e_tests.prompt2_qa_sk:informatika|0
+e_tests.prompt2_qa_sk:geografia|0
+e_tests.prompt2_qa_sk:chemia|0
+e_tests.prompt2_qa_sk:dejepis|0
+e_tests.prompt2_qa_sk:fyzika|0
+e_tests.prompt2_qa_sk:slovensky-jazyk|0
+slovak_tests.prompt1_qa_sk:monitor9|0
+slovak_tests.prompt1_qa_sk:maturita|0
+slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
+slovak_tests.prompt3_qa_sk:monitor9|0
+slovak_tests.prompt3_qa_sk:maturita|0
+slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
+slovak_tests.prompt2_qa_sk:monitor9|0
+slovak_tests.prompt2_qa_sk:maturita|0
+slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
+slohry.prompt2_declension_sk:declension_high|0
+slohry.prompt2_declension_sk:declension_mid|0
+slohry.prompt2_declension_sk:declension_low|0
+slohry.prompt1_declension_sk:declension_high|0
+slohry.prompt1_declension_sk:declension_mid|0
+slohry.prompt1_declension_sk:declension_low|0
+slohry.prompt3_declension_sk:declension_high|0
+slohry.prompt3_declension_sk:declension_mid|0
+slohry.prompt3_declension_sk:declension_low|0
+slohry.prompt3_gradation_sk:gradation_high|0
+slohry.prompt3_gradation_sk:gradation_mid|0
+slohry.prompt3_gradation_sk:gradation_low|0
+slohry.prompt1_gradation_sk:gradation_high|0
+slohry.prompt1_gradation_sk:gradation_mid|0
+slohry.prompt1_gradation_sk:gradation_low|0
+slohry.prompt2_gradation_sk:gradation_high|0
+slohry.prompt2_gradation_sk:gradation_mid|0
+slohry.prompt2_gradation_sk:gradation_low|0
+slohry.prompt1_identity_form_sk:identity_form_high|0
+slohry.prompt1_identity_form_sk:identity_form_mid|0
+slohry.prompt1_identity_form_sk:identity_form_low|0
+slohry.prompt2_identity_form_sk:identity_form_high|0
+slohry.prompt2_identity_form_sk:identity_form_mid|0
+slohry.prompt2_identity_form_sk:identity_form_low|0
+slohry.prompt3_identity_form_sk:identity_form_high|0
+slohry.prompt3_identity_form_sk:identity_form_mid|0
+slohry.prompt3_identity_form_sk:identity_form_low|0
+slohry.prompt2_inflection_sk:inflection_high|0
+slohry.prompt2_inflection_sk:inflection_mid|0
+slohry.prompt2_inflection_sk:inflection_low|0
+slohry.prompt1_inflection_sk:inflection_high|0
+slohry.prompt1_inflection_sk:inflection_mid|0
+slohry.prompt1_inflection_sk:inflection_low|0
+slohry.prompt3_inflection_sk:inflection_high|0
+slohry.prompt3_inflection_sk:inflection_mid|0
+slohry.prompt3_inflection_sk:inflection_low|0
+scio.prompt2_mqa_single_correct_sk:aj|0
+scio.prompt2_mqa_single_correct_sk:mat|0
+scio.prompt2_mqa_single_correct_sk:bio|0
+scio.prompt2_mqa_single_correct_sk:che|0
+scio.prompt2_mqa_single_correct_sk:vsp|0
+scio.prompt3_mqa_single_correct_sk:aj|0
+scio.prompt3_mqa_single_correct_sk:mat|0
+scio.prompt3_mqa_single_correct_sk:bio|0
+scio.prompt3_mqa_single_correct_sk:che|0
+scio.prompt3_mqa_single_correct_sk:vsp|0
+scio.prompt1_mqa_single_correct_sk:aj|0
+scio.prompt1_mqa_single_correct_sk:mat|0
+scio.prompt1_mqa_single_correct_sk:bio|0
+scio.prompt1_mqa_single_correct_sk:che|0
+scio.prompt1_mqa_single_correct_sk:vsp|0
+globalmmlu-lite.prompt3_mqa_single_correct_sk|0
+globalmmlu-lite.prompt1_mqa_single_correct_sk|0
+globalmmlu-lite.prompt2_mqa_single_correct_sk|0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available.txt b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available.txt
new file mode 100644
index 0000000..81d49a3
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available.txt
@@ -0,0 +1,225 @@
+arc.prompt3_mqa_single_correct_sk|0
+arc.prompt1_mqa_single_correct_sk|0
+arc.prompt2_mqa_single_correct_sk|0
+belebele.prompt3_mqa_single_correct_sk|0
+belebele.prompt1_mqa_single_correct_sk|0
+belebele.prompt2_mqa_single_correct_sk|0
+financial_exams.prompt1_completion_sk|0
+financial_exams.prompt3_completion_sk|0
+financial_exams.prompt2_completion_sk|0
+mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt3_mqa_single_correct_sk:anatomy|0
+mmlu.prompt1_mqa_single_correct_sk:anatomy|0
+mmlu.prompt2_mqa_single_correct_sk:anatomy|0
+mmlu.prompt3_mqa_single_correct_sk:astronomy|0
+mmlu.prompt1_mqa_single_correct_sk:astronomy|0
+mmlu.prompt2_mqa_single_correct_sk:astronomy|0
+mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt3_mqa_single_correct_sk:college_biology|0
+mmlu.prompt1_mqa_single_correct_sk:college_biology|0
+mmlu.prompt2_mqa_single_correct_sk:college_biology|0
+mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:college_physics|0
+mmlu.prompt1_mqa_single_correct_sk:college_physics|0
+mmlu.prompt2_mqa_single_correct_sk:college_physics|0
+mmlu.prompt3_mqa_single_correct_sk:computer_security|0
+mmlu.prompt1_mqa_single_correct_sk:computer_security|0
+mmlu.prompt2_mqa_single_correct_sk:computer_security|0
+mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt3_mqa_single_correct_sk:econometrics|0
+mmlu.prompt1_mqa_single_correct_sk:econometrics|0
+mmlu.prompt2_mqa_single_correct_sk:econometrics|0
+mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt3_mqa_single_correct_sk:global_facts|0
+mmlu.prompt1_mqa_single_correct_sk:global_facts|0
+mmlu.prompt2_mqa_single_correct_sk:global_facts|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt3_mqa_single_correct_sk:human_aging|0
+mmlu.prompt1_mqa_single_correct_sk:human_aging|0
+mmlu.prompt2_mqa_single_correct_sk:human_aging|0
+mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt3_mqa_single_correct_sk:international_law|0
+mmlu.prompt1_mqa_single_correct_sk:international_law|0
+mmlu.prompt2_mqa_single_correct_sk:international_law|0
+mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt3_mqa_single_correct_sk:management|0
+mmlu.prompt1_mqa_single_correct_sk:management|0
+mmlu.prompt2_mqa_single_correct_sk:management|0
+mmlu.prompt3_mqa_single_correct_sk:marketing|0
+mmlu.prompt1_mqa_single_correct_sk:marketing|0
+mmlu.prompt2_mqa_single_correct_sk:marketing|0
+mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt3_mqa_single_correct_sk:nutrition|0
+mmlu.prompt1_mqa_single_correct_sk:nutrition|0
+mmlu.prompt2_mqa_single_correct_sk:nutrition|0
+mmlu.prompt3_mqa_single_correct_sk:philosophy|0
+mmlu.prompt1_mqa_single_correct_sk:philosophy|0
+mmlu.prompt2_mqa_single_correct_sk:philosophy|0
+mmlu.prompt3_mqa_single_correct_sk:prehistory|0
+mmlu.prompt1_mqa_single_correct_sk:prehistory|0
+mmlu.prompt2_mqa_single_correct_sk:prehistory|0
+mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt3_mqa_single_correct_sk:professional_law|0
+mmlu.prompt1_mqa_single_correct_sk:professional_law|0
+mmlu.prompt2_mqa_single_correct_sk:professional_law|0
+mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:public_relations|0
+mmlu.prompt1_mqa_single_correct_sk:public_relations|0
+mmlu.prompt2_mqa_single_correct_sk:public_relations|0
+mmlu.prompt3_mqa_single_correct_sk:security_studies|0
+mmlu.prompt1_mqa_single_correct_sk:security_studies|0
+mmlu.prompt2_mqa_single_correct_sk:security_studies|0
+mmlu.prompt3_mqa_single_correct_sk:sociology|0
+mmlu.prompt1_mqa_single_correct_sk:sociology|0
+mmlu.prompt2_mqa_single_correct_sk:sociology|0
+mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt3_mqa_single_correct_sk:virology|0
+mmlu.prompt1_mqa_single_correct_sk:virology|0
+mmlu.prompt2_mqa_single_correct_sk:virology|0
+mmlu.prompt3_mqa_single_correct_sk:world_religions|0
+mmlu.prompt1_mqa_single_correct_sk:world_religions|0
+mmlu.prompt2_mqa_single_correct_sk:world_religions|0
+multi_wiki_qa.prompt3_qa_sk|0
+multi_wiki_qa.prompt2_qa_sk|0
+multi_wiki_qa.prompt1_qa_sk|0
+nli_sk.prompt1_nli_3_classes_sk|0
+nli_sk.prompt2_nli_3_classes_sk|0
+nli_sk.prompt3_nli_3_classes_sk|0
+senti_sk.prompt2_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_binary_sk|0
+senti_sk.prompt1_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_3_class_sk|0
+senti_sk.prompt1_sentiment_3_class_sk|0
+senti_sk.prompt2_sentiment_3_class_sk|0
+skj1.prompt3_skj1_sk|0
+skj1.prompt1_skj1_sk|0
+skj1.prompt2_skj1_sk|0
+sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_hate_speech_sk:hate-speech|0
+sklep.prompt3_hate_speech_sk:hate-speech|0
+sklep.prompt2_hate_speech_sk:hate-speech|0
+sklep.prompt1_nli_3_classes_sk:nli|0
+sklep.prompt2_nli_3_classes_sk:nli|0
+sklep.prompt3_nli_3_classes_sk:nli|0
+sklep.prompt1_nli_binary_sk:rte|0
+sklep.prompt3_nli_binary_sk:rte|0
+sklep.prompt2_nli_binary_sk:rte|0
+sklep.prompt3_qa_sk:question-answering|0
+sklep.prompt2_qa_sk:question-answering|0
+sklep.prompt1_qa_sk:question-answering|0
+toxic_sk.prompt2_toxicity_sk|0
+toxic_sk.prompt3_toxicity_sk|0
+toxic_sk.prompt1_toxicity_sk|0
+truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
+slovakcopa.prompt3_copa_sk|0
+slovakcopa.prompt1_copa_sk|0
+slovakcopa.prompt2_copa_sk|0
+movie_review.prompt3_sentiment_binary_sk|0
+movie_review.prompt2_sentiment_binary_sk|0
+movie_review.prompt1_sentiment_binary_sk|0
+globalmmlu-lite.prompt3_mqa_single_correct_sk|0
+globalmmlu-lite.prompt1_mqa_single_correct_sk|0
+globalmmlu-lite.prompt2_mqa_single_correct_sk|0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available_without_nli.txt b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available_without_nli.txt
new file mode 100644
index 0000000..45b85f6
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_available_without_nli.txt
@@ -0,0 +1,222 @@
+arc.prompt3_mqa_single_correct_sk|0
+arc.prompt1_mqa_single_correct_sk|0
+arc.prompt2_mqa_single_correct_sk|0
+belebele.prompt3_mqa_single_correct_sk|0
+belebele.prompt1_mqa_single_correct_sk|0
+belebele.prompt2_mqa_single_correct_sk|0
+financial_exams.prompt1_completion_sk|0
+financial_exams.prompt3_completion_sk|0
+financial_exams.prompt2_completion_sk|0
+mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0
+mmlu.prompt3_mqa_single_correct_sk:anatomy|0
+mmlu.prompt1_mqa_single_correct_sk:anatomy|0
+mmlu.prompt2_mqa_single_correct_sk:anatomy|0
+mmlu.prompt3_mqa_single_correct_sk:astronomy|0
+mmlu.prompt1_mqa_single_correct_sk:astronomy|0
+mmlu.prompt2_mqa_single_correct_sk:astronomy|0
+mmlu.prompt3_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt1_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt2_mqa_single_correct_sk:business_ethics|0
+mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0
+mmlu.prompt3_mqa_single_correct_sk:college_biology|0
+mmlu.prompt1_mqa_single_correct_sk:college_biology|0
+mmlu.prompt2_mqa_single_correct_sk:college_biology|0
+mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:college_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:college_physics|0
+mmlu.prompt1_mqa_single_correct_sk:college_physics|0
+mmlu.prompt2_mqa_single_correct_sk:college_physics|0
+mmlu.prompt3_mqa_single_correct_sk:computer_security|0
+mmlu.prompt1_mqa_single_correct_sk:computer_security|0
+mmlu.prompt2_mqa_single_correct_sk:computer_security|0
+mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0
+mmlu.prompt3_mqa_single_correct_sk:econometrics|0
+mmlu.prompt1_mqa_single_correct_sk:econometrics|0
+mmlu.prompt2_mqa_single_correct_sk:econometrics|0
+mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0
+mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt1_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt2_mqa_single_correct_sk:formal_logic|0
+mmlu.prompt3_mqa_single_correct_sk:global_facts|0
+mmlu.prompt1_mqa_single_correct_sk:global_facts|0
+mmlu.prompt2_mqa_single_correct_sk:global_facts|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0
+mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0
+mmlu.prompt3_mqa_single_correct_sk:human_aging|0
+mmlu.prompt1_mqa_single_correct_sk:human_aging|0
+mmlu.prompt2_mqa_single_correct_sk:human_aging|0
+mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0
+mmlu.prompt3_mqa_single_correct_sk:international_law|0
+mmlu.prompt1_mqa_single_correct_sk:international_law|0
+mmlu.prompt2_mqa_single_correct_sk:international_law|0
+mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0
+mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0
+mmlu.prompt3_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt1_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt2_mqa_single_correct_sk:machine_learning|0
+mmlu.prompt3_mqa_single_correct_sk:management|0
+mmlu.prompt1_mqa_single_correct_sk:management|0
+mmlu.prompt2_mqa_single_correct_sk:management|0
+mmlu.prompt3_mqa_single_correct_sk:marketing|0
+mmlu.prompt1_mqa_single_correct_sk:marketing|0
+mmlu.prompt2_mqa_single_correct_sk:marketing|0
+mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0
+mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0
+mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0
+mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0
+mmlu.prompt3_mqa_single_correct_sk:nutrition|0
+mmlu.prompt1_mqa_single_correct_sk:nutrition|0
+mmlu.prompt2_mqa_single_correct_sk:nutrition|0
+mmlu.prompt3_mqa_single_correct_sk:philosophy|0
+mmlu.prompt1_mqa_single_correct_sk:philosophy|0
+mmlu.prompt2_mqa_single_correct_sk:philosophy|0
+mmlu.prompt3_mqa_single_correct_sk:prehistory|0
+mmlu.prompt1_mqa_single_correct_sk:prehistory|0
+mmlu.prompt2_mqa_single_correct_sk:prehistory|0
+mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0
+mmlu.prompt3_mqa_single_correct_sk:professional_law|0
+mmlu.prompt1_mqa_single_correct_sk:professional_law|0
+mmlu.prompt2_mqa_single_correct_sk:professional_law|0
+mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0
+mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0
+mmlu.prompt3_mqa_single_correct_sk:public_relations|0
+mmlu.prompt1_mqa_single_correct_sk:public_relations|0
+mmlu.prompt2_mqa_single_correct_sk:public_relations|0
+mmlu.prompt3_mqa_single_correct_sk:security_studies|0
+mmlu.prompt1_mqa_single_correct_sk:security_studies|0
+mmlu.prompt2_mqa_single_correct_sk:security_studies|0
+mmlu.prompt3_mqa_single_correct_sk:sociology|0
+mmlu.prompt1_mqa_single_correct_sk:sociology|0
+mmlu.prompt2_mqa_single_correct_sk:sociology|0
+mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0
+mmlu.prompt3_mqa_single_correct_sk:virology|0
+mmlu.prompt1_mqa_single_correct_sk:virology|0
+mmlu.prompt2_mqa_single_correct_sk:virology|0
+mmlu.prompt3_mqa_single_correct_sk:world_religions|0
+mmlu.prompt1_mqa_single_correct_sk:world_religions|0
+mmlu.prompt2_mqa_single_correct_sk:world_religions|0
+multi_wiki_qa.prompt3_qa_sk|0
+multi_wiki_qa.prompt2_qa_sk|0
+multi_wiki_qa.prompt1_qa_sk|0
+senti_sk.prompt2_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_binary_sk|0
+senti_sk.prompt1_sentiment_binary_sk|0
+senti_sk.prompt3_sentiment_3_class_sk|0
+senti_sk.prompt1_sentiment_3_class_sk|0
+senti_sk.prompt2_sentiment_3_class_sk|0
+skj1.prompt3_skj1_sk|0
+skj1.prompt1_skj1_sk|0
+skj1.prompt2_skj1_sk|0
+sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0
+sklep.prompt1_hate_speech_sk:hate-speech|0
+sklep.prompt3_hate_speech_sk:hate-speech|0
+sklep.prompt2_hate_speech_sk:hate-speech|0
+sklep.prompt1_nli_3_classes_sk:nli|0
+sklep.prompt2_nli_3_classes_sk:nli|0
+sklep.prompt3_nli_3_classes_sk:nli|0
+sklep.prompt1_nli_binary_sk:rte|0
+sklep.prompt3_nli_binary_sk:rte|0
+sklep.prompt2_nli_binary_sk:rte|0
+sklep.prompt3_qa_sk:question-answering|0
+sklep.prompt2_qa_sk:question-answering|0
+sklep.prompt1_qa_sk:question-answering|0
+toxic_sk.prompt2_toxicity_sk|0
+toxic_sk.prompt3_toxicity_sk|0
+toxic_sk.prompt1_toxicity_sk|0
+truthfulqa.prompt3_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt1_mqa_single_correct_sk:mc1|0
+truthfulqa.prompt2_mqa_single_correct_sk:mc1|0
+slovakcopa.prompt3_copa_sk|0
+slovakcopa.prompt1_copa_sk|0
+slovakcopa.prompt2_copa_sk|0
+movie_review.prompt3_sentiment_binary_sk|0
+movie_review.prompt2_sentiment_binary_sk|0
+movie_review.prompt1_sentiment_binary_sk|0
+globalmmlu-lite.prompt3_mqa_single_correct_sk|0
+globalmmlu-lite.prompt1_mqa_single_correct_sk|0
+globalmmlu-lite.prompt2_mqa_single_correct_sk|0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_unavailable.txt b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_unavailable.txt
new file mode 100644
index 0000000..de2a1d0
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/configs/tasks_sk_zero_unavailable.txt
@@ -0,0 +1,187 @@
+chiby.prompt2_chiby_sk|0
+chiby.prompt1_chiby_sk|0
+chiby.prompt3_chiby_sk|0
+ifeval_sk.ifeval|0
+oneruler_sk.oneruler_slovak:cwe_4096|0
+oneruler_sk.oneruler_slovak:cwe_8192|0
+oneruler_sk.oneruler_slovak:cwe_16384|0
+oneruler_sk.oneruler_slovak:niah_multikey_4096|0
+oneruler_sk.oneruler_slovak:niah_multikey_8192|0
+oneruler_sk.oneruler_slovak:niah_multikey_16384|0
+oneruler_sk.oneruler_slovak:niah_multiquery_4096|0
+oneruler_sk.oneruler_slovak:niah_multiquery_8192|0
+oneruler_sk.oneruler_slovak:niah_multiquery_16384|0
+oneruler_sk.oneruler_slovak:niah_multivalue_4096|0
+oneruler_sk.oneruler_slovak:niah_multivalue_8192|0
+oneruler_sk.oneruler_slovak:niah_multivalue_16384|0
+oneruler_sk.oneruler_slovak:niah_none_4096|0
+oneruler_sk.oneruler_slovak:niah_none_8192|0
+oneruler_sk.oneruler_slovak:niah_none_16384|0
+oneruler_sk.oneruler_slovak:niah_single_4096|0
+oneruler_sk.oneruler_slovak:niah_single_8192|0
+oneruler_sk.oneruler_slovak:niah_single_16384|0
+slovaksum.prompt2_summarization_sk|0
+slovaksum.prompt3_summarization_sk|0
+slovaksum.prompt1_summarization_sk|0
+smesum.prompt2_summarization_sk|0
+smesum.prompt3_summarization_sk|0
+smesum.prompt1_summarization_sk|0
+slovak_trivia.prompt2_mqa_single_correct_sk|0
+slovak_trivia.prompt3_mqa_single_correct_sk|0
+slovak_trivia.prompt1_mqa_single_correct_sk|0
+analogies_sk.prompt1_analogy_sk|0
+analogies_sk.prompt2_analogy_sk|0
+analogies_sk.prompt3_analogy_sk|0
+driving_test.prompt3_mqa_single_correct_sk|0
+driving_test.prompt1_mqa_single_correct_sk|0
+driving_test.prompt2_mqa_single_correct_sk|0
+e_tests.prompt3_mqa_single_correct_sk:biologia|0
+e_tests.prompt3_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt3_mqa_single_correct_sk:informatika|0
+e_tests.prompt3_mqa_single_correct_sk:geografia|0
+e_tests.prompt3_mqa_single_correct_sk:matematika|0
+e_tests.prompt3_mqa_single_correct_sk:chemia|0
+e_tests.prompt3_mqa_single_correct_sk:dejepis|0
+e_tests.prompt3_mqa_single_correct_sk:fyzika|0
+e_tests.prompt3_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt1_mqa_single_correct_sk:biologia|0
+e_tests.prompt1_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt1_mqa_single_correct_sk:informatika|0
+e_tests.prompt1_mqa_single_correct_sk:geografia|0
+e_tests.prompt1_mqa_single_correct_sk:matematika|0
+e_tests.prompt1_mqa_single_correct_sk:chemia|0
+e_tests.prompt1_mqa_single_correct_sk:dejepis|0
+e_tests.prompt1_mqa_single_correct_sk:fyzika|0
+e_tests.prompt1_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt2_mqa_single_correct_sk:biologia|0
+e_tests.prompt2_mqa_single_correct_sk:obcianska-nauka|0
+e_tests.prompt2_mqa_single_correct_sk:informatika|0
+e_tests.prompt2_mqa_single_correct_sk:geografia|0
+e_tests.prompt2_mqa_single_correct_sk:matematika|0
+e_tests.prompt2_mqa_single_correct_sk:chemia|0
+e_tests.prompt2_mqa_single_correct_sk:dejepis|0
+e_tests.prompt2_mqa_single_correct_sk:fyzika|0
+e_tests.prompt2_mqa_single_correct_sk:slovensky-jazyk|0
+e_tests.prompt2_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt2_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt2_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt2_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt2_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt2_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt2_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt2_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt2_mqa_multiple_correct_sk:slovensky-jazyk|0
+e_tests.prompt1_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt1_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt1_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt1_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt1_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt1_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt1_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt1_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt1_mqa_multiple_correct_sk:slovensky-jazyk|0
+e_tests.prompt3_mqa_multiple_correct_sk:biologia|0
+e_tests.prompt3_mqa_multiple_correct_sk:obcianska-nauka|0
+e_tests.prompt3_mqa_multiple_correct_sk:informatika|0
+e_tests.prompt3_mqa_multiple_correct_sk:geografia|0
+e_tests.prompt3_mqa_multiple_correct_sk:matematika|0
+e_tests.prompt3_mqa_multiple_correct_sk:chemia|0
+e_tests.prompt3_mqa_multiple_correct_sk:dejepis|0
+e_tests.prompt3_mqa_multiple_correct_sk:fyzika|0
+e_tests.prompt3_mqa_multiple_correct_sk:slovensky-jazyk|0
+slovak_tests.prompt3_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt3_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt3_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt3_mqa_single_correct_sk:prijmacie_skusky|0
+slovak_tests.prompt1_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt1_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt1_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt1_mqa_single_correct_sk:prijmacie_skusky|0
+slovak_tests.prompt2_mqa_single_correct_sk:vyssie_studium|0
+slovak_tests.prompt2_mqa_single_correct_sk:monitor9|0
+slovak_tests.prompt2_mqa_single_correct_sk:maturita|0
+slovak_tests.prompt2_mqa_single_correct_sk:prijmacie_skusky|0
+e_tests.prompt1_qa_sk:biologia|0
+e_tests.prompt1_qa_sk:obcianska-nauka|0
+e_tests.prompt1_qa_sk:informatika|0
+e_tests.prompt1_qa_sk:geografia|0
+e_tests.prompt1_qa_sk:chemia|0
+e_tests.prompt1_qa_sk:dejepis|0
+e_tests.prompt1_qa_sk:fyzika|0
+e_tests.prompt1_qa_sk:slovensky-jazyk|0
+e_tests.prompt3_qa_sk:biologia|0
+e_tests.prompt3_qa_sk:obcianska-nauka|0
+e_tests.prompt3_qa_sk:informatika|0
+e_tests.prompt3_qa_sk:geografia|0
+e_tests.prompt3_qa_sk:chemia|0
+e_tests.prompt3_qa_sk:dejepis|0
+e_tests.prompt3_qa_sk:fyzika|0
+e_tests.prompt3_qa_sk:slovensky-jazyk|0
+e_tests.prompt2_qa_sk:biologia|0
+e_tests.prompt2_qa_sk:obcianska-nauka|0
+e_tests.prompt2_qa_sk:informatika|0
+e_tests.prompt2_qa_sk:geografia|0
+e_tests.prompt2_qa_sk:chemia|0
+e_tests.prompt2_qa_sk:dejepis|0
+e_tests.prompt2_qa_sk:fyzika|0
+e_tests.prompt2_qa_sk:slovensky-jazyk|0
+slovak_tests.prompt1_qa_sk:monitor9|0
+slovak_tests.prompt1_qa_sk:maturita|0
+slovak_tests.prompt1_qa_sk:prijmacie_skusky|0
+slovak_tests.prompt3_qa_sk:monitor9|0
+slovak_tests.prompt3_qa_sk:maturita|0
+slovak_tests.prompt3_qa_sk:prijmacie_skusky|0
+slovak_tests.prompt2_qa_sk:monitor9|0
+slovak_tests.prompt2_qa_sk:maturita|0
+slovak_tests.prompt2_qa_sk:prijmacie_skusky|0
+slohry.prompt2_declension_sk:declension_high|0
+slohry.prompt2_declension_sk:declension_mid|0
+slohry.prompt2_declension_sk:declension_low|0
+slohry.prompt1_declension_sk:declension_high|0
+slohry.prompt1_declension_sk:declension_mid|0
+slohry.prompt1_declension_sk:declension_low|0
+slohry.prompt3_declension_sk:declension_high|0
+slohry.prompt3_declension_sk:declension_mid|0
+slohry.prompt3_declension_sk:declension_low|0
+slohry.prompt3_gradation_sk:gradation_high|0
+slohry.prompt3_gradation_sk:gradation_mid|0
+slohry.prompt3_gradation_sk:gradation_low|0
+slohry.prompt1_gradation_sk:gradation_high|0
+slohry.prompt1_gradation_sk:gradation_mid|0
+slohry.prompt1_gradation_sk:gradation_low|0
+slohry.prompt2_gradation_sk:gradation_high|0
+slohry.prompt2_gradation_sk:gradation_mid|0
+slohry.prompt2_gradation_sk:gradation_low|0
+slohry.prompt1_identity_form_sk:identity_form_high|0
+slohry.prompt1_identity_form_sk:identity_form_mid|0
+slohry.prompt1_identity_form_sk:identity_form_low|0
+slohry.prompt2_identity_form_sk:identity_form_high|0
+slohry.prompt2_identity_form_sk:identity_form_mid|0
+slohry.prompt2_identity_form_sk:identity_form_low|0
+slohry.prompt3_identity_form_sk:identity_form_high|0
+slohry.prompt3_identity_form_sk:identity_form_mid|0
+slohry.prompt3_identity_form_sk:identity_form_low|0
+slohry.prompt2_inflection_sk:inflection_high|0
+slohry.prompt2_inflection_sk:inflection_mid|0
+slohry.prompt2_inflection_sk:inflection_low|0
+slohry.prompt1_inflection_sk:inflection_high|0
+slohry.prompt1_inflection_sk:inflection_mid|0
+slohry.prompt1_inflection_sk:inflection_low|0
+slohry.prompt3_inflection_sk:inflection_high|0
+slohry.prompt3_inflection_sk:inflection_mid|0
+slohry.prompt3_inflection_sk:inflection_low|0
+scio.prompt2_mqa_single_correct_sk:aj|0
+scio.prompt2_mqa_single_correct_sk:mat|0
+scio.prompt2_mqa_single_correct_sk:bio|0
+scio.prompt2_mqa_single_correct_sk:che|0
+scio.prompt2_mqa_single_correct_sk:vsp|0
+scio.prompt3_mqa_single_correct_sk:aj|0
+scio.prompt3_mqa_single_correct_sk:mat|0
+scio.prompt3_mqa_single_correct_sk:bio|0
+scio.prompt3_mqa_single_correct_sk:che|0
+scio.prompt3_mqa_single_correct_sk:vsp|0
+scio.prompt1_mqa_single_correct_sk:aj|0
+scio.prompt1_mqa_single_correct_sk:mat|0
+scio.prompt1_mqa_single_correct_sk:bio|0
+scio.prompt1_mqa_single_correct_sk:che|0
+scio.prompt1_mqa_single_correct_sk:vsp|0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log b/aya_300k_lr_experiments/benchmarks/skbench/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log
new file mode 100644
index 0000000..05a6f64
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log
@@ -0,0 +1,31318 @@
+Successfully patched TokenizersBackend._patch_mistral_regex to default fix_mistral_regex=True.
+Successfully patched VLLMModel._create_auto_tokenizer with litellm tokenizer.
+Successfully added skip_special_tokens field to GenerationParameters.
+Successfully added enforce_eager field to VLLMModelConfig (default=False).
+Successfully patched VLLMModel and AsyncVLLMModel to respect config.enforce_eager.
+Successfully patched PromptManager._prepare_chat_template.
+arc.prompt3_mqa_single_correct_sk|0,arc.prompt1_mqa_single_correct_sk|0,arc.prompt2_mqa_single_correct_sk|0,belebele.prompt3_mqa_single_correct_sk|0,belebele.prompt1_mqa_single_correct_sk|0,belebele.prompt2_mqa_single_correct_sk|0,financial_exams.prompt1_completion_sk|0,financial_exams.prompt3_completion_sk|0,financial_exams.prompt2_completion_sk|0,mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0,mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0,mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0,mmlu.prompt3_mqa_single_correct_sk:anatomy|0,mmlu.prompt1_mqa_single_correct_sk:anatomy|0,mmlu.prompt2_mqa_single_correct_sk:anatomy|0,mmlu.prompt3_mqa_single_correct_sk:astronomy|0,mmlu.prompt1_mqa_single_correct_sk:astronomy|0,mmlu.prompt2_mqa_single_correct_sk:astronomy|0,mmlu.prompt3_mqa_single_correct_sk:business_ethics|0,mmlu.prompt1_mqa_single_correct_sk:business_ethics|0,mmlu.prompt2_mqa_single_correct_sk:business_ethics|0,mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0,mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0,mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0,mmlu.prompt3_mqa_single_correct_sk:college_biology|0,mmlu.prompt1_mqa_single_correct_sk:college_biology|0,mmlu.prompt2_mqa_single_correct_sk:college_biology|0,mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0,mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0,mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0,mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0,mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0,mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0,mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0,mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0,mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0,mmlu.prompt3_mqa_single_correct_sk:college_medicine|0,mmlu.prompt1_mqa_single_correct_sk:college_medicine|0,mmlu.prompt2_mqa_single_correct_sk:college_medicine|0,mmlu.prompt3_mqa_single_correct_sk:college_physics|0,mmlu.prompt1_mqa_single_correct_sk:college_physics|0,mmlu.prompt2_mqa_single_correct_sk:college_physics|0,mmlu.prompt3_mqa_single_correct_sk:computer_security|0,mmlu.prompt1_mqa_single_correct_sk:computer_security|0,mmlu.prompt2_mqa_single_correct_sk:computer_security|0,mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0,mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0,mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0,mmlu.prompt3_mqa_single_correct_sk:econometrics|0,mmlu.prompt1_mqa_single_correct_sk:econometrics|0,mmlu.prompt2_mqa_single_correct_sk:econometrics|0,mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0,mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0,mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0,mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0,mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0,mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0,mmlu.prompt3_mqa_single_correct_sk:formal_logic|0,mmlu.prompt1_mqa_single_correct_sk:formal_logic|0,mmlu.prompt2_mqa_single_correct_sk:formal_logic|0,mmlu.prompt3_mqa_single_correct_sk:global_facts|0,mmlu.prompt1_mqa_single_correct_sk:global_facts|0,mmlu.prompt2_mqa_single_correct_sk:global_facts|0,mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0,mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0,mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0,mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0,mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0,mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0,mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0,mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0,mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0,mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0,mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0,mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0,mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0,mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0,mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0,mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0,mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0,mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0,mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0,mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0,mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0,mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0,mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0,mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0,mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0,mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0,mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0,mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0,mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0,mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0,mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0,mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0,mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0,mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0,mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0,mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0,mmlu.prompt3_mqa_single_correct_sk:human_aging|0,mmlu.prompt1_mqa_single_correct_sk:human_aging|0,mmlu.prompt2_mqa_single_correct_sk:human_aging|0,mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0,mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0,mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0,mmlu.prompt3_mqa_single_correct_sk:international_law|0,mmlu.prompt1_mqa_single_correct_sk:international_law|0,mmlu.prompt2_mqa_single_correct_sk:international_law|0,mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0,mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0,mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0,mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0,mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0,mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0,mmlu.prompt3_mqa_single_correct_sk:machine_learning|0,mmlu.prompt1_mqa_single_correct_sk:machine_learning|0,mmlu.prompt2_mqa_single_correct_sk:machine_learning|0,mmlu.prompt3_mqa_single_correct_sk:management|0,mmlu.prompt1_mqa_single_correct_sk:management|0,mmlu.prompt2_mqa_single_correct_sk:management|0,mmlu.prompt3_mqa_single_correct_sk:marketing|0,mmlu.prompt1_mqa_single_correct_sk:marketing|0,mmlu.prompt2_mqa_single_correct_sk:marketing|0,mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0,mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0,mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0,mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0,mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0,mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0,mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0,mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0,mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0,mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0,mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0,mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0,mmlu.prompt3_mqa_single_correct_sk:nutrition|0,mmlu.prompt1_mqa_single_correct_sk:nutrition|0,mmlu.prompt2_mqa_single_correct_sk:nutrition|0,mmlu.prompt3_mqa_single_correct_sk:philosophy|0,mmlu.prompt1_mqa_single_correct_sk:philosophy|0,mmlu.prompt2_mqa_single_correct_sk:philosophy|0,mmlu.prompt3_mqa_single_correct_sk:prehistory|0,mmlu.prompt1_mqa_single_correct_sk:prehistory|0,mmlu.prompt2_mqa_single_correct_sk:prehistory|0,mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0,mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0,mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0,mmlu.prompt3_mqa_single_correct_sk:professional_law|0,mmlu.prompt1_mqa_single_correct_sk:professional_law|0,mmlu.prompt2_mqa_single_correct_sk:professional_law|0,mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0,mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0,mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0,mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0,mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0,mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0,mmlu.prompt3_mqa_single_correct_sk:public_relations|0,mmlu.prompt1_mqa_single_correct_sk:public_relations|0,mmlu.prompt2_mqa_single_correct_sk:public_relations|0,mmlu.prompt3_mqa_single_correct_sk:security_studies|0,mmlu.prompt1_mqa_single_correct_sk:security_studies|0,mmlu.prompt2_mqa_single_correct_sk:security_studies|0,mmlu.prompt3_mqa_single_correct_sk:sociology|0,mmlu.prompt1_mqa_single_correct_sk:sociology|0,mmlu.prompt2_mqa_single_correct_sk:sociology|0,mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0,mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0,mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0,mmlu.prompt3_mqa_single_correct_sk:virology|0,mmlu.prompt1_mqa_single_correct_sk:virology|0,mmlu.prompt2_mqa_single_correct_sk:virology|0,mmlu.prompt3_mqa_single_correct_sk:world_religions|0,mmlu.prompt1_mqa_single_correct_sk:world_religions|0,mmlu.prompt2_mqa_single_correct_sk:world_religions|0,multi_wiki_qa.prompt3_qa_sk|0,multi_wiki_qa.prompt2_qa_sk|0,multi_wiki_qa.prompt1_qa_sk|0,senti_sk.prompt2_sentiment_binary_sk|0,senti_sk.prompt3_sentiment_binary_sk|0,senti_sk.prompt1_sentiment_binary_sk|0,senti_sk.prompt3_sentiment_3_class_sk|0,senti_sk.prompt1_sentiment_3_class_sk|0,senti_sk.prompt2_sentiment_3_class_sk|0,skj1.prompt3_skj1_sk|0,skj1.prompt1_skj1_sk|0,skj1.prompt2_skj1_sk|0,sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0,sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0,sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0,sklep.prompt1_hate_speech_sk:hate-speech|0,sklep.prompt3_hate_speech_sk:hate-speech|0,sklep.prompt2_hate_speech_sk:hate-speech|0,sklep.prompt1_nli_3_classes_sk:nli|0,sklep.prompt2_nli_3_classes_sk:nli|0,sklep.prompt3_nli_3_classes_sk:nli|0,sklep.prompt1_nli_binary_sk:rte|0,sklep.prompt3_nli_binary_sk:rte|0,sklep.prompt2_nli_binary_sk:rte|0,sklep.prompt3_qa_sk:question-answering|0,sklep.prompt2_qa_sk:question-answering|0,sklep.prompt1_qa_sk:question-answering|0,toxic_sk.prompt2_toxicity_sk|0,toxic_sk.prompt3_toxicity_sk|0,toxic_sk.prompt1_toxicity_sk|0,truthfulqa.prompt3_mqa_single_correct_sk:mc1|0,truthfulqa.prompt1_mqa_single_correct_sk:mc1|0,truthfulqa.prompt2_mqa_single_correct_sk:mc1|0,slovakcopa.prompt3_copa_sk|0,slovakcopa.prompt1_copa_sk|0,slovakcopa.prompt2_copa_sk|0,movie_review.prompt3_sentiment_binary_sk|0,movie_review.prompt2_sentiment_binary_sk|0,movie_review.prompt1_sentiment_binary_sk|0,globalmmlu-lite.prompt3_mqa_single_correct_sk|0,globalmmlu-lite.prompt1_mqa_single_correct_sk|0,globalmmlu-lite.prompt2_mqa_single_correct_sk|0/home/schwarc/diplomovka/sk-bench/.venv/lib/python3.13/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in 'list_models': model_name. Will not be supported from version '2.0'.
+
+Use `search` instead.
+ warnings.warn(message, FutureWarning)
+/home/schwarc/diplomovka/sk-bench/.venv/lib/python3.13/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in 'list_models': model_name. Will not be supported from version '2.0'.
+
+Use `search` instead.
+ warnings.warn(message, FutureWarning)
+Careful, the task financial_exams.prompt2_completion_sk is using evaluation data to build the few shot examples.
+Careful, the task belebele.prompt3_mqa_single_correct_sk is using evaluation data to build the few shot examples.
+Careful, the task belebele.prompt1_mqa_single_correct_sk is using evaluation data to build the few shot examples.
+Careful, the task belebele.prompt2_mqa_single_correct_sk is using evaluation data to build the few shot examples.
+Careful, the task skj1.prompt1_skj1_sk is using evaluation data to build the few shot examples.
+Careful, the task financial_exams.prompt3_completion_sk is using evaluation data to build the few shot examples.
+Careful, the task skj1.prompt2_skj1_sk is using evaluation data to build the few shot examples.
+Careful, the task financial_exams.prompt1_completion_sk is using evaluation data to build the few shot examples.
+Careful, the task skj1.prompt3_skj1_sk is using evaluation data to build the few shot examples.
+
+Applying chat template with reasoning effort: None
+
Filter: 0%| | 0/13062 [00:00= 0) for v in values)
+ row = {'task': task, 'sample_count': pf.metadata.num_rows,
+ 'truncation_values_present': sum(v is not None for v in values),
+ 'truncation_null_values': sum(v is None for v in values),
+ 'samples_with_truncation': sum(v is not None and v > 0 for v in values),
+ 'truncated_tokens_total': sum(v or 0 for v in values),
+ 'max_truncated_tokens_per_sample': max((v or 0 for v in values), default=0),
+ 'parquet_bytes': file.stat().st_size,
+ 'source_archive_member': original_paths.get(file.relative_to(source).as_posix(), file.relative_to(source).as_posix()),
+ 'extracted_relative_path': file.relative_to(source).as_posix()}
+ assert normalize(task) not in lookup
+ lookup[normalize(task)] = row
+ details.append(row)
+ assert set(lookup) == normalized_expected, 'DETAILS task coverage mismatch'
+ write_csv(out/'results/truncation_audit_per_task.csv', details)
+
+ metric_names = sorted({key for task in configs for key in d['results'][task]})
+ per_task = []
+ for task in sorted(configs):
+ metric = d['results'][task]
+ config = configs[task]
+ detail = lookup[normalize(task)]
+ primary = next((name for name in ['em_relaxed', 'em_norm_relaxed', 'exact_match_slk_prefix'] if name in metric), None)
+ family = task.split('.prompt')[0]
+ prompt = int(re.search(r'\.prompt([123])_', task).group(1))
+ task_group = re.sub(r'\.prompt[123]_', '.', task).removesuffix('|0')
+ row = {'task': task, 'family': family, 'task_group_without_prompt': task_group,
+ 'prompt': prompt, 'num_fewshots': config['num_fewshots'], 'sample_count': detail['sample_count'],
+ 'primary_metric': primary, 'primary_value': metric[primary],
+ 'generation_size': config['generation_size'], 'evaluation_splits': '|'.join(config['evaluation_splits']),
+ 'hf_repo': config['hf_repo'], 'hf_subset': config['hf_subset'], 'hf_revision': config['hf_revision']}
+ row.update({key: metric.get(key, '') for key in metric_names})
+ per_task.append(row)
+ write_csv(out/'results/per_task_metrics.csv', per_task)
+
+ # Each summary groups only identical metrics; values remain 0..1.
+ # Macro means weight task variants equally. Sample-weighted means weight
+ # their actual DETAILS rows. Repeated prompts are evaluation instances,
+ # never additional independent underlying test questions.
+ def summarize(group_columns, filename):
+ groups = collections.defaultdict(list)
+ for row in per_task:
+ for metric_name in metric_names:
+ if not metric_name.endswith('_stderr') and row[metric_name] != '':
+ groups[tuple(row[x] for x in group_columns)+(metric_name,)].append(row)
+ result = []
+ for key, rows in sorted(groups.items()):
+ metric_name = key[-1]
+ total = sum(x['sample_count'] for x in rows)
+ item = dict(zip(group_columns, key[:-1]))
+ item.update({'metric': metric_name, 'task_variants': len(rows),
+ 'prompt_count': len({x['prompt'] for x in rows}),
+ 'evaluated_instances': total,
+ 'task_macro_mean': statistics.mean(x[metric_name] for x in rows),
+ 'sample_weighted_mean': sum(x['sample_count']*x[metric_name] for x in rows)/total,
+ 'min_task_value': min(x[metric_name] for x in rows),
+ 'max_task_value': max(x[metric_name] for x in rows)})
+ result.append(item)
+ write_csv(out/'results'/filename, result)
+ return result
+ family_summary = summarize(['family'], 'family_summary.csv')
+ summarize(['family', 'prompt'], 'family_prompt_summary.csv')
+ three_prompt = summarize(['family', 'task_group_without_prompt'], 'three_prompt_summary.csv')
+ assert all(x['task_variants'] == x['prompt_count'] == 3 for x in three_prompt)
+
+ access = json.loads((base/'configs/dataset_access_report.json').read_text(encoding='utf-8'))
+ inaccessible = [x for x in access if x['status'] != 'OK']
+ with (base/'logs'/f'{run_name}.log').open(encoding='utf-8', newline='') as stream:
+ log_text = stream.read()
+ truncation_warnings = re.findall(r'context_size \+ max_new_tokens=\d+[^\r\n]*Truncating context to \d+ tokens\.', log_text)
+ warning_evidence = []
+ for line_number, line in enumerate(log_text.split('\n'), 1):
+ if 'Truncating context' in line:
+ warning_evidence.append({'line_number': line_number, 'message': line[line.index('context_size'):], 'source': f'logs/{run_name}.log'})
+ (out/'results/log_truncation_evidence.json').write_text(json.dumps(warning_evidence, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
+ summary = {'run_name': run_name, 'result_file': result_path.name,
+ 'task_list_count': len(tasks), 'configured_tasks': len(configs), 'scored_task_variants': len(per_task),
+ 'result_json_keys': len(d['results']),
+ 'extra_result_keys_excluded_from_task_summaries': sorted(set(d['results'])-set(configs)),
+ 'all_original_task_variants': len(all_tasks),
+ 'omitted_task_variants': len(omitted), 'inaccessible_repositories': len(inaccessible),
+ 'inaccessible_task_variants': sum(len(x['tasks']) for x in inaccessible),
+ 'additionally_excluded_nli_task_variants': 3,
+ 'families': dict(sorted(collections.Counter(row['family'] for row in per_task).items())),
+ 'prompt_counts': dict(sorted(collections.Counter(row['prompt'] for row in per_task).items())),
+ 'evaluated_instances': sum(row['sample_count'] for row in per_task),
+ 'instances_by_prompt': {str(prompt): sum(x['sample_count'] for x in per_task if x['prompt'] == prompt) for prompt in [1,2,3]},
+ 'details_files_audited': len(details), 'truncation_column': 'model_response.truncated_tokens_count',
+ 'truncation_values_present': sum(x['truncation_values_present'] for x in details),
+ 'truncation_null_values': sum(x['truncation_null_values'] for x in details),
+ 'details_reported_samples_with_truncation': sum(x['samples_with_truncation'] for x in details),
+ 'details_reported_truncated_tokens_total': sum(x['truncated_tokens_total'] for x in details),
+ 'details_reported_max_truncated_tokens_per_sample': max(x['max_truncated_tokens_per_sample'] for x in details),
+ 'context_truncation_observed_in_log': bool(truncation_warnings),
+ 'actual_truncated_sample_count': None,
+ 'console_context_truncation_warning_count': len(truncation_warnings),
+ 'console_context_truncation_warnings': truncation_warnings,
+ 'truncation_interpretation': 'All DETAILS counters are zero, but the console explicitly reports context truncation. Do not infer no truncation or an affected-sample count from these counters.',
+ 'model_config': d['config_general']['model_config'],
+ 'max_samples': d['config_general']['max_samples'],
+ 'generation_size_counts': dict(collections.Counter(str(x['generation_size']) for x in per_task)),
+ 'original_summary_general': d['summary_general'],
+ 'pyarrow_version': pyarrow.__version__, 'python_version': sys.version,
+ 'aggregation': 'Descriptive metric-specific task macro means and sample-weighted means; no official full-SK-bench overall score.',
+ 'limitations': ['Partial available zero-shot subset: 222 of 412 task variants.',
+ 'Only the validation-selected LR=0.00010 was benchmarked.',
+ 'Sample counts are evaluation instances; prompt variants reuse underlying questions.',
+ 'One console warning reports context truncation; all DETAILS counters are zero, so the recorded counters do not reliably enumerate affected inputs.',
+ 'The audit cannot recover the number of affected tasks/samples or quantify output generation-size termination.',
+ 'All-zero summary counters include non_truncated=0 and cannot prove absence of truncation.',
+ 'Raw DETAILS and generation-cache parquets are excluded from this curated tree.']}
+ (out/'results/audit_summary.json').write_text(json.dumps(summary, indent=2, ensure_ascii=False)+'\n', encoding='utf-8')
+ script_destination = out/'provenance/audit_skbench.py'
+ if pathlib.Path(__file__).resolve() != script_destination.resolve():
+ shutil.copyfile(pathlib.Path(__file__), script_destination)
+ print(json.dumps({key: summary[key] for key in ['scored_task_variants', 'evaluated_instances', 'instances_by_prompt', 'details_reported_samples_with_truncation', 'context_truncation_observed_in_log', 'actual_truncated_sample_count', 'truncation_null_values', 'families']}, indent=2))
+ print('Primary family macro means (metrics kept separate):')
+ for row in family_summary:
+ if row['metric'] in ['em_relaxed', 'em_norm_relaxed', 'exact_match_slk_prefix', 'f1_slk']:
+ print(row)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/provenance/copied_files_sha256.csv b/aya_300k_lr_experiments/benchmarks/skbench/provenance/copied_files_sha256.csv
new file mode 100644
index 0000000..8a6bbd7
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/provenance/copied_files_sha256.csv
@@ -0,0 +1,14 @@
+source_archive_member,curated_path,bytes,sha256
+skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/results/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010/results_2026-10-22T21-55-04.268099.json,results/results_2026-10-22T21-55-04.268099.json,638180,966123dc20b547860d04f3434f65752c681ca2d83baebee22873e553161ff0df
+skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log,logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log,4537535,23a857210201913c5506ac39553a86fd803ed0c057e1226fcd978030a40b6c78
+skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/tasks.txt,configs/tasks.txt,11415,297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
+skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/omitted_tasks.txt,configs/omitted_tasks.txt,8629,613b78472e6788eab826f31da1100c00b597ed9bbc8151af38ebb8ef5bf830af
+skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/uv.lock,provenance/uv.lock,690556,045d2840fda141981afba93bd4a1b9ddb59f0f0791c162cf66b2c88e6491fb41
+skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255/skbench_commit.txt,provenance/skbench_commit.txt,41,12655509d516c6a57225a072f708e5ca854600e7daa3bf5cdc95e6b36e94fe70
+archive_summary.json,provenance/archive_summary.json,6529,535390044cd67477ea1631527ac736f72108935f25a11a613b1709caf8154957
+skbench_evaluation/configs/dataset_access_report.json,configs/dataset_access_report.json,29828,8eea8c082d1eb914850128d48021a39c7e1fb7189578c5f1e496004af6ff593b
+skbench_evaluation/configs/export_lr_0p00010.yaml,configs/export_lr_0p00010.yaml,391,bff3e0b8abd130122c8d88be71647e0ea67e4a836b3caaed4e3bc307ef267dca
+skbench_evaluation/configs/tasks_sk_zero.txt,configs/tasks_sk_zero.txt,20044,25c3cdfeb1f46adb86b68be24a9d8e22439733501e2fffda1ce5034e610660a4
+skbench_evaluation/configs/tasks_sk_zero_available.txt,configs/tasks_sk_zero_available.txt,11517,5ce7f1666b2073afd93aab03a116937f48954cf51d313c567c198b31f0dfcaf2
+skbench_evaluation/configs/tasks_sk_zero_available_without_nli.txt,configs/tasks_sk_zero_available_without_nli.txt,11415,297c7185d8ddea6899d20287ed5cffb895116cf756121252a0e76070b248db76
+skbench_evaluation/configs/tasks_sk_zero_unavailable.txt,configs/tasks_sk_zero_unavailable.txt,8527,0071cffec8ab2446a5594d0b3a3116151e7fc375bb300394e9ea7cb91ed7df10
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/provenance/input_length_diagnostics.py b/aya_300k_lr_experiments/benchmarks/skbench/provenance/input_length_diagnostics.py
new file mode 100644
index 0000000..b95f6eb
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/provenance/input_length_diagnostics.py
@@ -0,0 +1,62 @@
+"""Inspect token-array and raw-prompt lengths; never export prompt content."""
+import argparse, csv, json, pathlib, re, shutil, statistics, sys
+from urllib.parse import unquote
+
+parser=argparse.ArgumentParser(description=__doc__)
+parser.add_argument('--source',type=pathlib.Path,required=True)
+parser.add_argument('--output',type=pathlib.Path,required=True)
+parser.add_argument('--pyarrow-path',type=pathlib.Path)
+args=parser.parse_args()
+if args.pyarrow_path: sys.path.insert(0,str(args.pyarrow_path.resolve()))
+import pyarrow.parquet as pq
+run=args.source/'skbench_evaluation/results/sk_zero_available_without_nli_lr_0p00010_20261022_185255'
+d=json.loads(next((run/'results').rglob('*.json')).read_text())
+limit=d['config_general']['model_config']['max_model_length']
+rows=[]
+outliers=[]
+for file in sorted((run/'details').rglob('*.parquet')):
+ task=re.match(r'details_(.+)_2026-10-22T21-55-04\.268099\.parquet',unquote(file.name)).group(1)
+ cfg=d['config_tasks'][task]
+ budget=limit-cfg['generation_size']
+ data=pq.ParquetFile(file).read(columns=['model_response.input_tokens','model_response.input'])
+ data=data.column('model_response').combine_chunks()
+ tokens=data.field('input_tokens').to_pylist()
+ texts=data.field('input').to_pylist()
+ lens=[None if value is None else len(value) for value in tokens]
+ chars=[None if value is None else len(value) for value in texts]
+ row={'task':task,'generation_size':cfg['generation_size'],'context_limit':limit,'input_budget':budget,
+ 'samples':len(lens),'null_token_arrays':sum(x is None for x in lens),'null_input_strings':sum(x is None for x in chars),
+ 'min_recorded_input_tokens':min(x for x in lens if x is not None),
+ 'max_recorded_input_tokens':max(x for x in lens if x is not None),
+ 'mean_recorded_input_tokens':statistics.mean(x for x in lens if x is not None),
+ 'arrays_at_input_budget':sum(x==budget for x in lens),
+ 'arrays_over_input_budget':sum(x is not None and x>budget for x in lens),
+ 'max_recorded_input_characters':max(x for x in chars if x is not None)}
+ rows.append(row)
+ for index,length in enumerate(lens):
+ if length is not None and length>=budget:
+ outliers.append({'task':task,'details_row_index_zero_based':index,'input_token_array_length':length,
+ 'generation_size':cfg['generation_size'],'input_budget':budget,
+ 'recorded_input_characters':chars[index]})
+for name,values in [('input_length_diagnostics_per_task.csv',rows),('input_length_budget_boundary_rows.csv',outliers)]:
+ with (args.output/name).open('w',newline='',encoding='utf-8') as f:
+ writer=csv.DictWriter(f,fieldnames=list(values[0]) if values else ['task'])
+ writer.writeheader();writer.writerows(values)
+with (args.source/'skbench_evaluation/logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log').open(encoding='utf-8',newline='') as stream:
+ log=stream.read()
+messages=[]
+for number,line in enumerate(log.split('\n'),1):
+ if re.search(r'truncat',line,re.I):
+ messages.append({'log_lf_line_number':number,'message':re.sub(r'\x1b\[[0-9;]*[A-Za-z]','',line).strip()[-1500:]})
+summary={'task_count':len(rows),'evaluated_instances':sum(x['samples'] for x in rows),
+ 'max_recorded_input_tokens':max(x['max_recorded_input_tokens'] for x in rows),
+ 'arrays_at_input_budget':sum(x['arrays_at_input_budget'] for x in rows),
+ 'arrays_over_input_budget':sum(x['arrays_over_input_budget'] for x in rows),
+ 'log_messages_matching_truncat_case_insensitive':messages,
+ 'limitation':'Input strings and token arrays are recorded artifacts, not a separately tokenized original source prompt. Their lengths cannot recover how much text was discarded or establish the number of truly truncated inputs.'}
+(args.output/'input_length_diagnostics_summary.json').write_text(json.dumps(summary,indent=2,ensure_ascii=False)+'\n',encoding='utf-8')
+print(json.dumps(summary,indent=2,ensure_ascii=False))
+print('BOUNDARY TASK COUNTS',json.dumps({task:sum(row['task']==task for row in outliers) for task in sorted({row['task'] for row in outliers})}))
+script_destination=args.output.parent/'provenance/input_length_diagnostics.py'
+if pathlib.Path(__file__).resolve()!=script_destination.resolve():
+ shutil.copyfile(pathlib.Path(__file__),script_destination)
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/provenance/requirements-audit.txt b/aya_300k_lr_experiments/benchmarks/skbench/provenance/requirements-audit.txt
new file mode 100644
index 0000000..8ed4982
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/provenance/requirements-audit.txt
@@ -0,0 +1 @@
+pyarrow==26.0.0
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/provenance/skbench_commit.txt b/aya_300k_lr_experiments/benchmarks/skbench/provenance/skbench_commit.txt
new file mode 100644
index 0000000..0cc84fb
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/provenance/skbench_commit.txt
@@ -0,0 +1 @@
+15823bbcc2d995e69441fcb7d34abb61bd8180c5
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/provenance/uv.lock b/aya_300k_lr_experiments/benchmarks/skbench/provenance/uv.lock
new file mode 100644
index 0000000..a734062
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/provenance/uv.lock
@@ -0,0 +1,6274 @@
+version = 1
+revision = 3
+requires-python = "==3.13.*"
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "sys_platform == 'darwin'",
+]
+
+[manifest]
+
+[[manifest.dependency-metadata]]
+name = "deep-gemm"
+version = "2.1.1.post3"
+
+[[package]]
+name = "absl-py"
+version = "2.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/64/c7/8de93764ad66968d19329a7e0c147a2bb3c7054c554d4a119111b8f9440f/absl_py-2.4.0.tar.gz", hash = "sha256:8c6af82722b35cf71e0f4d1d47dcaebfff286e27110a99fc359349b247dfb5d4", size = 116543, upload-time = "2026-01-28T10:17:05.322Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/18/a6/907a406bb7d359e6a63f99c313846d9eec4f7e6f7437809e03aa00fa3074/absl_py-2.4.0-py3-none-any.whl", hash = "sha256:88476fd881ca8aab94ffa78b7b6c632a782ab3ba1cd19c9bd423abc4fb4cd28d", size = 135750, upload-time = "2026-01-28T10:17:04.19Z" },
+]
+
+[[package]]
+name = "accelerate"
+version = "1.14.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "psutil" },
+ { name = "pyyaml" },
+ { name = "safetensors" },
+ { name = "torch" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/8d/75/94cd5d389649578aca399e5aa822637eec18319a1dadc400ffe2f9a7493f/accelerate-1.14.0.tar.gz", hash = "sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d", size = 412167, upload-time = "2026-06-11T13:45:52.326Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a8/db/253133d7e7cb40d3af384bb2f5c0b4a2b7fdcffbc95c688cc67a20a3c103/accelerate-1.14.0-py3-none-any.whl", hash = "sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6", size = 389246, upload-time = "2026-06-11T13:45:50.477Z" },
+]
+
+[[package]]
+name = "aenum"
+version = "3.1.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/f8/33e75863394f42e429bb553e05fda7c59763f0fd6848de847a25b3fbccf6/aenum-3.1.15.tar.gz", hash = "sha256:8cbd76cd18c4f870ff39b24284d3ea028fbe8731a58df3aa581e434c575b9559", size = 134730, upload-time = "2023-06-27T00:19:52.546Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d0/fa/ca0c66b388624ba9dbbf35aab3a9f326bfdf5e56a7237fe8f1b600da6864/aenum-3.1.15-py3-none-any.whl", hash = "sha256:e0dfaeea4c2bd362144b87377e2c61d91958c5ed0b4daf89cb6f45ae23af6288", size = 137633, upload-time = "2023-06-27T00:19:55.112Z" },
+]
+
+[[package]]
+name = "aioboto3"
+version = "15.5.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiobotocore", extra = ["boto3"] },
+ { name = "aiofiles" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a2/01/92e9ab00f36e2899315f49eefcd5b4685fbb19016c7f19a9edf06da80bb0/aioboto3-15.5.0.tar.gz", hash = "sha256:ea8d8787d315594842fbfcf2c4dce3bac2ad61be275bc8584b2ce9a3402a6979", size = 255069, upload-time = "2025-10-30T13:37:16.122Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e5/3e/e8f5b665bca646d43b916763c901e00a07e40f7746c9128bdc912a089424/aioboto3-15.5.0-py3-none-any.whl", hash = "sha256:cc880c4d6a8481dd7e05da89f41c384dbd841454fc1998ae25ca9c39201437a6", size = 35913, upload-time = "2025-10-30T13:37:14.549Z" },
+]
+
+[[package]]
+name = "aiobotocore"
+version = "2.25.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "aioitertools" },
+ { name = "botocore" },
+ { name = "jmespath" },
+ { name = "multidict" },
+ { name = "python-dateutil" },
+ { name = "wrapt" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/62/94/2e4ec48cf1abb89971cb2612d86f979a6240520f0a659b53a43116d344dc/aiobotocore-2.25.1.tar.gz", hash = "sha256:ea9be739bfd7ece8864f072ec99bb9ed5c7e78ebb2b0b15f29781fbe02daedbc", size = 120560, upload-time = "2025-10-28T22:33:21.787Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/95/2a/d275ec4ce5cd0096665043995a7d76f5d0524853c76a3d04656de49f8808/aiobotocore-2.25.1-py3-none-any.whl", hash = "sha256:eb6daebe3cbef5b39a0bb2a97cffbe9c7cb46b2fcc399ad141f369f3c2134b1f", size = 86039, upload-time = "2025-10-28T22:33:19.949Z" },
+]
+
+[package.optional-dependencies]
+boto3 = [
+ { name = "boto3" },
+]
+
+[[package]]
+name = "aiofiles"
+version = "25.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/41/c3/534eac40372d8ee36ef40df62ec129bee4fdb5ad9706e58a29be53b2c970/aiofiles-25.1.0.tar.gz", hash = "sha256:a8d728f0a29de45dc521f18f07297428d56992a742f0cd2701ba86e44d23d5b2", size = 46354, upload-time = "2025-10-09T20:51:04.358Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bc/8a/340a1555ae33d7354dbca4faa54948d76d89a27ceef032c8c3bc661d003e/aiofiles-25.1.0-py3-none-any.whl", hash = "sha256:abe311e527c862958650f9438e859c1fa7568a141b22abcd015e120e86a85695", size = 14668, upload-time = "2025-10-09T20:51:03.174Z" },
+]
+
+[[package]]
+name = "aiohappyeyeballs"
+version = "2.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ce/f4/eec0465c2f67b2664688d0240b3212d5196fd89e741df67ddb81f8d35658/aiohappyeyeballs-2.7.1.tar.gz", hash = "sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d", size = 24757, upload-time = "2026-07-01T17:11:55.501Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/43/1947f06babed6b3f1d7f38b0c767f52df66bfb2bc10b468c4a7de9eceff2/aiohappyeyeballs-2.7.1-py3-none-any.whl", hash = "sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472", size = 15038, upload-time = "2026-07-01T17:11:54.055Z" },
+]
+
+[[package]]
+name = "aiohttp"
+version = "3.14.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohappyeyeballs" },
+ { name = "aiosignal" },
+ { name = "attrs" },
+ { name = "frozenlist" },
+ { name = "multidict" },
+ { name = "propcache" },
+ { name = "yarl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/82/78/8ea7308cac6934de8c74a14f3d5f65d1c89287426688be79538d0e5c013d/aiohttp-3.14.1.tar.gz", hash = "sha256:307f2cff90a764d329e77040603fa032db89c5c24fdad50c4c15334cba744035", size = 7955794, upload-time = "2026-06-07T21:09:35.529Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bc/97/bd137012dd97e1649162b099135a80e1fd59aaa807b2430fc448d1029aff/aiohttp-3.14.1-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:b3a03285a7f9c7b016324574a6d92a1c895da6b978cb8f1deee3ac72bc6da178", size = 506882, upload-time = "2026-06-07T21:07:15.501Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/79/e5cc690e9d922a66887ceeaca53a8ffd5a7b0be3816142b7abc433742d89/aiohttp-3.14.1-cp313-cp313-android_21_x86_64.whl", hash = "sha256:2a73f487ab8ef5abbb24b7aa9b73e98eaba9e9e031804ff2416f02eca315ccaf", size = 515270, upload-time = "2026-06-07T21:07:17.53Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/22/a73ccbf9dbd6e26dda0b24d5fd5db7da92ee3383a79f47677ffb834c5c5b/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:915fbb7b41b115192259f8c9ae58f3ddc444d2b5579917270211858e606a4afd", size = 485841, upload-time = "2026-06-07T21:07:19.555Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/b9/57ed8eaf596321c2ad747bd480fb1700dbd7177c60dfc9e4c187f629662e/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:7fb4bdf95b0561a79f259f9d28fbc109728c5ee7f27aff6391f0ca703a329abe", size = 492088, upload-time = "2026-06-07T21:07:21.581Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c0/5ebe5270a7c140d7c6f79dcb018640225f14d406c149e4eec04a7d82fe71/aiohttp-3.14.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:1b9748363260121d2927704f5d4fc498150669ca3ae93625986ee89c8f80dcd4", size = 501564, upload-time = "2026-06-07T21:07:23.388Z" },
+ { url = "https://files.pythonhosted.org/packages/75/7f/8cdaa24fc7983865e0915153b96a9ac5bcdd3548d64c5a27d17cecccad2d/aiohttp-3.14.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:86a6dab78b0e43e2897a3bbe15745aa60dc5423ca437b7b0b164c069bf91b876", size = 751998, upload-time = "2026-06-07T21:07:25.046Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/f4/c4227aacfacc5cb0cc2d119b65301d177912a6842cd64e120c47af76064f/aiohttp-3.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4dfd6e47d3c44c2279907607f73a4240b88c69eb8b90da7e2441a8045dfd21da", size = 510918, upload-time = "2026-06-07T21:07:27.28Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/01/a2d5f96cd4e74424864d30bc0a7e44d0a12dacdcfa91b5b2d1bd3dca6bf3/aiohttp-3.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:317acd9f8602858dc7d59679812c376c7f0b97bcbbf16e0d6237f54141d8a8a6", size = 508657, upload-time = "2026-06-07T21:07:29.252Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/ed/3c0fb5c500fdd8e7ebc10d1889c04384fffa1a9163eac1356088ca9da1b1/aiohttp-3.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bd869c427324e5cb15195793de951295710db28be7d818247f3097b4ab5d4b96", size = 1757907, upload-time = "2026-06-07T21:07:31.03Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/ab/d4c924d9bd5be3050c226612413ce68cb54c70d2c31b661bfc8d9a5b6a70/aiohttp-3.14.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:93b032b5ec3255473c143627d21a69ac74ae12f7f33974cb587c564d11b1066f", size = 1737565, upload-time = "2026-06-07T21:07:33.031Z" },
+ { url = "https://files.pythonhosted.org/packages/19/2a/37326821ff779084020cdc33224d20b19f42f4183a500ff92022a739eda7/aiohttp-3.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f234b4deb12f3ad59127e037bc57c40c21e45b45282df7d3a55a0f409f595296", size = 1799018, upload-time = "2026-06-07T21:07:35.003Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/4f/6e947ba73e4ce09070761c05ed3a8ceb7c21f5e46798671d8b2aac0e4626/aiohttp-3.14.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9af6779bfb46abf124068327abcdf9ce95c9ef8287a3e8da76ccf2d0f16c28fa", size = 1894416, upload-time = "2026-06-07T21:07:36.956Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/6e/dbf1d0625dc711fb2851f4f3c3055c39ed58bae92082d8c627dbe6013736/aiohttp-3.14.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:faccab372e66bc76d5731525e7f1143c922271725b9d38c9f97edcc66266b451", size = 1783881, upload-time = "2026-06-07T21:07:39.063Z" },
+ { url = "https://files.pythonhosted.org/packages/44/c2/5e25098a67268ed369483ae7d1a58bd0a13d03aab860d2a0e4a6eb25b046/aiohttp-3.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f380468b09d2a81633ee863b0ec5648d364bd17bb8ecfb8c2f387f7ac1faf42c", size = 1587572, upload-time = "2026-06-07T21:07:41.058Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/bd/cf9cee17e140f942a3de73e658a543aa8fbf35a5fc67a9d2538d52d77f0b/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:97e704dcd26271f5bda3fa07c3ce0fb76d6d3f8659f4baa1a24442cc9ba177ca", size = 1722137, upload-time = "2026-06-07T21:07:43.014Z" },
+ { url = "https://files.pythonhosted.org/packages/89/6d/5684f8c59045c96f81a18cefbc1fbbd79d25b88f1c622f2a5c5c08fcb632/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:269b76ac5394092b95bc4a098f4fc6c191c083c3bd12775d1e30e663132f6a09", size = 1755953, upload-time = "2026-06-07T21:07:45.933Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/40/35caf3170f8359760740a7d9aa0fff2e344bef98e1d1186f5a0f6dec17e6/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c0b3e614340c889d575451696374c9d17affd54cd607ca0babed8f8c37b9397", size = 1766479, upload-time = "2026-06-07T21:07:48.047Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/a1/b0c61e7a137f0d81de49a82023a6df73c3c16d6fefb0f8e4a93d21639002/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:5663ee9257cfa1add7253a7da3035a02f31b6600ec48261585e1800a81533080", size = 1580077, upload-time = "2026-06-07T21:07:50.069Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/41/194ea4623693009fcefebef7aef63c141754f153e9cd0d39d3b9e36c175c/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:603a2c834142172ffddc054067f5ec0ca65d57a0aa98a71bc81952573208e345", size = 1791688, upload-time = "2026-06-07T21:07:52.106Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/45/4de841f005cfe1fd63e2a2fe011262c515e2a62aa6994b15947e7d717ac9/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cb21957bb8aca671c1765e32f58164cf0c50e6bf41c0bbbd16da20732ecaf588", size = 1761094, upload-time = "2026-06-07T21:07:54.113Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/ae/dbce10533d3896d544d5053939ed75b7dc31a1b0973d959b1b5ae21028d6/aiohttp-3.14.1-cp313-cp313-win32.whl", hash = "sha256:e509a55f681e6158c20f70f102f9cf61fb20fbc382272bc6d94b7343f2582780", size = 452662, upload-time = "2026-06-07T21:07:56.06Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/d9/0bf1a19362c32f06229da5e7ddfcec91f93474d6307f7a2d3135e9c674dc/aiohttp-3.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:1ac8531b638959718e18c2207fbfe297819875da46a740b29dfa29beba64355a", size = 479748, upload-time = "2026-06-07T21:07:58.319Z" },
+ { url = "https://files.pythonhosted.org/packages/22/0a/62e7232dc9484fbec112ceb32efb6a624cc7994ec6e2b019286f17c4e8f2/aiohttp-3.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:250d14af67f6b6a1a4a811049b1afa69d61d617fca6bf33149b3ab1a6dbcf7b8", size = 447723, upload-time = "2026-06-07T21:08:00.154Z" },
+]
+
+[[package]]
+name = "aioitertools"
+version = "0.13.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fd/3c/53c4a17a05fb9ea2313ee1777ff53f5e001aefd5cc85aa2f4c2d982e1e38/aioitertools-0.13.0.tar.gz", hash = "sha256:620bd241acc0bbb9ec819f1ab215866871b4bbd1f73836a55f799200ee86950c", size = 19322, upload-time = "2025-11-06T22:17:07.609Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/10/a1/510b0a7fadc6f43a6ce50152e69dbd86415240835868bb0bd9b5b88b1e06/aioitertools-0.13.0-py3-none-any.whl", hash = "sha256:0be0292b856f08dfac90e31f4739432f4cb6d7520ab9eb73e143f4f2fa5259be", size = 24182, upload-time = "2025-11-06T22:17:06.502Z" },
+]
+
+[[package]]
+name = "aiosignal"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "frozenlist" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/61/62/06741b579156360248d1ec624842ad0edf697050bbaf7c3e46394e106ad1/aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7", size = 25007, upload-time = "2025-07-03T22:54:43.528Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" },
+]
+
+[[package]]
+name = "annotated-doc"
+version = "0.0.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/57/ba/046ceea27344560984e26a590f90bc7f4a75b06701f653222458922b558c/annotated_doc-0.0.4.tar.gz", hash = "sha256:fbcda96e87e9c92ad167c2e53839e57503ecfda18804ea28102353485033faa4", size = 7288, upload-time = "2025-11-10T22:07:42.062Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/d3/26bf1008eb3d2daa8ef4cacc7f3bfdc11818d111f7e2d0201bc6e3b49d45/annotated_doc-0.0.4-py3-none-any.whl", hash = "sha256:571ac1dc6991c450b25a9c2d84a3705e2ae7a53467b5d111c24fa8baabbed320", size = 5303, upload-time = "2025-11-10T22:07:40.673Z" },
+]
+
+[[package]]
+name = "annotated-types"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" },
+]
+
+[[package]]
+name = "anthropic"
+version = "0.116.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "distro" },
+ { name = "docstring-parser" },
+ { name = "httpx" },
+ { name = "jiter" },
+ { name = "pydantic" },
+ { name = "sniffio" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/66/a2/d31f14e28d49bae983a3634e38dfb4b31c50110b5e403596c5c6a20b23f8/anthropic-0.116.0.tar.gz", hash = "sha256:5fc248fbb9fe03ef686f8a774f81586bca31a043260aab88b387ea3660f4a396", size = 949149, upload-time = "2026-07-02T19:08:10.534Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c7/dd/2a1e81cf1b163acc340afc4ec74ed1d86f5eed1a809fabdeed3e0997b346/anthropic-0.116.0-py3-none-any.whl", hash = "sha256:6c0a7698e8d652455da3499978279bb2588c7264d0a35be3666009a4258c8256", size = 956896, upload-time = "2026-07-02T19:08:08.756Z" },
+]
+
+[[package]]
+name = "antlr4-python3-runtime"
+version = "4.13.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/33/5f/2cdf6f7aca3b20d3f316e9f505292e1f256a32089bd702034c29ebde6242/antlr4_python3_runtime-4.13.2.tar.gz", hash = "sha256:909b647e1d2fc2b70180ac586df3933e38919c85f98ccc656a96cd3f25ef3916", size = 117467, upload-time = "2024-08-03T19:00:12.757Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/89/03/a851e84fcbb85214dc637b6378121ef9a0dd61b4c65264675d8a5c9b1ae7/antlr4_python3_runtime-4.13.2-py3-none-any.whl", hash = "sha256:fe3835eb8d33daece0e799090eda89719dbccee7aa39ef94eed3818cafa5a7e8", size = 144462, upload-time = "2024-08-03T19:00:11.134Z" },
+]
+
+[[package]]
+name = "anyio"
+version = "4.14.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "idna" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" },
+]
+
+[[package]]
+name = "apache-tvm-ffi"
+version = "0.1.11"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6d/3d/4b9226cd45aa800a6904603dda9b323d728f3c3869952a673f3483b78b19/apache_tvm_ffi-0.1.11.tar.gz", hash = "sha256:153cd2c5a9717804cb0bcd9b2709f22a1e5f80ed05b5a490faf5949b136eedba", size = 2798354, upload-time = "2026-05-04T17:48:43.852Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/9d/0f81ca556e5836b3ca64818cdae3f47dc7822bd35d22ddef7a54106d801d/apache_tvm_ffi-0.1.11-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:6ae51cc7df415b5f373a9df4baa1165a65608e519bea81e7dd23428f00eeb689", size = 2418793, upload-time = "2026-05-04T17:47:57.879Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/a9/f48e5dd4ae1f6f0c5ffac259c0a9531b7d6a7c0a4c45bc2229d55de6adf8/apache_tvm_ffi-0.1.11-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:da2c8d07fdc737d1ba75f4de25c29f156905b9dc980f1da90c395b4db525f522", size = 2605176, upload-time = "2026-05-04T17:47:59.676Z" },
+ { url = "https://files.pythonhosted.org/packages/36/99/2848df4e8ed5bf51df1d286d1718510584fa61e88adbc9c5b23d71b38f7c/apache_tvm_ffi-0.1.11-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:78aa1857b04a2ea718317041ab3f01288b3d496e6036eb1b99ebdc9da0fdaef5", size = 2725887, upload-time = "2026-05-04T17:48:01.381Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/80/963c991934a4eb0fa0c0178f51963333fe14a96b732009da642b6bf6b42e/apache_tvm_ffi-0.1.11-cp312-abi3-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a8b845c8dff498fb981c1dda36c954549204191b485a385845e604966594d0b2", size = 2513121, upload-time = "2026-05-04T17:48:03.43Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/18/95569107ee83619d61a3bb0d28743a0599f85c5161981e3e098c82c2b185/apache_tvm_ffi-0.1.11-cp312-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2843f084cdc94dedacd8b257a395a2b71b8a3dc7fc99711b148bf1d161983128", size = 2697683, upload-time = "2026-05-04T17:48:05.222Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/99/f352cf1cce8f6f05584c4adf11de9eca07e6d217229bad6af35fb372926c/apache_tvm_ffi-0.1.11-cp312-abi3-win_amd64.whl", hash = "sha256:bd67e03759d25ff59f4e0ed9c8630a16872afc9dd8792f46ac3c927554015e60", size = 2365545, upload-time = "2026-05-04T17:48:07.295Z" },
+]
+
+[[package]]
+name = "appnope"
+version = "0.1.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/35/5d/752690df9ef5b76e169e68d6a129fa6d08a7100ca7f754c89495db3c6019/appnope-0.1.4.tar.gz", hash = "sha256:1de3860566df9caf38f01f86f65e0e13e379af54f9e4bee1e66b48f2efffd1ee", size = 4170, upload-time = "2024-02-06T09:43:11.258Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/81/29/5ecc3a15d5a33e31b26c11426c45c501e439cb865d0bff96315d86443b78/appnope-0.1.4-py2.py3-none-any.whl", hash = "sha256:502575ee11cd7a28c0205f379b525beefebab9d161b7c964670864014ed7213c", size = 4321, upload-time = "2024-02-06T09:43:09.663Z" },
+]
+
+[[package]]
+name = "argon2-cffi"
+version = "25.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "argon2-cffi-bindings" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0e/89/ce5af8a7d472a67cc819d5d998aa8c82c5d860608c4db9f46f1162d7dab9/argon2_cffi-25.1.0.tar.gz", hash = "sha256:694ae5cc8a42f4c4e2bf2ca0e64e51e23a040c6a517a85074683d3959e1346c1", size = 45706, upload-time = "2025-06-03T06:55:32.073Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4f/d3/a8b22fa575b297cd6e3e3b0155c7e25db170edf1c74783d6a31a2490b8d9/argon2_cffi-25.1.0-py3-none-any.whl", hash = "sha256:fdc8b074db390fccb6eb4a3604ae7231f219aa669a2652e0f20e16ba513d5741", size = 14657, upload-time = "2025-06-03T06:55:30.804Z" },
+]
+
+[[package]]
+name = "argon2-cffi-bindings"
+version = "25.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cffi" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5c/2d/db8af0df73c1cf454f71b2bbe5e356b8c1f8041c979f505b3d3186e520a9/argon2_cffi_bindings-25.1.0.tar.gz", hash = "sha256:b957f3e6ea4d55d820e40ff76f450952807013d361a65d7f28acc0acbf29229d", size = 1783441, upload-time = "2025-07-30T10:02:05.147Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1d/57/96b8b9f93166147826da5f90376e784a10582dd39a393c99bb62cfcf52f0/argon2_cffi_bindings-25.1.0-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:aecba1723ae35330a008418a91ea6cfcedf6d31e5fbaa056a166462ff066d500", size = 54121, upload-time = "2025-07-30T10:01:50.815Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/08/a9bebdb2e0e602dde230bdde8021b29f71f7841bd54801bcfd514acb5dcf/argon2_cffi_bindings-25.1.0-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:2630b6240b495dfab90aebe159ff784d08ea999aa4b0d17efa734055a07d2f44", size = 29177, upload-time = "2025-07-30T10:01:51.681Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/02/d297943bcacf05e4f2a94ab6f462831dc20158614e5d067c35d4e63b9acb/argon2_cffi_bindings-25.1.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:7aef0c91e2c0fbca6fc68e7555aa60ef7008a739cbe045541e438373bc54d2b0", size = 31090, upload-time = "2025-07-30T10:01:53.184Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/93/44365f3d75053e53893ec6d733e4a5e3147502663554b4d864587c7828a7/argon2_cffi_bindings-25.1.0-cp39-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1e021e87faa76ae0d413b619fe2b65ab9a037f24c60a1e6cc43457ae20de6dc6", size = 81246, upload-time = "2025-07-30T10:01:54.145Z" },
+ { url = "https://files.pythonhosted.org/packages/09/52/94108adfdd6e2ddf58be64f959a0b9c7d4ef2fa71086c38356d22dc501ea/argon2_cffi_bindings-25.1.0-cp39-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d3e924cfc503018a714f94a49a149fdc0b644eaead5d1f089330399134fa028a", size = 87126, upload-time = "2025-07-30T10:01:55.074Z" },
+ { url = "https://files.pythonhosted.org/packages/72/70/7a2993a12b0ffa2a9271259b79cc616e2389ed1a4d93842fac5a1f923ffd/argon2_cffi_bindings-25.1.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c87b72589133f0346a1cb8d5ecca4b933e3c9b64656c9d175270a000e73b288d", size = 80343, upload-time = "2025-07-30T10:01:56.007Z" },
+ { url = "https://files.pythonhosted.org/packages/78/9a/4e5157d893ffc712b74dbd868c7f62365618266982b64accab26bab01edc/argon2_cffi_bindings-25.1.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1db89609c06afa1a214a69a462ea741cf735b29a57530478c06eb81dd403de99", size = 86777, upload-time = "2025-07-30T10:01:56.943Z" },
+ { url = "https://files.pythonhosted.org/packages/74/cd/15777dfde1c29d96de7f18edf4cc94c385646852e7c7b0320aa91ccca583/argon2_cffi_bindings-25.1.0-cp39-abi3-win32.whl", hash = "sha256:473bcb5f82924b1becbb637b63303ec8d10e84c8d241119419897a26116515d2", size = 27180, upload-time = "2025-07-30T10:01:57.759Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/c6/a759ece8f1829d1f162261226fbfd2c6832b3ff7657384045286d2afa384/argon2_cffi_bindings-25.1.0-cp39-abi3-win_amd64.whl", hash = "sha256:a98cd7d17e9f7ce244c0803cad3c23a7d379c301ba618a5fa76a67d116618b98", size = 31715, upload-time = "2025-07-30T10:01:58.56Z" },
+ { url = "https://files.pythonhosted.org/packages/42/b9/f8d6fa329ab25128b7e98fd83a3cb34d9db5b059a9847eddb840a0af45dd/argon2_cffi_bindings-25.1.0-cp39-abi3-win_arm64.whl", hash = "sha256:b0fdbcf513833809c882823f98dc2f931cf659d9a1429616ac3adebb49f5db94", size = 27149, upload-time = "2025-07-30T10:01:59.329Z" },
+]
+
+[[package]]
+name = "arrow"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "python-dateutil" },
+ { name = "tzdata" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b9/33/032cdc44182491aa708d06a68b62434140d8c50820a087fac7af37703357/arrow-1.4.0.tar.gz", hash = "sha256:ed0cc050e98001b8779e84d461b0098c4ac597e88704a655582b21d116e526d7", size = 152931, upload-time = "2025-10-18T17:46:46.761Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ed/c9/d7977eaacb9df673210491da99e6a247e93df98c715fc43fd136ce1d3d33/arrow-1.4.0-py3-none-any.whl", hash = "sha256:749f0769958ebdc79c173ff0b0670d59051a535fa26e8eba02953dc19eb43205", size = 68797, upload-time = "2025-10-18T17:46:45.663Z" },
+]
+
+[[package]]
+name = "ast-serialize"
+version = "0.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/58/ad/0d70a3a2d6e01968d985415259e8ec7ad3f777903f9b1c1f3c8c44642c60/ast_serialize-0.6.0.tar.gz", hash = "sha256:aadd3ffcf4858c9726bf3515f7b199c7eadbe504f96028e4a87172c0da65a8fe", size = 61489, upload-time = "2026-06-30T20:02:55.555Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/52/19/ac8348ae8711c9b5ae834634f635780cab62a0f5e6f988882e048b89c2ae/ast_serialize-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:093cb8bb91b720d8523580498d031791bb1bbaa048599c3d21085d380e11a596", size = 1185367, upload-time = "2026-06-30T20:02:30.427Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/f6/ec7ec652c51db77c2f61d8573338e13e4704303265ccc658cb4031d9f354/ast_serialize-0.6.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:e61580a69faf47e3689795367ed211f2a10fd741478cc0f36a0f128793360aad", size = 1178657, upload-time = "2026-06-30T20:02:31.964Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/02/613a7534a41d0122f37d1e0c64aa8ac78bfb831f8c92f6db057a311abb3c/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:305802f2ce2a7c4e87835078ea85c58b586ddda8095b92fe2ead9364ae19c80a", size = 1238620, upload-time = "2026-06-30T20:02:33.664Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/21/087957bba486242afc52f49b2d9e21c9dad00289356cf9efe67084015a9d/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c7b8b8f0c42f752ea00b2b7d7c090b3f80d9c1c5c75cadf16423790a0cc74081", size = 1236075, upload-time = "2026-06-30T20:02:34.936Z" },
+ { url = "https://files.pythonhosted.org/packages/82/04/78128bbb170071c2c72a210a181f1c00e11cc1cec60a8beef747b07f9201/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:cd5b91b9e6f2356ace3a556963b0cd783b395fbbb0bb17b4defc283415466e77", size = 1441348, upload-time = "2026-06-30T20:02:36.245Z" },
+ { url = "https://files.pythonhosted.org/packages/64/64/62fb99d6faf199b4c3e5b08a07136e9a0d7664bb249c6de3670e5b63e9b6/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4d6ef91590258ada18909b9caea344dac4de2013906b035473cd674a43f4b790", size = 1258580, upload-time = "2026-06-30T20:02:37.53Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/87/b4d6c38e0ccd5e85dc54cecdf933a152c60b28fe5d993a6d8a72fa6d5896/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dcbed41e9386059fc0261d602445ede0976c2ecec2939688bcbcb9ed0b6f28b7", size = 1261693, upload-time = "2026-06-30T20:02:39.123Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/4b/3676ca2191f39bafb75f93f99b2f429ec464586158fece2165f3572805dc/ast_serialize-0.6.0-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:cdc4e6f930b9090c2f92c9036ad12ffb8e6e44d4a5ba06f1458a05d60f203f7b", size = 1252517, upload-time = "2026-06-30T20:02:40.511Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/58/494ef8c4b4acb2f4a265ac934caf45f792a08fe27d6b853de35ad991941a/ast_serialize-0.6.0-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:897ac47b5637be41c0c07061c8a912fafa967ef1dc73fa115e4bfa70882a093b", size = 1304843, upload-time = "2026-06-30T20:02:41.961Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/f2/13736d920ab3d49bbee80ef1a277dd7b7aaf3b3545efd9d2a8114fe05525/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c4af9a1386166e40ed01464991806f89038a2d89782576c7774876fa77034e32", size = 1413698, upload-time = "2026-06-30T20:02:44.179Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/5a/e046f3899e2acba4677d7427b76431443a1aa1a0e583dfb05b55b69d55cf/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:c901adbd750029b9ac4ad3d6aa56853e0ad4875119fbf52b7b8298afc223828b", size = 1512209, upload-time = "2026-06-30T20:02:45.584Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/c7/e42aaca7bb2d22a7c06d5a8c7930086c5a334e93d716e6fa5e6647a4515f/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae22a366b752ab4496191525b78b097b5b72d531752e3c1dd7e383a8f2c8a1a", size = 1508464, upload-time = "2026-06-30T20:02:46.942Z" },
+ { url = "https://files.pythonhosted.org/packages/95/93/5524a3dc6c3f593de3228ed9cbef73afa047625b7000ec21b7f58e6eb4d4/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4ed29121da8b3fdc291002801a1de0f76248fa07dce89157a5f277842cf6126e", size = 1457164, upload-time = "2026-06-30T20:02:48.294Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/c0/36a6ffb4d653cf621427b4c4928671f53ad800c453474de2b82564a44ad9/ast_serialize-0.6.0-cp39-abi3-pyemscripten_2026_0_wasm32.whl", hash = "sha256:b1dac4e09d341c1300ba69cdcbe62867b32a8c75d90db9bf4d083bec3b039f0b", size = 863014, upload-time = "2026-06-30T20:02:49.742Z" },
+ { url = "https://files.pythonhosted.org/packages/09/c7/7d5ad8b49e1278e1c2a1e0274bd7850560b3f09313aa00c13bc8d5544792/ast_serialize-0.6.0-cp39-abi3-win32.whl", hash = "sha256:82c312a7844d2fdeb4d5c48bd3d215bf940dafd4704e1a9bcf252a99010a99b1", size = 1063165, upload-time = "2026-06-30T20:02:50.98Z" },
+ { url = "https://files.pythonhosted.org/packages/47/ae/6710c14ecb276031cf10249f6adf5a59e2d3fdb3b5183bd59f70524067ee/ast_serialize-0.6.0-cp39-abi3-win_amd64.whl", hash = "sha256:113b58346f9ceb664352032770caca817d4a3c86f611c6088e6ef65ddaa70f0e", size = 1101444, upload-time = "2026-06-30T20:02:52.554Z" },
+ { url = "https://files.pythonhosted.org/packages/66/40/c53deb2cd0c9b0fb636d24d9f40924cf2e65028e6b20b10cd5c1eeb2c730/ast_serialize-0.6.0-cp39-abi3-win_arm64.whl", hash = "sha256:ccd132fe8db56f61fe743b1f644d01b8d65b83248a8da506f3132bda86d6ed5e", size = 1072965, upload-time = "2026-06-30T20:02:54.097Z" },
+]
+
+[[package]]
+name = "astor"
+version = "0.8.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/5a/21/75b771132fee241dfe601d39ade629548a9626d1d39f333fde31bc46febe/astor-0.8.1.tar.gz", hash = "sha256:6a6effda93f4e1ce9f618779b2dd1d9d84f1e32812c23a29b3fff6fd7f63fa5e", size = 35090, upload-time = "2019-12-10T01:50:35.51Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c3/88/97eef84f48fa04fbd6750e62dcceafba6c63c81b7ac1420856c8dcc0a3f9/astor-0.8.1-py2.py3-none-any.whl", hash = "sha256:070a54e890cefb5b3739d19f30f5a5ec840ffc9c50ffa7d23cc9fc1a38ebbfc5", size = 27488, upload-time = "2019-12-10T01:50:33.628Z" },
+]
+
+[[package]]
+name = "asttokens"
+version = "3.0.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/be/a5/8e3f9b6771b0b408517c82d97aed8f2036509bc247d46114925e32fe33f0/asttokens-3.0.1.tar.gz", hash = "sha256:71a4ee5de0bde6a31d64f6b13f2293ac190344478f081c3d1bccfcf5eacb0cb7", size = 62308, upload-time = "2025-11-15T16:43:48.578Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/39/e7eaf1799466a4aef85b6a4fe7bd175ad2b1c6345066aa33f1f58d4b18d0/asttokens-3.0.1-py3-none-any.whl", hash = "sha256:15a3ebc0f43c2d0a50eeafea25e19046c68398e487b9f1f5b517f7c0f40f976a", size = 27047, upload-time = "2025-11-15T16:43:16.109Z" },
+]
+
+[[package]]
+name = "async-lru"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e8/1f/989ecfef8e64109a489fff357450cb73fa73a865a92bd8c272170a6922c2/async_lru-2.3.0.tar.gz", hash = "sha256:89bdb258a0140d7313cf8f4031d816a042202faa61d0ab310a0a538baa1c24b6", size = 16332, upload-time = "2026-03-19T01:04:32.413Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e5/e2/c2e3abf398f80732e58b03be77bde9022550d221dd8781bf586bd4d97cc1/async_lru-2.3.0-py3-none-any.whl", hash = "sha256:eea27b01841909316f2cc739807acea1c623df2be8c5cfad7583286397bb8315", size = 8403, upload-time = "2026-03-19T01:04:30.883Z" },
+]
+
+[[package]]
+name = "attrs"
+version = "26.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" },
+]
+
+[[package]]
+name = "babel"
+version = "2.18.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" },
+]
+
+[[package]]
+name = "beautifulsoup4"
+version = "4.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "soupsieve" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/43/65/318323f98dbee45d42dff61d8f047181bc6f2268a9068cfad035a46be5af/beautifulsoup4-4.15.0.tar.gz", hash = "sha256:288e3ca7d54b06f2ac191970bc275c1939cb46d450b255bf6718b04aa37ab4f7", size = 632571, upload-time = "2026-06-07T16:44:20.453Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/c6/92fcd42f1ba33e1184263f25bfabf3d27c383410470f169e4b8163bf9c17/beautifulsoup4-4.15.0-py3-none-any.whl", hash = "sha256:d6f88de62e1d4e38ecb1077eb9724cd0eff29d2a08ca16a401e9b9e93f117cf9", size = 109924, upload-time = "2026-06-07T16:44:21.566Z" },
+]
+
+[[package]]
+name = "blake3"
+version = "1.0.9"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/6a/4cc5a9dd40fd8a6d283fd3761e5f59c490109571ef8e3c73245417e5a305/blake3-1.0.9.tar.gz", hash = "sha256:5fa374fa5070ca084368776c19b420157eb0f2d3f091343d6bc59189929d62e2", size = 116872, upload-time = "2026-06-22T18:02:25.366Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/1b/95b473d649f5322e69674622a307ffdb4f0b63adb0a0adcbc5cb8a8833c2/blake3-1.0.9-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:69ff5aebc7650954443aa701feff2028d7c7ea5b5e18ee265f15e2104e892328", size = 343869, upload-time = "2026-06-22T18:00:51.936Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/9d/adec22c719d8451af1dc9e624bf5907008ef1e0afa51aa69fd1e8c91e60e/blake3-1.0.9-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0cdfeff65488089ef86f7587c76055ff72b28d28d10e427b547f5711477c376d", size = 328482, upload-time = "2026-06-22T18:00:53.39Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/aa/0a6967ff9a6ae182419a681aed54f7338b34a1f71372e90f787a2afa42e6/blake3-1.0.9-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:766f1555cbe614f14f399c2fbec0983568d20edb36837ba04040807eb9e1a609", size = 373616, upload-time = "2026-06-22T18:00:54.701Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/51/5d4e198bf3ae902c6697ad6ec77d7210736ad8f680980e8b648dcfcd09a0/blake3-1.0.9-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:128a62136c9a39c7cb9fdaa5fb38471f2418853da7f5a89f31495735d0ba6f2c", size = 374149, upload-time = "2026-06-22T18:00:56.015Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/62/d3c7c364925b3f10828e5137376f3947f112c32188e899b42f09c2fde98a/blake3-1.0.9-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d1ea0bf17b184b03444007646d902207d2b4d4f3e91a0cac3836552d83db74b9", size = 446151, upload-time = "2026-06-22T18:00:57.378Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/01/55b89389c5036c9d24b1d762d6265e91552e10b76a3c99fece3c4a7a4783/blake3-1.0.9-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:73a48f7e9f0e047f51a445d9b0361ab1907bdc72b6857815a84dacd2e59556f8", size = 487256, upload-time = "2026-06-22T18:00:58.763Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/7a/a21b52253292ad3e4df63ea4a01ce11d3ee8f4a8a8d80eaf0c7ce92a62bd/blake3-1.0.9-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b27550ada40f839aca64c66127940e4318bb6ef3e291890ef913017f6f637448", size = 383977, upload-time = "2026-06-22T18:01:00.192Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/f0/fe7188201a29ee9b042616c786a98afd864d537ca96198e64c3fe4ff13a9/blake3-1.0.9-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:66c84dbc2a31eda88b55bbf5c5b711037bf0698eba0fd1faf06bdaf313c39048", size = 383615, upload-time = "2026-06-22T18:01:01.557Z" },
+ { url = "https://files.pythonhosted.org/packages/22/08/f6a213b950e30fe9ef7d7fc061ec388e66ed62643570226882e6f7136ea3/blake3-1.0.9-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:dab59b324aa65c09e937d6c43de5de85ec9581627f4e79dcc9806d85b54a1c34", size = 380288, upload-time = "2026-06-22T18:01:03.025Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/fb/b171e47c1b835483bcf1545ebc289458165f8dc0f5c7f74a9176d7e9af03/blake3-1.0.9-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:eca281fedcbe5c56655bd5a4176e6036eddbbe57df96114a03838fce08b1e0ca", size = 549122, upload-time = "2026-06-22T18:01:04.486Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/d8/7bf71c2c85a0951e406971f151435e0751716907e3924c6c48a2d6dae0db/blake3-1.0.9-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:3cbe7f190164896dc3908e920716ee66bc31d40f1a0fb603ed59ac53290fb9cf", size = 591183, upload-time = "2026-06-22T18:01:06.259Z" },
+ { url = "https://files.pythonhosted.org/packages/20/85/34c3ea03cc90b2516628494ab3e0a98aec4ca8b04d037840ccd390e480ca/blake3-1.0.9-cp313-cp313-win32.whl", hash = "sha256:508ccaf8f9377cc47e6026c2897fdc37de61faeb1420dc023b6379cc2474eb65", size = 229053, upload-time = "2026-06-22T18:01:07.638Z" },
+ { url = "https://files.pythonhosted.org/packages/db/2e/f09e8ed426f360aa2005206466ceab2f707486eb5d9db7051dbcbae056d1/blake3-1.0.9-cp313-cp313-win_amd64.whl", hash = "sha256:caded2806d2cbeed638c5e2517ed8b2a94165b3452fda35e72896142d22070e0", size = 217589, upload-time = "2026-06-22T18:01:08.922Z" },
+]
+
+[[package]]
+name = "bleach"
+version = "6.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "webencodings" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/48/3c/e12ac860709702bd5ebeb9b56a4fe334f1001246ee1b8f2b7ee28912df7d/bleach-6.4.0.tar.gz", hash = "sha256:4202482733d85cedd04e59fcb2f89f4e4c7c385a78d3c3c23c30446843a37452", size = 204857, upload-time = "2026-06-05T13:01:13.734Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/58/9d/40b6267367182187139a4000b82a3b287d84d745bccd808e75d916920e9d/bleach-6.4.0-py3-none-any.whl", hash = "sha256:4b6b6a54fff2e69a3dde9d21cc6301220bee3c3cb792187d11403fd795031081", size = 165109, upload-time = "2026-06-05T13:01:12.504Z" },
+]
+
+[package.optional-dependencies]
+css = [
+ { name = "tinycss2" },
+]
+
+[[package]]
+name = "blis"
+version = "1.3.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d0/d0/d8cc8c9a4488a787e7fa430f6055e5bd1ddb22c340a751d9e901b82e2efe/blis-1.3.3.tar.gz", hash = "sha256:034d4560ff3cc43e8aa37e188451b0440e3261d989bb8a42ceee865607715ecd", size = 2644873, upload-time = "2025-11-17T12:28:30.511Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e6/f7/d26e62d9be3d70473a63e0a5d30bae49c2fe138bebac224adddcdef8a7ce/blis-1.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:1e647341f958421a86b028a2efe16ce19c67dba2a05f79e8f7e80b1ff45328aa", size = 6928322, upload-time = "2025-11-17T12:27:57.965Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/78/750d12da388f714958eb2f2fd177652323bbe7ec528365c37129edd6eb84/blis-1.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d563160f874abb78a57e346f07312c5323f7ad67b6370052b6b17087ef234a8e", size = 1229635, upload-time = "2025-11-17T12:28:00.118Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/36/eac4199c5b200a5f3e93cad197da8d26d909f218eb444c4f552647c95240/blis-1.3.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:30b8a5b90cb6cb81d1ada9ae05aa55fb8e70d9a0ae9db40d2401bb9c1c8f14c4", size = 2815650, upload-time = "2025-11-17T12:28:02.544Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/51/472e7b36a6bedb5242a9757e7486f702c3619eff76e256735d0c8b1679c6/blis-1.3.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e9f5c53b277f6ac5b3ca30bc12ebab7ea16c8f8c36b14428abb56924213dc127", size = 11359008, upload-time = "2025-11-17T12:28:04.589Z" },
+ { url = "https://files.pythonhosted.org/packages/84/da/d0dfb6d6e6321ae44df0321384c32c322bd07b15740d7422727a1a49fc5d/blis-1.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6297e7616c158b305c9a8a4e47ca5fc9b0785194dd96c903b1a1591a7ca21ddf", size = 3011959, upload-time = "2025-11-17T12:28:06.862Z" },
+ { url = "https://files.pythonhosted.org/packages/20/c5/2b0b5e556fa0364ed671051ea078a6d6d7b979b1cfef78d64ad3ca5f0c7f/blis-1.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3f966ca74f89f8a33e568b9a1d71992fc9a0d29a423e047f0a212643e21b5458", size = 14232456, upload-time = "2025-11-17T12:28:08.779Z" },
+ { url = "https://files.pythonhosted.org/packages/31/07/4cdc81a47bf862c0b06d91f1bc6782064e8b69ac9b5d4ff51d97e4ff03da/blis-1.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:7a0fc4b237a3a453bdc3c7ab48d91439fcd2d013b665c46948d9eaf9c3e45a97", size = 6192624, upload-time = "2025-11-17T12:28:14.197Z" },
+]
+
+[[package]]
+name = "boto3"
+version = "1.40.61"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "botocore" },
+ { name = "jmespath" },
+ { name = "s3transfer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ed/f9/6ef8feb52c3cce5ec3967a535a6114b57ac7949fd166b0f3090c2b06e4e5/boto3-1.40.61.tar.gz", hash = "sha256:d6c56277251adf6c2bdd25249feae625abe4966831676689ff23b4694dea5b12", size = 111535, upload-time = "2025-10-28T19:26:57.247Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/61/24/3bf865b07d15fea85b63504856e137029b6acbc73762496064219cdb265d/boto3-1.40.61-py3-none-any.whl", hash = "sha256:6b9c57b2a922b5d8c17766e29ed792586a818098efe84def27c8f582b33f898c", size = 139321, upload-time = "2025-10-28T19:26:55.007Z" },
+]
+
+[[package]]
+name = "botocore"
+version = "1.40.61"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jmespath" },
+ { name = "python-dateutil" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/a3/81d3a47c2dbfd76f185d3b894f2ad01a75096c006a2dd91f237dca182188/botocore-1.40.61.tar.gz", hash = "sha256:a2487ad69b090f9cccd64cf07c7021cd80ee9c0655ad974f87045b02f3ef52cd", size = 14393956, upload-time = "2025-10-28T19:26:46.108Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/c5/f6ce561004db45f0b847c2cd9b19c67c6bf348a82018a48cb718be6b58b0/botocore-1.40.61-py3-none-any.whl", hash = "sha256:17ebae412692fd4824f99cde0f08d50126dc97954008e5ba2b522eb049238aa7", size = 14055973, upload-time = "2025-10-28T19:26:42.15Z" },
+]
+
+[[package]]
+name = "cachetools"
+version = "7.1.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f4/8b/0d3945a13955303b81272f759a0331e54c5c793da455e6f5706b89d2639c/cachetools-7.1.4.tar.gz", hash = "sha256:437f55a4e0c1b01a4f3077cc470e6991d47430970e36fbcb77e2be0df4fc1cd6", size = 40085, upload-time = "2026-05-21T22:40:43.376Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8c/7b/1fc1c09cc0756cf25861a3be10565915953876da48bb228fb9a672b20a42/cachetools-7.1.4-py3-none-any.whl", hash = "sha256:323dc4127934744db5b54eb4924482d7edafbf9554e820d1531c2e08c0e4ef54", size = 16761, upload-time = "2026-05-21T22:40:41.845Z" },
+]
+
+[[package]]
+name = "catalogue"
+version = "2.0.10"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/38/b4/244d58127e1cdf04cf2dc7d9566f0d24ef01d5ce21811bab088ecc62b5ea/catalogue-2.0.10.tar.gz", hash = "sha256:4f56daa940913d3f09d589c191c74e5a6d51762b3a9e37dd53b7437afd6cda15", size = 19561, upload-time = "2023-09-25T06:29:24.962Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/96/d32b941a501ab566a16358d68b6eb4e4acc373fab3c3c4d7d9e649f7b4bb/catalogue-2.0.10-py3-none-any.whl", hash = "sha256:58c2de0020aa90f4a2da7dfad161bf7b3b054c86a5f09fcedc0b2b740c109a9f", size = 17325, upload-time = "2023-09-25T06:29:23.337Z" },
+]
+
+[[package]]
+name = "cbor2"
+version = "6.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/75/af/473c241e41c142ea06ebef8d1f660fa6ff928fb97210e7bec8ee5974f8cd/cbor2-6.1.2.tar.gz", hash = "sha256:6b43037a66947dee5af0abb1a4c3a13b3abac5a4a3f32f9771efbbcd030fd909", size = 86760, upload-time = "2026-06-02T19:01:29.333Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2b/dc/bc045c8f36317e4e5f7a60d94b36833139909fc32e3a65f44bc61a36def0/cbor2-6.1.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f1aa38c422d87ea61849b2a823b10b64053fb4da8763f19ac78ea9a69d682b2a", size = 408846, upload-time = "2026-06-02T19:00:55.476Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/36/d66f5f0dd98ecbdcfc7da1fbd423f7b3782a27719f0062a560476f00b334/cbor2-6.1.2-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:ff7d0bd8ff432832338a8d2430aee34f8a082342480ff537c0ba90e2b8ff7894", size = 454624, upload-time = "2026-06-02T19:00:56.744Z" },
+ { url = "https://files.pythonhosted.org/packages/38/6b/4884b9cf03db14dc5007825d5d1bf8678a75c49d4268d8e0c1c6e9580104/cbor2-6.1.2-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:c1eedf3290d88a5f663bd8b4b8f0f0e2103d0594c293fa5f4e62e53100972309", size = 466585, upload-time = "2026-06-02T19:00:58.209Z" },
+ { url = "https://files.pythonhosted.org/packages/50/f6/36a15beb3915f56a79d6e9213c6d40c0f5cb90cd3462923f555d78068847/cbor2-6.1.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3049b04bddf9a5a2d0e5bb25dccdaf4552fcaf607b404e249d4f78f010fcc7d0", size = 521678, upload-time = "2026-06-02T19:00:59.524Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/3f/e899313371ebeb7a191d751de97ccd8242abc24bbc9d8e2c58e04475cfb0/cbor2-6.1.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:96eb687a62040401668f06a85de8f47361ef44574de1493899e0ec678109fc04", size = 534044, upload-time = "2026-06-02T19:01:00.875Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/5e/1a872acdeb1ab9a884ec3460f73a43e02154dc20d8ccb627bbd60f4c0ea1/cbor2-6.1.2-cp313-cp313-win32.whl", hash = "sha256:03440b505882280023db1fedcee6844804e9968bb50f9eb4ff12aaf27777fcfe", size = 282328, upload-time = "2026-06-02T19:01:02.347Z" },
+ { url = "https://files.pythonhosted.org/packages/70/79/29721bc15d38889e7bec214ede2346ee15970bedcc5e6ce1fa30f21e9a4e/cbor2-6.1.2-cp313-cp313-win_amd64.whl", hash = "sha256:d2c8da2c0f821827dcc9eb59a5c9351791a8aa3b389a2ea7ca64c4f97bcb94cf", size = 300313, upload-time = "2026-06-02T19:01:03.69Z" },
+ { url = "https://files.pythonhosted.org/packages/07/98/a13b424fb2f14fe332b57f71f479953b2f291a051f797d42ddab9fcd2027/cbor2-6.1.2-cp313-cp313-win_arm64.whl", hash = "sha256:8e1478d3b980ddfcaf56e27cecbfe13057e0f67d5e8240fe8a398815acb9c4bf", size = 288725, upload-time = "2026-06-02T19:01:04.933Z" },
+]
+
+[[package]]
+name = "certifi"
+version = "2026.6.17"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c9/c7/424b75da314c1045981bd9777432fad05a9e0c69daa4ed7e308bbaffe405/certifi-2026.6.17.tar.gz", hash = "sha256:024c88eeec92ca068db80f02b8b07c9cef7b9fe261d1d535abfd5abd6f6af432", size = 134594, upload-time = "2026-06-17T10:31:07.894Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ef/2f/c5464532e965badff2f4c4c1a3a83f5697f0d7c407ed0cda44aaa99bb451/certifi-2026.6.17-py3-none-any.whl", hash = "sha256:2227dcbaafe0d2f59279d1762ddddc37783ed4354594f194ffc31d20f41fc3db", size = 133289, upload-time = "2026-06-17T10:31:06.348Z" },
+]
+
+[[package]]
+name = "cffi"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pycparser", marker = "implementation_name != 'PyPy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/eb/56/b1ba7935a17738ae8453301356628e8147c79dbb825bcbc73dc7401f9846/cffi-2.0.0.tar.gz", hash = "sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529", size = 523588, upload-time = "2025-09-08T23:24:04.541Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4b/8d/a0a47a0c9e413a658623d014e91e74a50cdd2c423f7ccfd44086ef767f90/cffi-2.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:00bdf7acc5f795150faa6957054fbbca2439db2f775ce831222b66f192f03beb", size = 185230, upload-time = "2025-09-08T23:23:00.879Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/d2/a6c0296814556c68ee32009d9c2ad4f85f2707cdecfd7727951ec228005d/cffi-2.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:45d5e886156860dc35862657e1494b9bae8dfa63bf56796f2fb56e1679fc0bca", size = 181043, upload-time = "2025-09-08T23:23:02.231Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/1e/d22cc63332bd59b06481ceaac49d6c507598642e2230f201649058a7e704/cffi-2.0.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:07b271772c100085dd28b74fa0cd81c8fb1a3ba18b21e03d7c27f3436a10606b", size = 212446, upload-time = "2025-09-08T23:23:03.472Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/f5/a2c23eb03b61a0b8747f211eb716446c826ad66818ddc7810cc2cc19b3f2/cffi-2.0.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d48a880098c96020b02d5a1f7d9251308510ce8858940e6fa99ece33f610838b", size = 220101, upload-time = "2025-09-08T23:23:04.792Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/7f/e6647792fc5850d634695bc0e6ab4111ae88e89981d35ac269956605feba/cffi-2.0.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f93fd8e5c8c0a4aa1f424d6173f14a892044054871c771f8566e4008eaa359d2", size = 207948, upload-time = "2025-09-08T23:23:06.127Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/1e/a5a1bd6f1fb30f22573f76533de12a00bf274abcdc55c8edab639078abb6/cffi-2.0.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:dd4f05f54a52fb558f1ba9f528228066954fee3ebe629fc1660d874d040ae5a3", size = 206422, upload-time = "2025-09-08T23:23:07.753Z" },
+ { url = "https://files.pythonhosted.org/packages/98/df/0a1755e750013a2081e863e7cd37e0cdd02664372c754e5560099eb7aa44/cffi-2.0.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c8d3b5532fc71b7a77c09192b4a5a200ea992702734a2e9279a37f2478236f26", size = 219499, upload-time = "2025-09-08T23:23:09.648Z" },
+ { url = "https://files.pythonhosted.org/packages/50/e1/a969e687fcf9ea58e6e2a928ad5e2dd88cc12f6f0ab477e9971f2309b57c/cffi-2.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d9b29c1f0ae438d5ee9acb31cadee00a58c46cc9c0b2f9038c6b0b3470877a8c", size = 222928, upload-time = "2025-09-08T23:23:10.928Z" },
+ { url = "https://files.pythonhosted.org/packages/36/54/0362578dd2c9e557a28ac77698ed67323ed5b9775ca9d3fe73fe191bb5d8/cffi-2.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6d50360be4546678fc1b79ffe7a66265e28667840010348dd69a314145807a1b", size = 221302, upload-time = "2025-09-08T23:23:12.42Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/6d/bf9bda840d5f1dfdbf0feca87fbdb64a918a69bca42cfa0ba7b137c48cb8/cffi-2.0.0-cp313-cp313-win32.whl", hash = "sha256:74a03b9698e198d47562765773b4a8309919089150a0bb17d829ad7b44b60d27", size = 172909, upload-time = "2025-09-08T23:23:14.32Z" },
+ { url = "https://files.pythonhosted.org/packages/37/18/6519e1ee6f5a1e579e04b9ddb6f1676c17368a7aba48299c3759bbc3c8b3/cffi-2.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:19f705ada2530c1167abacb171925dd886168931e0a7b78f5bffcae5c6b5be75", size = 183402, upload-time = "2025-09-08T23:23:15.535Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/0e/02ceeec9a7d6ee63bb596121c2c8e9b3a9e150936f4fbef6ca1943e6137c/cffi-2.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:256f80b80ca3853f90c21b23ee78cd008713787b1b1e93eae9f3d6a7134abd91", size = 177780, upload-time = "2025-09-08T23:23:16.761Z" },
+]
+
+[[package]]
+name = "cfgv"
+version = "3.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4e/b5/721b8799b04bf9afe054a3899c6cf4e880fcf8563cc71c15610242490a0c/cfgv-3.5.0.tar.gz", hash = "sha256:d5b1034354820651caa73ede66a6294d6e95c1b00acc5e9b098e917404669132", size = 7334, upload-time = "2025-11-19T20:55:51.612Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl", hash = "sha256:a8dc6b26ad22ff227d2634a65cb388215ce6cc96bbcc5cfde7641ae87e8dacc0", size = 7445, upload-time = "2025-11-19T20:55:50.744Z" },
+]
+
+[[package]]
+name = "chardet"
+version = "6.0.0.post1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7f/42/fb9436c103a881a377e34b9f58d77b5f503461c702ff654ebe86151bcfe9/chardet-6.0.0.post1.tar.gz", hash = "sha256:6b78048c3c97c7b2ed1fbad7a18f76f5a6547f7d34dbab536cc13887c9a92fa4", size = 12521798, upload-time = "2026-02-22T15:09:17.925Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/66/42/5de54f632c2de53cd3415b3703383d5fff43a94cbc0567ef362515261a21/chardet-6.0.0.post1-py3-none-any.whl", hash = "sha256:c894a36800549adf7bb5f2af47033281b75fdfcd2aa0f0243be0ad22a52e2dcb", size = 627245, upload-time = "2026-02-22T15:09:15.876Z" },
+]
+
+[[package]]
+name = "charset-normalizer"
+version = "3.4.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e7/a1/67fe25fac3c7642725500a3f6cfe5821ad557c3abb11c9d20d12c7008d3e/charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5", size = 144271, upload-time = "2026-04-02T09:28:39.342Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c1/3b/66777e39d3ae1ddc77ee606be4ec6d8cbd4c801f65e5a1b6f2b11b8346dd/charset_normalizer-3.4.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f496c9c3cc02230093d8330875c4c3cdfc3b73612a5fd921c65d39cbcef08063", size = 309627, upload-time = "2026-04-02T09:26:45.198Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/4e/b7f84e617b4854ade48a1b7915c8ccfadeba444d2a18c291f696e37f0d3b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea948db76d31190bf08bd371623927ee1339d5f2a0b4b1b4a4439a65298703c", size = 207008, upload-time = "2026-04-02T09:26:46.824Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/bb/ec73c0257c9e11b268f018f068f5d00aa0ef8c8b09f7753ebd5f2880e248/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a277ab8928b9f299723bc1a2dabb1265911b1a76341f90a510368ca44ad9ab66", size = 228303, upload-time = "2026-04-02T09:26:48.397Z" },
+ { url = "https://files.pythonhosted.org/packages/85/fb/32d1f5033484494619f701e719429c69b766bfc4dbc61aa9e9c8c166528b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3bec022aec2c514d9cf199522a802bd007cd588ab17ab2525f20f9c34d067c18", size = 224282, upload-time = "2026-04-02T09:26:49.684Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/07/330e3a0dda4c404d6da83b327270906e9654a24f6c546dc886a0eb0ffb23/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e044c39e41b92c845bc815e5ae4230804e8e7bc29e399b0437d64222d92809dd", size = 215595, upload-time = "2026-04-02T09:26:50.915Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/7c/fc890655786e423f02556e0216d4b8c6bcb6bdfa890160dc66bf52dee468/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:f495a1652cf3fbab2eb0639776dad966c2fb874d79d87ca07f9d5f059b8bd215", size = 201986, upload-time = "2026-04-02T09:26:52.197Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/97/bfb18b3db2aed3b90cf54dc292ad79fdd5ad65c4eae454099475cbeadd0d/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e712b419df8ba5e42b226c510472b37bd57b38e897d3eca5e8cfd410a29fa859", size = 211711, upload-time = "2026-04-02T09:26:53.49Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/a5/a581c13798546a7fd557c82614a5c65a13df2157e9ad6373166d2a3e645d/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7804338df6fcc08105c7745f1502ba68d900f45fd770d5bdd5288ddccb8a42d8", size = 210036, upload-time = "2026-04-02T09:26:54.975Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/bf/b3ab5bcb478e4193d517644b0fb2bf5497fbceeaa7a1bc0f4d5b50953861/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:481551899c856c704d58119b5025793fa6730adda3571971af568f66d2424bb5", size = 202998, upload-time = "2026-04-02T09:26:56.303Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/4e/23efd79b65d314fa320ec6017b4b5834d5c12a58ba4610aa353af2e2f577/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:f59099f9b66f0d7145115e6f80dd8b1d847176df89b234a5a6b3f00437aa0832", size = 230056, upload-time = "2026-04-02T09:26:57.554Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/9f/1e1941bc3f0e01df116e68dc37a55c4d249df5e6fa77f008841aef68264f/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6", size = 211537, upload-time = "2026-04-02T09:26:58.843Z" },
+ { url = "https://files.pythonhosted.org/packages/80/0f/088cbb3020d44428964a6c97fe1edfb1b9550396bf6d278330281e8b709c/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:3dedcc22d73ec993f42055eff4fcfed9318d1eeb9a6606c55892a26964964e48", size = 226176, upload-time = "2026-04-02T09:27:00.437Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/9f/130394f9bbe06f4f63e22641d32fc9b202b7e251c9aef4db044324dac493/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:64f02c6841d7d83f832cd97ccf8eb8a906d06eb95d5276069175c696b024b60a", size = 217723, upload-time = "2026-04-02T09:27:02.021Z" },
+ { url = "https://files.pythonhosted.org/packages/73/55/c469897448a06e49f8fa03f6caae97074fde823f432a98f979cc42b90e69/charset_normalizer-3.4.7-cp313-cp313-win32.whl", hash = "sha256:4042d5c8f957e15221d423ba781e85d553722fc4113f523f2feb7b188cc34c5e", size = 148085, upload-time = "2026-04-02T09:27:03.192Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/78/1b74c5bbb3f99b77a1715c91b3e0b5bdb6fe302d95ace4f5b1bec37b0167/charset_normalizer-3.4.7-cp313-cp313-win_amd64.whl", hash = "sha256:3946fa46a0cf3e4c8cb1cc52f56bb536310d34f25f01ca9b6c16afa767dab110", size = 158819, upload-time = "2026-04-02T09:27:04.454Z" },
+ { url = "https://files.pythonhosted.org/packages/68/86/46bd42279d323deb8687c4a5a811fd548cb7d1de10cf6535d099877a9a9f/charset_normalizer-3.4.7-cp313-cp313-win_arm64.whl", hash = "sha256:80d04837f55fc81da168b98de4f4b797ef007fc8a79ab71c6ec9bc4dd662b15b", size = 147915, upload-time = "2026-04-02T09:27:05.971Z" },
+ { url = "https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d", size = 61958, upload-time = "2026-04-02T09:28:37.794Z" },
+]
+
+[[package]]
+name = "click"
+version = "8.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" },
+]
+
+[[package]]
+name = "cloudpathlib"
+version = "0.24.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/06/19/58bc6b5d7d0f81c7209b05445af477e147c486552f96665a5912211839b9/cloudpathlib-0.24.0.tar.gz", hash = "sha256:c521a984e77b47e656fe78e20a7e3e260e0ab45fc69e33ac01094227c979e34a", size = 53600, upload-time = "2026-04-30T00:54:43.265Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c2/5b/ba933f896d9b0b07608d575a8501e2b4e32166b60d84c430a4a7285ebe64/cloudpathlib-0.24.0-py3-none-any.whl", hash = "sha256:b1c51e2d2ec7dc4fed6538991f4aea849d6cf11a7e6b9069f86e461aa1f9b5b4", size = 63214, upload-time = "2026-04-30T00:54:42.06Z" },
+]
+
+[[package]]
+name = "cloudpickle"
+version = "3.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/27/fb/576f067976d320f5f0114a8d9fa1215425441bb35627b1993e5afd8111e5/cloudpickle-3.1.2.tar.gz", hash = "sha256:7fda9eb655c9c230dab534f1983763de5835249750e85fbcef43aaa30a9a2414", size = 22330, upload-time = "2025-11-03T09:25:26.604Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl", hash = "sha256:9acb47f6afd73f60dc1df93bb801b472f05ff42fa6c84167d25cb206be1fbf4a", size = 22228, upload-time = "2025-11-03T09:25:25.534Z" },
+]
+
+[[package]]
+name = "colorama"
+version = "0.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
+]
+
+[[package]]
+name = "colorlog"
+version = "6.10.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a2/61/f083b5ac52e505dfc1c624eafbf8c7589a0d7f32daa398d2e7590efa5fda/colorlog-6.10.1.tar.gz", hash = "sha256:eb4ae5cb65fe7fec7773c2306061a8e63e02efc2c72eba9d27b0fa23c94f1321", size = 17162, upload-time = "2025-10-16T16:14:11.978Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6d/c1/e419ef3723a074172b68aaa89c9f3de486ed4c2399e2dbd8113a4fdcaf9e/colorlog-6.10.1-py3-none-any.whl", hash = "sha256:2d7e8348291948af66122cff006c9f8da6255d224e7cf8e37d8de2df3bad8c9c", size = 11743, upload-time = "2025-10-16T16:14:10.512Z" },
+]
+
+[[package]]
+name = "comm"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4c/13/7d740c5849255756bc17888787313b61fd38a0a8304fc4f073dfc46122aa/comm-0.2.3.tar.gz", hash = "sha256:2dc8048c10962d55d7ad693be1e7045d891b7ce8d999c97963a5e3e99c055971", size = 6319, upload-time = "2025-07-25T14:02:04.452Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/60/97/891a0971e1e4a8c5d2b20bbe0e524dc04548d2307fee33cdeba148fd4fc7/comm-0.2.3-py3-none-any.whl", hash = "sha256:c615d91d75f7f04f095b30d1c1711babd43bdc6419c1be9886a85f2f4e489417", size = 7294, upload-time = "2025-07-25T14:02:02.896Z" },
+]
+
+[[package]]
+name = "compressed-tensors"
+version = "0.17.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "loguru" },
+ { name = "pydantic" },
+ { name = "torch" },
+ { name = "transformers" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2c/9e/d7f18bd9a0354088abc11a0c1f2c7698f7c49e5a709faedf6a46e388f693/compressed_tensors-0.17.0.tar.gz", hash = "sha256:15c20d06bdbcf35b51fc99fd125e7b9be1e1855567c33b7a46dfac26ad6fb126", size = 257091, upload-time = "2026-06-03T16:49:17.208Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/35/63/6edf0415b072fff0bf8b546074dea3f0f9b148e49b601ac98bdc60a76c68/compressed_tensors-0.17.0-py3-none-any.whl", hash = "sha256:4a1b89b508f7efb8ffb4eee8a6e69e0452d9b080cae130146025c64fbe9fa9aa", size = 211714, upload-time = "2026-06-03T16:49:15.672Z" },
+]
+
+[[package]]
+name = "confection"
+version = "1.3.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ca/65/efd0fe8a936fc8ca2978cb7b82581fb20d901c6039e746a808f746b7647b/confection-1.3.3.tar.gz", hash = "sha256:f0f6810d567ff73993fe74d218ca5e1ffb6a44fb03f391257fc5d033546cbfaa", size = 54895, upload-time = "2026-03-24T18:45:24.331Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8d/e4/d66708bdf0d92fb4d49b22cdff4b10cec38aca5dcd7e81d909bb55c65cd7/confection-1.3.3-py3-none-any.whl", hash = "sha256:b9fef9ee84b237ef4611ec3eb5797b70e13063e6310ad9f15536373f5e313c82", size = 35902, upload-time = "2026-03-24T18:45:22.664Z" },
+]
+
+[[package]]
+name = "contourpy"
+version = "1.3.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/58/01/1253e6698a07380cd31a736d248a3f2a50a7c88779a1813da27503cadc2a/contourpy-1.3.3.tar.gz", hash = "sha256:083e12155b210502d0bca491432bb04d56dc3432f95a979b429f2848c3dbe880", size = 13466174, upload-time = "2025-07-26T12:03:12.549Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/68/35/0167aad910bbdb9599272bd96d01a9ec6852f36b9455cf2ca67bd4cc2d23/contourpy-1.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:177fb367556747a686509d6fef71d221a4b198a3905fe824430e5ea0fda54eb5", size = 293257, upload-time = "2025-07-26T12:01:39.367Z" },
+ { url = "https://files.pythonhosted.org/packages/96/e4/7adcd9c8362745b2210728f209bfbcf7d91ba868a2c5f40d8b58f54c509b/contourpy-1.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d002b6f00d73d69333dac9d0b8d5e84d9724ff9ef044fd63c5986e62b7c9e1b1", size = 274034, upload-time = "2025-07-26T12:01:40.645Z" },
+ { url = "https://files.pythonhosted.org/packages/73/23/90e31ceeed1de63058a02cb04b12f2de4b40e3bef5e082a7c18d9c8ae281/contourpy-1.3.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:348ac1f5d4f1d66d3322420f01d42e43122f43616e0f194fc1c9f5d830c5b286", size = 334672, upload-time = "2025-07-26T12:01:41.942Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/93/b43d8acbe67392e659e1d984700e79eb67e2acb2bd7f62012b583a7f1b55/contourpy-1.3.3-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:655456777ff65c2c548b7c454af9c6f33f16c8884f11083244b5819cc214f1b5", size = 381234, upload-time = "2025-07-26T12:01:43.499Z" },
+ { url = "https://files.pythonhosted.org/packages/46/3b/bec82a3ea06f66711520f75a40c8fc0b113b2a75edb36aa633eb11c4f50f/contourpy-1.3.3-cp313-cp313-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:644a6853d15b2512d67881586bd03f462c7ab755db95f16f14d7e238f2852c67", size = 385169, upload-time = "2025-07-26T12:01:45.219Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/32/e0f13a1c5b0f8572d0ec6ae2f6c677b7991fafd95da523159c19eff0696a/contourpy-1.3.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4debd64f124ca62069f313a9cb86656ff087786016d76927ae2cf37846b006c9", size = 362859, upload-time = "2025-07-26T12:01:46.519Z" },
+ { url = "https://files.pythonhosted.org/packages/33/71/e2a7945b7de4e58af42d708a219f3b2f4cff7386e6b6ab0a0fa0033c49a9/contourpy-1.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a15459b0f4615b00bbd1e91f1b9e19b7e63aea7483d03d804186f278c0af2659", size = 1332062, upload-time = "2025-07-26T12:01:48.964Z" },
+ { url = "https://files.pythonhosted.org/packages/12/fc/4e87ac754220ccc0e807284f88e943d6d43b43843614f0a8afa469801db0/contourpy-1.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ca0fdcd73925568ca027e0b17ab07aad764be4706d0a925b89227e447d9737b7", size = 1403932, upload-time = "2025-07-26T12:01:51.979Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/2e/adc197a37443f934594112222ac1aa7dc9a98faf9c3842884df9a9d8751d/contourpy-1.3.3-cp313-cp313-win32.whl", hash = "sha256:b20c7c9a3bf701366556e1b1984ed2d0cedf999903c51311417cf5f591d8c78d", size = 185024, upload-time = "2025-07-26T12:01:53.245Z" },
+ { url = "https://files.pythonhosted.org/packages/18/0b/0098c214843213759692cc638fce7de5c289200a830e5035d1791d7a2338/contourpy-1.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:1cadd8b8969f060ba45ed7c1b714fe69185812ab43bd6b86a9123fe8f99c3263", size = 226578, upload-time = "2025-07-26T12:01:54.422Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/9a/2f6024a0c5995243cd63afdeb3651c984f0d2bc727fd98066d40e141ad73/contourpy-1.3.3-cp313-cp313-win_arm64.whl", hash = "sha256:fd914713266421b7536de2bfa8181aa8c699432b6763a0ea64195ebe28bff6a9", size = 193524, upload-time = "2025-07-26T12:01:55.73Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/b3/f8a1a86bd3298513f500e5b1f5fd92b69896449f6cab6a146a5d52715479/contourpy-1.3.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:88df9880d507169449d434c293467418b9f6cbe82edd19284aa0409e7fdb933d", size = 306730, upload-time = "2025-07-26T12:01:57.051Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/11/4780db94ae62fc0c2053909b65dc3246bd7cecfc4f8a20d957ad43aa4ad8/contourpy-1.3.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d06bb1f751ba5d417047db62bca3c8fde202b8c11fb50742ab3ab962c81e8216", size = 287897, upload-time = "2025-07-26T12:01:58.663Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/15/e59f5f3ffdd6f3d4daa3e47114c53daabcb18574a26c21f03dc9e4e42ff0/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e4e6b05a45525357e382909a4c1600444e2a45b4795163d3b22669285591c1ae", size = 326751, upload-time = "2025-07-26T12:02:00.343Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/81/03b45cfad088e4770b1dcf72ea78d3802d04200009fb364d18a493857210/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ab3074b48c4e2cf1a960e6bbeb7f04566bf36b1861d5c9d4d8ac04b82e38ba20", size = 375486, upload-time = "2025-07-26T12:02:02.128Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/ba/49923366492ffbdd4486e970d421b289a670ae8cf539c1ea9a09822b371a/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6c3d53c796f8647d6deb1abe867daeb66dcc8a97e8455efa729516b997b8ed99", size = 388106, upload-time = "2025-07-26T12:02:03.615Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/52/5b00ea89525f8f143651f9f03a0df371d3cbd2fccd21ca9b768c7a6500c2/contourpy-1.3.3-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50ed930df7289ff2a8d7afeb9603f8289e5704755c7e5c3bbd929c90c817164b", size = 352548, upload-time = "2025-07-26T12:02:05.165Z" },
+ { url = "https://files.pythonhosted.org/packages/32/1d/a209ec1a3a3452d490f6b14dd92e72280c99ae3d1e73da74f8277d4ee08f/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4feffb6537d64b84877da813a5c30f1422ea5739566abf0bd18065ac040e120a", size = 1322297, upload-time = "2025-07-26T12:02:07.379Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/9e/46f0e8ebdd884ca0e8877e46a3f4e633f6c9c8c4f3f6e72be3fe075994aa/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2b7e9480ffe2b0cd2e787e4df64270e3a0440d9db8dc823312e2c940c167df7e", size = 1391023, upload-time = "2025-07-26T12:02:10.171Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/70/f308384a3ae9cd2209e0849f33c913f658d3326900d0ff5d378d6a1422d2/contourpy-1.3.3-cp313-cp313t-win32.whl", hash = "sha256:283edd842a01e3dcd435b1c5116798d661378d83d36d337b8dde1d16a5fc9ba3", size = 196157, upload-time = "2025-07-26T12:02:11.488Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/dd/880f890a6663b84d9e34a6f88cded89d78f0091e0045a284427cb6b18521/contourpy-1.3.3-cp313-cp313t-win_amd64.whl", hash = "sha256:87acf5963fc2b34825e5b6b048f40e3635dd547f590b04d2ab317c2619ef7ae8", size = 240570, upload-time = "2025-07-26T12:02:12.754Z" },
+ { url = "https://files.pythonhosted.org/packages/80/99/2adc7d8ffead633234817ef8e9a87115c8a11927a94478f6bb3d3f4d4f7d/contourpy-1.3.3-cp313-cp313t-win_arm64.whl", hash = "sha256:3c30273eb2a55024ff31ba7d052dde990d7d8e5450f4bbb6e913558b3d6c2301", size = 199713, upload-time = "2025-07-26T12:02:14.4Z" },
+]
+
+[[package]]
+name = "cryptography"
+version = "49.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cffi", marker = "platform_python_implementation != 'PyPy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1f/99/d1c90d6041656cc6ee229dc99cd67fd0cd5aec3c5f7d72fffc27cc750054/cryptography-49.0.0.tar.gz", hash = "sha256:f89660a348f4f78a92366240a61404e337586ef7f5909a2fef59ca88ef505493", size = 854345, upload-time = "2026-06-12T20:02:30.512Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9b/22/adf66990e63584a68dfb50c24f48a125c07b1699899381c8151e63ed458c/cryptography-49.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:966fe0e9c67490071f14c0d2b1cb2dfb3023c5ce39457343931415f08382f2db", size = 4032100, upload-time = "2026-06-12T20:02:32.143Z" },
+ { url = "https://files.pythonhosted.org/packages/09/41/3797cfaf69cae04a13ee78ebd83f0678d9c02b4779d21ce24445326f1a69/cryptography-49.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:36d1709f992593689b45bda411498d62c6e365f2ca00b84657d4dadd24de16db", size = 4692978, upload-time = "2026-06-12T20:01:21.305Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/8b/43011f7ebe515a8aa20d61f290a326cd890c2e738e16e59eaff8d9c3a412/cryptography-49.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0e959b578856a3924bc0cbb710fc12c387b9412a951389f3ca61704a9e25f325", size = 4716422, upload-time = "2026-06-12T20:01:48.566Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/91/01ce7303a4579e6d3a6abef01bd322848e9ea7a219adcabc5048b9033571/cryptography-49.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:53ecee2e23f7169b6117e99fc8a944e5e50f79e69758a83b52a00cb98ab2b2d2", size = 4700503, upload-time = "2026-06-12T20:02:47.091Z" },
+ { url = "https://files.pythonhosted.org/packages/62/99/a2c95cf8293f07491e9e27c20cc4dcd18176d944e674679adeb1d0173fd6/cryptography-49.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2eda353d8a27bcbcaa4cbed18994a74ab4d19a2ca897db188ea269ab9b71419b", size = 5309779, upload-time = "2026-06-12T20:02:08.987Z" },
+ { url = "https://files.pythonhosted.org/packages/20/2c/0622f20ff02b2ef32558733443805dc82fd4c275be01b2d19d14676f3a1b/cryptography-49.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2afe9051da7ae7bd5905da5a949280c7d2bb75682e188f650a9d0f2756b834c6", size = 4749683, upload-time = "2026-06-12T20:02:03.335Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/5b/c5246635d5fd3b64e0d45ae10e99fd32fe9676a79915ccfe5a61ba9af1a5/cryptography-49.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:0b82e28ee398a386f0807bba7884d30f25218855690f45115831bcce5d90822c", size = 4337874, upload-time = "2026-06-12T20:02:54.323Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/88/05563c7fe2e914e87d1a536d06fe83e66b4e1d95cb593e05aea375531da8/cryptography-49.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:ccac2bfebc306b862133e3bb71f3f6ee8bb525240089b2d952e4144b3a6d5da7", size = 4700283, upload-time = "2026-06-12T20:01:34.822Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/b6/d7696e4e890d6ae1469935164c9e5215c557671cb78d6e3f458ccceaa632/cryptography-49.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:d0527ce944105f257f605a827d6ebead966c752038b6e8656abb9c5edee6fc68", size = 5265844, upload-time = "2026-06-12T20:01:24.09Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/3c/f3ad17eecc1a57b0ba236dc01f90e783c51f4a2f35f64777cc4f47a184b2/cryptography-49.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:cbc77da8c523d5abd028635ba850a6966fcee2c82e2bf65a41d1d8afe0f98be9", size = 4749290, upload-time = "2026-06-12T20:01:30.848Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/01/339573cf1023163a400b0b5d16f6d507de413b9f60be6fd1b77feeaf6737/cryptography-49.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b87e65d263b3e5d3bb92a57e2a6638e2f31110fa7aa890c7b2dbba42248d0a3f", size = 4834612, upload-time = "2026-06-12T20:01:29.246Z" },
+ { url = "https://files.pythonhosted.org/packages/71/fd/577302e213a1be9468f92d1afef66fcf1ef83d516819d9992ca547f592bd/cryptography-49.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:66ec79c3904820572d7e987abdf304281f141d37ad9a489b8e97066e7b9b6459", size = 4980804, upload-time = "2026-06-12T20:01:42.853Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/09/f42b1d190c5ba75f72062a387f8030d1d75f6ab035788f1d9c4b01de6525/cryptography-49.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:e5dfc1e64de5677cec922ffa8da89c546d0415bf6efdf081842e5d44c84e1f0e", size = 3810026, upload-time = "2026-06-12T20:02:39.262Z" },
+ { url = "https://files.pythonhosted.org/packages/19/2a/5bb823f5bedcf80718cea7fbc95ec5515cca3769633c4b01a32be7f30e7c/cryptography-49.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ec5e529fb80935c94fe7b729f9972b50e351a0e6b50aa294fd5cabb109fcc29a", size = 4025947, upload-time = "2026-06-12T20:01:25.745Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/df/40577043ca124e17012f408ddddaeb213b856336ac82ddb3bc915f39e29f/cryptography-49.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f78ff2c9ed8dc2d036b0f4d640e22522213d047c1b14e61205a7e55c80a494d4", size = 4692429, upload-time = "2026-06-12T20:01:53.628Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/99/2d13299eb3dd27b02dcfaafcc91d6b5cb3329f7cbd6d8f51921acd566c1a/cryptography-49.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:35b151772baff2c74cba7fa290ceaff4c3b11c0c881eb93eb5dbc05a7cfbba18", size = 4700968, upload-time = "2026-06-12T20:02:45.383Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/4d/9c0cd02f95e2602dd5e563da149ee0830abef3537be8b34dc56281ebe27a/cryptography-49.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0f21641cf4b30fca7aee061ced0ec7ad7b073518088b7c9969a297c0ae796c69", size = 4697758, upload-time = "2026-06-12T20:01:41.13Z" },
+ { url = "https://files.pythonhosted.org/packages/24/01/186c825898477d77e2324d5360fefe622ff1d8d1963ec0554e2cada8ec77/cryptography-49.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:9e82dcc8e56052715fb18b2429e3bca4823b1629136a2084fc45a9a5cecb9b64", size = 5298863, upload-time = "2026-06-12T20:02:24.579Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/7b/62cbbab75d0659865bf0273790031544a0b16c8072d258f9428dcd8190dc/cryptography-49.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:6f2debedf9ca60cf1d5bd466475638af5130f89965605cd818484d19987d3a21", size = 4735983, upload-time = "2026-06-12T20:01:50.14Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/72/3e798c064bc39e471008075d0f9bc9daf77a80879c092e4a8e170c585ed4/cryptography-49.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:8c25ceb16df5b9435f3f6a9829204985b0e0cbee3b48aacd432c7d2c850b44d9", size = 4334173, upload-time = "2026-06-12T20:01:44.743Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/ee/6fca21d1ac73e06f8bef71940abfd4d2f6472b4bca284d770f32bd4086f6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:28d8b15e6275f12c8a207dc309dfa957903c927d08d0cc937ee3f63f200693cc", size = 4697298, upload-time = "2026-06-12T20:02:20.918Z" },
+ { url = "https://files.pythonhosted.org/packages/67/d0/a5fcd3515f0bae49a7b6d0413cc1bdccdcc1fc0047037a0d480642cdc5d6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6fc361c34fb6aac015ce19435876635e5c6d21db31998b0920f675f131e043b8", size = 5254338, upload-time = "2026-06-12T20:02:22.737Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/84/84fe36f19caf857d61cb7fc9c63035a47ffabd84ea12d1d393148efa3615/cryptography-49.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:2400ef9c9e2299a25614eb1dea3db54a69b1349efd043bfac9c67630d136df36", size = 4735650, upload-time = "2026-06-12T20:02:41.389Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/a0/db537264e234f7273a73ec020873d6d6b39dfd8a53db78b550ca8320440e/cryptography-49.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:67e1d20ad9ef3a563c59ef22e7a8a0b8210bd26604369ea4a30a7c66aefe504e", size = 4834820, upload-time = "2026-06-12T20:01:51.847Z" },
+ { url = "https://files.pythonhosted.org/packages/93/77/8df9eb486495979bccecd1062e2eaf435250e84437040295b57d09048b0b/cryptography-49.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:42b0684e0e40cf26122427802486f6d93aea593612603a94fbf260c7eb1e9c1b", size = 4967968, upload-time = "2026-06-12T20:02:12.524Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/e6/f60198ea8d9dfa15fff9ed4ca02ce362f6eadd9ba757dcc50634c4257b63/cryptography-49.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:026ac7423e6fa66872d3bf889be5974507da3944f866f704fa200eadacd00001", size = 3785547, upload-time = "2026-06-12T20:02:26.847Z" },
+]
+
+[[package]]
+name = "cuda-bindings"
+version = "12.9.7"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform == 'darwin'",
+]
+dependencies = [
+ { name = "cuda-pathfinder", marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/43/b2/753fe88151001d0dc23f56a8e119fe06b991b0d1a885fa02f9852b12f523/cuda_bindings-12.9.7-cp313-cp313-win_amd64.whl", hash = "sha256:5bd89dcb78475a6d8a4620ea94b74edf0cbbeacee6d1622d8f94452c1e8d3f15", size = 7360097, upload-time = "2026-05-27T18:44:22.405Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/c8/d79a20ba396e7ab2dfdd4b72b62356972b25b88aee2ded49a70c797ddea1/cuda_bindings-12.9.7-cp313-cp313t-win_amd64.whl", hash = "sha256:64f7ade7a7a3b69001489753acc21706d9dbda32db8deb68a767a0a0aab30b68", size = 7780136, upload-time = "2026-05-27T18:44:28.121Z" },
+]
+
+[[package]]
+name = "cuda-bindings"
+version = "13.3.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "cuda-pathfinder", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cc/6e/2394f8163360f8391f8f1b7e72d300a82724edb81a7b7084c799fbd4c91f/cuda_bindings-13.3.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9efb21c1ee64981e184b9e0ba5eb3179e5ba3d4b51665a6cb52b8ef3d01a7cbf", size = 5920504, upload-time = "2026-05-29T23:11:56.883Z" },
+ { url = "https://files.pythonhosted.org/packages/34/c2/ef9b6a63f7dc432712a462c816662e662e00d38caa9b861c8c2588195d03/cuda_bindings-13.3.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2732904099e0a4d4db774a5fc6d91ee95fae065b4d2ecabb4968c5fe2406c9d7", size = 6476660, upload-time = "2026-05-29T23:11:59.188Z" },
+]
+
+[[package]]
+name = "cuda-core"
+version = "1.0.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "cuda-pathfinder", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "numpy", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/57/f9/a6676b1fa555fad5748a945f4b530b51b898b4771a1e5d9f3520d3f415ea/cuda_core-1.0.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c427e5025096d96fcd5092fdc85d5d5e4ac3dea007914e90472ed52f27220446", size = 4749800, upload-time = "2026-05-12T20:11:38.012Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/9d/4534a9564a812ee95b43db7324f9b25cbffda001bb348bb5b3f90dad50b9/cuda_core-1.0.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b392178202c652368883dbe3773cee14f3e1ed6b8bf45d1a1bcdd37c73604e06", size = 5078597, upload-time = "2026-05-12T20:11:40.836Z" },
+]
+
+[[package]]
+name = "cuda-core"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform == 'darwin'",
+]
+dependencies = [
+ { name = "cuda-pathfinder", marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6d/cd/efc25f86f280ba658b5955f91ea9ca26523d9f05b29d2d69d561c513ec08/cuda_core-1.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:ff1585f687e4081c4731e24076b0af9dd0a55eb5e4a90c608289effa708718fb", size = 5579371, upload-time = "2026-07-29T23:05:40.016Z" },
+]
+
+[[package]]
+name = "cuda-pathfinder"
+version = "1.5.6"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/53/8fc9b0cdc5b7f62746e6a01b85b6461e5ae27f871010a5fcf8fa6950766d/cuda_pathfinder-1.5.6-py3-none-any.whl", hash = "sha256:7e4c07c117b78ba1fb35dac4c444d21f3677b1b1ff56175c53a8e3025c5b43c0", size = 52972, upload-time = "2026-06-30T00:58:04.34Z" },
+]
+
+[[package]]
+name = "cuda-python"
+version = "12.9.7"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform == 'darwin'",
+]
+dependencies = [
+ { name = "cuda-bindings", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c1/9d/05e753afbaac3f92691059b3ba875589c98a425d69e5808cec32b31b580c/cuda_python-12.9.7-py3-none-any.whl", hash = "sha256:23a1fc406d491eef7a7e985095725cb7b20a04a7bd9b7a66400e5c86e082e0aa", size = 7597, upload-time = "2026-05-27T19:50:32.605Z" },
+]
+
+[[package]]
+name = "cuda-python"
+version = "13.3.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "cuda-bindings", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "cuda-core", version = "1.0.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "cuda-pathfinder", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/31/7ff3f7768eded7535c621abc2fecb9d181a34ea4cae3afe682feb796f242/cuda_python-13.3.1-py3-none-any.whl", hash = "sha256:280b014139ab447b6dd70a377db1596f310d6e887d9d342e6651b919ec145fb3", size = 8295, upload-time = "2026-05-29T23:28:47.012Z" },
+]
+
+[[package]]
+name = "cuda-tile"
+version = "1.5.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/00/46/60aea981ee7cc0b159eb08c42b795e8e95ae8a7fb451e4565cad0f43cca0/cuda_tile-1.5.0-cp313-cp313-manylinux2014_aarch64.whl", hash = "sha256:cfa4a5ef920d9c1fee702611b25bde449915f12bf929c1b8a3faee0c9b03f750", size = 322643, upload-time = "2026-07-08T01:49:26.107Z" },
+ { url = "https://files.pythonhosted.org/packages/26/d5/ae03d2b70ed8d6c21ca809ddc98227ad07988e7fe67e7e41d888c0b13d32/cuda_tile-1.5.0-cp313-cp313-manylinux2014_x86_64.whl", hash = "sha256:e7cb56186b0cc98166b72c7e5a3764c236151aa8d53e0b37a153766b79ea005a", size = 324995, upload-time = "2026-07-08T01:49:19.931Z" },
+ { url = "https://files.pythonhosted.org/packages/da/a6/e3f6e9aefcc94d548e5d69f01a02ba5da7c5e200702eba3eb7a4f572a883/cuda_tile-1.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:c400918faa8492d84c4da3da81ce01ed30d81ed780f9ddbc1a9bdd588058b776", size = 304825, upload-time = "2026-07-08T01:49:37.749Z" },
+]
+
+[[package]]
+name = "cuda-toolkit"
+version = "13.0.3.0"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/c7/a79086a62c98befcdb8349656c6f114e2db3b8b2422f6e25c97a7f2a9a3c/cuda_toolkit-13.0.3.0-py2.py3-none-any.whl", hash = "sha256:d693caaa261214ddd7dbb60d68e71cbed884e68c2be7509778f3051da0b91c3f", size = 2512, upload-time = "2026-04-14T00:50:08.173Z" },
+]
+
+[package.optional-dependencies]
+cublas = [
+ { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+ { name = "nvidia-cuda-nvrtc", version = "13.0.88", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+cudart = [
+ { name = "nvidia-cuda-runtime", version = "13.0.96", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+cufft = [
+ { name = "nvidia-cufft", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+cufile = [
+ { name = "nvidia-cufile", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+cupti = [
+ { name = "nvidia-cuda-cupti", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+curand = [
+ { name = "nvidia-curand", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+cusolver = [
+ { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+ { name = "nvidia-cusolver", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+ { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+cusparse = [
+ { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+nvjitlink = [
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+nvrtc = [
+ { name = "nvidia-cuda-nvrtc", version = "13.0.88", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+nvtx = [
+ { name = "nvidia-nvtx", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
+]
+
+[[package]]
+name = "cycler"
+version = "0.12.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615, upload-time = "2023-10-07T05:32:18.335Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" },
+]
+
+[[package]]
+name = "cymem"
+version = "2.0.13"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/2f0fbb32535c3731b7c2974c569fb9325e0a38ed5565a08e1139a3b71e82/cymem-2.0.13.tar.gz", hash = "sha256:1c91a92ae8c7104275ac26bd4d29b08ccd3e7faff5893d3858cb6fadf1bc1588", size = 12320, upload-time = "2025-11-14T14:58:36.902Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ce/0f/95a4d1e3bebfdfa7829252369357cf9a764f67569328cd9221f21e2c952e/cymem-2.0.13-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:891fd9030293a8b652dc7fb9fdc79a910a6c76fc679cd775e6741b819ffea476", size = 43478, upload-time = "2025-11-14T14:57:42.682Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/a0/8fc929cc29ae466b7b4efc23ece99cbd3ea34992ccff319089c624d667fd/cymem-2.0.13-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:89c4889bd16513ce1644ccfe1e7c473ba7ca150f0621e66feac3a571bde09e7e", size = 42695, upload-time = "2025-11-14T14:57:43.741Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/b3/deeb01354ebaf384438083ffe0310209ef903db3e7ba5a8f584b06d28387/cymem-2.0.13-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:45dcaba0f48bef9cc3d8b0b92058640244a95a9f12542210b51318da97c2cf28", size = 250573, upload-time = "2025-11-14T14:57:44.81Z" },
+ { url = "https://files.pythonhosted.org/packages/36/36/bc980b9a14409f3356309c45a8d88d58797d02002a9d794dd6c84e809d3a/cymem-2.0.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e96848faaafccc0abd631f1c5fb194eac0caee4f5a8777fdbb3e349d3a21741c", size = 254572, upload-time = "2025-11-14T14:57:46.023Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/dd/a12522952624685bd0f8968e26d2ed6d059c967413ce6eb52292f538f1b0/cymem-2.0.13-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e02d3e2c3bfeb21185d5a4a70790d9df40629a87d8d7617dc22b4e864f665fa3", size = 248060, upload-time = "2025-11-14T14:57:47.605Z" },
+ { url = "https://files.pythonhosted.org/packages/08/11/5dc933ddfeb2dfea747a0b935cb965b9a7580b324d96fc5f5a1b5ff8df29/cymem-2.0.13-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:fece5229fd5ecdcd7a0738affb8c59890e13073ae5626544e13825f26c019d3c", size = 254601, upload-time = "2025-11-14T14:57:48.861Z" },
+ { url = "https://files.pythonhosted.org/packages/70/66/d23b06166864fa94e13a98e5922986ce774832936473578febce64448d75/cymem-2.0.13-cp313-cp313-win_amd64.whl", hash = "sha256:38aefeb269597c1a0c2ddf1567dd8605489b661fa0369c6406c1acd433b4c7ba", size = 40103, upload-time = "2025-11-14T14:57:50.396Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/9e/c7b21271ab88a21760f3afdec84d2bc09ffa9e6c8d774ad9d4f1afab0416/cymem-2.0.13-cp313-cp313-win_arm64.whl", hash = "sha256:717270dcfd8c8096b479c42708b151002ff98e434a7b6f1f916387a6c791e2ad", size = 36016, upload-time = "2025-11-14T14:57:51.611Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/28/d3b03427edc04ae04910edf1c24b993881c3ba93a9729a42bcbb816a1808/cymem-2.0.13-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:7e1a863a7f144ffb345397813701509cfc74fc9ed360a4d92799805b4b865dd1", size = 46429, upload-time = "2025-11-14T14:57:52.582Z" },
+ { url = "https://files.pythonhosted.org/packages/35/a9/7ed53e481f47ebfb922b0b42e980cec83e98ccb2137dc597ea156642440c/cymem-2.0.13-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:c16cb80efc017b054f78998c6b4b013cef509c7b3d802707ce1f85a1d68361bf", size = 46205, upload-time = "2025-11-14T14:57:53.64Z" },
+ { url = "https://files.pythonhosted.org/packages/61/39/a3d6ad073cf7f0fbbb8bbf09698c3c8fac11be3f791d710239a4e8dd3438/cymem-2.0.13-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0d78a27c88b26c89bd1ece247d1d5939dba05a1dae6305aad8fd8056b17ddb51", size = 296083, upload-time = "2025-11-14T14:57:55.922Z" },
+ { url = "https://files.pythonhosted.org/packages/36/0c/20697c8bc19f624a595833e566f37d7bcb9167b0ce69de896eba7cfc9c2d/cymem-2.0.13-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6d36710760f817194dacb09d9fc45cb6a5062ed75e85f0ef7ad7aeeb13d80cc3", size = 286159, upload-time = "2025-11-14T14:57:57.106Z" },
+ { url = "https://files.pythonhosted.org/packages/82/d4/9326e3422d1c2d2b4a8fb859bdcce80138f6ab721ddafa4cba328a505c71/cymem-2.0.13-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c8f30971cadd5dcf73bcfbbc5849b1f1e1f40db8cd846c4aa7d3b5e035c7b583", size = 288186, upload-time = "2025-11-14T14:57:58.334Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/bc/68da7dd749b72884dc22e898562f335002d70306069d496376e5ff3b6153/cymem-2.0.13-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:9d441d0e45798ec1fd330373bf7ffa6b795f229275f64016b6a193e6e2a51522", size = 290353, upload-time = "2025-11-14T14:58:00.562Z" },
+ { url = "https://files.pythonhosted.org/packages/50/23/dbf2ad6ecd19b99b3aab6203b1a06608bbd04a09c522d836b854f2f30f73/cymem-2.0.13-cp313-cp313t-win_amd64.whl", hash = "sha256:d1c950eebb9f0f15e3ef3591313482a5a611d16fc12d545e2018cd607f40f472", size = 44764, upload-time = "2025-11-14T14:58:01.793Z" },
+ { url = "https://files.pythonhosted.org/packages/54/3f/35701c13e1fc7b0895198c8b20068c569a841e0daf8e0b14d1dc0816b28f/cymem-2.0.13-cp313-cp313t-win_arm64.whl", hash = "sha256:042e8611ef862c34a97b13241f5d0da86d58aca3cecc45c533496678e75c5a1f", size = 38964, upload-time = "2025-11-14T14:58:02.87Z" },
+]
+
+[[package]]
+name = "dataproperty"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mbstrdecoder" },
+ { name = "typepy", extra = ["datetime"] },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f1/6f/a801320bb388d965be9c370ec753cc33120e6cbe0069fa05644f05821975/dataproperty-1.1.1.tar.gz", hash = "sha256:a83af82a234edda5378a36fb092bc90dd554646c5e58202a310acf468ae81bc8", size = 42954, upload-time = "2026-05-09T10:33:42.212Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/03/41/eab7fe313820578b341a2a1d6aeeedd2c38ec1e3f3d51e57e2735b5beac0/dataproperty-1.1.1-py3-none-any.whl", hash = "sha256:cf026aa002dbd6c57c619ec6741ffd61ae7bf2f20481951d8af2dff44480340e", size = 27691, upload-time = "2026-05-09T10:33:40.468Z" },
+]
+
+[[package]]
+name = "datasets"
+version = "5.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dill" },
+ { name = "filelock" },
+ { name = "fsspec", extra = ["http"] },
+ { name = "httpx" },
+ { name = "huggingface-hub" },
+ { name = "multiprocess" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pandas" },
+ { name = "pyarrow" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "tqdm" },
+ { name = "xxhash" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d9/85/ce4f780c32f7e36d71257f1c27e8ba898ebe379cb54f211f5f2013f2c219/datasets-5.0.0.tar.gz", hash = "sha256:83dbbbdb07a33b82192b8c419deb18739b138ee2ce1a322d55ce6b100954ec1a", size = 631708, upload-time = "2026-06-05T13:18:26.124Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/66/73034ad30b59f13439b75e620989dacba4c047256e358ba7c2e9ec98ea22/datasets-5.0.0-py3-none-any.whl", hash = "sha256:7dd34927a0fd7046e98aad5cb9430e699c373238a15befa7b9bf22b991a7fee6", size = 555084, upload-time = "2026-06-05T13:18:24.435Z" },
+]
+
+[[package]]
+name = "debugpy"
+version = "1.8.21"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f2/aa/12037145b7a56eaa5b29b41872f7a21b538e807e13f32c4d3c46e59be084/debugpy-1.8.21.tar.gz", hash = "sha256:a3c53278e84c94e11bd87c53970ec391d1a67396c8b22609fcac576520e611a6", size = 1697577, upload-time = "2026-06-01T19:30:35.156Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/6b/d817e1f8cc77aa055d37fba092e0febfdff40fe652d8d53d4cd7a86ad98d/debugpy-1.8.21-cp313-cp313-macosx_15_0_universal2.whl", hash = "sha256:13678151fc401e2d68c9880b91e28714f797d40422994572b24560ef80910a88", size = 2477398, upload-time = "2026-06-01T19:30:57.644Z" },
+ { url = "https://files.pythonhosted.org/packages/48/57/412421516afc3055fa577516f00beec3d663f9b0ab330639547ae6c57720/debugpy-1.8.21-cp313-cp313-manylinux_2_34_x86_64.whl", hash = "sha256:ecbd158386c31ffe71d46f72d44d56e66331ab9b16cad649156d514368f23ab2", size = 3962096, upload-time = "2026-06-01T19:30:59.235Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/62/2c616337cf6ba7b07ebbc97f02c6c945a8e2f76b365e33ee809c32ee36d1/debugpy-1.8.21-cp313-cp313-win32.whl", hash = "sha256:2c2ae706dec41d99a9ca1f7ebc987a83e65578363be6f6b3ac9067504917fae1", size = 5336288, upload-time = "2026-06-01T19:31:00.79Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/99/9175103392f84c4b1bf7622888cdc68da07f0ff7d9e581266428f6776033/debugpy-1.8.21-cp313-cp313-win_amd64.whl", hash = "sha256:aa648733047443eb1d07682c4ef287d36a54507b643ffdf38b09a3ef002c72a0", size = 5376567, upload-time = "2026-06-01T19:31:02.56Z" },
+ { url = "https://files.pythonhosted.org/packages/95/51/67e7cf11a53e40694f720457d5b3a1cdaaa3d5a9a633e482f225456b93ff/debugpy-1.8.21-py2.py3-none-any.whl", hash = "sha256:b1e37d333663c8851516a47364ef473da127f9caebe4417e6df6f5825a7e9a92", size = 5352888, upload-time = "2026-06-01T19:31:25.186Z" },
+]
+
+[[package]]
+name = "decorator"
+version = "5.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/60/8b/32f9823da46cde7df2087faa08cd98d01b908f8dcab982cdba9c84e85355/decorator-5.3.1.tar.gz", hash = "sha256:4cbcdd55a6efadb9dbea26b858f4fb3264567b52d69ca0d25b721b553f60ea82", size = 58084, upload-time = "2026-05-18T06:03:28.057Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/7f/798705f5296a58ca505d600456748d1be48078eac8a7050d8a98bc9edb89/decorator-5.3.1-py3-none-any.whl", hash = "sha256:f47fe6fdbd2edd623ecfe36875d37aba411624e2670dd395dddae1358689bb3c", size = 10365, upload-time = "2026-05-18T06:03:26.517Z" },
+]
+
+[[package]]
+name = "deep-gemm"
+version = "2.1.1.post3"
+source = { git = "https://github.com/deepseek-ai/DeepGEMM.git?rev=v2.1.1.post3#c9f8b34dcdacc20aa746b786f983492c51072870" }
+
+[[package]]
+name = "defusedxml"
+version = "0.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0f/d5/c66da9b79e5bdb124974bfe172b4daf3c984ebd9c2a06e2b8a4dc7331c72/defusedxml-0.7.1.tar.gz", hash = "sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69", size = 75520, upload-time = "2021-03-08T10:59:26.269Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61", size = 25604, upload-time = "2021-03-08T10:59:24.45Z" },
+]
+
+[[package]]
+name = "depyf"
+version = "0.20.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "astor" },
+ { name = "dill" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/88/35/83fb0178212279aa0af031031905804c6de5618435d229f41ed21bb9ad2c/depyf-0.20.0.tar.gz", hash = "sha256:fb7683bd72c44f67b56029df2c47721e9a02ffa4d7b19095f1c54c4ebf797a98", size = 6168761, upload-time = "2025-10-13T12:33:38.589Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cf/65/4df6936130b56e1429114e663e7c1576cf845f3aef1b2dd200c0a5d19dba/depyf-0.20.0-py3-none-any.whl", hash = "sha256:d31effad4261cebecb58955d832e448ace88f432328f95f82fd99c30fd9308d4", size = 39381, upload-time = "2025-10-13T12:33:33.647Z" },
+]
+
+[[package]]
+name = "detect-installer"
+version = "0.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/5f/ce/6897d812825e9d4c53e3c7112726e800cc5231b013b2223bf64f653ff362/detect_installer-0.1.0.tar.gz", hash = "sha256:00ad7ba0a36e3cf7d08a40d3643011746dbc112597c7d475cc91c416710ca4e7", size = 3049, upload-time = "2026-02-23T10:40:22.567Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cc/34/8cc73273414405086c58852916e4031812a6a30fe04c057e37ad99397b7f/detect_installer-0.1.0-py3-none-any.whl", hash = "sha256:034fb20fd665c36e6ba52b8821525ea07fb4f7f938cac459df889fb33801528a", size = 4539, upload-time = "2026-02-23T10:40:23.807Z" },
+]
+
+[[package]]
+name = "dill"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/81/e1/56027a71e31b02ddc53c7d65b01e68edf64dea2932122fe7746a516f75d5/dill-0.4.1.tar.gz", hash = "sha256:423092df4182177d4d8ba8290c8a5b640c66ab35ec7da59ccfa00f6fa3eea5fa", size = 187315, upload-time = "2026-01-19T02:36:56.85Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl", hash = "sha256:1e1ce33e978ae97fcfcff5638477032b801c46c7c65cf717f95fbc2248f79a9d", size = 120019, upload-time = "2026-01-19T02:36:55.663Z" },
+]
+
+[[package]]
+name = "diskcache"
+version = "5.6.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/3f/21/1c1ffc1a039ddcc459db43cc108658f32c57d271d7289a2794e401d0fdb6/diskcache-5.6.3.tar.gz", hash = "sha256:2c3a3fa2743d8535d832ec61c2054a1641f41775aa7c556758a109941e33e4fc", size = 67916, upload-time = "2023-08-31T06:12:00.316Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" },
+]
+
+[[package]]
+name = "distlib"
+version = "0.4.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c9/02/bd72be9134d25ed783ecbbc38a539ffaefbf90c78418c7fb7229600dbac7/distlib-0.4.3.tar.gz", hash = "sha256:f152097224a0ae24be5a0f6bae1b9359af82133bce63f98a95f86cae1aede9ed", size = 615141, upload-time = "2026-06-12T08:04:52.847Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/02/08/9c41fb51ab5b43eb21674aff13df270e8ba6c4b29c8624e328dc7a9482af/distlib-0.4.3-py2.py3-none-any.whl", hash = "sha256:4b0ce306c966eb73bc3a7b6abad017c556dadd92c44701562cd528ac7fde4d5b", size = 470628, upload-time = "2026-06-12T08:04:50.506Z" },
+]
+
+[[package]]
+name = "distro"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed", size = 60722, upload-time = "2023-12-24T09:54:32.31Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" },
+]
+
+[[package]]
+name = "dnspython"
+version = "2.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8c/8b/57666417c0f90f08bcafa776861060426765fdb422eb10212086fb811d26/dnspython-2.8.0.tar.gz", hash = "sha256:181d3c6996452cb1189c4046c61599b84a5a86e099562ffde77d26984ff26d0f", size = 368251, upload-time = "2025-09-07T18:58:00.022Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ba/5a/18ad964b0086c6e62e2e7500f7edc89e3faa45033c71c1893d34eed2b2de/dnspython-2.8.0-py3-none-any.whl", hash = "sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af", size = 331094, upload-time = "2025-09-07T18:57:58.071Z" },
+]
+
+[[package]]
+name = "docstring-parser"
+version = "0.18.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e0/4d/f332313098c1de1b2d2ff91cf2674415cc7cddab2ca1b01ae29774bd5fdf/docstring_parser-0.18.0.tar.gz", hash = "sha256:292510982205c12b1248696f44959db3cdd1740237a968ea1e2e7a900eeb2015", size = 29341, upload-time = "2026-04-14T04:09:19.867Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a7/5f/ed01f9a3cdffbd5a008556fc7b2a08ddb1cc6ace7effa7340604b1d16699/docstring_parser-0.18.0-py3-none-any.whl", hash = "sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b", size = 22484, upload-time = "2026-04-14T04:09:18.638Z" },
+]
+
+[[package]]
+name = "einops"
+version = "0.8.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2c/77/850bef8d72ffb9219f0b1aac23fbc1bf7d038ee6ea666f331fa273031aa2/einops-0.8.2.tar.gz", hash = "sha256:609da665570e5e265e27283aab09e7f279ade90c4f01bcfca111f3d3e13f2827", size = 56261, upload-time = "2026-01-26T04:13:17.638Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
+]
+
+[[package]]
+name = "email-validator"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dnspython" },
+ { name = "idna" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f5/22/900cb125c76b7aaa450ce02fd727f452243f2e91a61af068b40adba60ea9/email_validator-2.3.0.tar.gz", hash = "sha256:9fc05c37f2f6cf439ff414f8fc46d917929974a82244c20eb10231ba60c54426", size = 51238, upload-time = "2025-08-26T13:09:06.831Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/de/15/545e2b6cf2e3be84bc1ed85613edd75b8aea69807a71c26f4ca6a9258e82/email_validator-2.3.0-py3-none-any.whl", hash = "sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4", size = 35604, upload-time = "2025-08-26T13:09:05.858Z" },
+]
+
+[[package]]
+name = "emoji"
+version = "2.15.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/78/0d2db9382c92a163d7095fc08efff7800880f830a152cfced40161e7638d/emoji-2.15.0.tar.gz", hash = "sha256:eae4ab7d86456a70a00a985125a03263a5eac54cd55e51d7e184b1ed3b6757e4", size = 615483, upload-time = "2025-09-21T12:13:02.755Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e1/5e/4b5aaaabddfacfe36ba7768817bd1f71a7a810a43705e531f3ae4c690767/emoji-2.15.0-py3-none-any.whl", hash = "sha256:205296793d66a89d88af4688fa57fd6496732eb48917a87175a023c8138995eb", size = 608433, upload-time = "2025-09-21T12:13:01.197Z" },
+]
+
+[[package]]
+name = "executing"
+version = "2.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cc/28/c14e053b6762b1044f34a13aab6859bbf40456d37d23aa286ac24cfd9a5d/executing-2.2.1.tar.gz", hash = "sha256:3632cc370565f6648cc328b32435bd120a1e4ebb20c77e3fdde9a13cd1e533c4", size = 1129488, upload-time = "2025-09-01T09:48:10.866Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c1/ea/53f2148663b321f21b5a606bd5f191517cf40b7072c0497d3c92c4a13b1e/executing-2.2.1-py2.py3-none-any.whl", hash = "sha256:760643d3452b4d777d295bb167ccc74c64a81df23fb5e08eff250c425a4b2017", size = 28317, upload-time = "2025-09-01T09:48:08.5Z" },
+]
+
+[[package]]
+name = "fastapi"
+version = "0.136.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "annotated-doc" },
+ { name = "pydantic" },
+ { name = "starlette" },
+ { name = "typing-extensions" },
+ { name = "typing-inspection" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/81/2d/ff8d91d7b564d464629a0fd50a4489c97fcb836ac230bf3a7269232a9b1f/fastapi-0.136.3.tar.gz", hash = "sha256:e487fae93ad408e6f47641ee4dfe389864fd7bec92e547ea8498fc13f43e83ab", size = 396410, upload-time = "2026-05-23T18:53:15.192Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e0/82/45359b62a067409bd929ae8a56b8ed13e5a8c8a61194b3c236920999ab83/fastapi-0.136.3-py3-none-any.whl", hash = "sha256:3d2a69bdf04b7e9f3afa292c3bc7a98816bbfafa10bc9b45f3f3700d2f761620", size = 117481, upload-time = "2026-05-23T18:53:16.924Z" },
+]
+
+[package.optional-dependencies]
+standard = [
+ { name = "email-validator" },
+ { name = "fastapi-cli", extra = ["standard"] },
+ { name = "fastar" },
+ { name = "httpx" },
+ { name = "jinja2" },
+ { name = "pydantic-extra-types" },
+ { name = "pydantic-settings" },
+ { name = "python-multipart" },
+ { name = "uvicorn", extra = ["standard"] },
+]
+
+[[package]]
+name = "fastapi-cli"
+version = "0.0.27"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "rich-toolkit" },
+ { name = "typer" },
+ { name = "uvicorn", extra = ["standard"] },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/37/d0/ee5678346811967b8d096d5d5604e71b50d6bf5a2abfbdb331157e2bbaa9/fastapi_cli-0.0.27.tar.gz", hash = "sha256:1dffb1e40c0c88f2e0171a8a252a2b615c1e63ff8c05626649e4badd6a84336a", size = 23630, upload-time = "2026-06-18T14:48:43.421Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1a/ab/0a709f9488fe62647db80f8a277fb0ee62e85adc6746abf477ed373c9eb7/fastapi_cli-0.0.27-py3-none-any.whl", hash = "sha256:2e389a40f318e29fec8cb1e289f267f17c048876fb82dbfa869a10b16740495d", size = 13070, upload-time = "2026-06-18T14:48:44.311Z" },
+]
+
+[package.optional-dependencies]
+standard = [
+ { name = "fastapi-cloud-cli" },
+ { name = "uvicorn", extra = ["standard"] },
+]
+
+[[package]]
+name = "fastapi-cloud-cli"
+version = "0.22.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "detect-installer" },
+ { name = "fastar" },
+ { name = "httpx" },
+ { name = "pydantic", extra = ["email"] },
+ { name = "rich-toolkit" },
+ { name = "rignore" },
+ { name = "sentry-sdk" },
+ { name = "typer" },
+ { name = "uvicorn", extra = ["standard"] },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0b/e5/bee77aa542ec66bcc55b458d606f3356a58f5bb9f2c59006f6ff53a3869b/fastapi_cloud_cli-0.22.1.tar.gz", hash = "sha256:50d80de6ce397a4959e6f3509574edac65d0a6998655215c95d077b18ec2f4b1", size = 94501, upload-time = "2026-07-01T22:09:03.358Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/75/0c/9e3069ff571142e571d68156aa4c1ab1d51c0b87f21bbe0f44c10029b19b/fastapi_cloud_cli-0.22.1-py3-none-any.whl", hash = "sha256:4ba307b97b08282d1efb2daef5f1e88af23e02fe2286c25273c1794e399df509", size = 77739, upload-time = "2026-07-01T22:09:02.359Z" },
+]
+
+[[package]]
+name = "fastar"
+version = "0.11.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/03/0f/0aeb3fc50046617702acc0078b277b58367fd62eb727b9ec733ae0e8bbcc/fastar-0.11.0.tar.gz", hash = "sha256:aa7f100f7313c03fdb20f1385927ba95671071ba308ad0c1763fef295e1895ce", size = 70238, upload-time = "2026-04-13T17:11:17.143Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c9/d6/3be260037e86fb694e88d47f583bac3a0188c99cee1a6b257ac26cb6b53c/fastar-0.11.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:33f544b08b4541b678e53749b4552a44720d96761fb79c172b005b1089c443ed", size = 707975, upload-time = "2026-04-13T17:09:58.866Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/cd/7867aefb1784662554a335f2952c75a50f0c70585ed0d2210d6cc15e5627/fastar-0.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:91c1c792447e4a642745f347ff9847c52af39633071c57ee67ed53c157fc3506", size = 628460, upload-time = "2026-04-13T17:09:43.776Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/2b/d11d84bdd5e0e377771b955755771e3460b290da5809cb78c1b735ee2228/fastar-0.11.0-cp313-cp313-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:881247e6b6eaea59fc6569f9b61447aa6b9fc2ee864e048b4643d69c52745805", size = 863054, upload-time = "2026-04-13T17:09:13.048Z" },
+ { url = "https://files.pythonhosted.org/packages/25/39/d3f428b318fa940b1b6e785b8d54fc895dfb5d5b945ef8d5442ffa904fb2/fastar-0.11.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:863b7929845c9fec92ef6c8d59579cf46af5136655e5342f8df5cebe46cab06c", size = 760247, upload-time = "2026-04-13T17:07:57.396Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/04/03949aee82aabb8ede06ac5a4a5579ffaf98a8fe59ce958494508ff15513/fastar-0.11.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:96b4a57df12bf3211662627a3ea29d62ecb314a2434a0d0843f9fc23e47536e5", size = 756512, upload-time = "2026-04-13T17:08:12.415Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/0c/2ca1ae0a3828ca51047962d932b80daca2522db73e8cb9d040cb6ebe28d5/fastar-0.11.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ceef1c2c4df7b7b8ebd3f5d718bbf457b9bbdf25ce0bd07870211ec4fbd9aff4", size = 922183, upload-time = "2026-04-13T17:08:27.187Z" },
+ { url = "https://files.pythonhosted.org/packages/65/68/7fe808b1f73a68e686f25434f538c6dc10ef4dfb3db0ace22cd861744bf8/fastar-0.11.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b8e545918441910a779659d4759ad0eef349e935fbdb4668a666d3681567eb05", size = 816394, upload-time = "2026-04-13T17:08:57.657Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/17/07d086080f8a83b8d7966955e29bcdbd6a060f5bd949dc9d5abd3658cead/fastar-0.11.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:28095bb8f821e85fc2764e1a55f03e5e2876dee2abe7cd0ee9420d929905d643", size = 818983, upload-time = "2026-04-13T17:09:28.46Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/e2/2c4edf0910af2e814ff6d65b77a91196d472ca8a9fb2033bd983f6856caa/fastar-0.11.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:0fafb95ecbe70f666a5e9b35dd63974ccdc9bb3d99ccdbd4014a823ec3e659b5", size = 884689, upload-time = "2026-04-13T17:08:42.763Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/ba/04fdcbd6558e60de4ced3b55230fac47675d181252582b2fcec3c74608e5/fastar-0.11.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:af48fed039b94016629dcdad1c95c90c486326dd068de2b0a4df419ee09b6821", size = 970677, upload-time = "2026-04-13T17:10:15.124Z" },
+ { url = "https://files.pythonhosted.org/packages/df/b3/2b860a9658550167dbd5824c85e88d0b4b912bf493e42a6322544d6e483d/fastar-0.11.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:74cd96163f39b8638ab4e8d49708ca887959672a22871d8170d01f067319533b", size = 1034026, upload-time = "2026-04-13T17:10:32.318Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/9b/fa42ea1188b144bac4b1b60753dfd449974a4d5eda132029ee7711569f94/fastar-0.11.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:4e8b993cb5613bab495ed482810bedc0986633fcb9a3b55c37ec88e0d6714f6a", size = 1071147, upload-time = "2026-04-13T17:10:48.833Z" },
+ { url = "https://files.pythonhosted.org/packages/95/c8/d2e501556dca9f1fbc9246111a31792fb49ad908fa4927f34938a97a3604/fastar-0.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dfe39d91fc28e37e06162d94afe01050220edb7df554acb5b702b5503e564816", size = 1028377, upload-time = "2026-04-13T17:11:06.374Z" },
+ { url = "https://files.pythonhosted.org/packages/db/33/5f11f23eca0a569cd052507bc45dda2e5468697f8665728d25be44120f7d/fastar-0.11.0-cp313-cp313-win32.whl", hash = "sha256:c5f63d4d99ff4bfb37c659982ec413358bdee747005348756cc50a04d412d989", size = 454089, upload-time = "2026-04-13T17:11:46.821Z" },
+ { url = "https://files.pythonhosted.org/packages/da/2f/35ff03c939cba7a255a9132367873fec6c355fd06a7f84fedcbaf4c8129f/fastar-0.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:8690ed1928d31ded3ada308e1086525fb3871f5fa81e1b69601a3f7774004583", size = 486312, upload-time = "2026-04-13T17:11:32.86Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/71/ee9246cbfcbfd4144558f35e7e9a306ffe0a7564730a5188c45f21d2dab8/fastar-0.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:d977ded9d98a0719a305e0a4d5ee811f1d3e856d853a50acb8ae833c3cd6d5d2", size = 461975, upload-time = "2026-04-13T17:11:22.589Z" },
+]
+
+[[package]]
+name = "fastjsonschema"
+version = "2.21.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/20/b5/23b216d9d985a956623b6bd12d4086b60f0059b27799f23016af04a74ea1/fastjsonschema-2.21.2.tar.gz", hash = "sha256:b1eb43748041c880796cd077f1a07c3d94e93ae84bba5ed36800a33554ae05de", size = 374130, upload-time = "2025-08-14T18:49:36.666Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cb/a8/20d0723294217e47de6d9e2e40fd4a9d2f7c4b6ef974babd482a59743694/fastjsonschema-2.21.2-py3-none-any.whl", hash = "sha256:1c797122d0a86c5cace2e54bf4e819c36223b552017172f32c5c024a6b77e463", size = 24024, upload-time = "2025-08-14T18:49:34.776Z" },
+]
+
+[[package]]
+name = "fastsafetensors"
+version = "0.3.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/07/4c/f17bd54c933fd23648ce1e4272adf27d8d7e99b788c8859544bbd39f02e7/fastsafetensors-0.3.3.tar.gz", hash = "sha256:ba4fb59be8a6adbc91723848c3c6f57a9a9a5d2247d9768f84511380d01e554c", size = 77817, upload-time = "2026-07-07T07:21:47.121Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/16/a2/fd30fe6ed5825cb4642bdd102d3c5a32d2d82c91bdb45ae61a9571278cf8/fastsafetensors-0.3.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f969b2c818942748ff1bc619e6d575a7a7156a7d488d4c41779fba8c4e1891dc", size = 1887148, upload-time = "2026-07-07T07:21:38.414Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/9d/d3db53dd5b5069b44f2b8f6fc5a943f4101178a6f74b48a7fc72dc68f142/fastsafetensors-0.3.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bbf9fbc5f74d6a3f333777eb222aa63453795dacd7ae7f559316a5398d4942ef", size = 1915579, upload-time = "2026-07-07T07:21:40.047Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/d3/4193c0ae305edb94a45f4b0789cf705106b87a1546ca5dafb211c8ab274a/fastsafetensors-0.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:b61a5f6e0e0b0da591f2790d3fbc6eef7f6e8597087977b946069e7bfbb2ec7a", size = 424967, upload-time = "2026-07-07T07:21:41.243Z" },
+]
+
+[[package]]
+name = "fastuuid"
+version = "0.14.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/7d/d9daedf0f2ebcacd20d599928f8913e9d2aea1d56d2d355a93bfa2b611d7/fastuuid-0.14.0.tar.gz", hash = "sha256:178947fc2f995b38497a74172adee64fdeb8b7ec18f2a5934d037641ba265d26", size = 18232, upload-time = "2025-10-19T22:19:22.402Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a5/83/ae12dd39b9a39b55d7f90abb8971f1a5f3c321fd72d5aa83f90dc67fe9ed/fastuuid-0.14.0-cp313-cp313-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:77a09cb7427e7af74c594e409f7731a0cf887221de2f698e1ca0ebf0f3139021", size = 510720, upload-time = "2025-10-19T22:42:34.633Z" },
+ { url = "https://files.pythonhosted.org/packages/53/b0/a4b03ff5d00f563cc7546b933c28cb3f2a07344b2aec5834e874f7d44143/fastuuid-0.14.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:9bd57289daf7b153bfa3e8013446aa144ce5e8c825e9e366d455155ede5ea2dc", size = 262024, upload-time = "2025-10-19T22:30:25.482Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/6d/64aee0a0f6a58eeabadd582e55d0d7d70258ffdd01d093b30c53d668303b/fastuuid-0.14.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ac60fc860cdf3c3f327374db87ab8e064c86566ca8c49d2e30df15eda1b0c2d5", size = 251679, upload-time = "2025-10-19T22:36:14.096Z" },
+ { url = "https://files.pythonhosted.org/packages/60/f5/a7e9cda8369e4f7919d36552db9b2ae21db7915083bc6336f1b0082c8b2e/fastuuid-0.14.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ab32f74bd56565b186f036e33129da77db8be09178cd2f5206a5d4035fb2a23f", size = 277862, upload-time = "2025-10-19T22:36:23.302Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/d3/8ce11827c783affffd5bd4d6378b28eb6cc6d2ddf41474006b8d62e7448e/fastuuid-0.14.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:33e678459cf4addaedd9936bbb038e35b3f6b2061330fd8f2f6a1d80414c0f87", size = 278278, upload-time = "2025-10-19T22:29:43.809Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/51/680fb6352d0bbade04036da46264a8001f74b7484e2fd1f4da9e3db1c666/fastuuid-0.14.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1e3cc56742f76cd25ecb98e4b82a25f978ccffba02e4bdce8aba857b6d85d87b", size = 301788, upload-time = "2025-10-19T22:36:06.825Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/7c/2014b5785bd8ebdab04ec857635ebd84d5ee4950186a577db9eff0fb8ff6/fastuuid-0.14.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:cb9a030f609194b679e1660f7e32733b7a0f332d519c5d5a6a0a580991290022", size = 459819, upload-time = "2025-10-19T22:35:31.623Z" },
+ { url = "https://files.pythonhosted.org/packages/01/d2/524d4ceeba9160e7a9bc2ea3e8f4ccf1ad78f3bde34090ca0c51f09a5e91/fastuuid-0.14.0-cp313-cp313-musllinux_1_1_i686.whl", hash = "sha256:09098762aad4f8da3a888eb9ae01c84430c907a297b97166b8abc07b640f2995", size = 478546, upload-time = "2025-10-19T22:26:03.023Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/17/354d04951ce114bf4afc78e27a18cfbd6ee319ab1829c2d5fb5e94063ac6/fastuuid-0.14.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:1383fff584fa249b16329a059c68ad45d030d5a4b70fb7c73a08d98fd53bcdab", size = 450921, upload-time = "2025-10-19T22:31:02.151Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/be/d7be8670151d16d88f15bb121c5b66cdb5ea6a0c2a362d0dcf30276ade53/fastuuid-0.14.0-cp313-cp313-win32.whl", hash = "sha256:a0809f8cc5731c066c909047f9a314d5f536c871a7a22e815cc4967c110ac9ad", size = 154559, upload-time = "2025-10-19T22:36:36.011Z" },
+ { url = "https://files.pythonhosted.org/packages/22/1d/5573ef3624ceb7abf4a46073d3554e37191c868abc3aecd5289a72f9810a/fastuuid-0.14.0-cp313-cp313-win_amd64.whl", hash = "sha256:0df14e92e7ad3276327631c9e7cec09e32572ce82089c55cb1bb8df71cf394ed", size = 156539, upload-time = "2025-10-19T22:33:35.898Z" },
+]
+
+[[package]]
+name = "filelock"
+version = "3.29.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e3/ee/29c668c50888588c432a702f7c2e8ee8a0c9e5286028d91f170308d6b2e9/filelock-3.29.5.tar.gz", hash = "sha256:6e6034c57a00a020e767f2614a5539863f056de7e7991d6d1473aef7ff73f156", size = 68927, upload-time = "2026-07-03T03:50:31.818Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4a/e3/f1fae3647d170919c2cf2a898e77e7d1a4e5c7cae0aed7bb4bd3f5ebff6f/filelock-3.29.5-py3-none-any.whl", hash = "sha256:8af830889ba3a0ffcefbd6c7d2af8a54012058103771f2e10848222f476a1693", size = 45073, upload-time = "2026-07-03T03:50:30.445Z" },
+]
+
+[[package]]
+name = "flashinfer-python"
+version = "0.6.16.post3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "apache-tvm-ffi" },
+ { name = "click" },
+ { name = "cuda-python", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-python", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "cuda-tile" },
+ { name = "einops" },
+ { name = "nccl4py" },
+ { name = "ninja" },
+ { name = "numpy" },
+ { name = "nvidia-cudnn-frontend" },
+ { name = "nvidia-cutlass-dsl" },
+ { name = "nvidia-ml-py" },
+ { name = "packaging" },
+ { name = "requests" },
+ { name = "tabulate" },
+ { name = "torch" },
+ { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c2/46/17045cb47b7b93fcb4e5303e398e6b129b86239930ac0875942a83d8b962/flashinfer_python-0.6.16.post3.tar.gz", hash = "sha256:9b146cfcc1454c80f3f99444db48013b3eadaeb32f2a399bd76c9471ecb72f04", size = 11076656, upload-time = "2026-08-08T01:14:16.408Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9d/dc/5367cb601fc9190cc75b4c141f5f7e9a224d1c26a1e983151823e02a25b3/flashinfer_python-0.6.16.post3-py3-none-any.whl", hash = "sha256:caf686b9b079abe1c9d65ab505698bd325e8072de40afd822f2c74f2ac3bc601", size = 15836034, upload-time = "2026-08-08T01:14:13.709Z" },
+]
+
+[[package]]
+name = "fonttools"
+version = "4.63.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/84/69/c97f2c18e0db87d2c7b15da1974dace76ae938f1cfa22e2727a648b7ed43/fonttools-4.63.0.tar.gz", hash = "sha256:caeb583deeb5168e694b65cda8b4ee62abedfa66cf88488734466f2366b9c4e0", size = 3597189, upload-time = "2026-05-14T12:04:30.958Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0f/8d/d8fec3dcde2963f8c908fb315e5ff2cd0ac34f82394bbbf73a2aa5145ce3/fonttools-4.63.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:cd7e9857e5e63738b9d9fd707bc1f59c8b09e5177726d23664db393c59bb08bd", size = 2876062, upload-time = "2026-05-14T12:03:32.554Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/71/d935dc54e4ff121bfdd11e08702db63a7e6f25af21d8a3d7b7212df53641/fonttools-4.63.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c2a2a42198b696a6f48fad91709afb55176e66a5e566131219dba372fb7f8c59", size = 2424594, upload-time = "2026-05-14T12:03:34.86Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/40/e76320afa1df918e146155ef239b1719ee266092e96f5423bfd075affba1/fonttools-4.63.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1e874792a8212b44583ea02189d9e693906b2f78b261f372f95d6c563210ac1d", size = 5024840, upload-time = "2026-05-14T12:03:36.745Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/36/0b805d8c485f872f65a509cbe3b58a5d0d17bee855333b54a150c79d3061/fonttools-4.63.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:22135da48a348785c5e2d5d2d9d6bec5ed44adacbaeb9db12d9493bf6c6bfa68", size = 4975801, upload-time = "2026-05-14T12:03:38.833Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/26/2cee03d0aa083ab022da5c07aff9ed3f689da1defb81ad6917c9627896da/fonttools-4.63.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ccf41f2efdf56994d22d73bef4ced1052161958169428d06ba9724ea9e9a64be", size = 4965009, upload-time = "2026-05-14T12:03:41.494Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/48/cc4b66d9058c0d0982c833fad10127c4b0e9324606aafa41382295ca4102/fonttools-4.63.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9ced0bd02ac751dd6319b0da88aaef24414e3b0dbc32bb4f24944821a3741a27", size = 5105892, upload-time = "2026-05-14T12:03:43.525Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/1f/a98a30a814b9ddef3a2e706025f90b9e0bc94890e6cb15254bc86547d11a/fonttools-4.63.0-cp313-cp313-win32.whl", hash = "sha256:85be818f5506e8a7753153def2c9550178f0ecae6a47b5e0e8dbb23f7cc90380", size = 2291313, upload-time = "2026-05-14T12:03:45.594Z" },
+ { url = "https://files.pythonhosted.org/packages/92/46/5177b01f3b4abfdd4409f31cca4ab279c9343a26efbe9ec78c97fc612e02/fonttools-4.63.0-cp313-cp313-win_amd64.whl", hash = "sha256:ba04cb5891d4c0c21b6da95eda8d7b090021508a294fff33464fc7d241e0856b", size = 2342299, upload-time = "2026-05-14T12:03:47.414Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" },
+]
+
+[[package]]
+name = "fqdn"
+version = "1.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/30/3e/a80a8c077fd798951169626cde3e239adeba7dab75deb3555716415bd9b0/fqdn-1.5.1.tar.gz", hash = "sha256:105ed3677e767fb5ca086a0c1f4bb66ebc3c100be518f0e0d755d9eae164d89f", size = 6015, upload-time = "2021-03-11T07:16:29.08Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cf/58/8acf1b3e91c58313ce5cb67df61001fc9dcd21be4fadb76c1a2d540e09ed/fqdn-1.5.1-py3-none-any.whl", hash = "sha256:3a179af3761e4df6eb2e026ff9e1a3033d3587bf980a0b1b2e1e5d08d7358014", size = 9121, upload-time = "2021-03-11T07:16:28.351Z" },
+]
+
+[[package]]
+name = "frozendict"
+version = "2.4.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/90/b2/2a3d1374b7780999d3184e171e25439a8358c47b481f68be883c14086b4c/frozendict-2.4.7.tar.gz", hash = "sha256:e478fb2a1391a56c8a6e10cc97c4a9002b410ecd1ac28c18d780661762e271bd", size = 317082, upload-time = "2025-11-11T22:40:14.251Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/74/f94141b38a51a553efef7f510fc213894161ae49b88bffd037f8d2a7cb2f/frozendict-2.4.7-py3-none-any.whl", hash = "sha256:972af65924ea25cf5b4d9326d549e69a9a4918d8a76a9d3a7cd174d98b237550", size = 16264, upload-time = "2025-11-11T22:40:12.836Z" },
+]
+
+[[package]]
+name = "frozenlist"
+version = "1.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875, upload-time = "2025-10-06T05:38:17.865Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2d/40/0832c31a37d60f60ed79e9dfb5a92e1e2af4f40a16a29abcc7992af9edff/frozenlist-1.8.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a", size = 85717, upload-time = "2025-10-06T05:36:27.341Z" },
+ { url = "https://files.pythonhosted.org/packages/30/ba/b0b3de23f40bc55a7057bd38434e25c34fa48e17f20ee273bbde5e0650f3/frozenlist-1.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7", size = 49651, upload-time = "2025-10-06T05:36:28.855Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/ab/6e5080ee374f875296c4243c381bbdef97a9ac39c6e3ce1d5f7d42cb78d6/frozenlist-1.8.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40", size = 49417, upload-time = "2025-10-06T05:36:29.877Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/4e/e4691508f9477ce67da2015d8c00acd751e6287739123113a9fca6f1604e/frozenlist-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027", size = 234391, upload-time = "2025-10-06T05:36:31.301Z" },
+ { url = "https://files.pythonhosted.org/packages/40/76/c202df58e3acdf12969a7895fd6f3bc016c642e6726aa63bd3025e0fc71c/frozenlist-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822", size = 233048, upload-time = "2025-10-06T05:36:32.531Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/c0/8746afb90f17b73ca5979c7a3958116e105ff796e718575175319b5bb4ce/frozenlist-1.8.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121", size = 226549, upload-time = "2025-10-06T05:36:33.706Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/eb/4c7eefc718ff72f9b6c4893291abaae5fbc0c82226a32dcd8ef4f7a5dbef/frozenlist-1.8.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5", size = 239833, upload-time = "2025-10-06T05:36:34.947Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/4e/e5c02187cf704224f8b21bee886f3d713ca379535f16893233b9d672ea71/frozenlist-1.8.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e", size = 245363, upload-time = "2025-10-06T05:36:36.534Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/96/cb85ec608464472e82ad37a17f844889c36100eed57bea094518bf270692/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11", size = 229314, upload-time = "2025-10-06T05:36:38.582Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/6f/4ae69c550e4cee66b57887daeebe006fe985917c01d0fff9caab9883f6d0/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1", size = 243365, upload-time = "2025-10-06T05:36:40.152Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/58/afd56de246cf11780a40a2c28dc7cbabbf06337cc8ddb1c780a2d97e88d8/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1", size = 237763, upload-time = "2025-10-06T05:36:41.355Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/36/cdfaf6ed42e2644740d4a10452d8e97fa1c062e2a8006e4b09f1b5fd7d63/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8", size = 240110, upload-time = "2025-10-06T05:36:42.716Z" },
+ { url = "https://files.pythonhosted.org/packages/03/a8/9ea226fbefad669f11b52e864c55f0bd57d3c8d7eb07e9f2e9a0b39502e1/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed", size = 233717, upload-time = "2025-10-06T05:36:44.251Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/0b/1b5531611e83ba7d13ccc9988967ea1b51186af64c42b7a7af465dcc9568/frozenlist-1.8.0-cp313-cp313-win32.whl", hash = "sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496", size = 39628, upload-time = "2025-10-06T05:36:45.423Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/cf/174c91dbc9cc49bc7b7aab74d8b734e974d1faa8f191c74af9b7e80848e6/frozenlist-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231", size = 43882, upload-time = "2025-10-06T05:36:46.796Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/17/502cd212cbfa96eb1388614fe39a3fc9ab87dbbe042b66f97acb57474834/frozenlist-1.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62", size = 39676, upload-time = "2025-10-06T05:36:47.8Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/5c/3bbfaa920dfab09e76946a5d2833a7cbdf7b9b4a91c714666ac4855b88b4/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94", size = 89235, upload-time = "2025-10-06T05:36:48.78Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/d6/f03961ef72166cec1687e84e8925838442b615bd0b8854b54923ce5b7b8a/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c", size = 50742, upload-time = "2025-10-06T05:36:49.837Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/bb/a6d12b7ba4c3337667d0e421f7181c82dda448ce4e7ad7ecd249a16fa806/frozenlist-1.8.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52", size = 51725, upload-time = "2025-10-06T05:36:50.851Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/71/d1fed0ffe2c2ccd70b43714c6cab0f4188f09f8a67a7914a6b46ee30f274/frozenlist-1.8.0-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51", size = 284533, upload-time = "2025-10-06T05:36:51.898Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/1f/fb1685a7b009d89f9bf78a42d94461bc06581f6e718c39344754a5d9bada/frozenlist-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65", size = 292506, upload-time = "2025-10-06T05:36:53.101Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/3b/b991fe1612703f7e0d05c0cf734c1b77aaf7c7d321df4572e8d36e7048c8/frozenlist-1.8.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82", size = 274161, upload-time = "2025-10-06T05:36:54.309Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/ec/c5c618767bcdf66e88945ec0157d7f6c4a1322f1473392319b7a2501ded7/frozenlist-1.8.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714", size = 294676, upload-time = "2025-10-06T05:36:55.566Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/ce/3934758637d8f8a88d11f0585d6495ef54b2044ed6ec84492a91fa3b27aa/frozenlist-1.8.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d", size = 300638, upload-time = "2025-10-06T05:36:56.758Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/4f/a7e4d0d467298f42de4b41cbc7ddaf19d3cfeabaf9ff97c20c6c7ee409f9/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506", size = 283067, upload-time = "2025-10-06T05:36:57.965Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/48/c7b163063d55a83772b268e6d1affb960771b0e203b632cfe09522d67ea5/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51", size = 292101, upload-time = "2025-10-06T05:36:59.237Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/d0/2366d3c4ecdc2fd391e0afa6e11500bfba0ea772764d631bbf82f0136c9d/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e", size = 289901, upload-time = "2025-10-06T05:37:00.811Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/94/daff920e82c1b70e3618a2ac39fbc01ae3e2ff6124e80739ce5d71c9b920/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0", size = 289395, upload-time = "2025-10-06T05:37:02.115Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/20/bba307ab4235a09fdcd3cc5508dbabd17c4634a1af4b96e0f69bfe551ebd/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41", size = 283659, upload-time = "2025-10-06T05:37:03.711Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/00/04ca1c3a7a124b6de4f8a9a17cc2fcad138b4608e7a3fc5877804b8715d7/frozenlist-1.8.0-cp313-cp313t-win32.whl", hash = "sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b", size = 43492, upload-time = "2025-10-06T05:37:04.915Z" },
+ { url = "https://files.pythonhosted.org/packages/59/5e/c69f733a86a94ab10f68e496dc6b7e8bc078ebb415281d5698313e3af3a1/frozenlist-1.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888", size = 48034, upload-time = "2025-10-06T05:37:06.343Z" },
+ { url = "https://files.pythonhosted.org/packages/16/6c/be9d79775d8abe79b05fa6d23da99ad6e7763a1d080fbae7290b286093fd/frozenlist-1.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042", size = 41749, upload-time = "2025-10-06T05:37:07.431Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/9a/e35b4a917281c0b8419d4207f4334c8e8c5dbf4f3f5f9ada73958d937dcc/frozenlist-1.8.0-py3-none-any.whl", hash = "sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d", size = 13409, upload-time = "2025-10-06T05:38:16.721Z" },
+]
+
+[[package]]
+name = "fsspec"
+version = "2025.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/de/e0/bab50af11c2d75c9c4a2a26a5254573c0bd97cea152254401510950486fa/fsspec-2025.9.0.tar.gz", hash = "sha256:19fd429483d25d28b65ec68f9f4adc16c17ea2c7c7bf54ec61360d478fb19c19", size = 304847, upload-time = "2025-09-02T19:10:49.215Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/47/71/70db47e4f6ce3e5c37a607355f80da8860a33226be640226ac52cb05ef2e/fsspec-2025.9.0-py3-none-any.whl", hash = "sha256:530dc2a2af60a414a832059574df4a6e10cce927f6f4a78209390fe38955cfb7", size = 199289, upload-time = "2025-09-02T19:10:47.708Z" },
+]
+
+[package.optional-dependencies]
+http = [
+ { name = "aiohttp" },
+]
+
+[[package]]
+name = "gitdb"
+version = "4.0.12"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "smmap" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/72/94/63b0fc47eb32792c7ba1fe1b694daec9a63620db1e313033d18140c2320a/gitdb-4.0.12.tar.gz", hash = "sha256:5ef71f855d191a3326fcfbc0d5da835f26b13fbcba60c32c21091c349ffdb571", size = 394684, upload-time = "2025-01-02T07:20:46.413Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" },
+]
+
+[[package]]
+name = "gitpython"
+version = "3.1.50"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "gitdb" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/33/f6/354ae6491228b5eb40e10d89c4d13c651fe1cf7556e35ebdded50cff57ce/gitpython-3.1.50.tar.gz", hash = "sha256:80da2d12504d52e1f998772dc5baf6e553f8d2fcfe1fcc226c9d9a2ee3372dcc", size = 219798, upload-time = "2026-05-06T04:01:26.571Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/20/7a/1c6e3562dfd8950adbb11ffbc65d21e7c89d01a6e4f137fa981056de25c5/gitpython-3.1.50-py3-none-any.whl", hash = "sha256:d352abe2908d07355014abdd21ddf798c2a961469239afec4962e9da884858f9", size = 212507, upload-time = "2026-05-06T04:01:23.799Z" },
+]
+
+[[package]]
+name = "googleapis-common-protos"
+version = "1.75.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "protobuf" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b5/c8/f439cffde755cffa462bfbb156278fa6f9d09119719af9814b858fd4f81f/googleapis_common_protos-1.75.0.tar.gz", hash = "sha256:53a062ff3c32552fbd62c11fe23768b78e4ddf0494d5e5fd97d3f4689c75fbbd", size = 151035, upload-time = "2026-05-07T08:04:49.423Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/c8/e2645aa8ed02fd4c7a2f59d68783b65b1f3cbdfe39a6308e156509d1fee8/googleapis_common_protos-1.75.0-py3-none-any.whl", hash = "sha256:961ed60399c457ceb0ee8f285a84c870aabc9c6a832b9d37bb281b5bebde43ed", size = 300631, upload-time = "2026-05-07T08:03:30.345Z" },
+]
+
+[[package]]
+name = "grpcio"
+version = "1.81.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b0/b5/1ff353970a87eda4c98251e34d2dfd214abd4982dc89119c9252a2a482d2/grpcio-1.81.1.tar.gz", hash = "sha256:6fa10a767143a5e82e8eaab53918af0cd8909a57a27f8cb2288b80a613ac671b", size = 13026582, upload-time = "2026-06-11T12:46:51.673Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7a/42/dcc2e4b600538ef18327c0839d56b7d3c3812337c5d710df5877dbb39b1e/grpcio-1.81.1-cp313-cp313-linux_armv7l.whl", hash = "sha256:b10e1ff4756ed27d5a29d7fc79cfce7ef1ff56ad20025b89bac7cf79e09abbbe", size = 6054466, upload-time = "2026-06-11T12:45:48.43Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/4a/a36e03210183a8a7d4c80c3936acee679f4bd77d5861f369db47b2cc5f05/grpcio-1.81.1-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:819edbdcb42ab8598b494bcf0222684bbb7a3c772bd1b1f0be7e029a6063c28e", size = 12048795, upload-time = "2026-06-11T12:45:54.011Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/d5/d68e30b29098f63beab6fe501100fe82674ff142b32c672532da86a99b3a/grpcio-1.81.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c5bf2dc311127d91230cc79b92188c082634a06cf66c5234db49a43b910183b0", size = 6599094, upload-time = "2026-06-11T12:45:57.799Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/b3/e837954d279754f638a11cca5dcf6b24a005efb398984cefaf7735945a54/grpcio-1.81.1-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:e8ca6a1fcdb2943c9cbc1804a1baf3acb6071d72a471591678ded84218006e14", size = 7307182, upload-time = "2026-06-11T12:46:00.568Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/1e/b47957057e729adc6cdf519a47f8be2562b7140e280f1418443eb4022192/grpcio-1.81.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e64dd101d380a115cc5a0c7856788adb535f1a4e21fc543775602f8be95180ae", size = 6810962, upload-time = "2026-06-11T12:46:03.312Z" },
+ { url = "https://files.pythonhosted.org/packages/40/26/569868e364e05b19ec8f969da53d230bcd89c962cd198f7c29943155c4d3/grpcio-1.81.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:98a07f9bf591e3a8919797bee1c53f026ba4acd587e5a4404c8e57c9ec36b2a5", size = 7415698, upload-time = "2026-06-11T12:46:06.005Z" },
+ { url = "https://files.pythonhosted.org/packages/36/0c/5440a0582cb5653fc42a6e262eeb22700943313f8076f9dc927491b20a59/grpcio-1.81.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:c261d74b1a945cf895a9d6eccd1685a8e837531beaab782da4d630a8d12deffb", size = 8407779, upload-time = "2026-06-11T12:46:08.84Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/aa/66fe9f39871d766987d869a03ee0842a026f499c7b1e62decb9e78a8088e/grpcio-1.81.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:58ad1131c300d3c9b933802b3cc4dc69d380822935ba50b28703156ea826fbf7", size = 7844521, upload-time = "2026-06-11T12:46:12.171Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/9e/69bb7194861bcd28fb3193261d4f9c3831b4446993f002cf59068943e7ab/grpcio-1.81.1-cp313-cp313-win32.whl", hash = "sha256:78e29211f26da2fdd0e9c6d2b79f489476140cf7029b6a64808ade7ca4156a42", size = 4182786, upload-time = "2026-06-11T12:46:15.192Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/20/3da8bb0d637feccdc3e1e419bb511ce93651ce7d54164f95de22cc0b8b34/grpcio-1.81.1-cp313-cp313-win_amd64.whl", hash = "sha256:edb59506291b647a30884b1d51a599d605f40b20af4a7dc3d33786a47a31de60", size = 4928648, upload-time = "2026-06-11T12:46:17.823Z" },
+]
+
+[[package]]
+name = "h11"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" },
+]
+
+[[package]]
+name = "hf"
+version = "1.29.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/cb/8e8e878128a3ea229d21058ba184e6dbc3f98c89d4600c2f9d8ef8923210/hf-1.29.0-py3-none-any.whl", hash = "sha256:67c3c4b665fb64c88526310d2e81bc203895d9f59ae19135f089f83883140483", size = 2417, upload-time = "2026-08-27T12:18:43.47Z" },
+]
+
+[[package]]
+name = "hf-xet"
+version = "1.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/ab/522a2ab67f27971a9d48ca666d4fca85ef7d5282d142e31fd087e27b1bbe/hf_xet-1.6.0.tar.gz", hash = "sha256:2e58454a340b3556dfa4972d5451aff4fba8dd42a236600ba1a1d2b1514f0fef", size = 920527, upload-time = "2026-08-03T22:33:13.243Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/50/7afa2c9c787405864fc47a0d1bbc02c62e9101947ed43c1f43899fc7d91d/hf_xet-1.6.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:633dc0cd71d32da58ab8c03ad38e2fac452c15c2b0a2866ebf6ededfe0a5061d", size = 4071729, upload-time = "2026-08-03T22:33:00.721Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/69/55b8dcf636142ae660fec1869fcac14c4da2e8412e14d6eee1523be77e9f/hf_xet-1.6.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:f0906082d9932ae0c0057fa194041c22b4e2cdb46b2592ef3b91f020d62a081a", size = 3876287, upload-time = "2026-08-03T22:33:02.251Z" },
+ { url = "https://files.pythonhosted.org/packages/67/4e/a28359bf1c1ecf11eba22123168c138698f7cb576ac678f5a2e16cd5da08/hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d62671bb130879cef0ee4c9ebe47a14af6c66ec53e6d84dc15936e5ffdfac82f", size = 4464663, upload-time = "2026-08-03T22:33:03.802Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/69/1f0cbc2fb22ae6082d094f743d1b8945a3f36f6089cb95f42b7ee348cda7/hf_xet-1.6.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0e6e21fa3cdfcdcd76748564bf593870a5e013f47d97cf10aed63aa222cff5b7", size = 4262538, upload-time = "2026-08-03T22:33:05.287Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/3a/4f4f2301ade26e404462d3336fa11f7958d914cabbabdd6e03c3c5d5658c/hf_xet-1.6.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:4fc74352a17015bd0ee90038bc9efe38db894cde45f268b6712b04fce8cd0acb", size = 4460520, upload-time = "2026-08-03T22:33:06.81Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/5f/311725e2a905534dfee2dcb5b08414f249147f1f12252bfc2bd24caa075c/hf_xet-1.6.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8fb4f71cba6129110c3374a33f919001ff130488fc23553698e34cc1c2a1198c", size = 4675937, upload-time = "2026-08-03T22:33:08.616Z" },
+ { url = "https://files.pythonhosted.org/packages/98/b7/8c59a66d15205024662f1d66968136f13893f96df1ddc5087e2e281fc95f/hf_xet-1.6.0-cp38-abi3-win_amd64.whl", hash = "sha256:fb4fadde1b2b70bf4c0c14a6dccbe7194b1c28947fefd5bbe3fed9d940676c3b", size = 4033128, upload-time = "2026-08-03T22:33:10.171Z" },
+ { url = "https://files.pythonhosted.org/packages/73/63/ca511b6f802f28cf3489b280fe77475bcca8de85e81a6299d7916b5b5555/hf_xet-1.6.0-cp38-abi3-win_arm64.whl", hash = "sha256:3dc3e35441ba395006af5aaacc40ef2e603c51ef46c3530b9156185f00935ea3", size = 3859359, upload-time = "2026-08-03T22:33:11.725Z" },
+]
+
+[[package]]
+name = "httpcore"
+version = "1.0.9"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "h11" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" },
+]
+
+[[package]]
+name = "httptools"
+version = "0.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/43/e5/d471fcb0e14523fe1c3f4ba58ca52480e7bd70ad7109a3846bc75892f7fb/httptools-0.8.0.tar.gz", hash = "sha256:6b2a32f18d97e16e90827d7a819ffa8dbd8cc245fc4e1fa9d1095b54ef4bd999", size = 271342, upload-time = "2026-05-25T22:17:48.841Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5e/e5/8cfcabc5546e8022f168be28bcdaa128a240a0befdd03b59d558b4f18bd6/httptools-0.8.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:614ceea8ea606848bece2338ac03b3ce5324bcb4be8dc7d377ed708012fa4db8", size = 205148, upload-time = "2026-05-25T22:17:16.333Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/0e/0fb14848c19a686c8062ff9067c1a48793e3224b47bc5b201535b6036fce/httptools-0.8.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2d689918c15a013c65ef52d9fd495d766893ab831a2c8d89f2ac5940a5df847c", size = 111368, upload-time = "2026-05-25T22:17:17.586Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/1b/46f1cecf06b9bbde8e4b8c88034ac7908989e5ff7a3a388ef38392949c1f/httptools-0.8.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:eb3028cca2fc0a6d720e52ef61d8ebb62fcbfeb1de56874546d858d3f25a26b7", size = 486447, upload-time = "2026-05-25T22:17:18.564Z" },
+ { url = "https://files.pythonhosted.org/packages/77/00/258bfc0837221f81d9725c45f9b948a6a6b2994a147a4fb66e85100c668f/httptools-0.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:88bdd940f2b5d487b4d032c6afa5489a7dc4694410d43de3c38c4fb3af0dc45d", size = 482448, upload-time = "2026-05-25T22:17:19.912Z" },
+ { url = "https://files.pythonhosted.org/packages/04/ab/d1cef3b5523f4d272a70f42a776c3169a2dddfe3a54de4b2ce4a36341528/httptools-0.8.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6a43c9dd399758ccc0531acb0a3c4a6c299ee893ee9400e9c893b7bdcfae0681", size = 464460, upload-time = "2026-05-25T22:17:20.882Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/48/5d1d072442277bb2b3434e0e60690b8e8c23840ef7de8b6ea54040a536d3/httptools-0.8.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0770728beb05094c809b98e814edff5fef69d26ad7d21185f2f6d5884a0ba683", size = 471312, upload-time = "2026-05-25T22:17:22.085Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/66/b96623b27e51a68199ef4efdda0613cced9233fe3062ac74e50749c5ad37/httptools-0.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:7685df791fad561384bfb139e77fde27a1ffd93134e016f95a0db424ffbf77b1", size = 90117, upload-time = "2026-05-25T22:17:23.074Z" },
+]
+
+[[package]]
+name = "httpx"
+version = "0.28.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "certifi" },
+ { name = "httpcore" },
+ { name = "idna" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" },
+]
+
+[[package]]
+name = "httpx-sse"
+version = "0.4.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0f/4c/751061ffa58615a32c31b2d82e8482be8dd4a89154f003147acee90f2be9/httpx_sse-0.4.3.tar.gz", hash = "sha256:9b1ed0127459a66014aec3c56bebd93da3c1bc8bb6618c8082039a44889a755d", size = 15943, upload-time = "2025-10-10T21:48:22.271Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" },
+]
+
+[[package]]
+name = "huggingface-hub"
+version = "1.29.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "filelock" },
+ { name = "fsspec" },
+ { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" },
+ { name = "httpx" },
+ { name = "packaging" },
+ { name = "pyyaml" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/64/35/42316e8f6908b6d21bc8df017cc6efba94fb5edbf99b64e28dd142325e20/huggingface_hub-1.29.0.tar.gz", hash = "sha256:6ebb385a581435325cf6d5c5b233d5d4bc91175834d99fd65dae14379b36e9ad", size = 963121, upload-time = "2026-08-27T12:18:37.432Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4e/a5/47c2ea9b228ccbcba8467e9a64823146e8ebbad29855e591d8f5eedcc9c7/huggingface_hub-1.29.0-py3-none-any.whl", hash = "sha256:b00f7782afc14db4bc6572763810a635bdfbab8623d957bfb553bd18e03852cd", size = 795768, upload-time = "2026-08-27T12:18:35.431Z" },
+]
+
+[package.optional-dependencies]
+hf-xet = [
+ { name = "hf-xet" },
+]
+
+[[package]]
+name = "humming-kernels"
+version = "0.1.12"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-bindings", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-bindings", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "jinja2" },
+ { name = "numpy" },
+ { name = "nvidia-ml-py" },
+ { name = "safetensors" },
+ { name = "tabulate" },
+ { name = "torch" },
+ { name = "tqdm" },
+ { name = "triton", version = "3.6.0", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "triton", version = "3.7.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/93/b9/53797fff4b1b31b4567453a365646d724469b1bb0d76c07f8746617d0cb6/humming_kernels-0.1.12.tar.gz", hash = "sha256:c96c37c95f74379067d2d2d0c5154d12ee9a8cfa29a7c3370a7f3e34f207ca72", size = 259323, upload-time = "2026-07-31T14:28:23.124Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/58/0f/01f871dc26f8d7e1683df9464890d48cfd5c4e46c4a264d7ee0868749197/humming_kernels-0.1.12-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:988e8b9da41e3679ae2816bdfb51acab53c05cc56254b5ca98e847f4efcf24fd", size = 318019, upload-time = "2026-07-31T14:28:19.925Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/a2/f96912ee7d68f56e61c15555657ffa057b8afbaa21241c63342b5969bb74/humming_kernels-0.1.12-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:cd3ef712a93f3a9075ea99de2c72bcd3ec89dab3759b3a248d869f5507b60331", size = 322656, upload-time = "2026-07-31T14:28:21.511Z" },
+]
+
+[package.optional-dependencies]
+cu13 = [
+ { name = "nvidia-cuda-cccl" },
+ { name = "nvidia-cuda-nvcc" },
+ { name = "nvidia-cuda-nvrtc", version = "13.0.88", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cuda-nvrtc", version = "13.3.33", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "nvidia-cuda-runtime", version = "13.0.96", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cuda-runtime", version = "13.3.29", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+]
+
+[[package]]
+name = "identify"
+version = "2.6.19"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/52/63/51723b5f116cc04b061cb6f5a561790abf249d25931d515cd375e063e0f4/identify-2.6.19.tar.gz", hash = "sha256:6be5020c38fcb07da56c53733538a3081ea5aa70d36a156f83044bfbf9173842", size = 99567, upload-time = "2026-04-17T18:39:50.265Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/94/84/d9273cd09688070a6523c4aee4663a8538721b2b755c4962aafae0011e72/identify-2.6.19-py2.py3-none-any.whl", hash = "sha256:20e6a87f786f768c092a721ad107fc9df0eb89347be9396cadf3f4abbd1fb78a", size = 99397, upload-time = "2026-04-17T18:39:49.221Z" },
+]
+
+[[package]]
+name = "idna"
+version = "3.18"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" },
+]
+
+[[package]]
+name = "ijson"
+version = "3.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f4/57/60d1a6a512f2f0508d0bc8b4f1cc5616fd3196619b66bd6a01f9155a1292/ijson-3.5.0.tar.gz", hash = "sha256:94688760720e3f5212731b3cb8d30267f9a045fb38fb3870254e7b9504246f31", size = 68658, upload-time = "2026-02-24T03:58:30.974Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/71/d67e764a712c3590627480643a3b51efcc3afa4ef3cb54ee4c989073c97e/ijson-3.5.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:e9cedc10e40dd6023c351ed8bfc7dcfce58204f15c321c3c1546b9c7b12562a4", size = 88544, upload-time = "2026-02-24T03:57:21.293Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/39/f1c299371686153fa3cf5c0736b96247a87a1bee1b7145e6d21f359c505a/ijson-3.5.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:3647649f782ee06c97490b43680371186651f3f69bebe64c6083ee7615d185e5", size = 60495, upload-time = "2026-02-24T03:57:22.501Z" },
+ { url = "https://files.pythonhosted.org/packages/16/94/b1438e204d75e01541bebe3e668fe3e68612d210e9931ae1611062dd0a56/ijson-3.5.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:90e74be1dce05fce73451c62d1118671f78f47c9f6be3991c82b91063bf01fc9", size = 60325, upload-time = "2026-02-24T03:57:23.332Z" },
+ { url = "https://files.pythonhosted.org/packages/30/e2/4aa9c116fa86cc8b0f574f3c3a47409edc1cd4face05d0e589a5a176b05d/ijson-3.5.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:78e9ad73e7be2dd80627504bd5cbf512348c55ce2c06e362ed7683b5220e8568", size = 138774, upload-time = "2026-02-24T03:57:24.683Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/d2/738b88752a70c3be1505faa4dcd7110668c2712e582a6a36488ed1e295d4/ijson-3.5.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9577449313cc94be89a4fe4b3e716c65f09cc19636d5a6b2861c4e80dddebd58", size = 149820, upload-time = "2026-02-24T03:57:26.062Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/df/0b3ab9f393ca8f72ea03bc896ba9fdc987e90ae08cdb51c32a4ee0c14d5e/ijson-3.5.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3e4c1178fb50aff5f5701a30a5152ead82a14e189ce0f6102fa1b5f10b2f54ff", size = 149747, upload-time = "2026-02-24T03:57:27.308Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/a3/b0037119f75131b78cb00acc2657b1a9d0435475f1f2c5f8f5a170b66b9c/ijson-3.5.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0eb402ab026ffb37a918d75af2b7260fe6cfbce13232cc83728a714dd30bd81d", size = 151027, upload-time = "2026-02-24T03:57:28.522Z" },
+ { url = "https://files.pythonhosted.org/packages/22/a0/cb344de1862bf09d8f769c9d25c944078c87dd59a1b496feec5ad96309a4/ijson-3.5.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:5b08ee08355f9f729612a8eb9bf69cc14f9310c3b2a487c6f1c3c65d85216ec4", size = 142996, upload-time = "2026-02-24T03:57:29.774Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/32/a8ffd67182e02ea61f70f62daf43ded4fa8a830a2520a851d2782460aba8/ijson-3.5.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:bda62b6d48442903e7bf56152108afb7f0f1293c2b9bef2f2c369defea76ab18", size = 152068, upload-time = "2026-02-24T03:57:30.969Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/d1/3578df8e75d446aab0ae92e27f641341f586b85e1988536adebc65300cb4/ijson-3.5.0-cp313-cp313-win32.whl", hash = "sha256:8d073d9b13574cfa11083cc7267c238b7a6ed563c2661e79192da4a25f09c82c", size = 53065, upload-time = "2026-02-24T03:57:31.93Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/a2/f7cdaf5896710da3e69e982e44f015a83d168aa0f3a89b6f074b5426779d/ijson-3.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:2419f9e32e0968a876b04d8f26aeac042abd16f582810b576936bbc4c6015069", size = 55499, upload-time = "2026-02-24T03:57:32.773Z" },
+ { url = "https://files.pythonhosted.org/packages/42/65/13e2492d17e19a2084523e18716dc2809159f2287fd2700c735f311e76c4/ijson-3.5.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:4d4b0cd676b8c842f7648c1a783448fac5cd3b98289abd83711b3e275e143524", size = 93019, upload-time = "2026-02-24T03:57:33.976Z" },
+ { url = "https://files.pythonhosted.org/packages/33/92/483fc97ece0c3f1cecabf48f6a7a36e89d19369eec462faaeaa34c788992/ijson-3.5.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:252dec3680a48bb82d475e36b4ae1b3a9d7eb690b951bb98a76c5fe519e30188", size = 62714, upload-time = "2026-02-24T03:57:34.819Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/88/793fe020a0fe9d9eed4c285cf4a5cfdb0a935708b3bde0d72f35c794b513/ijson-3.5.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:aa1b5dca97d323931fde2501172337384c958914d81a9dac7f00f0d4bfc76bc7", size = 62460, upload-time = "2026-02-24T03:57:35.874Z" },
+ { url = "https://files.pythonhosted.org/packages/51/69/f1a2690aa8d4df1f4e262b385e65a933ffdc250b091531bac9a449c19e16/ijson-3.5.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7a5ec7fd86d606094bba6f6f8f87494897102fa4584ef653f3005c51a784c320", size = 199273, upload-time = "2026-02-24T03:57:37.07Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/a2/f1346d5299e79b988ab472dc773d5381ec2d57c23cb2f1af3ede4a810e62/ijson-3.5.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:009f41443e1521847701c6d87fa3923c0b1961be3c7e7de90947c8cb92ea7c44", size = 216884, upload-time = "2026-02-24T03:57:38.346Z" },
+ { url = "https://files.pythonhosted.org/packages/28/3c/8b637e869be87799e6c2c3c275a30a546f086b1aed77e2b7f11512168c5a/ijson-3.5.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e4c3651d1f9fe2839a93fdf8fd1d5ca3a54975349894249f3b1b572bcc4bd577", size = 207306, upload-time = "2026-02-24T03:57:39.718Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/7c/18b1c1df6951ca056782d7580ec40cea4ff9a27a0947d92640d1cc8c4ae3/ijson-3.5.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:945b7abcfcfeae2cde17d8d900870f03536494245dda7ad4f8d056faa303256c", size = 211364, upload-time = "2026-02-24T03:57:40.953Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/55/e795812e82851574a9dba8a53fde045378f531ef14110c6fb55dbd23b443/ijson-3.5.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:0574b0a841ff97495c13e9d7260fbf3d85358b061f540c52a123db9dbbaa2ed6", size = 200608, upload-time = "2026-02-24T03:57:42.272Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/cd/013c85b4749b57a4cb4c2670014d1b32b8db4ab1a7be92ea7aeb5d7fe7b5/ijson-3.5.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f969ffb2b89c5cdf686652d7fb66252bc72126fa54d416317411497276056a18", size = 205127, upload-time = "2026-02-24T03:57:43.286Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/7c/faf643733e3ab677f180018f6a855c4ef70b7c46540987424c563c959e42/ijson-3.5.0-cp313-cp313t-win32.whl", hash = "sha256:59d3f9f46deed1332ad669518b8099920512a78bda64c1f021fcd2aff2b36693", size = 55282, upload-time = "2026-02-24T03:57:44.353Z" },
+ { url = "https://files.pythonhosted.org/packages/69/22/94ddb47c24b491377aca06cd8fc9202cad6ab50619842457d2beefde21ea/ijson-3.5.0-cp313-cp313t-win_amd64.whl", hash = "sha256:5c2839fa233746d8aad3b8cd2354e441613f5df66d721d59da4a09394bd1db2b", size = 58016, upload-time = "2026-02-24T03:57:45.237Z" },
+]
+
+[[package]]
+name = "immutabledict"
+version = "4.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1d/e6/718471048fea0366c3e3d1df3acfd914ca66d571cdffcf6d37bbcd725708/immutabledict-4.3.1.tar.gz", hash = "sha256:f844a669106cfdc73f47b1a9da003782fb17dc955a54c80972e0d93d1c63c514", size = 7806, upload-time = "2026-02-15T10:32:34.668Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a3/ce/f9018bf69ae91b273b6391a095e7c93fa5e1617f25b6ba81ad4b20c9df10/immutabledict-4.3.1-py3-none-any.whl", hash = "sha256:c9facdc0ff30fdb8e35bd16532026cac472a549e182c94fa201b51b25e4bf7bf", size = 5000, upload-time = "2026-02-15T10:32:33.672Z" },
+]
+
+[[package]]
+name = "importlib-metadata"
+version = "8.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "zipp" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e7/72/c600ae4f68c28fc19f9c31b9403053e5dbb8cace2e6842c7b7c3e4d42fe9/importlib_metadata-8.9.0.tar.gz", hash = "sha256:58850626cef4bd2df100378b0f2aea9724a7b92f10770d547725b047078f99ee", size = 56140, upload-time = "2026-03-20T16:56:26.362Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7d/f9/97f2ca8bb3ec6e4b1d64f983ebe98b9a192faddff67fac3d6303a537e670/importlib_metadata-8.9.0-py3-none-any.whl", hash = "sha256:e0f761b6ea91ced3b0844c14c9d955224d538105921f8e6754c00f6ca79fba7f", size = 27220, upload-time = "2026-03-20T16:56:25.07Z" },
+]
+
+[[package]]
+name = "iniconfig"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" },
+]
+
+[[package]]
+name = "inspect-ai"
+version = "0.3.145"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aioboto3" },
+ { name = "aiohttp" },
+ { name = "anyio" },
+ { name = "beautifulsoup4" },
+ { name = "boto3" },
+ { name = "click" },
+ { name = "debugpy" },
+ { name = "docstring-parser" },
+ { name = "frozendict" },
+ { name = "fsspec" },
+ { name = "httpx" },
+ { name = "ijson" },
+ { name = "jsonlines" },
+ { name = "jsonpatch" },
+ { name = "jsonpath-ng" },
+ { name = "jsonref" },
+ { name = "jsonschema" },
+ { name = "mmh3" },
+ { name = "nest-asyncio" },
+ { name = "numpy" },
+ { name = "platformdirs" },
+ { name = "psutil" },
+ { name = "pydantic" },
+ { name = "python-dotenv" },
+ { name = "pyyaml" },
+ { name = "rich" },
+ { name = "s3fs" },
+ { name = "semver" },
+ { name = "shortuuid" },
+ { name = "sniffio" },
+ { name = "tenacity" },
+ { name = "textual" },
+ { name = "typing-extensions" },
+ { name = "universal-pathlib" },
+ { name = "zipp" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/87/5d/7d5ef4f052a6ec7c521d1764fb8dfcd0f98b9e7e81124826299711291b9f/inspect_ai-0.3.145.tar.gz", hash = "sha256:75acc41e5ed37ea181e248b8ded3cb761d6c2fe92f92c911942eee5276f313ca", size = 43273241, upload-time = "2025-11-09T22:04:44.1Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/09/2df83d92a4488dfc6b01ecbd7b53e7b480490b63d603abdad68196d51ca8/inspect_ai-0.3.145-py3-none-any.whl", hash = "sha256:6abff3f25b9e83e304d436fb77a4c8c471af298b678a2484743011b40a316c97", size = 34599281, upload-time = "2025-11-09T22:04:06.005Z" },
+]
+
+[[package]]
+name = "interegular"
+version = "0.3.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/dc/9d/8b6dde58a028a3962ce17e84d5fe73758df61378e00ef8ac3d85da34b0ff/interegular-0.3.3.tar.gz", hash = "sha256:d9b697b21b34884711399ba0f0376914b81899ce670032486d0d048344a76600", size = 24705, upload-time = "2024-01-06T23:01:22.372Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c4/01/72d6472f80651673716d1deda2a5bbb633e563ecf94f4479da5519d69d25/interegular-0.3.3-py37-none-any.whl", hash = "sha256:b0c07007d48c89d6d19f7204972d369b2a77222722e126b6aa63aa721dc3b19c", size = 23635, upload-time = "2024-01-06T23:01:20.829Z" },
+]
+
+[[package]]
+name = "ipykernel"
+version = "7.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "appnope", marker = "sys_platform == 'darwin'" },
+ { name = "comm" },
+ { name = "debugpy" },
+ { name = "ipython" },
+ { name = "jupyter-client" },
+ { name = "jupyter-core" },
+ { name = "matplotlib-inline" },
+ { name = "nest-asyncio2" },
+ { name = "packaging" },
+ { name = "psutil" },
+ { name = "pyzmq" },
+ { name = "tornado" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3d/c4/e4a38f579de4225a561305666f7541cdabb30075def2aa1ac17bd73c1fb5/ipykernel-7.3.0.tar.gz", hash = "sha256:9acaaaf97d16355166e4085afe9d225bfbdf2b7ef520f9df3be8f2b248275e09", size = 184899, upload-time = "2026-06-10T08:41:25.481Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3d/02/77b271f5dc58bfbc0b577c877b2365d1ffea2afe66a80c13f2312820348c/ipykernel-7.3.0-py3-none-any.whl", hash = "sha256:897eb64da762549ef610698fca5e9675195ec6ac8ec7f19d81ce1ca20c876057", size = 120583, upload-time = "2026-06-10T08:41:23.648Z" },
+]
+
+[[package]]
+name = "ipython"
+version = "9.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "decorator" },
+ { name = "ipython-pygments-lexers" },
+ { name = "jedi" },
+ { name = "matplotlib-inline" },
+ { name = "pexpect", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "prompt-toolkit" },
+ { name = "psutil", marker = "sys_platform != 'cygwin' and sys_platform != 'emscripten'" },
+ { name = "pygments" },
+ { name = "stack-data" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/53/59/165d3b4d75cc34add3122c4417ecb229085140ac573103c223cd01dde96f/ipython-9.15.0.tar.gz", hash = "sha256:da2819ce2aa83135257df830660b1176d986c3d2876db24df01974fa955b2756", size = 4442580, upload-time = "2026-06-26T11:03:35.913Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/40/3a/948263ca3b9d65bb2b1b0c521b3a49fad5d59ada58724bd87d2bd5ff3f36/ipython-9.15.0-py3-none-any.whl", hash = "sha256:515ad9c3cdf0c932a5a9f6245419e8aba706b7bd03c3e1d3a1c83d9351d6aa6e", size = 630895, upload-time = "2026-06-26T11:03:33.809Z" },
+]
+
+[[package]]
+name = "ipython-pygments-lexers"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ef/4c/5dd1d8af08107f88c7f741ead7a40854b8ac24ddf9ae850afbcf698aa552/ipython_pygments_lexers-1.1.1.tar.gz", hash = "sha256:09c0138009e56b6854f9535736f4171d855c8c08a563a0dcd8022f78355c7e81", size = 8393, upload-time = "2025-01-17T11:24:34.505Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d9/33/1f075bf72b0b747cb3288d011319aaf64083cf2efef8354174e3ed4540e2/ipython_pygments_lexers-1.1.1-py3-none-any.whl", hash = "sha256:a9462224a505ade19a605f71f8fa63c2048833ce50abc86768a0d81d876dc81c", size = 8074, upload-time = "2025-01-17T11:24:33.271Z" },
+]
+
+[[package]]
+name = "ipywidgets"
+version = "8.1.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "comm" },
+ { name = "ipython" },
+ { name = "jupyterlab-widgets" },
+ { name = "traitlets" },
+ { name = "widgetsnbextension" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4c/ae/c5ce1edc1afe042eadb445e95b0671b03cee61895264357956e61c0d2ac0/ipywidgets-8.1.8.tar.gz", hash = "sha256:61f969306b95f85fba6b6986b7fe45d73124d1d9e3023a8068710d47a22ea668", size = 116739, upload-time = "2025-11-01T21:18:12.393Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/56/6d/0d9848617b9f753b87f214f1c682592f7ca42de085f564352f10f0843026/ipywidgets-8.1.8-py3-none-any.whl", hash = "sha256:ecaca67aed704a338f88f67b1181b58f821ab5dc89c1f0f5ef99db43c1c2921e", size = 139808, upload-time = "2025-11-01T21:18:10.956Z" },
+]
+
+[[package]]
+name = "isoduration"
+version = "20.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "arrow" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7c/1a/3c8edc664e06e6bd06cce40c6b22da5f1429aa4224d0c590f3be21c91ead/isoduration-20.11.0.tar.gz", hash = "sha256:ac2f9015137935279eac671f94f89eb00584f940f5dc49462a0c4ee692ba1bd9", size = 11649, upload-time = "2020-11-01T11:00:00.312Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7b/55/e5326141505c5d5e34c5e0935d2908a74e4561eca44108fbfb9c13d2911a/isoduration-20.11.0-py3-none-any.whl", hash = "sha256:b2904c2a4228c3d44f409c8ae8e2370eb21a26f7ac2ec5446df141dde3452042", size = 11321, upload-time = "2020-11-01T10:59:58.02Z" },
+]
+
+[[package]]
+name = "jedi"
+version = "0.20.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "parso" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/46/b7/a3635f6a2d7cf5b5dd98064fc1d5fbbafcb25477bcea204a3a92145d158b/jedi-0.20.0.tar.gz", hash = "sha256:c3f4ccbd276696f4b19c54618d4fb18f9fc24b0aef02acf704b23f487daa1011", size = 3119416, upload-time = "2026-05-01T23:38:47.814Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9a/93/242e2eab5fe682ffcb8b0084bde703a41d51e17ee0f3a31ff0d9d813620a/jedi-0.20.0-py2.py3-none-any.whl", hash = "sha256:7bdd9c2634f56713299976f4cbd59cb3fa92165cc5e05ea811fb253480728b67", size = 4884812, upload-time = "2026-05-01T23:38:43.919Z" },
+]
+
+[[package]]
+name = "jieba"
+version = "0.42.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c6/cb/18eeb235f833b726522d7ebed54f2278ce28ba9438e3135ab0278d9792a2/jieba-0.42.1.tar.gz", hash = "sha256:055ca12f62674fafed09427f176506079bc135638a14e23e25be909131928db2", size = 19214172, upload-time = "2020-01-20T14:27:23.5Z" }
+
+[[package]]
+name = "jinja2"
+version = "3.1.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markupsafe" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" },
+]
+
+[[package]]
+name = "jiter"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1d/1f/10936e16d8860c70698a1aa939a46aa0224813b782bce4e000e637da0b2d/jiter-0.16.0.tar.gz", hash = "sha256:7b24c3492c5f4f84a37946ad9cf504910cf6a782d6a4e0689b6673c5894b4a1c", size = 176431, upload-time = "2026-06-29T13:05:13.657Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/91/c0/555fc60473d30d66894ba825e63615e3be7524fac23858356afa7a38906c/jiter-0.16.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:41977aa5654023948c2dae2a81cbf9c43343954bef1cd59a154dd15a4d84c195", size = 306203, upload-time = "2026-06-29T13:03:36.243Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/2b/c3eaf16f5d7c9bad66ea32f40a95bd169b29a91217fcc7f081375157e99c/jiter-0.16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d28bb3c26762358dadf3e5bf0bccd29ae987d65e6988d2e6f49829c76b003c09", size = 306489, upload-time = "2026-06-29T13:03:37.846Z" },
+ { url = "https://files.pythonhosted.org/packages/96/3f/02fdfc6705cad96127d883af5c34e4867f554f29ec7705ec1a46156400a9/jiter-0.16.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0542a7189c26920778658fc8fcf2af8bae05bae9924577f71804acef37996536", size = 335453, upload-time = "2026-06-29T13:03:39.221Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/a6/e4bda5920d4b0d7c5dfb7174ce4a6b2e4d3e11c9162c452ef0eab4cdbdbd/jiter-0.16.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8fb8de1e23a0cb2a7f53c335049c7b72b6db41aa6227cdcc0972a1de5cb39450", size = 361625, upload-time = "2026-06-29T13:03:40.597Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/97/4e6b59b2c6e55cbb3e183595f81ad65dcfb21c915fee5e19e335df21bc55/jiter-0.16.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b72d0b2990ca754a9102779ac98d8597b7cb31678958562214a007f909eab78e", size = 456958, upload-time = "2026-06-29T13:03:42.074Z" },
+ { url = "https://files.pythonhosted.org/packages/15/e0/97e9557686d2f94f4b93786eccb7eed28e9228ad132ea8237f44727314a7/jiter-0.16.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d5f91b1c27fc22a57993d5a5cb8a627cb8ed4b10502716fac1ffbfe1d19d84e8", size = 372017, upload-time = "2026-06-29T13:03:43.658Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/94/db768b6938e0df35c86beeba3dfbbb025c9ee5c19e1aa271f2396e50864d/jiter-0.16.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c682bea068a90b764577bdb78a60a4c1d1606daf9cd4c893832a37c7cc9d9026", size = 343320, upload-time = "2026-06-29T13:03:45.226Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/d6/5a59d938244a30735fe62d9433fd325f9021ea29d89780ea4596ea93bc89/jiter-0.16.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:8d031aabecc4f1b6276adfb42e3aabb77c89d468bf616600e8d3a11328929053", size = 350520, upload-time = "2026-06-29T13:03:46.671Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f8/c4a857f49c9af125f6bbcac7e3eee7f7978ed89682833062e2dbf62576b1/jiter-0.16.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:eab2cd170150e70153de16896a1774e3a1dca80154c56b54d7a812c479a7165e", size = 387550, upload-time = "2026-06-29T13:03:48.361Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/d6/5fbc2f7d6b67b754caa61a993a2e626e815dec47ffc2f9e35f01adfebec7/jiter-0.16.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:6edb63a46e65a82c26800a868e49b2cac30dd5a4218b88d74bc2c848c8ad60bb", size = 515424, upload-time = "2026-06-29T13:03:49.881Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/54/284f0164b64a5fed915fea6ba7e9ba9b3d8d37c67d59cf2e3bb99d45cdfe/jiter-0.16.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:659039cc50b5addcc35fcc87ae2c1833b7c0a8e5326ef631a75e4478447bcf84", size = 546981, upload-time = "2026-06-29T13:03:51.363Z" },
+ { url = "https://files.pythonhosted.org/packages/13/c5/2a467585a576594384e1d2c43e1224deaafc085f24e243529cf98beef8e1/jiter-0.16.0-cp313-cp313-win32.whl", hash = "sha256:c9c53be232c2e206ef9cdbad81a48bfa74c3d3f08bcf8124630a8a748aad993e", size = 202853, upload-time = "2026-06-29T13:03:53.015Z" },
+ { url = "https://files.pythonhosted.org/packages/88/6a/de61d04b9eec69c71719968d2f716532a3bc121170c44a39e14979c6be81/jiter-0.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:baad945ed47f163ad833314f8e3288c396118934f94e7bbb9e243ce4b341a4fd", size = 196160, upload-time = "2026-06-29T13:03:54.447Z" },
+ { url = "https://files.pythonhosted.org/packages/19/4b/b390ed59bafb3f31d008d1218578f10327714484b334439947f7e5b11e7f/jiter-0.16.0-cp313-cp313-win_arm64.whl", hash = "sha256:3c1fd2dbe1b0af19e987f03fe66c5f5bd105a2229c1aff4ab14890b24f41d21a", size = 189862, upload-time = "2026-06-29T13:03:55.754Z" },
+]
+
+[[package]]
+name = "jmespath"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d3/59/322338183ecda247fb5d1763a6cbe46eff7222eaeebafd9fa65d4bf5cb11/jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d", size = 27377, upload-time = "2026-01-22T16:35:26.279Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/14/2f/967ba146e6d58cf6a652da73885f52fc68001525b4197effc174321d70b4/jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64", size = 20419, upload-time = "2026-01-22T16:35:24.919Z" },
+]
+
+[[package]]
+name = "joblib"
+version = "1.5.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/41/f2/d34e8b3a08a9cc79a50b2208a93dce981fe615b64d5a4d4abee421d898df/joblib-1.5.3.tar.gz", hash = "sha256:8561a3269e6801106863fd0d6d84bb737be9e7631e33aaed3fb9ce5953688da3", size = 331603, upload-time = "2025-12-15T08:41:46.427Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl", hash = "sha256:5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713", size = 309071, upload-time = "2025-12-15T08:41:44.973Z" },
+]
+
+[[package]]
+name = "json5"
+version = "0.15.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e4/7d/05c46a96a78147ae3bf99c2f4169ce144a70220b8d6fcd56f6ec368b8ce9/json5-0.15.0.tar.gz", hash = "sha256:7424d1f1eb1d56da6e3d70643f53619862b4ce81440bdb8ecfd6f875e5ba4a71", size = 53278, upload-time = "2026-06-19T20:08:27.716Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/eb/be/59527c99478aade6bb33a68d72e6e18dd4e6ff6eacfc7d01bdb15bc76912/json5-0.15.0-py3-none-any.whl", hash = "sha256:56636a30c0e8a4665fe2179c0212f32eae3796dea89ea6f649b9436ecdb39618", size = 36570, upload-time = "2026-06-19T20:08:26.748Z" },
+]
+
+[[package]]
+name = "jsonlines"
+version = "4.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/35/87/bcda8e46c88d0e34cad2f09ee2d0c7f5957bccdb9791b0b934ec84d84be4/jsonlines-4.0.0.tar.gz", hash = "sha256:0c6d2c09117550c089995247f605ae4cf77dd1533041d366351f6f298822ea74", size = 11359, upload-time = "2023-09-01T12:34:44.187Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f8/62/d9ba6323b9202dd2fe166beab8a86d29465c41a0288cbe229fac60c1ab8d/jsonlines-4.0.0-py3-none-any.whl", hash = "sha256:185b334ff2ca5a91362993f42e83588a360cf95ce4b71a73548502bda52a7c55", size = 8701, upload-time = "2023-09-01T12:34:42.563Z" },
+]
+
+[[package]]
+name = "jsonpatch"
+version = "1.33"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jsonpointer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/78/18813351fe5d63acad16aec57f94ec2b70a09e53ca98145589e185423873/jsonpatch-1.33.tar.gz", hash = "sha256:9fcd4009c41e6d12348b4a0ff2563ba56a2923a7dfee731d004e212e1ee5030c", size = 21699, upload-time = "2023-06-26T12:07:29.144Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/73/07/02e16ed01e04a374e644b575638ec7987ae846d25ad97bcc9945a3ee4b0e/jsonpatch-1.33-py2.py3-none-any.whl", hash = "sha256:0ae28c0cd062bbd8b8ecc26d7d164fbbea9652a1a3693f3b956c1eae5145dade", size = 12898, upload-time = "2023-06-16T21:01:28.466Z" },
+]
+
+[[package]]
+name = "jsonpath-ng"
+version = "1.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/32/58/250751940d75c8019659e15482d548a4aa3b6ce122c515102a4bfdac50e3/jsonpath_ng-1.8.0.tar.gz", hash = "sha256:54252968134b5e549ea5b872f1df1168bd7defe1a52fed5a358c194e1943ddc3", size = 74513, upload-time = "2026-02-24T14:42:06.182Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/03/99/33c7d78a3fb70d545fd5411ac67a651c81602cc09c9cf0df383733f068c5/jsonpath_ng-1.8.0-py3-none-any.whl", hash = "sha256:b8dde192f8af58d646fc031fac9c99fe4d00326afc4148f1f043c601a8cfe138", size = 67844, upload-time = "2026-02-28T00:53:19.637Z" },
+]
+
+[[package]]
+name = "jsonpointer"
+version = "3.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/18/c7/af399a2e7a67fd18d63c40c5e62d3af4e67b836a2107468b6a5ea24c4304/jsonpointer-3.1.1.tar.gz", hash = "sha256:0b801c7db33a904024f6004d526dcc53bbb8a4a0f4e32bfd10beadf60adf1900", size = 9068, upload-time = "2026-03-23T22:32:32.458Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/6a/a83720e953b1682d2d109d3c2dbb0bc9bf28cc1cbc205be4ef4be5da709d/jsonpointer-3.1.1-py3-none-any.whl", hash = "sha256:8ff8b95779d071ba472cf5bc913028df06031797532f08a7d5b602d8b2a488ca", size = 7659, upload-time = "2026-03-23T22:32:31.568Z" },
+]
+
+[[package]]
+name = "jsonref"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/0d/c1f3277e90ccdb50d33ed5ba1ec5b3f0a242ed8c1b1a85d3afeb68464dca/jsonref-1.1.0.tar.gz", hash = "sha256:32fe8e1d85af0fdefbebce950af85590b22b60f9e95443176adbde4e1ecea552", size = 8814, upload-time = "2023-01-16T16:10:04.455Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/ec/e1db9922bceb168197a558a2b8c03a7963f1afe93517ddd3cf99f202f996/jsonref-1.1.0-py3-none-any.whl", hash = "sha256:590dc7773df6c21cbf948b5dac07a72a251db28b0238ceecce0a2abfa8ec30a9", size = 9425, upload-time = "2023-01-16T16:10:02.255Z" },
+]
+
+[[package]]
+name = "jsonschema"
+version = "4.26.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "jsonschema-specifications" },
+ { name = "referencing" },
+ { name = "rpds-py" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" },
+]
+
+[package.optional-dependencies]
+format-nongpl = [
+ { name = "fqdn" },
+ { name = "idna" },
+ { name = "isoduration" },
+ { name = "jsonpointer" },
+ { name = "rfc3339-validator" },
+ { name = "rfc3986-validator" },
+ { name = "rfc3987-syntax" },
+ { name = "uri-template" },
+ { name = "webcolors" },
+]
+
+[[package]]
+name = "jsonschema-specifications"
+version = "2025.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "referencing" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" },
+]
+
+[[package]]
+name = "jupyter"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ipykernel" },
+ { name = "ipywidgets" },
+ { name = "jupyter-console" },
+ { name = "jupyterlab" },
+ { name = "nbconvert" },
+ { name = "notebook" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/58/f3/af28ea964ab8bc1e472dba2e82627d36d470c51f5cd38c37502eeffaa25e/jupyter-1.1.1.tar.gz", hash = "sha256:d55467bceabdea49d7e3624af7e33d59c37fff53ed3a350e1ac957bed731de7a", size = 5714959, upload-time = "2024-08-30T07:15:48.299Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/64/285f20a31679bf547b75602702f7800e74dbabae36ef324f716c02804753/jupyter-1.1.1-py2.py3-none-any.whl", hash = "sha256:7a59533c22af65439b24bbe60373a4e95af8f16ac65a6c00820ad378e3f7cc83", size = 2657, upload-time = "2024-08-30T07:15:47.045Z" },
+]
+
+[[package]]
+name = "jupyter-builder"
+version = "1.0.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-core" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fb/45/d0df8b43c10a61529c0f4a8af5e19ebe108f0c3af8f57e0fc358969907af/jupyter_builder-1.0.2.tar.gz", hash = "sha256:6155d78a5325010532a6419ffcba89eac643fd1aa56ea83115e661924d6f6aab", size = 968638, upload-time = "2026-06-12T02:33:25.767Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/28/b6/c418e0b3256f67c04933566b80bfce947350682db92c4b786a8653db32d6/jupyter_builder-1.0.2-py3-none-any.whl", hash = "sha256:b024f65d36e1d530542db597b00dd513261aa59842e0d0fbbb1015a9f1935e9c", size = 910789, upload-time = "2026-06-12T02:33:23.317Z" },
+]
+
+[[package]]
+name = "jupyter-client"
+version = "8.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-core" },
+ { name = "python-dateutil" },
+ { name = "pyzmq" },
+ { name = "tornado" },
+ { name = "traitlets" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7d/dc/5512503b088997c2250b8bf18258fba9d9ce5ead641183700960d3c9d342/jupyter_client-8.9.1.tar.gz", hash = "sha256:a58f730dd9e728ba16ba1d62ebccf7ffe1ebbdbce4e95cfae941b7321ae1f4fa", size = 359256, upload-time = "2026-06-09T13:15:01.033Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/6f/56d39bf385c5c27988aebaf0c18a2a17e960575740100973511018bd904e/jupyter_client-8.9.1-py3-none-any.whl", hash = "sha256:0b7a295bc46e8751e9adae84781f726c851c1d911bd793edc4a3bde942e3da81", size = 109828, upload-time = "2026-06-09T13:14:58.835Z" },
+]
+
+[[package]]
+name = "jupyter-console"
+version = "6.6.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ipykernel" },
+ { name = "ipython" },
+ { name = "jupyter-client" },
+ { name = "jupyter-core" },
+ { name = "prompt-toolkit" },
+ { name = "pygments" },
+ { name = "pyzmq" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bd/2d/e2fd31e2fc41c14e2bcb6c976ab732597e907523f6b2420305f9fc7fdbdb/jupyter_console-6.6.3.tar.gz", hash = "sha256:566a4bf31c87adbfadf22cdf846e3069b59a71ed5da71d6ba4d8aaad14a53539", size = 34363, upload-time = "2023-03-06T14:13:31.02Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ca/77/71d78d58f15c22db16328a476426f7ac4a60d3a5a7ba3b9627ee2f7903d4/jupyter_console-6.6.3-py3-none-any.whl", hash = "sha256:309d33409fcc92ffdad25f0bcdf9a4a9daa61b6f341177570fdac03de5352485", size = 24510, upload-time = "2023-03-06T14:13:28.229Z" },
+]
+
+[[package]]
+name = "jupyter-core"
+version = "5.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "platformdirs" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/02/49/9d1284d0dc65e2c757b74c6687b6d319b02f822ad039e5c512df9194d9dd/jupyter_core-5.9.1.tar.gz", hash = "sha256:4d09aaff303b9566c3ce657f580bd089ff5c91f5f89cf7d8846c3cdf465b5508", size = 89814, upload-time = "2025-10-16T19:19:18.444Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/e7/80988e32bf6f73919a113473a604f5a8f09094de312b9d52b79c2df7612b/jupyter_core-5.9.1-py3-none-any.whl", hash = "sha256:ebf87fdc6073d142e114c72c9e29a9d7ca03fad818c5d300ce2adc1fb0743407", size = 29032, upload-time = "2025-10-16T19:19:16.783Z" },
+]
+
+[[package]]
+name = "jupyter-events"
+version = "0.12.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jsonschema", extra = ["format-nongpl"] },
+ { name = "packaging" },
+ { name = "python-json-logger" },
+ { name = "pyyaml" },
+ { name = "referencing" },
+ { name = "rfc3339-validator" },
+ { name = "rfc3986-validator" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/18/f8/475c4241b2b75af0deaae453ed003c6c851766dbc44d332d8baf245dc931/jupyter_events-0.12.1.tar.gz", hash = "sha256:faff25f77218335752f35f23c5fe6e4a392a7bd99a5939ccb9b8fbf594636cf3", size = 62854, upload-time = "2026-04-20T23:17:50.66Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/eb/6c/6fcde0c8f616ed360ffd3587f7db9e225a7e62b583a04494d2f069cf64ea/jupyter_events-0.12.1-py3-none-any.whl", hash = "sha256:c366585253f537a627da52fa7ca7410c5b5301fe893f511e7b077c2d93ec8bcf", size = 19512, upload-time = "2026-04-20T23:17:48.927Z" },
+]
+
+[[package]]
+name = "jupyter-lsp"
+version = "2.3.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-server" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/36/ff/1e4a61f5170a9a1d978f3ac3872449de6c01fc71eaf89657824c878b1549/jupyter_lsp-2.3.1.tar.gz", hash = "sha256:fdf8a4aa7d85813976d6e29e95e6a2c8f752701f926f2715305249a3829805a6", size = 55677, upload-time = "2026-04-02T08:10:06.749Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/23/e8/9d61dcbd1dce8ef418f06befd4ac084b4720429c26b0b1222bc218685eff/jupyter_lsp-2.3.1-py3-none-any.whl", hash = "sha256:71b954d834e85ff3096400554f2eefaf7fe37053036f9a782b0f7c5e42dadb81", size = 77513, upload-time = "2026-04-02T08:10:01.753Z" },
+]
+
+[[package]]
+name = "jupyter-server"
+version = "2.20.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "argon2-cffi" },
+ { name = "jinja2" },
+ { name = "jupyter-client" },
+ { name = "jupyter-core" },
+ { name = "jupyter-events" },
+ { name = "jupyter-server-terminals" },
+ { name = "nbconvert" },
+ { name = "nbformat" },
+ { name = "packaging" },
+ { name = "prometheus-client" },
+ { name = "pywinpty", marker = "os_name == 'nt' and sys_platform != 'darwin'" },
+ { name = "pyzmq" },
+ { name = "send2trash" },
+ { name = "terminado" },
+ { name = "tornado" },
+ { name = "traitlets" },
+ { name = "websocket-client" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6b/dc/db3a582633170186f8c8b31298d7eb26ad0eb031a1f53476c258b64eed05/jupyter_server-2.20.0.tar.gz", hash = "sha256:b5778ba337d8015a3dc2b80803ecdd5ac18d3797fddf61a50ea5fb472b4ebe14", size = 756523, upload-time = "2026-06-17T12:09:09.435Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f3/71/8c002223e873a870f5c41dc69b0a7c922301123e4a31d5d01ecb700aef77/jupyter_server-2.20.0-py3-none-any.whl", hash = "sha256:c3b67c93c471e947c18b5026f04f21614218adb706df8f48227d3ee8e0a7cdcc", size = 393143, upload-time = "2026-06-17T12:09:07.234Z" },
+]
+
+[[package]]
+name = "jupyter-server-terminals"
+version = "0.5.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pywinpty", marker = "os_name == 'nt' and sys_platform != 'darwin'" },
+ { name = "terminado" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f4/a7/bcd0a9b0cbba88986fe944aaaf91bfda603e5a50bda8ed15123f381a3b2f/jupyter_server_terminals-0.5.4.tar.gz", hash = "sha256:bbda128ed41d0be9020349f9f1f2a4ab9952a73ed5f5ac9f1419794761fb87f5", size = 31770, upload-time = "2026-01-14T16:53:20.213Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/2d/6674563f71c6320841fc300911a55143925112a72a883e2ca71fba4c618d/jupyter_server_terminals-0.5.4-py3-none-any.whl", hash = "sha256:55be353fc74a80bc7f3b20e6be50a55a61cd525626f578dcb66a5708e2007d14", size = 13704, upload-time = "2026-01-14T16:53:18.738Z" },
+]
+
+[[package]]
+name = "jupyterlab"
+version = "4.6.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "async-lru" },
+ { name = "httpx" },
+ { name = "ipykernel" },
+ { name = "jinja2" },
+ { name = "jupyter-builder" },
+ { name = "jupyter-core" },
+ { name = "jupyter-lsp" },
+ { name = "jupyter-server" },
+ { name = "jupyterlab-server" },
+ { name = "notebook-shim" },
+ { name = "packaging" },
+ { name = "tornado" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bc/2a/d6af53bfd45a43a5bfe7e40ba47ee7a8921a807daf4bb708e3a295bbb54d/jupyterlab-4.6.1.tar.gz", hash = "sha256:75315982ed28427edaa62bb85eadb5105e4043a757643c910efd787fe6ed0837", size = 28179125, upload-time = "2026-06-29T12:48:45.402Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5a/81/90ac6cc31d248e83a0d1eab343a5e6e68bc783d3f74fbe61640f42a61da4/jupyterlab-4.6.1-py3-none-any.whl", hash = "sha256:85a58546c831f3dce6cf919468c26874c9065e99c42279fb4abb8e1b552a98bb", size = 17164660, upload-time = "2026-06-29T12:48:41.21Z" },
+]
+
+[[package]]
+name = "jupyterlab-pygments"
+version = "0.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/90/51/9187be60d989df97f5f0aba133fa54e7300f17616e065d1ada7d7646b6d6/jupyterlab_pygments-0.3.0.tar.gz", hash = "sha256:721aca4d9029252b11cfa9d185e5b5af4d54772bb8072f9b7036f4170054d35d", size = 512900, upload-time = "2023-11-23T09:26:37.44Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b1/dd/ead9d8ea85bf202d90cc513b533f9c363121c7792674f78e0d8a854b63b4/jupyterlab_pygments-0.3.0-py3-none-any.whl", hash = "sha256:841a89020971da1d8693f1a99997aefc5dc424bb1b251fd6322462a1b8842780", size = 15884, upload-time = "2023-11-23T09:26:34.325Z" },
+]
+
+[[package]]
+name = "jupyterlab-server"
+version = "2.28.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "babel" },
+ { name = "jinja2" },
+ { name = "json5" },
+ { name = "jsonschema" },
+ { name = "jupyter-server" },
+ { name = "packaging" },
+ { name = "requests" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d6/2c/90153f189e421e93c4bb4f9e3f59802a1f01abd2ac5cf40b152d7f735232/jupyterlab_server-2.28.0.tar.gz", hash = "sha256:35baa81898b15f93573e2deca50d11ac0ae407ebb688299d3a5213265033712c", size = 76996, upload-time = "2025-10-22T13:59:18.37Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e0/07/a000fe835f76b7e1143242ab1122e6362ef1c03f23f83a045c38859c2ae0/jupyterlab_server-2.28.0-py3-none-any.whl", hash = "sha256:e4355b148fdcf34d312bbbc80f22467d6d20460e8b8736bf235577dd18506968", size = 59830, upload-time = "2025-10-22T13:59:16.767Z" },
+]
+
+[[package]]
+name = "jupyterlab-widgets"
+version = "3.0.16"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/2d/ef58fed122b268c69c0aa099da20bc67657cdfb2e222688d5731bd5b971d/jupyterlab_widgets-3.0.16.tar.gz", hash = "sha256:423da05071d55cf27a9e602216d35a3a65a3e41cdf9c5d3b643b814ce38c19e0", size = 897423, upload-time = "2025-11-01T21:11:29.724Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ab/b5/36c712098e6191d1b4e349304ef73a8d06aed77e56ceaac8c0a306c7bda1/jupyterlab_widgets-3.0.16-py3-none-any.whl", hash = "sha256:45fa36d9c6422cf2559198e4db481aa243c7a32d9926b500781c830c80f7ecf8", size = 914926, upload-time = "2025-11-01T21:11:28.008Z" },
+]
+
+[[package]]
+name = "kiwisolver"
+version = "1.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/67/9c61eccb13f0bdca9307614e782fec49ffdde0f7a2314935d489fa93cd9c/kiwisolver-1.5.0.tar.gz", hash = "sha256:d4193f3d9dc3f6f79aaed0e5637f45d98850ebf01f7ca20e69457f3e8946b66a", size = 103482, upload-time = "2026-03-09T13:15:53.382Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9d/69/024d6711d5ba575aa65d5538042e99964104e97fa153a9f10bc369182bc2/kiwisolver-1.5.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:fd40bb9cd0891c4c3cb1ddf83f8bbfa15731a248fdc8162669405451e2724b09", size = 123166, upload-time = "2026-03-09T13:13:48.032Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/48/adbb40df306f587054a348831220812b9b1d787aff714cfbc8556e38fccd/kiwisolver-1.5.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c0e1403fd7c26d77c1f03e096dc58a5c726503fa0db0456678b8668f76f521e3", size = 66395, upload-time = "2026-03-09T13:13:49.365Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/3a/d0a972b34e1c63e2409413104216cd1caa02c5a37cb668d1687d466c1c45/kiwisolver-1.5.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:dda366d548e89a90d88a86c692377d18d8bd64b39c1fb2b92cb31370e2896bbd", size = 64065, upload-time = "2026-03-09T13:13:50.562Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/0a/7b98e1e119878a27ba8618ca1e18b14f992ff1eda40f47bccccf4de44121/kiwisolver-1.5.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:332b4f0145c30b5f5ad9374881133e5aa64320428a57c2c2b61e9d891a51c2f3", size = 1477903, upload-time = "2026-03-09T13:13:52.084Z" },
+ { url = "https://files.pythonhosted.org/packages/18/d8/55638d89ffd27799d5cc3d8aa28e12f4ce7a64d67b285114dbedc8ea4136/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c50b89ffd3e1a911c69a1dd3de7173c0cd10b130f56222e57898683841e4f96", size = 1278751, upload-time = "2026-03-09T13:13:54.673Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/97/b4c8d0d18421ecceba20ad8701358453b88e32414e6f6950b5a4bad54e65/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4db576bb8c3ef9365f8b40fe0f671644de6736ae2c27a2c62d7d8a1b4329f099", size = 1296793, upload-time = "2026-03-09T13:13:56.287Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/10/f862f94b6389d8957448ec9df59450b81bec4abb318805375c401a1e6892/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0b85aad90cea8ac6797a53b5d5f2e967334fa4d1149f031c4537569972596cb8", size = 1346041, upload-time = "2026-03-09T13:13:58.269Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/6a/f1650af35821eaf09de398ec0bc2aefc8f211f0cda50204c9f1673741ba9/kiwisolver-1.5.0-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:d36ca54cb4c6c4686f7cbb7b817f66f5911c12ddb519450bbe86707155028f87", size = 987292, upload-time = "2026-03-09T13:13:59.871Z" },
+ { url = "https://files.pythonhosted.org/packages/de/19/d7fb82984b9238115fe629c915007be608ebd23dc8629703d917dbfaffd4/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:38f4a703656f493b0ad185211ccfca7f0386120f022066b018eb5296d8613e23", size = 2227865, upload-time = "2026-03-09T13:14:01.401Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/b9/46b7f386589fd222dac9e9de9c956ce5bcefe2ee73b4e79891381dda8654/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3ac2360e93cb41be81121755c6462cff3beaa9967188c866e5fce5cf13170859", size = 2324369, upload-time = "2026-03-09T13:14:02.972Z" },
+ { url = "https://files.pythonhosted.org/packages/92/8b/95e237cf3d9c642960153c769ddcbe278f182c8affb20cecc1cc983e7cc5/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c95cab08d1965db3d84a121f1c7ce7479bdd4072c9b3dafd8fecce48a2e6b902", size = 1977989, upload-time = "2026-03-09T13:14:04.503Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/95/980c9df53501892784997820136c01f62bc1865e31b82b9560f980c0e649/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:fc20894c3d21194d8041a28b65622d5b86db786da6e3cfe73f0c762951a61167", size = 2491645, upload-time = "2026-03-09T13:14:06.106Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/32/900647fd0840abebe1561792c6b31e6a7c0e278fc3973d30572a965ca14c/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7a32f72973f0f950c1920475d5c5ea3d971b81b6f0ec53b8d0a956cc965f22e0", size = 2295237, upload-time = "2026-03-09T13:14:08.891Z" },
+ { url = "https://files.pythonhosted.org/packages/be/8a/be60e3bbcf513cc5a50f4a3e88e1dcecebb79c1ad607a7222877becaa101/kiwisolver-1.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:0bf3acf1419fa93064a4c2189ac0b58e3be7872bf6ee6177b0d4c63dc4cea276", size = 73573, upload-time = "2026-03-09T13:14:12.327Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/d2/64be2e429eb4fca7f7e1c52a91b12663aeaf25de3895e5cca0f47ef2a8d0/kiwisolver-1.5.0-cp313-cp313-win_arm64.whl", hash = "sha256:fa8eb9ecdb7efb0b226acec134e0d709e87a909fa4971a54c0c4f6e88635484c", size = 64998, upload-time = "2026-03-09T13:14:13.469Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/69/ce68dd0c85755ae2de490bf015b62f2cea5f6b14ff00a463f9d0774449ff/kiwisolver-1.5.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:db485b3847d182b908b483b2ed133c66d88d49cacf98fd278fadafe11b4478d1", size = 125700, upload-time = "2026-03-09T13:14:14.636Z" },
+ { url = "https://files.pythonhosted.org/packages/74/aa/937aac021cf9d4349990d47eb319309a51355ed1dbdc9c077cdc9224cb11/kiwisolver-1.5.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:be12f931839a3bdfe28b584db0e640a65a8bcbc24560ae3fdb025a449b3d754e", size = 67537, upload-time = "2026-03-09T13:14:15.808Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/20/3a87fbece2c40ad0f6f0aefa93542559159c5f99831d596050e8afae7a9f/kiwisolver-1.5.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:16b85d37c2cbb3253226d26e64663f755d88a03439a9c47df6246b35defbdfb7", size = 65514, upload-time = "2026-03-09T13:14:18.035Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/7f/f943879cda9007c45e1f7dba216d705c3a18d6b35830e488b6c6a4e7cdf0/kiwisolver-1.5.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4432b835675f0ea7414aab3d37d119f7226d24869b7a829caeab49ebda407b0c", size = 1584848, upload-time = "2026-03-09T13:14:19.745Z" },
+ { url = "https://files.pythonhosted.org/packages/37/f8/4d4f85cc1870c127c88d950913370dd76138482161cd07eabbc450deff01/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b0feb50971481a2cc44d94e88bdb02cdd497618252ae226b8eb1201b957e368", size = 1391542, upload-time = "2026-03-09T13:14:21.54Z" },
+ { url = "https://files.pythonhosted.org/packages/04/0b/65dd2916c84d252b244bd405303220f729e7c17c9d7d33dca6feeff9ffc4/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:56fa888f10d0f367155e76ce849fa1166fc9730d13bd2d65a2aa13b6f5424489", size = 1404447, upload-time = "2026-03-09T13:14:23.205Z" },
+ { url = "https://files.pythonhosted.org/packages/39/5c/2606a373247babce9b1d056c03a04b65f3cf5290a8eac5d7bdead0a17e21/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:940dda65d5e764406b9fb92761cbf462e4e63f712ab60ed98f70552e496f3bf1", size = 1455918, upload-time = "2026-03-09T13:14:24.74Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/d1/c6078b5756670658e9192a2ef11e939c92918833d2745f85cd14a6004bdf/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_39_riscv64.whl", hash = "sha256:89fc958c702ee9a745e4700378f5d23fddbc46ff89e8fdbf5395c24d5c1452a3", size = 1072856, upload-time = "2026-03-09T13:14:26.597Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/c8/7def6ddf16eb2b3741d8b172bdaa9af882b03c78e9b0772975408801fa63/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9027d773c4ff81487181a925945743413f6069634d0b122d0b37684ccf4f1e18", size = 2333580, upload-time = "2026-03-09T13:14:28.237Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/87/2ac1fce0eb1e616fcd3c35caa23e665e9b1948bb984f4764790924594128/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5b233ea3e165e43e35dba1d2b8ecc21cf070b45b65ae17dd2747d2713d942021", size = 2423018, upload-time = "2026-03-09T13:14:30.018Z" },
+ { url = "https://files.pythonhosted.org/packages/67/13/c6700ccc6cc218716bfcda4935e4b2997039869b4ad8a94f364c5a3b8e63/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ce9bf03dad3b46408c08649c6fbd6ca28a9fce0eb32fdfffa6775a13103b5310", size = 2062804, upload-time = "2026-03-09T13:14:32.888Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/bd/877056304626943ff0f1f44c08f584300c199b887cb3176cd7e34f1515f1/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:fc4d3f1fb9ca0ae9f97b095963bc6326f1dbfd3779d6679a1e016b9baaa153d3", size = 2597482, upload-time = "2026-03-09T13:14:34.971Z" },
+ { url = "https://files.pythonhosted.org/packages/75/19/c60626c47bf0f8ac5dcf72c6c98e266d714f2fbbfd50cf6dab5ede3aaa50/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f443b4825c50a51ee68585522ab4a1d1257fac65896f282b4c6763337ac9f5d2", size = 2394328, upload-time = "2026-03-09T13:14:36.816Z" },
+ { url = "https://files.pythonhosted.org/packages/47/84/6a6d5e5bb8273756c27b7d810d47f7ef2f1f9b9fd23c9ee9a3f8c75c9cef/kiwisolver-1.5.0-cp313-cp313t-win_arm64.whl", hash = "sha256:893ff3a711d1b515ba9da14ee090519bad4610ed1962fbe298a434e8c5f8db53", size = 68410, upload-time = "2026-03-09T13:14:38.695Z" },
+]
+
+[[package]]
+name = "langcodes"
+version = "3.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a9/75/f9edc5d72945019312f359e69ded9f82392a81d49c5051ed3209b100c0d2/langcodes-3.5.1.tar.gz", hash = "sha256:40bff315e01b01d11c2ae3928dd4f5cbd74dd38f9bd912c12b9a3606c143f731", size = 191084, upload-time = "2025-12-02T16:22:01.627Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dd/c1/d10b371bcba7abce05e2b33910e39c33cfa496a53f13640b7b8e10bb4d2b/langcodes-3.5.1-py3-none-any.whl", hash = "sha256:b6a9c25c603804e2d169165091d0cdb23934610524a21d226e4f463e8e958a72", size = 183050, upload-time = "2025-12-02T16:21:59.954Z" },
+]
+
+[[package]]
+name = "langdetect"
+version = "1.0.9"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0e/72/a3add0e4eec4eb9e2569554f7c70f4a3c27712f40e3284d483e88094cc0e/langdetect-1.0.9.tar.gz", hash = "sha256:cbc1fef89f8d062739774bd51eda3da3274006b3661d199c2655f6b3f6d605a0", size = 981474, upload-time = "2021-05-07T07:54:13.562Z" }
+
+[[package]]
+name = "lark"
+version = "1.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/af/60/bc7622aefb2aee1c0b4ba23c1446d3e30225c8770b38d7aedbfb65ca9d5a/lark-1.2.2.tar.gz", hash = "sha256:ca807d0162cd16cef15a8feecb862d7319e7a09bdb13aef927968e45040fed80", size = 252132, upload-time = "2024-08-13T19:49:00.652Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2d/00/d90b10b962b4277f5e64a78b6609968859ff86889f5b898c1a778c06ec00/lark-1.2.2-py3-none-any.whl", hash = "sha256:c2276486b02f0f1b90be155f2c8ba4a8e194d42775786db622faccd652d8e80c", size = 111036, upload-time = "2024-08-13T19:48:58.603Z" },
+]
+
+[[package]]
+name = "latex2sympy2-extended"
+version = "1.0.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "antlr4-python3-runtime" },
+ { name = "sympy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e6/99/57fe9239572c898888a3dba3dffe5f326bf19bd934a1b20af9ebd2ceb2e0/latex2sympy2_extended-1.0.6.tar.gz", hash = "sha256:1f994f3cb49c613387437980610326ae7392d41da151523ab16faa521d406083", size = 82727, upload-time = "2025-02-06T11:47:40.309Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/15/f3/ea8c08d9c279e9088a039cc9a7eaf9d036fff216d79b608483e7fd906fe0/latex2sympy2_extended-1.0.6-py3-none-any.whl", hash = "sha256:036da4b24c73c4b13566ef4e347be08232c6539ddd75b38f083cc5821e738bba", size = 82014, upload-time = "2025-02-06T11:47:37.415Z" },
+]
+
+[[package]]
+name = "librt"
+version = "0.12.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c6/e0/dbd0f2a68a1c1a1991eb7921ff6014465d56608cdc9a9fb468a616210a37/librt-0.12.0.tar.gz", hash = "sha256:cb26faedbd09c6130e9c1b64d8000efec5076ffd18d606c6cd1cf02730e6d8b0", size = 203841, upload-time = "2026-06-30T16:14:29.671Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f2/87/568d948c8079c9ff3c9e8110cf85f1eb70218e1209af29d0b7b89aa4a60c/librt-0.12.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8d9a55760a34ae5ce70434aabb6a6c61c6c44a0ec58ca1cfd9cd86e4745d417d", size = 146808, upload-time = "2026-06-30T16:13:05.417Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/1d/bea471ecea210088847bb5f3c4b4b424d596518934c06679b78ca85d6e63/librt-0.12.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ff0b197e338b4cf432873e0d6ef025213fdea85311ec4d87d2ea88c28adf2409", size = 145503, upload-time = "2026-06-30T16:13:07.023Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/9e/984ad422b56de95fdce158f06b051655373784ebea0aba9a7fcbc41614d1/librt-0.12.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7e69f120a20b69e2539d603bbd4d62db38399b10f8bf73a1cf445038a621e8af", size = 488421, upload-time = "2026-06-30T16:13:08.492Z" },
+ { url = "https://files.pythonhosted.org/packages/50/03/1a2f94009b07ea71f8e1a4cfe53370565b56da9caa341b89e0699325e9f5/librt-0.12.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:fde3cde595e947fc8e755b0a21f919a1622483d07c662d00496e040773d22591", size = 483488, upload-time = "2026-06-30T16:13:10.169Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/3b/084bdc295823fbb6ab91670047adf8f420787f9e8794bf2d140b66dc196b/librt-0.12.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d977447315fa09ea4e8c7ae9b4e22f7659b5128161c1fd55ff786b5349f73503", size = 518428, upload-time = "2026-06-30T16:13:11.681Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/22/5a307390b93a115ffbecd95c64eecb4e56269680e45e9415ada7285f2cf4/librt-0.12.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7ffac8a67e4143cea9a549d4822b93bc0bbaad73fc25aa0ab0ba5ec27d178677", size = 509744, upload-time = "2026-06-30T16:13:13.217Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/90/83f3cb6184f5d669660717b4b2e317c9ddaccf7ca5bb97f2196deac1a3b7/librt-0.12.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:94af1ed773ff104ef08ef3d669a0ba9d3a5916c609eb698cffe5d5476d66ff9b", size = 527749, upload-time = "2026-06-30T16:13:15.277Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/3b/f162be5cc88d47378e3a20776fe425fa1c2bece755da15e2783ebf06d3d6/librt-0.12.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:548199d21d22fb26398dfbbe0ba953a52465c66f3a49f38e6fddce1b127faf53", size = 532582, upload-time = "2026-06-30T16:13:17.074Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/28/6c5d2f6b7232fd24f284fc4cab37a459fe69a9096a09942f44cc5c55e073/librt-0.12.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c8f1f413b966a9dd3ecf80cd337b0ad7bb3de2474a4ff448ed3ebabfc3f803fc", size = 522235, upload-time = "2026-06-30T16:13:18.823Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/1c/bd115360587fdc22c8ae8fac14c040a556b442e2965d4370d2cf274c8b95/librt-0.12.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:55f13f95b629be5b6ab38918e439bf14169d6f9a8deaae55e0c14e12fb0c74b9", size = 559055, upload-time = "2026-06-30T16:13:20.509Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/5a/c26f49f576437014825a86faea3cec60c1ed17f976abd567b6c12b8e35a7/librt-0.12.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:8b2dc079dfe29e77a47a19073d2040fa4879aa3656501f1650f8402ddce0313c", size = 79809, upload-time = "2026-06-30T16:13:22.401Z" },
+ { url = "https://files.pythonhosted.org/packages/69/0b/a55244261d9ad7375ac039b8af06d42602722e2e8b8d8d6b86e4a3888c02/librt-0.12.0-cp313-cp313-win32.whl", hash = "sha256:da58944be8270f2bfee628a9a2a60c1cf6a12c8bea8e2c9b6edf3e5414ca7793", size = 99308, upload-time = "2026-06-30T16:13:23.661Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/bf/ed9465e58d44c5a5637795547d0841c8934aab905ea452cac1adf14672cf/librt-0.12.0-cp313-cp313-win_amd64.whl", hash = "sha256:1db4be3037e4ce065a071fa7deee93e78ebc25f448340a02a6c1c0b82c37e383", size = 119438, upload-time = "2026-06-30T16:13:25.188Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/44/3cad652aeb892e6e8ffe48d0fafa2bc652f28ec7ed3f4403fcbb1be4f948/librt-0.12.0-cp313-cp313-win_arm64.whl", hash = "sha256:05fd2542892ad770b5dd45003fd080477cf220b611d3ee59b0792097eb0873a9", size = 105118, upload-time = "2026-06-30T16:13:26.533Z" },
+]
+
+[[package]]
+name = "lighteval"
+version = "0.13.1.dev0"
+source = { git = "https://github.com/huggingface/lighteval.git?rev=10b9104eb16778b174e9e240140974a039018981#10b9104eb16778b174e9e240140974a039018981" }
+dependencies = [
+ { name = "accelerate" },
+ { name = "aenum" },
+ { name = "colorlog" },
+ { name = "datasets" },
+ { name = "fsspec" },
+ { name = "gitpython" },
+ { name = "hf-xet" },
+ { name = "httpx" },
+ { name = "huggingface-hub", extra = ["hf-xet"] },
+ { name = "inspect-ai" },
+ { name = "langcodes" },
+ { name = "latex2sympy2-extended" },
+ { name = "nltk" },
+ { name = "numpy" },
+ { name = "openai" },
+ { name = "protobuf" },
+ { name = "pycountry" },
+ { name = "pydantic" },
+ { name = "pytablewriter" },
+ { name = "rich" },
+ { name = "rouge-score" },
+ { name = "sacrebleu" },
+ { name = "scikit-learn" },
+ { name = "sentencepiece" },
+ { name = "termcolor" },
+ { name = "torch" },
+ { name = "transformers" },
+ { name = "typer" },
+]
+
+[package.optional-dependencies]
+multilingual = [
+ { name = "jieba" },
+ { name = "pyvi" },
+ { name = "spacy", extra = ["ja", "ko", "th"] },
+ { name = "stanza" },
+]
+vllm = [
+ { name = "more-itertools" },
+ { name = "ray" },
+ { name = "vllm" },
+]
+
+[[package]]
+name = "linkify-it-py"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "uc-micro-py" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2e/c9/06ea13676ef354f0af6169587ae292d3e2406e212876a413bf9eece4eb23/linkify_it_py-2.1.0.tar.gz", hash = "sha256:43360231720999c10e9328dc3691160e27a718e280673d444c38d7d3aaa3b98b", size = 29158, upload-time = "2026-03-01T07:48:47.683Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b4/de/88b3be5c31b22333b3ca2f6ff1de4e863d8fe45aaea7485f591970ec1d3e/linkify_it_py-2.1.0-py3-none-any.whl", hash = "sha256:0d252c1594ecba2ecedc444053db5d3a9b7ec1b0dd929c8f1d74dce89f86c05e", size = 19878, upload-time = "2026-03-01T07:48:46.098Z" },
+]
+
+[[package]]
+name = "litellm"
+version = "1.90.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "click" },
+ { name = "fastuuid" },
+ { name = "httpx" },
+ { name = "importlib-metadata" },
+ { name = "jinja2" },
+ { name = "jsonschema" },
+ { name = "openai" },
+ { name = "pydantic" },
+ { name = "python-dotenv" },
+ { name = "tiktoken" },
+ { name = "tokenizers" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/88/00/4187e33818d0de19fd602214ce15a6e1cfa5bf04fb50a6e59606214a92df/litellm-1.90.2.tar.gz", hash = "sha256:b536603894ba2a0ccd14a6f1ebeb5f46cc35b19d4537e8fda7bfdfc7757f19d3", size = 14819154, upload-time = "2026-07-01T02:29:58.397Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/24/30/3afa7b212ce1f9bb8b6fa5f58c631f437c1d7b3a025e4e4ed6b01b3d28ad/litellm-1.90.2-py3-none-any.whl", hash = "sha256:6fb46f485150cc861be62a62d670f94d33adbd26991091befeb51bcdfdad34f6", size = 16612720, upload-time = "2026-07-01T02:29:55.215Z" },
+]
+
+[package.optional-dependencies]
+caching = [
+ { name = "diskcache" },
+]
+
+[[package]]
+name = "llguidance"
+version = "1.7.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/da/91/6bc8bb503dc259e46d253b5424385a54fe06c38a4c7a12befe69a3c2455a/llguidance-1.7.6.tar.gz", hash = "sha256:db7febbe412ed2015501904646750071d7e00e6df7f85c4b956ad4f206fd2df7", size = 1156574, upload-time = "2026-06-03T20:13:25.316Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fa/1d/5a9a13421b1f3f1c1acf82beb63ed72fa4d302e65099b72f4a4fe5a098ab/llguidance-1.7.6-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:eabf4572c8731734c0444c353b9ea06bc5c156986d2ff0a4ec0499159271381f", size = 3227892, upload-time = "2026-06-03T20:13:09.533Z" },
+ { url = "https://files.pythonhosted.org/packages/46/fe/bb185f11bad82f2637e3cd8cbf6b200cbb6ed56ac395de47ea05a60d4649/llguidance-1.7.6-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:9c54c899db8cb4b4fba128a7d844730066576c70d806c95ada92b2bd2d6ab498", size = 3138127, upload-time = "2026-06-03T20:13:11.649Z" },
+ { url = "https://files.pythonhosted.org/packages/51/b9/dc76d7716e04dc7b3427cae52eaa32bd20771382d4d1dd9f4538a9dd2086/llguidance-1.7.6-cp39-abi3-manylinux_2_31_aarch64.whl", hash = "sha256:e70fa25ed550c2b50c2fd70baa9e2808b4ecb859d01e453bd5459aff62ba38c3", size = 2899993, upload-time = "2026-06-03T20:13:13.563Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/64/d74336f22242ef94356a456057d4ff1be7c1bc9c7dbc867171c6982a5512/llguidance-1.7.6-cp39-abi3-manylinux_2_31_x86_64.whl", hash = "sha256:ceec951d29a74309984e3be0fe7f5f56c1362434cd937abd517b259a60908b1e", size = 3074809, upload-time = "2026-06-03T20:13:15.498Z" },
+ { url = "https://files.pythonhosted.org/packages/49/37/99d700f0e2c83acf25a8d8946b2bee9f5eac47bc530bfbd53ba3126c667f/llguidance-1.7.6-cp39-abi3-win_amd64.whl", hash = "sha256:ace7e81cd31950a87186356ab24bd7f75fbc10a05ca9d9f7f8748f931963f763", size = 2879207, upload-time = "2026-06-03T20:13:23.341Z" },
+]
+
+[[package]]
+name = "llvmlite"
+version = "0.47.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/01/88/a8952b6d5c21e74cbf158515b779666f692846502623e9e3c39d8e8ba25f/llvmlite-0.47.0.tar.gz", hash = "sha256:62031ce968ec74e95092184d4b0e857e444f8fdff0b8f9213707699570c33ccc", size = 193614, upload-time = "2026-03-31T18:29:53.497Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/6f/4615353e016799f80fa52ccb270a843c413b22361fadda2589b2922fb9b0/llvmlite-0.47.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:a3c6a735d4e1041808434f9d440faa3d78d9b4af2ee64d05a66f351883b6ceec", size = 37232771, upload-time = "2026-03-31T18:29:01.324Z" },
+ { url = "https://files.pythonhosted.org/packages/31/b8/69f5565f1a280d032525878a86511eebed0645818492feeb169dfb20ae8e/llvmlite-0.47.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2699a74321189e812d476a43d6d7f652f51811e7b5aad9d9bba842a1c7927acb", size = 56275178, upload-time = "2026-03-31T18:29:05.748Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/da/b32cafcb926fb0ce2aa25553bf32cb8764af31438f40e2481df08884c947/llvmlite-0.47.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6c6951e2b29930227963e53ee152441f0e14be92e9d4231852102d986c761e40", size = 55128632, upload-time = "2026-03-31T18:29:11.235Z" },
+ { url = "https://files.pythonhosted.org/packages/46/9f/4898b44e4042c60fafcb1162dfb7014f6f15b1ec19bf29cfea6bf26df90d/llvmlite-0.47.0-cp313-cp313-win_amd64.whl", hash = "sha256:c2e9adf8698d813a9a5efb2d4370caf344dbc1e145019851fee6a6f319ba760e", size = 38138695, upload-time = "2026-03-31T18:29:15.43Z" },
+]
+
+[[package]]
+name = "lm-format-enforcer"
+version = "0.11.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "interegular" },
+ { name = "packaging" },
+ { name = "pydantic" },
+ { name = "pyyaml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/84/d5/41cd417ba7dfdbbcfe46cebf81fb3dfd7c591b89897560ad05bb410a465d/lm_format_enforcer-0.11.3.tar.gz", hash = "sha256:e68081c108719cce284a9bcc889709b26ffb085a1945b5eba3a12cfa96d528da", size = 40258, upload-time = "2025-08-24T19:37:47.527Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/ef/11292bb0b85cf4c93447cab5a29f64576ed14d3ab4280e35ddd23486594a/lm_format_enforcer-0.11.3-py3-none-any.whl", hash = "sha256:cf586350875def1ae7a8fba84fcbbfc8371424b6c9d05c1fcba70aa233fbf06f", size = 45418, upload-time = "2025-08-24T19:37:46.325Z" },
+]
+
+[[package]]
+name = "loguru"
+version = "0.7.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "win32-setctime", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3a/05/a1dae3dffd1116099471c643b8924f5aa6524411dc6c63fdae648c4f1aca/loguru-0.7.3.tar.gz", hash = "sha256:19480589e77d47b8d85b2c827ad95d49bf31b0dcde16593892eb51dd18706eb6", size = 63559, upload-time = "2024-12-06T11:20:56.608Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },
+]
+
+[[package]]
+name = "lxml"
+version = "6.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/05/3b/aab6728cae887456f409b4d75e8a01856e4f04bd510de38052a47768b680/lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40", size = 4197430, upload-time = "2026-05-18T19:19:06.424Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a5/eb/7e6f37c5584ccbb2ff267f56fd0339016938c1c8684cfefab9b33ffc2f36/lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736", size = 8559780, upload-time = "2026-05-18T19:17:57.661Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/36/587c2521cf23a2cd6c9c22108aa7528f683a1f195ed7ccd23a4b1786ad36/lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9", size = 4618006, upload-time = "2026-05-18T19:18:04.452Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/ca/ab7bfe2bf4c972af5e7878262845ead3a24a929a9b04bc11c7c1ece6c82a/lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354", size = 4924139, upload-time = "2026-05-18T19:19:04.873Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/55/a0c72851dfee5ecc689f949723a73dea457758912542cb955b108eaf0d8f/lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca", size = 5082329, upload-time = "2026-05-18T19:19:09.728Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/b6/0608f7d61a3b96cc67e5648a3d906e31a5082093e10e7be65b3886289938/lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099", size = 4993564, upload-time = "2026-05-18T19:19:13.608Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/66/ae227524b066d29d55bf0b453d93d2d793c40218657d643dcbbca13b8faf/lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6", size = 5613467, upload-time = "2026-05-18T19:19:16.228Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/76/dbe4a00b50385e40194231dcfe5a12c059de7cf90e89c83407d2b085b719/lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085", size = 5228304, upload-time = "2026-05-18T19:19:19.354Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/01/00b1b8442ed2041793336868ba0b9ea4b13d7da7c085c6404c207a63bf79/lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e", size = 5341607, upload-time = "2026-05-18T19:19:22.297Z" },
+ { url = "https://files.pythonhosted.org/packages/63/36/1ad29931e9a4638bb707869f01d423a6c815f82152138d1a40dfcfde2b95/lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f", size = 4700168, upload-time = "2026-05-18T19:19:25.133Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/d1/a9536cecf9be18a0dc72d32bead283a2332d1ffebd2dd3ac70ce444686e5/lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c", size = 5232487, upload-time = "2026-05-18T19:19:28.603Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/77/b4fb1e03bf5d130e879214d3100092e386418807fb74dd0adc4b0a48f351/lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b", size = 5044231, upload-time = "2026-05-18T19:18:42.246Z" },
+ { url = "https://files.pythonhosted.org/packages/26/4c/d00daeeb0a5530c4028a9232aa1b93db3ef4ed2158c116ea73c79a9765b3/lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2", size = 4769450, upload-time = "2026-05-18T19:18:48.013Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/6a/715a3a8d156ce42f29cf014706f5410c2ff3b02267774110fc23266409fe/lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5", size = 5635874, upload-time = "2026-05-18T19:18:51.914Z" },
+ { url = "https://files.pythonhosted.org/packages/45/37/0544bc21dde2a88f3a17b504e6fc79c0e01d25a33c2f6079724e9e72b9c7/lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785", size = 5223987, upload-time = "2026-05-18T19:18:59.715Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/f8/f6a5e8185bcb28c2befae3d31f8e3df3b811cb0f47746517a81279fcafe1/lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947", size = 5250276, upload-time = "2026-05-18T19:19:03.834Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/f2/1a2b9f1b7a49d45495369be7ef9ad05b262930f2eab3e3145706fca8083f/lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca", size = 3596903, upload-time = "2026-05-18T19:17:29.863Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/99/f4ffb024f238eec2131aaa09f3278fb6129cf892741bf68e1fc1afb8c100/lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660", size = 3995869, upload-time = "2026-05-18T19:18:02.596Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/53/70eb8c5c6037f27448f1e3c54ebede9545a801ae63f0a7254afca4fe8e45/lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc", size = 3658490, upload-time = "2026-05-19T19:22:53.846Z" },
+]
+
+[[package]]
+name = "markdown-it-py"
+version = "4.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mdurl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" },
+]
+
+[package.optional-dependencies]
+linkify = [
+ { name = "linkify-it-py" },
+]
+
+[[package]]
+name = "markupsafe"
+version = "3.0.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/2f/907b9c7bbba283e68f20259574b13d005c121a0fa4c175f9bed27c4597ff/markupsafe-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795", size = 11622, upload-time = "2025-09-27T18:36:41.777Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/d9/5f7756922cdd676869eca1c4e3c0cd0df60ed30199ffd775e319089cb3ed/markupsafe-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219", size = 12029, upload-time = "2025-09-27T18:36:43.257Z" },
+ { url = "https://files.pythonhosted.org/packages/00/07/575a68c754943058c78f30db02ee03a64b3c638586fba6a6dd56830b30a3/markupsafe-3.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6", size = 24374, upload-time = "2025-09-27T18:36:44.508Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/21/9b05698b46f218fc0e118e1f8168395c65c8a2c750ae2bab54fc4bd4e0e8/markupsafe-3.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676", size = 22980, upload-time = "2025-09-27T18:36:45.385Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/71/544260864f893f18b6827315b988c146b559391e6e7e8f7252839b1b846a/markupsafe-3.0.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9", size = 21990, upload-time = "2025-09-27T18:36:46.916Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/28/b50fc2f74d1ad761af2f5dcce7492648b983d00a65b8c0e0cb457c82ebbe/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1", size = 23784, upload-time = "2025-09-27T18:36:47.884Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/76/104b2aa106a208da8b17a2fb72e033a5a9d7073c68f7e508b94916ed47a9/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc", size = 21588, upload-time = "2025-09-27T18:36:48.82Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/99/16a5eb2d140087ebd97180d95249b00a03aa87e29cc224056274f2e45fd6/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12", size = 23041, upload-time = "2025-09-27T18:36:49.797Z" },
+ { url = "https://files.pythonhosted.org/packages/19/bc/e7140ed90c5d61d77cea142eed9f9c303f4c4806f60a1044c13e3f1471d0/markupsafe-3.0.3-cp313-cp313-win32.whl", hash = "sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed", size = 14543, upload-time = "2025-09-27T18:36:51.584Z" },
+ { url = "https://files.pythonhosted.org/packages/05/73/c4abe620b841b6b791f2edc248f556900667a5a1cf023a6646967ae98335/markupsafe-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5", size = 15113, upload-time = "2025-09-27T18:36:52.537Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/3a/fa34a0f7cfef23cf9500d68cb7c32dd64ffd58a12b09225fb03dd37d5b80/markupsafe-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485", size = 13911, upload-time = "2025-09-27T18:36:53.513Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/d7/e05cd7efe43a88a17a37b3ae96e79a19e846f3f456fe79c57ca61356ef01/markupsafe-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73", size = 11658, upload-time = "2025-09-27T18:36:54.819Z" },
+ { url = "https://files.pythonhosted.org/packages/99/9e/e412117548182ce2148bdeacdda3bb494260c0b0184360fe0d56389b523b/markupsafe-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37", size = 12066, upload-time = "2025-09-27T18:36:55.714Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/e6/fa0ffcda717ef64a5108eaa7b4f5ed28d56122c9a6d70ab8b72f9f715c80/markupsafe-3.0.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19", size = 25639, upload-time = "2025-09-27T18:36:56.908Z" },
+ { url = "https://files.pythonhosted.org/packages/96/ec/2102e881fe9d25fc16cb4b25d5f5cde50970967ffa5dddafdb771237062d/markupsafe-3.0.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025", size = 23569, upload-time = "2025-09-27T18:36:57.913Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/30/6f2fce1f1f205fc9323255b216ca8a235b15860c34b6798f810f05828e32/markupsafe-3.0.3-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6", size = 23284, upload-time = "2025-09-27T18:36:58.833Z" },
+ { url = "https://files.pythonhosted.org/packages/58/47/4a0ccea4ab9f5dcb6f79c0236d954acb382202721e704223a8aafa38b5c8/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f", size = 24801, upload-time = "2025-09-27T18:36:59.739Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/70/3780e9b72180b6fecb83a4814d84c3bf4b4ae4bf0b19c27196104149734c/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb", size = 22769, upload-time = "2025-09-27T18:37:00.719Z" },
+ { url = "https://files.pythonhosted.org/packages/98/c5/c03c7f4125180fc215220c035beac6b9cb684bc7a067c84fc69414d315f5/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009", size = 23642, upload-time = "2025-09-27T18:37:01.673Z" },
+ { url = "https://files.pythonhosted.org/packages/80/d6/2d1b89f6ca4bff1036499b1e29a1d02d282259f3681540e16563f27ebc23/markupsafe-3.0.3-cp313-cp313t-win32.whl", hash = "sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354", size = 14612, upload-time = "2025-09-27T18:37:02.639Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/98/e48a4bfba0a0ffcf9925fe2d69240bfaa19c6f7507b8cd09c70684a53c1e/markupsafe-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218", size = 15200, upload-time = "2025-09-27T18:37:03.582Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/72/e3cc540f351f316e9ed0f092757459afbc595824ca724cbc5a5d4263713f/markupsafe-3.0.3-cp313-cp313t-win_arm64.whl", hash = "sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287", size = 13973, upload-time = "2025-09-27T18:37:04.929Z" },
+]
+
+[[package]]
+name = "matplotlib"
+version = "3.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "contourpy" },
+ { name = "cycler" },
+ { name = "fonttools" },
+ { name = "kiwisolver" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pillow" },
+ { name = "pyparsing" },
+ { name = "python-dateutil" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1f/24/080c99d223d158d3a8902769269ab6da5b50f7a0e6e072513907e02b7a6c/matplotlib-3.11.0.tar.gz", hash = "sha256:68c0c7be01b30dcca3638934f7f591df73401235cbdbf0d1ab1c71e7db7f8b57", size = 33251176, upload-time = "2026-06-12T02:29:15.508Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/55/41/aa47f156b061d14c98b906f76c428507397708ec63ff94f410ae1752b426/matplotlib-3.11.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6ce3b839b34ae1f430b4616893a2945a2999debaa7e94e7e29a2a8bbf286f7b5", size = 9450532, upload-time = "2026-06-12T02:28:06.769Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/4f/5a9eb0375e81413953febf8af7b012a6b6357f53438a15c4f5ad86c6bbb5/matplotlib-3.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:373db8f91214e8ccaf35ac833cc1dd59dd961e148bbd55dd027141591dde1313", size = 9279760, upload-time = "2026-06-12T02:28:09.152Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/c0/1117d53077e3ac3152503a84e9cf7a5c239576805ee71276e80c2aaa7471/matplotlib-3.11.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:be152b7570324dc8d01574cc9474dd2d803237acf528bcbb5b211fa347461a09", size = 10031623, upload-time = "2026-06-12T02:28:11.26Z" },
+ { url = "https://files.pythonhosted.org/packages/92/7e/e937138daffad65b71bf831a377809dcbc830fb4f31a31e067dc1faa2575/matplotlib-3.11.0-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:126f256df600652d7e4b394cf3164ff75210a00038f287c95a012a6f58d0e83f", size = 10839372, upload-time = "2026-06-12T02:28:14.102Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/c2/438ecc197ffb8023b6b9922915542f2172f5fd45b76703b0b4fc47322243/matplotlib-3.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:03acfeddf87b0dddb11b081ef7740ad445a3ca8bcb6b8e3011b08f2cf802b75c", size = 10924099, upload-time = "2026-06-12T02:28:16.383Z" },
+ { url = "https://files.pythonhosted.org/packages/40/2e/395883da416f378b3ed2c9f3e843ac477eae1ce731b671b79adaa6f0bacd/matplotlib-3.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:ab3722f04f3ff34c23b5012c5873d2894174e06c3822fcdac3610965a5ac7d06", size = 9329727, upload-time = "2026-06-12T02:28:18.581Z" },
+ { url = "https://files.pythonhosted.org/packages/61/82/2c388956abf8bf392dfb5b8917c502f1082df6a941b781ab8c8e5ba2474b/matplotlib-3.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:c945824670fb8915b4ac879e5e61f3c58e0913022f70a0de4c082b17372f8771", size = 9003506, upload-time = "2026-06-12T02:28:20.474Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/c1/34454baa44da7975ada82e9aea37105ec47059514dc967d3be14426ba8dc/matplotlib-3.11.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3489c3dc487669b4a980bc3068f87856de7a1564248d3f6c629efb2a58b03f24", size = 9499838, upload-time = "2026-06-12T02:28:22.713Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/c3/98fe79a398cf232219f090163a7fa7e6766e9f2e0ad26df54d6f8934d8ee/matplotlib-3.11.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:6a98f5476ce784a50ce09998f4ae1e6a9f25043cef8a480c98949902eda74620", size = 9332298, upload-time = "2026-06-12T02:28:24.796Z" },
+ { url = "https://files.pythonhosted.org/packages/95/e4/b4b7c33151e74e5c802f3cde1ba807ebfc38401e329b44e215a5888dd76d/matplotlib-3.11.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:565af866fd63e4bd3f987d580afe27c44c2552a3b3305f4ecbb85133601ea6f3", size = 10045491, upload-time = "2026-06-12T02:28:27.141Z" },
+ { url = "https://files.pythonhosted.org/packages/71/28/394548efd68354110c1a1be11fe6b6e559e06d1a23da35908a0e316c55a9/matplotlib-3.11.0-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e6b3e64dea5062c570f04358e2711859f3531b459f29516274fbad889079e4f3", size = 10857059, upload-time = "2026-06-12T02:28:29.222Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/44/e7922e6e2a4d63bdfbc9dc4a53e3850ab438d46cf42e6779bb15ec92c948/matplotlib-3.11.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:942b37c5db1899610bd1543ce8e13e4ecff9a4633e7f63bb6aa9205d2644ebd1", size = 10939576, upload-time = "2026-06-12T02:28:31.66Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/be/b1ca96003a441d619b727fee21d671fdff7a5ce2f1bb797b2521aa2f679a/matplotlib-3.11.0-cp313-cp313t-win_amd64.whl", hash = "sha256:c08e649a6313e1291e713623b97a38e5bb4aa580b2a100a94a3309bc6b9c8eb3", size = 9379519, upload-time = "2026-06-12T02:28:33.888Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/72/4bf3b91821c34596dd6a7bdac5836d94f744144c8208939ef49d8ec43f7e/matplotlib-3.11.0-cp313-cp313t-win_arm64.whl", hash = "sha256:2746cd2c113742ff6ce37a864c5ac5fd7aa644568f445e66166e457ac78e40e0", size = 9055456, upload-time = "2026-06-12T02:28:35.878Z" },
+]
+
+[[package]]
+name = "matplotlib-inline"
+version = "0.2.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bd/c0/9f7c9a46090390368a4d7bcb76bb87a4a36c421e4c0792cdb53486ffac7a/matplotlib_inline-0.2.2.tar.gz", hash = "sha256:72f3fe8fce36b70d4a5b612f899090cd0401deddc4ea90e1572b9f4bfb058c79", size = 8150, upload-time = "2026-05-08T17:33:33.49Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/09/5b161152e2d90f7b87f781c2e1267494aef9c32498df793f73ad0a0a494a/matplotlib_inline-0.2.2-py3-none-any.whl", hash = "sha256:3c821cf1c209f59fb2d2d64abbf5b23b67bcb2210d663f9918dd851c6da1fcf6", size = 9534, upload-time = "2026-05-08T17:33:32.055Z" },
+]
+
+[[package]]
+name = "mbstrdecoder"
+version = "1.1.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "chardet" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4b/9c/dd6e38d747a62ead27f9abef32f4ca4311d4e40ac28e76bcc9ffb5dd0329/mbstrdecoder-1.1.5.tar.gz", hash = "sha256:8cbfba26938befd8a35e3cc06ca0632f61320b7b2be7df32550b895e1725b1ce", size = 14529, upload-time = "2026-05-05T04:17:58.23Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4c/eb/711270faab7b7df702339a2c68b31fd3ed4fffc68b0e99e5bdf49b1e87e4/mbstrdecoder-1.1.5-py3-none-any.whl", hash = "sha256:4a50fe113d4abecfd86e8f716b2e413cce03d63af83ec3c7cdbe81dec0e519ed", size = 7966, upload-time = "2026-05-05T04:17:56.78Z" },
+]
+
+[[package]]
+name = "mcp"
+version = "1.28.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "httpx" },
+ { name = "httpx-sse" },
+ { name = "jsonschema" },
+ { name = "pydantic" },
+ { name = "pydantic-settings" },
+ { name = "pyjwt", extra = ["crypto"] },
+ { name = "python-multipart" },
+ { name = "pywin32", marker = "sys_platform == 'win32'" },
+ { name = "sse-starlette" },
+ { name = "starlette" },
+ { name = "typing-extensions" },
+ { name = "typing-inspection" },
+ { name = "uvicorn", marker = "sys_platform != 'emscripten'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6e/77/9450b8f251a13affb6281997d0523c4615f8a8b35d0b21ff30db3a5aac9d/mcp-1.28.1.tar.gz", hash = "sha256:d51e36a5f5644faea4f85ea649bfffa6bc6c26770d42798ad6a3de3d2ba69683", size = 638501, upload-time = "2026-06-26T12:57:29.093Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e2/5e/d118fce19f87a2e7d8101c35c8ae0ec289098a4df0ff244cec23e415aca0/mcp-1.28.1-py3-none-any.whl", hash = "sha256:2726bca5e7193f61c5dde8b12500a6de2d9acf6d1a1c0be9e8c2e706437991df", size = 222620, upload-time = "2026-06-26T12:57:27.218Z" },
+]
+
+[[package]]
+name = "mdit-py-plugins"
+version = "0.6.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown-it-py" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/59/fc/f8d0863f8862f25602c0404d75568e89fb6b4109804645e5cdfb1be5cf56/mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0", size = 56114, upload-time = "2026-05-13T09:03:38.91Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a5/69/6da5581c6a7fede7dc261bf4e67d6adca4196f176b43288b55b3db395b6e/mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d", size = 66663, upload-time = "2026-05-13T09:03:37.76Z" },
+]
+
+[[package]]
+name = "mdurl"
+version = "0.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
+]
+
+[[package]]
+name = "mistral-common"
+version = "1.11.7"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jsonschema" },
+ { name = "numpy" },
+ { name = "pillow" },
+ { name = "pydantic" },
+ { name = "pydantic-extra-types", extra = ["pycountry"] },
+ { name = "requests" },
+ { name = "tiktoken" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6f/d0/61b2c24be62a8e2f0e46a1c16de23de386c8644408da249bc66768a6681b/mistral_common-1.11.7.tar.gz", hash = "sha256:d3b79583595cf6d96a2ab33e42cb8449768383147b8c56cac5a4f193be19d20d", size = 6387178, upload-time = "2026-07-23T09:21:17.206Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ef/a4/bc2850eb33cc2d633a21f51530756350dca325ee69b07c9202552f2bbadb/mistral_common-1.11.7-py3-none-any.whl", hash = "sha256:a9511b88eacacbe7dacddd9d3498c1739f56847b7fdddbd5a22e7844fd9def95", size = 6553583, upload-time = "2026-07-23T09:21:19.818Z" },
+]
+
+[package.optional-dependencies]
+image = [
+ { name = "opencv-python-headless" },
+]
+
+[[package]]
+name = "mistune"
+version = "3.3.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/04/5f/007786743f962224423753b78f7d7acb0f2ade46d1604f2e0fa2bedf9020/mistune-3.3.2.tar.gz", hash = "sha256:e12ee4f1e74336e91aa1141e35f913b337c40bdf7c0cc49f21fb853a27e8b62f", size = 111284, upload-time = "2026-06-23T00:29:28.568Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/44/43/894c2cbbcbdf53b57d1257a249811abe2ee9ab7ef76af301b40f1c054533/mistune-3.3.2-py3-none-any.whl", hash = "sha256:a678a56387d487db7368ede4647cb2ba1deff22ce61f92343e4ebe0ddfce4f2d", size = 61554, upload-time = "2026-06-23T00:29:27.088Z" },
+]
+
+[[package]]
+name = "ml-dtypes"
+version = "0.5.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0e/4a/c27b42ed9b1c7d13d9ba8b6905dece787d6259152f2309338aed29b2447b/ml_dtypes-0.5.4.tar.gz", hash = "sha256:8ab06a50fb9bf9666dd0fe5dfb4676fa2b0ac0f31ecff72a6c3af8e22c063453", size = 692314, upload-time = "2025-11-17T22:32:31.031Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d9/a1/4008f14bbc616cfb1ac5b39ea485f9c63031c4634ab3f4cf72e7541f816a/ml_dtypes-0.5.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8c760d85a2f82e2bed75867079188c9d18dae2ee77c25a54d60e9cc79be1bc48", size = 676888, upload-time = "2025-11-17T22:31:56.907Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/b7/dff378afc2b0d5a7d6cd9d3209b60474d9819d1189d347521e1688a60a53/ml_dtypes-0.5.4-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ce756d3a10d0c4067172804c9cc276ba9cc0ff47af9078ad439b075d1abdc29b", size = 5036993, upload-time = "2025-11-17T22:31:58.497Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/33/40cd74219417e78b97c47802037cf2d87b91973e18bb968a7da48a96ea44/ml_dtypes-0.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:533ce891ba774eabf607172254f2e7260ba5f57bdd64030c9a4fcfbd99815d0d", size = 5010956, upload-time = "2025-11-17T22:31:59.931Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/8b/200088c6859d8221454825959df35b5244fa9bdf263fd0249ac5fb75e281/ml_dtypes-0.5.4-cp313-cp313-win_amd64.whl", hash = "sha256:f21c9219ef48ca5ee78402d5cc831bd58ea27ce89beda894428bc67a52da5328", size = 212224, upload-time = "2025-11-17T22:32:01.349Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/75/dfc3775cb36367816e678f69a7843f6f03bd4e2bcd79941e01ea960a068e/ml_dtypes-0.5.4-cp313-cp313-win_arm64.whl", hash = "sha256:35f29491a3e478407f7047b8a4834e4640a77d2737e0b294d049746507af5175", size = 160798, upload-time = "2025-11-17T22:32:02.864Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/74/e9ddb35fd1dd43b1106c20ced3f53c2e8e7fc7598c15638e9f80677f81d4/ml_dtypes-0.5.4-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:304ad47faa395415b9ccbcc06a0350800bc50eda70f0e45326796e27c62f18b6", size = 702083, upload-time = "2025-11-17T22:32:04.08Z" },
+ { url = "https://files.pythonhosted.org/packages/74/f5/667060b0aed1aa63166b22897fdf16dca9eb704e6b4bbf86848d5a181aa7/ml_dtypes-0.5.4-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6a0df4223b514d799b8a1629c65ddc351b3efa833ccf7f8ea0cf654a61d1e35d", size = 5354111, upload-time = "2025-11-17T22:32:05.546Z" },
+ { url = "https://files.pythonhosted.org/packages/40/49/0f8c498a28c0efa5f5c95a9e374c83ec1385ca41d0e85e7cf40e5d519a21/ml_dtypes-0.5.4-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:531eff30e4d368cb6255bc2328d070e35836aa4f282a0fb5f3a0cd7260257298", size = 5366453, upload-time = "2025-11-17T22:32:07.115Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/27/12607423d0a9c6bbbcc780ad19f1f6baa2b68b18ce4bddcdc122c4c68dc9/ml_dtypes-0.5.4-cp313-cp313t-win_amd64.whl", hash = "sha256:cb73dccfc991691c444acc8c0012bee8f2470da826a92e3a20bb333b1a7894e6", size = 225612, upload-time = "2025-11-17T22:32:08.615Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/80/5a5929e92c72936d5b19872c5fb8fc09327c1da67b3b68c6a13139e77e20/ml_dtypes-0.5.4-cp313-cp313t-win_arm64.whl", hash = "sha256:3bbbe120b915090d9dd1375e4684dd17a20a2491ef25d640a908281da85e73f1", size = 164145, upload-time = "2025-11-17T22:32:09.782Z" },
+]
+
+[[package]]
+name = "mmh3"
+version = "5.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/91/1a/edb23803a168f070ded7a3014c6d706f63b90c84ccc024f89d794a3b7a6d/mmh3-5.2.1.tar.gz", hash = "sha256:bbea5b775f0ac84945191fb83f845a6fd9a21a03ea7f2e187defac7e401616ad", size = 33775, upload-time = "2026-03-05T15:55:57.716Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/25/a5/9daa0508a1569a54130f6198d5462a92deda870043624aa3ea72721aa765/mmh3-5.2.1-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:723b2681ed4cc07d3401bbea9c201ad4f2a4ca6ba8cddaff6789f715dd2b391e", size = 40832, upload-time = "2026-03-05T15:54:43.212Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/6b/3230c6d80c1f4b766dedf280a92c2241e99f87c1504ff74205ec8cebe451/mmh3-5.2.1-cp313-cp313-android_21_x86_64.whl", hash = "sha256:3619473a0e0d329fd4aec8075628f8f616be2da41605300696206d6f36920c3d", size = 41964, upload-time = "2026-03-05T15:54:44.204Z" },
+ { url = "https://files.pythonhosted.org/packages/62/fb/648bfddb74a872004b6ee751551bfdda783fe6d70d2e9723bad84dbe5311/mmh3-5.2.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:e48d4dbe0f88e53081da605ae68644e5182752803bbc2beb228cca7f1c4454d6", size = 39114, upload-time = "2026-03-05T15:54:45.205Z" },
+ { url = "https://files.pythonhosted.org/packages/95/c2/ab7901f87af438468b496728d11264cb397b3574d41506e71b92128e0373/mmh3-5.2.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:a482ac121de6973897c92c2f31defc6bafb11c83825109275cffce54bb64933f", size = 39819, upload-time = "2026-03-05T15:54:46.509Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/ed/6f88dda0df67de1612f2e130ffea34cf84aaee5bff5b0aff4dbff2babe34/mmh3-5.2.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:17fbb47f0885ace8327ce1235d0416dc86a211dcd8cc1e703f41523be32cfec8", size = 40330, upload-time = "2026-03-05T15:54:47.864Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/66/7516d23f53cdf90f43fce24ab80c28f45e6851d78b46bef8c02084edf583/mmh3-5.2.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:d51fde50a77f81330523562e3c2734ffdca9c4c9e9d355478117905e1cfe16c6", size = 56078, upload-time = "2026-03-05T15:54:48.9Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/34/4d152fdf4a91a132cb226b671f11c6b796eada9ab78080fb5ce1e95adaab/mmh3-5.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:19bbd3b841174ae6ed588536ab5e1b1fe83d046e668602c20266547298d939a9", size = 40498, upload-time = "2026-03-05T15:54:49.942Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/4c/8e3af1b6d85a299767ec97bd923f12b06267089c1472c27c1696870d1175/mmh3-5.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:be77c402d5e882b6fbacfd90823f13da8e0a69658405a39a569c6b58fdb17b03", size = 40033, upload-time = "2026-03-05T15:54:50.994Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/f2/966ea560e32578d453c9e9db53d602cbb1d0da27317e232afa7c38ceba11/mmh3-5.2.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:fd96476f04db5ceba1cfa0f21228f67c1f7402296f0e73fee3513aa680ad237b", size = 97320, upload-time = "2026-03-05T15:54:52.072Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/0d/2c5f9893b38aeb6b034d1a44ecd55a010148054f6a516abe53b5e4057297/mmh3-5.2.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:707151644085dd0f20fe4f4b573d28e5130c4aaa5f587e95b60989c5926653b5", size = 103299, upload-time = "2026-03-05T15:54:53.569Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/fc/2ebaef4a4d4376f89761274dc274035ffd96006ab496b4ee5af9b08f21a9/mmh3-5.2.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3737303ca9ea0f7cb83028781148fcda4f1dac7821db0c47672971dabcf63593", size = 106222, upload-time = "2026-03-05T15:54:55.092Z" },
+ { url = "https://files.pythonhosted.org/packages/57/09/ea7ffe126d0ba0406622602a2d05e1e1a6841cc92fc322eb576c95b27fad/mmh3-5.2.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2778fed822d7db23ac5008b181441af0c869455b2e7d001f4019636ac31b6fe4", size = 113048, upload-time = "2026-03-05T15:54:56.305Z" },
+ { url = "https://files.pythonhosted.org/packages/85/57/9447032edf93a64aa9bef4d9aa596400b1756f40411890f77a284f6293ca/mmh3-5.2.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d57dea657357230cc780e13920d7fa7db059d58fe721c80020f94476da4ca0a1", size = 120742, upload-time = "2026-03-05T15:54:57.453Z" },
+ { url = "https://files.pythonhosted.org/packages/53/82/a86cc87cc88c92e9e1a598fee509f0409435b57879a6129bf3b3e40513c7/mmh3-5.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:169e0d178cb59314456ab30772429a802b25d13227088085b0d49b9fe1533104", size = 99132, upload-time = "2026-03-05T15:54:58.583Z" },
+ { url = "https://files.pythonhosted.org/packages/54/f7/6b16eb1b40ee89bb740698735574536bc20d6cdafc65ae702ea235578e05/mmh3-5.2.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:7e4e1f580033335c6f76d1e0d6b56baf009d1a64d6a4816347e4271ba951f46d", size = 98686, upload-time = "2026-03-05T15:55:00.078Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/88/a601e9f32ad1410f438a6d0544298ea621f989bd34a0731a7190f7dec799/mmh3-5.2.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:2bd9f19f7f1fcebd74e830f4af0f28adad4975d40d80620be19ffb2b2af56c9f", size = 106479, upload-time = "2026-03-05T15:55:01.532Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/5c/ce29ae3dfc4feec4007a437a1b7435fb9507532a25147602cd5b52be86db/mmh3-5.2.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:c88653877aeb514c089d1b3d473451677b8b9a6d1497dbddf1ae7934518b06d2", size = 110030, upload-time = "2026-03-05T15:55:02.934Z" },
+ { url = "https://files.pythonhosted.org/packages/13/30/ae444ef2ff87c805d525da4fa63d27cda4fe8a48e77003a036b8461cfd5c/mmh3-5.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:fceef7fe67c81e1585198215e42ad3fdba3a25644beda8fbdaf85f4d7b93175a", size = 97536, upload-time = "2026-03-05T15:55:04.135Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/f9/dc3787ee5c813cc27fe79f45ad4500d9b5437f23a7402435cc34e07c7718/mmh3-5.2.1-cp313-cp313-win32.whl", hash = "sha256:54b64fb2433bc71488e7a449603bf8bd31fbcf9cb56fbe1eb6d459e90b86c37b", size = 40769, upload-time = "2026-03-05T15:55:05.277Z" },
+ { url = "https://files.pythonhosted.org/packages/43/67/850e0b5a1e97799822ebfc4ca0e8c6ece3ed8baf7dcdf64de817dfdda2ca/mmh3-5.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:cae6383181f1e345317742d2ddd88f9e7d2682fa4c9432e3a74e47d92dce0229", size = 41563, upload-time = "2026-03-05T15:55:06.283Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/cc/98c90b28e1da5458e19fbfaf4adb5289208d3bfccd45dd14eab216a2f0bb/mmh3-5.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:022aa1a528604e6c83d0a7705fdef0b5355d897a9e0fa3a8d26709ceaa06965d", size = 39310, upload-time = "2026-03-05T15:55:07.323Z" },
+]
+
+[[package]]
+name = "model-hosting-container-standards"
+version = "0.1.16"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "fastapi" },
+ { name = "httpx" },
+ { name = "jmespath" },
+ { name = "pydantic" },
+ { name = "setuptools" },
+ { name = "starlette" },
+ { name = "supervisor" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2d/5f/bc0d0fce1bd0a35378696aa13b21feffa18d9cda837f4e1be124e45ee090/model_hosting_container_standards-0.1.16.tar.gz", hash = "sha256:d34589633900e53c3ee5f7c78280a7cf7e4f6532c35e763341a262fc85cbe84a", size = 94130, upload-time = "2026-06-15T21:29:34.771Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/07/ef/6eabeb251d2a0598cb5f9a274159e05ae07a1e3fe6a1473bf6035793252a/model_hosting_container_standards-0.1.16-py3-none-any.whl", hash = "sha256:47f4f65713120bc3a69feb022981a38db9e557aedf88dbd72077f20588caa12b", size = 125666, upload-time = "2026-06-15T21:29:33.415Z" },
+]
+
+[[package]]
+name = "more-itertools"
+version = "11.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/de/1d/f4da6f02cdffe04d6362210b807146a26044c88d839208aec273bb0d9184/more_itertools-11.1.0.tar.gz", hash = "sha256:48e8f4d9e7e5878571ecf6f2b4e57634f93cd474cc8cfbd2376f2d11b396e30d", size = 145772, upload-time = "2026-05-22T14:14:29.909Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e8/3d/1087453384dbde46a8c7f9356eead2c58be8a7bf156bca40243377c85715/more_itertools-11.1.0-py3-none-any.whl", hash = "sha256:4b65538ae22f6fed0ce4874efd317463a7489796a0939fa66824dd542125a192", size = 72226, upload-time = "2026-05-22T14:14:28.824Z" },
+]
+
+[[package]]
+name = "mpmath"
+version = "1.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f", size = 508106, upload-time = "2023-03-07T16:47:11.061Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" },
+]
+
+[[package]]
+name = "msgpack"
+version = "1.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/31/f9/c0a1c127f9049db9155afc316952ea571720dd01833ff5e4d7e8e6352dbb/msgpack-1.2.1.tar.gz", hash = "sha256:04c721c2c7448767e9e3f2520a475663d8ee0f09c31890f6d2bd70fd636a9647", size = 183960, upload-time = "2026-06-18T16:13:52.594Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b0/ac/dcddcab6f6c20ecb387ca5e980371cdb3f87ff69aeca388be97eebc4c074/msgpack-1.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0a70e3cf2804a300d921bb0940426e35f4e489a23adfb77a808892241db0a064", size = 83151, upload-time = "2026-06-18T16:13:12.173Z" },
+ { url = "https://files.pythonhosted.org/packages/64/71/fbcfa83a1d6a9c6091942d1cfd070962244664b87427a9a49a6897b1b219/msgpack-1.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:491cc39455ca765fad51fb451bf2915eb2cf41192ab5801ce8d67c1d614fe056", size = 82351, upload-time = "2026-06-18T16:13:13.194Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/10/ddf7b06db879e8792d13934ddda09ff20bd2a583fd84c9b59aae9b0e650b/msgpack-1.2.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f310233ef7fb9c14e201c93639fe5f5260b005f56f0b29048e999c30935596cc", size = 407518, upload-time = "2026-06-18T16:13:14.233Z" },
+ { url = "https://files.pythonhosted.org/packages/79/d3/36a46a8ed992b781acbc05928bd5bee3c810cb0c3563bf81a7b0c04a1a76/msgpack-1.2.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:787c9bebb5833e8f6fc8abca3c0597683d8d87f56a8842b6b89c75a5f3176e2d", size = 416405, upload-time = "2026-06-18T16:13:15.435Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/84/e8e9598b557c0ba6ddae901a73780a4c75ac667dddf59414b1e56a42fb34/msgpack-1.2.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dc871b997a9370d855b7394465f2f350e847a5b806dd38dcc9c989e7d87da155", size = 376257, upload-time = "2026-06-18T16:13:17.022Z" },
+ { url = "https://files.pythonhosted.org/packages/40/16/738fe6d875ad7e2a9429c165322a4ec088f4f273cdfae63d96a89c467961/msgpack-1.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:85f57e960d877f2977f6430896191b04a21f8901b3b4baf2e4604329f4db5402", size = 397469, upload-time = "2026-06-18T16:13:18.287Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/be/6d5952df75a7f24f35833af764c3a6860780364cb3a0030beb8099e1b2b4/msgpack-1.2.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:1233ee2dd0cefba127583de50ea654677277047d238303521db35def3d7b2e7c", size = 372802, upload-time = "2026-06-18T16:13:19.685Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/39/e2ef7dbf0473bcb8dc7c50bf782a892d67414877b63e47fc88eb189ef5e6/msgpack-1.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e3dc2feb0876209d9c38aa56cb1de169bd6c4348f1aa48271f241226590993e6", size = 411273, upload-time = "2026-06-18T16:13:21.028Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/c5/133f4512a56e983a93445c836c9d94d88f3bc2e0980ff4b9e577bd8416ce/msgpack-1.2.1-cp313-cp313-win32.whl", hash = "sha256:6d09badf350af2be9d189184e04e64cf54ad93569ab3d96fca58bd3e84aad707", size = 64471, upload-time = "2026-06-18T16:13:22.293Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/98/577e10b055096a7dd40732358cabaf7180a20c79ed1dcdbb618e4b9deac7/msgpack-1.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:33f14fba63278b714efe6ad07e50ea5f03d91537aa6a1c5f1ceca4cf44013ca9", size = 71274, upload-time = "2026-06-18T16:13:23.455Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/ee/0c0048e7cfbef23c6a94791b8959ab28155232e7956de8a305b5ff588f05/msgpack-1.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:afc5febcd4c99effbc02b528e49d6fd0760b2b7d48c05239e345a5fa6e743d9a", size = 64795, upload-time = "2026-06-18T16:13:24.687Z" },
+]
+
+[[package]]
+name = "msgspec"
+version = "0.21.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e3/60/f79b9b013a16fa3a58350c9295ddc6789f2e335f36ea61ed10a21b215364/msgspec-0.21.1.tar.gz", hash = "sha256:2313508e394b0d208f8f56892ca9b2799e2561329de9763b19619595a6c0f72c", size = 319193, upload-time = "2026-04-12T21:44:50.394Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/74/f11ede02839b19ff459f88e3145df5d711626ca84da4e23520cebf819367/msgspec-0.21.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:764173717a01743f007e9f74520ed281f24672c604514f7d76c1c3a10e8edb66", size = 196176, upload-time = "2026-04-12T21:44:17.613Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/40/4476c1bd341418a046c4955aff632ec769315d1e3cb94e6acf86d461f9ed/msgspec-0.21.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:344c7cd0eaed1fb81d7959f99100ef71ec9b536881a376f11b9a6c4803365697", size = 188524, upload-time = "2026-04-12T21:44:18.815Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/d9/9e9d7d7e5061b47540d03d640fab9b3965ba7ae49c1b2154861c8f007518/msgspec-0.21.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:48943e278b3854c2f89f955ddc6f9f430d3f0784b16e47d10604ee0463cd21f5", size = 218880, upload-time = "2026-04-12T21:44:20.028Z" },
+ { url = "https://files.pythonhosted.org/packages/74/66/2bb344f34abb4b57e60c7c9c761994e0417b9718ec1460bf00c296f2a7ea/msgspec-0.21.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9aa659ebb0101b1cbc31461212b87e341d961f0ab0772aaf068a99e001ec4aa", size = 225050, upload-time = "2026-04-12T21:44:21.577Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/84/7c1e412f76092277bf760cef12b7979d03314d259ab5b5cafde5d0c1722d/msgspec-0.21.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7b27d1a8ead2b6f5b0c4f2d07b8be1ccfcc041c8a0e704781edebe3ae13c484", size = 222713, upload-time = "2026-04-12T21:44:22.83Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/27/0bba04b2b4ef05f3d068429410bc71d2cea925f1596a8f41152cccd5edb8/msgspec-0.21.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:38fe93e86b61328fe544cb7fd871fad5a27c8734bfda90f65e5dbe288ae50f61", size = 227259, upload-time = "2026-04-12T21:44:24.11Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/2d/09574b0eea02fed2c2c1383dbaae2c7f79dc16dcd6487a886000afb5d7c4/msgspec-0.21.1-cp313-cp313-win_amd64.whl", hash = "sha256:8bc666331c35fcce05a7cd2d6221adbe0f6058f8e750711413d22793c080ac6a", size = 189857, upload-time = "2026-04-12T21:44:25.359Z" },
+ { url = "https://files.pythonhosted.org/packages/46/34/105b1576ad182879914f0c821f17ee1d13abb165cb060448f96fe2aff078/msgspec-0.21.1-cp313-cp313-win_arm64.whl", hash = "sha256:42bb1241e0750c1a4346f2aa84db26c5ffd99a4eb3a954927d9f149ff2f42898", size = 175403, upload-time = "2026-04-12T21:44:26.608Z" },
+]
+
+[[package]]
+name = "multidict"
+version = "6.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1a/c2/c2d94cbe6ac1753f3fc980da97b3d930efe1da3af3c9f5125354436c073d/multidict-6.7.1.tar.gz", hash = "sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d", size = 102010, upload-time = "2026-01-26T02:46:45.979Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f2/22/929c141d6c0dba87d3e1d38fbdf1ba8baba86b7776469f2bc2d3227a1e67/multidict-6.7.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23", size = 76174, upload-time = "2026-01-26T02:44:18.509Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/75/bc704ae15fee974f8fccd871305e254754167dce5f9e42d88a2def741a1d/multidict-6.7.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2", size = 45116, upload-time = "2026-01-26T02:44:19.745Z" },
+ { url = "https://files.pythonhosted.org/packages/79/76/55cd7186f498ed080a18440c9013011eb548f77ae1b297206d030eb1180a/multidict-6.7.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445", size = 43524, upload-time = "2026-01-26T02:44:21.571Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/3c/414842ef8d5a1628d68edee29ba0e5bcf235dbfb3ccd3ea303a7fe8c72ff/multidict-6.7.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177", size = 249368, upload-time = "2026-01-26T02:44:22.803Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/32/befed7f74c458b4a525e60519fe8d87eef72bb1e99924fa2b0f9d97a221e/multidict-6.7.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23", size = 256952, upload-time = "2026-01-26T02:44:24.306Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d6/c878a44ba877f366630c860fdf74bfb203c33778f12b6ac274936853c451/multidict-6.7.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060", size = 240317, upload-time = "2026-01-26T02:44:25.772Z" },
+ { url = "https://files.pythonhosted.org/packages/68/49/57421b4d7ad2e9e60e25922b08ceb37e077b90444bde6ead629095327a6f/multidict-6.7.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d", size = 267132, upload-time = "2026-01-26T02:44:27.648Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/fe/ec0edd52ddbcea2a2e89e174f0206444a61440b40f39704e64dc807a70bd/multidict-6.7.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed", size = 268140, upload-time = "2026-01-26T02:44:29.588Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/73/6e1b01cbeb458807aa0831742232dbdd1fa92bfa33f52a3f176b4ff3dc11/multidict-6.7.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429", size = 254277, upload-time = "2026-01-26T02:44:30.902Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/b2/5fb8c124d7561a4974c342bc8c778b471ebbeb3cc17df696f034a7e9afe7/multidict-6.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6", size = 252291, upload-time = "2026-01-26T02:44:32.31Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/96/51d4e4e06bcce92577fcd488e22600bd38e4fd59c20cb49434d054903bd2/multidict-6.7.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9", size = 250156, upload-time = "2026-01-26T02:44:33.734Z" },
+ { url = "https://files.pythonhosted.org/packages/db/6b/420e173eec5fba721a50e2a9f89eda89d9c98fded1124f8d5c675f7a0c0f/multidict-6.7.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c", size = 249742, upload-time = "2026-01-26T02:44:35.222Z" },
+ { url = "https://files.pythonhosted.org/packages/44/a3/ec5b5bd98f306bc2aa297b8c6f11a46714a56b1e6ef5ebda50a4f5d7c5fb/multidict-6.7.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84", size = 262221, upload-time = "2026-01-26T02:44:36.604Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/f7/e8c0d0da0cd1e28d10e624604e1a36bcc3353aaebdfdc3a43c72bc683a12/multidict-6.7.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d", size = 258664, upload-time = "2026-01-26T02:44:38.008Z" },
+ { url = "https://files.pythonhosted.org/packages/52/da/151a44e8016dd33feed44f730bd856a66257c1ee7aed4f44b649fb7edeb3/multidict-6.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33", size = 249490, upload-time = "2026-01-26T02:44:39.386Z" },
+ { url = "https://files.pythonhosted.org/packages/87/af/a3b86bf9630b732897f6fc3f4c4714b90aa4361983ccbdcd6c0339b21b0c/multidict-6.7.1-cp313-cp313-win32.whl", hash = "sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3", size = 41695, upload-time = "2026-01-26T02:44:41.318Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/35/e994121b0e90e46134673422dd564623f93304614f5d11886b1b3e06f503/multidict-6.7.1-cp313-cp313-win_amd64.whl", hash = "sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5", size = 45884, upload-time = "2026-01-26T02:44:42.488Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/61/42d3e5dbf661242a69c97ea363f2d7b46c567da8eadef8890022be6e2ab0/multidict-6.7.1-cp313-cp313-win_arm64.whl", hash = "sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df", size = 43122, upload-time = "2026-01-26T02:44:43.664Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/b3/e6b21c6c4f314bb956016b0b3ef2162590a529b84cb831c257519e7fde44/multidict-6.7.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1", size = 83175, upload-time = "2026-01-26T02:44:44.894Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/76/23ecd2abfe0957b234f6c960f4ade497f55f2c16aeb684d4ecdbf1c95791/multidict-6.7.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963", size = 48460, upload-time = "2026-01-26T02:44:46.106Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/57/a0ed92b23f3a042c36bc4227b72b97eca803f5f1801c1ab77c8a212d455e/multidict-6.7.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34", size = 46930, upload-time = "2026-01-26T02:44:47.278Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/66/02ec7ace29162e447f6382c495dc95826bf931d3818799bbef11e8f7df1a/multidict-6.7.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65", size = 242582, upload-time = "2026-01-26T02:44:48.604Z" },
+ { url = "https://files.pythonhosted.org/packages/58/18/64f5a795e7677670e872673aca234162514696274597b3708b2c0d276cce/multidict-6.7.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292", size = 250031, upload-time = "2026-01-26T02:44:50.544Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/ed/e192291dbbe51a8290c5686f482084d31bcd9d09af24f63358c3d42fd284/multidict-6.7.1-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43", size = 228596, upload-time = "2026-01-26T02:44:51.951Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/7e/3562a15a60cf747397e7f2180b0a11dc0c38d9175a650e75fa1b4d325e15/multidict-6.7.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca", size = 257492, upload-time = "2026-01-26T02:44:53.902Z" },
+ { url = "https://files.pythonhosted.org/packages/24/02/7d0f9eae92b5249bb50ac1595b295f10e263dd0078ebb55115c31e0eaccd/multidict-6.7.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd", size = 255899, upload-time = "2026-01-26T02:44:55.316Z" },
+ { url = "https://files.pythonhosted.org/packages/00/e3/9b60ed9e23e64c73a5cde95269ef1330678e9c6e34dd4eb6b431b85b5a10/multidict-6.7.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7", size = 247970, upload-time = "2026-01-26T02:44:56.783Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/06/538e58a63ed5cfb0bd4517e346b91da32fde409d839720f664e9a4ae4f9d/multidict-6.7.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3", size = 245060, upload-time = "2026-01-26T02:44:58.195Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/2f/d743a3045a97c895d401e9bd29aaa09b94f5cbdf1bd561609e5a6c431c70/multidict-6.7.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4", size = 235888, upload-time = "2026-01-26T02:44:59.57Z" },
+ { url = "https://files.pythonhosted.org/packages/38/83/5a325cac191ab28b63c52f14f1131f3b0a55ba3b9aa65a6d0bf2a9b921a0/multidict-6.7.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8", size = 243554, upload-time = "2026-01-26T02:45:01.054Z" },
+ { url = "https://files.pythonhosted.org/packages/20/1f/9d2327086bd15da2725ef6aae624208e2ef828ed99892b17f60c344e57ed/multidict-6.7.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c", size = 252341, upload-time = "2026-01-26T02:45:02.484Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/2c/2a1aa0280cf579d0f6eed8ee5211c4f1730bd7e06c636ba2ee6aafda302e/multidict-6.7.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52", size = 246391, upload-time = "2026-01-26T02:45:03.862Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/03/7ca022ffc36c5a3f6e03b179a5ceb829be9da5783e6fe395f347c0794680/multidict-6.7.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108", size = 243422, upload-time = "2026-01-26T02:45:05.296Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/1d/b31650eab6c5778aceed46ba735bd97f7c7d2f54b319fa916c0f96e7805b/multidict-6.7.1-cp313-cp313t-win32.whl", hash = "sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32", size = 47770, upload-time = "2026-01-26T02:45:06.754Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/5b/2d2d1d522e51285bd61b1e20df8f47ae1a9d80839db0b24ea783b3832832/multidict-6.7.1-cp313-cp313t-win_amd64.whl", hash = "sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8", size = 53109, upload-time = "2026-01-26T02:45:08.044Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/a3/cc409ba012c83ca024a308516703cf339bdc4b696195644a7215a5164a24/multidict-6.7.1-cp313-cp313t-win_arm64.whl", hash = "sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118", size = 45573, upload-time = "2026-01-26T02:45:09.349Z" },
+ { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" },
+]
+
+[[package]]
+name = "multiprocess"
+version = "0.70.19"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dill" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" },
+ { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" },
+ { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/74/d2c27e03cb84251dfe7249b8e82923643c6d48fa4883b9476b025e7dc7eb/multiprocess-0.70.19-py313-none-any.whl", hash = "sha256:8d5eb4ec5017ba2fab4e34a747c6d2c2b6fecfe9e7236e77988db91580ada952", size = 156414, upload-time = "2026-01-19T06:47:35.915Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" },
+]
+
+[[package]]
+name = "murmurhash"
+version = "1.0.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/23/2e/88c147931ea9725d634840d538622e94122bceaf346233349b7b5c62964b/murmurhash-1.0.15.tar.gz", hash = "sha256:58e2b27b7847f9e2a6edf10b47a8c8dd70a4705f45dccb7bf76aeadacf56ba01", size = 13291, upload-time = "2025-11-14T09:51:15.272Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/29/2f/ba300b5f04dae0409202d6285668b8a9d3ade43a846abee3ef611cb388d5/murmurhash-1.0.15-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:fe50dc70e52786759358fd1471e309b94dddfffb9320d9dfea233c7684c894ba", size = 27861, upload-time = "2025-11-14T09:50:23.804Z" },
+ { url = "https://files.pythonhosted.org/packages/34/02/29c19d268e6f4ea1ed2a462c901eed1ed35b454e2cbc57da592fad663ac6/murmurhash-1.0.15-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1349a7c23f6092e7998ddc5bd28546cc31a595afc61e9fdb3afc423feec3d7ad", size = 27840, upload-time = "2025-11-14T09:50:25.146Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/63/58e2de2b5232cd294c64092688c422196e74f9fa8b3958bdf02d33df24b9/murmurhash-1.0.15-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b3ba6d05de2613535b5a9227d4ad8ef40a540465f64660d4a8800634ae10e04f", size = 133080, upload-time = "2025-11-14T09:50:26.566Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/9a/d13e2e9f8ba1ced06840921a50f7cece0a475453284158a3018b72679761/murmurhash-1.0.15-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fa1b70b3cc2801ab44179c65827bbd12009c68b34e9d9ce7125b6a0bd35af63c", size = 132648, upload-time = "2025-11-14T09:50:27.788Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/e1/47994f1813fa205c84977b0ff51ae6709f8539af052c7491a5f863d82bdc/murmurhash-1.0.15-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:213d710fb6f4ef3bc11abbfad0fa94a75ffb675b7dc158c123471e5de869f9af", size = 131502, upload-time = "2025-11-14T09:50:29.339Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/ea/90c1fd00b4aeb704fb5e84cd666b33ffd7f245155048071ffbb51d2bb57d/murmurhash-1.0.15-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b65a5c4e7f5d71f7ccac2d2b60bdf7092d7976270878cfec59d5a66a533db823", size = 132736, upload-time = "2025-11-14T09:50:30.545Z" },
+ { url = "https://files.pythonhosted.org/packages/00/db/da73462dbfa77f6433b128d2120ba7ba300f8c06dc4f4e022c38d240a5f5/murmurhash-1.0.15-cp313-cp313-win_amd64.whl", hash = "sha256:9aba94c5d841e1904cd110e94ceb7f49cfb60a874bbfb27e0373622998fb7c7c", size = 25682, upload-time = "2025-11-14T09:50:31.624Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/83/032729ef14971b938fbef41ee125fc8800020ee229bd35178b6ede8ee934/murmurhash-1.0.15-cp313-cp313-win_arm64.whl", hash = "sha256:263807eca40d08c7b702413e45cca75ecb5883aa337237dc5addb660f1483378", size = 23370, upload-time = "2025-11-14T09:50:33.264Z" },
+ { url = "https://files.pythonhosted.org/packages/10/83/7547d9205e9bd2f8e5dfd0b682cc9277594f98909f228eb359489baec1df/murmurhash-1.0.15-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:694fd42a74b7ce257169d14c24aa616aa6cd4ccf8abe50eca0557e08da99d055", size = 29955, upload-time = "2025-11-14T09:50:34.488Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/c7/3afd5de7a5b3ae07fe2d3a3271b327ee1489c58ba2b2f2159bd31a25edb9/murmurhash-1.0.15-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:a2ea4546ba426390beff3cd10db8f0152fdc9072c4f2583ec7d8aa9f3e4ac070", size = 30108, upload-time = "2025-11-14T09:50:35.53Z" },
+ { url = "https://files.pythonhosted.org/packages/02/69/d6637ee67d78ebb2538c00411f28ea5c154886bbe1db16c49435a8a4ab16/murmurhash-1.0.15-cp313-cp313t-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:34e5a91139c40b10f98d0b297907f5d5267b4b1b2e5dd2eb74a021824f751b98", size = 164054, upload-time = "2025-11-14T09:50:36.591Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/4c/89e590165b4c7da6bf941441212a721a270195332d3aacfdfdf527d466ca/murmurhash-1.0.15-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:dc35606868a5961cf42e79314ca0bddf5a400ce377b14d83192057928d6252ec", size = 168153, upload-time = "2025-11-14T09:50:37.856Z" },
+ { url = "https://files.pythonhosted.org/packages/07/7a/95c42df0c21d2e413b9fcd17317a7587351daeb264dc29c6aec1fdbd26f8/murmurhash-1.0.15-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:43cc6ac3b91ca0f7a5ae9c063ba4d6c26972c97fd7c25280ecc666413e4c5535", size = 164345, upload-time = "2025-11-14T09:50:39.346Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/22/9d02c880a88b83bb3ce7d6a38fb727373ab78d82e5f3d8d9fc5612219f90/murmurhash-1.0.15-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:847d712136cb462f0e4bd6229ee2d9eb996d8854eb8312dff3d20c8f5181fda5", size = 161990, upload-time = "2025-11-14T09:50:40.689Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/e3/750232524e0dc262e8dcede6536dafc766faadd9a52f1d23746b02948ad8/murmurhash-1.0.15-cp313-cp313t-win_amd64.whl", hash = "sha256:2680851af6901dbe66cc4aa7ef8e263de47e6e1b425ae324caa571bdf18f8d58", size = 28812, upload-time = "2025-11-14T09:50:41.971Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/89/4ad9d215ef6ade89f27a72dc4e86b98ef1a43534cc3e6a6900a362a0bf0a/murmurhash-1.0.15-cp313-cp313t-win_arm64.whl", hash = "sha256:189a8de4d657b5da9efd66601b0636330b08262b3a55431f2379097c986995d0", size = 25398, upload-time = "2025-11-14T09:50:43.023Z" },
+]
+
+[[package]]
+name = "mypy"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ast-serialize" },
+ { name = "librt", marker = "platform_python_implementation != 'PyPy'" },
+ { name = "mypy-extensions" },
+ { name = "pathspec" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/82/15/cca9d88503549ed6fedeaa1d448cdddd542ee8a490232d732e278036fbf2/mypy-2.1.0.tar.gz", hash = "sha256:81e76ad12c2d804512e9b13240d1588316531bfba07558286078bfbce9613633", size = 3898359, upload-time = "2026-05-11T18:37:36.237Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6e/dd/c7191469c777f07689c032a8f7326e393ea34c92d6d76eb7ce5ba57ea66d/mypy-2.1.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:35aac3bb114e03888f535d5eb51b8bafbb3266586b599da1940f9b1be3ec5bd5", size = 14852174, upload-time = "2026-05-11T18:31:38.929Z" },
+ { url = "https://files.pythonhosted.org/packages/55/8c/aed55408879043d72bb9135f4d0d19a02b886dd569631e113e3d2706cb8d/mypy-2.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8de55a8c861f2a49331f807be98d90caeceeef520bde13d43a160207f8af613e", size = 13651542, upload-time = "2026-05-11T18:36:04.636Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/8e/f371a824b1f1fa8ea6e3dbb8703d232977d572be2329554a3bc4d960302f/mypy-2.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5fdf2941a07434af755837d9880f7d7d25f1dacb1af9dcd4b9b66f2220a3024e", size = 14033929, upload-time = "2026-05-11T18:35:55.742Z" },
+ { url = "https://files.pythonhosted.org/packages/94/21/f54be870d6dd53a82c674407e0f8eed7174b05ec78d42e5abd7b42e84fd5/mypy-2.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e195b817c13f02352a9c124301f9f30f078405444679b6753c1b96b6eed37285", size = 15039200, upload-time = "2026-05-11T18:33:10.281Z" },
+ { url = "https://files.pythonhosted.org/packages/17/99/bf21748626a40ce59fd29a39386ab46afec88b7bd2f0fa6c3a97c995523f/mypy-2.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5431d42af987ebd92ba2f71d45c85ed41d8e6ca9f5fd209a69f68f707d2469e5", size = 15272690, upload-time = "2026-05-11T18:32:07.205Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/d7/9e90d2cf47100bea550ed2bc7b0d4de3a62181d84d5e37da0003e8462637/mypy-2.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:767fe8c66dc3e01e19e1737d4c38ebefead16125e1b8e58ad421903b376f5c65", size = 11147435, upload-time = "2026-05-11T18:33:56.477Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/46/e5c449e858798e35ffc90946282a27c62a77be743fe17480e4977374eb91/mypy-2.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:ecfe70d43775ab99562ab128ce49854a362044c9f894961f68f898c23cb7429d", size = 10035052, upload-time = "2026-05-11T18:32:30.049Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/2a/13ca1f292f6db1b98ff495ef3467736b331621c5917cad984b7043e7348d/mypy-2.1.0-py3-none-any.whl", hash = "sha256:a663814603a5c563fb87a4f96fb473eeb30d1f5a4885afcf44f9db000a366289", size = 2693302, upload-time = "2026-05-11T18:31:29.246Z" },
+]
+
+[[package]]
+name = "mypy-extensions"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" },
+]
+
+[[package]]
+name = "narwhals"
+version = "2.23.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e8/ac/66ed1fc6e38a0c0f330627ec5c5d597990d6159b6712b82af0ad2c65f06c/narwhals-2.23.0.tar.gz", hash = "sha256:13e7ff5b4bb4a2f77b907c2e4d8a76e273dfc1323a3c997440a2f9fd26aed408", size = 656209, upload-time = "2026-07-01T11:21:53.278Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f4/4e/afc8c31605cb8be1d3bb4438c4d979daa104dab6306cd2b87abe9c3a7299/narwhals-2.23.0-py3-none-any.whl", hash = "sha256:769e7b9ab102c93d8fa019f6b4cd1a657909b04a20bf6210e5a35aae06814ae9", size = 458938, upload-time = "2026-07-01T11:21:51.677Z" },
+]
+
+[[package]]
+name = "natto-py"
+version = "1.0.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cffi" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/18/b5/ce97638848783d99e36388aa3cf88df5e67fa491232630a647bd091ecf07/natto-py-1.0.1.tar.gz", hash = "sha256:760103bb397232ee033c9924d1357e32b142bbe132fc6a43b8cf82dd6b654e86", size = 36832, upload-time = "2022-09-07T04:25:11.406Z" }
+
+[[package]]
+name = "nbclient"
+version = "0.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-client" },
+ { name = "jupyter-core" },
+ { name = "nbformat" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/a5/b3bae4b590c0cbcada2c63a34f7580024e834a8ba213e949a2f906705787/nbclient-0.11.0.tar.gz", hash = "sha256:04a134a5b087f2c5887f228aca155db50169b8cd9334dee6942c8e927e56081a", size = 62535, upload-time = "2026-06-05T07:52:41.746Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/36/c9/94d73e5a01c5b926c3fa2496e97d7a8dc28ed5a77c0b2ed712f1a62e6694/nbclient-0.11.0-py3-none-any.whl", hash = "sha256:ef7fa0d59d6e1d41103933d8a445a18d5de860ca6b613b87b8574accdb3c2895", size = 25288, upload-time = "2026-06-05T07:52:40.115Z" },
+]
+
+[[package]]
+name = "nbconvert"
+version = "7.17.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "beautifulsoup4" },
+ { name = "bleach", extra = ["css"] },
+ { name = "defusedxml" },
+ { name = "jinja2" },
+ { name = "jupyter-core" },
+ { name = "jupyterlab-pygments" },
+ { name = "markupsafe" },
+ { name = "mistune" },
+ { name = "nbclient" },
+ { name = "nbformat" },
+ { name = "packaging" },
+ { name = "pandocfilters" },
+ { name = "pygments" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/01/b1/708e53fe2e429c103c6e6e159106bcf0357ac41aa4c28772bd8402339051/nbconvert-7.17.1.tar.gz", hash = "sha256:34d0d0a7e73ce3cbab6c5aae8f4f468797280b01fd8bd2ca746da8569eddd7d2", size = 865311, upload-time = "2026-04-08T00:44:14.914Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/67/f8/bb0a9d5f46819c821dc1f004aa2cc29b1d91453297dbf5ff20470f00f193/nbconvert-7.17.1-py3-none-any.whl", hash = "sha256:aa85c087b435e7bf1ffd03319f658e285f2b89eccab33bc1ba7025495ab3e7c8", size = 261927, upload-time = "2026-04-08T00:44:12.845Z" },
+]
+
+[[package]]
+name = "nbformat"
+version = "5.10.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "fastjsonschema" },
+ { name = "jsonschema" },
+ { name = "jupyter-core" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6d/fd/91545e604bc3dad7dca9ed03284086039b294c6b3d75c0d2fa45f9e9caf3/nbformat-5.10.4.tar.gz", hash = "sha256:322168b14f937a5d11362988ecac2a4952d3d8e3a2cbeb2319584631226d5b3a", size = 142749, upload-time = "2024-04-04T11:20:37.371Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a9/82/0340caa499416c78e5d8f5f05947ae4bc3cba53c9f038ab6e9ed964e22f1/nbformat-5.10.4-py3-none-any.whl", hash = "sha256:3b48d6c8fbca4b299bf3982ea7db1af21580e4fec269ad087b9e81588891200b", size = 78454, upload-time = "2024-04-04T11:20:34.895Z" },
+]
+
+[[package]]
+name = "nccl4py"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-core", version = "1.0.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "cuda-core", version = "1.1.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-pathfinder" },
+ { name = "numpy" },
+ { name = "packaging" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6a/1c/412099397e727a5a03864121959546e7aee90034c543b2ea637d201a79c4/nccl4py-0.4.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3d31180a94b4220b50c5dd1627965303055733cf0f790bd9275afa95fa84ac5c", size = 2830787, upload-time = "2026-08-11T23:47:28.741Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/71/4199004ba5db045053c0a13beab790e0facbf78b139656b459ac4791cee1/nccl4py-0.4.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c39c7b59dca0460cd7e34473dbb4aa7fbce4a3380d237817f1d3ed629f98ef9f", size = 2877926, upload-time = "2026-08-11T23:47:55.221Z" },
+]
+
+[[package]]
+name = "nest-asyncio"
+version = "1.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" },
+]
+
+[[package]]
+name = "nest-asyncio2"
+version = "1.7.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b4/73/731debf26e27e0a0323d7bda270dc2f634b398e38f040a09da1f4351d0aa/nest_asyncio2-1.7.2.tar.gz", hash = "sha256:1921d70b92cc4612c374928d081552efb59b83d91b2b789d935c665fa01729a8", size = 14743, upload-time = "2026-02-13T00:34:04.386Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c5/3c/3179b85b0e1c3659f0369940200cd6d0fa900e6cefcc7ea0bc6dd0e29ffb/nest_asyncio2-1.7.2-py3-none-any.whl", hash = "sha256:f5dfa702f3f81f6a03857e9a19e2ba578c0946a4ad417b4c50a24d7ba641fe01", size = 7843, upload-time = "2026-02-13T00:34:02.691Z" },
+]
+
+[[package]]
+name = "networkx"
+version = "3.6.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" },
+]
+
+[[package]]
+name = "ninja"
+version = "1.13.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/43/73/79a0b22fc731989c708068427579e840a6cf4e937fe7ae5c5d0b7356ac22/ninja-1.13.0.tar.gz", hash = "sha256:4a40ce995ded54d9dc24f8ea37ff3bf62ad192b547f6c7126e7e25045e76f978", size = 242558, upload-time = "2025-08-11T15:10:19.421Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3c/74/d02409ed2aa865e051b7edda22ad416a39d81a84980f544f8de717cab133/ninja-1.13.0-py3-none-macosx_10_9_universal2.whl", hash = "sha256:fa2a8bfc62e31b08f83127d1613d10821775a0eb334197154c4d6067b7068ff1", size = 310125, upload-time = "2025-08-11T15:09:50.971Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/de/6e1cd6b84b412ac1ef327b76f0641aeb5dcc01e9d3f9eee0286d0c34fd93/ninja-1.13.0-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3d00c692fb717fd511abeb44b8c5d00340c36938c12d6538ba989fe764e79630", size = 177467, upload-time = "2025-08-11T15:09:52.767Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/83/49320fb6e58ae3c079381e333575fdbcf1cca3506ee160a2dcce775046fa/ninja-1.13.0-py3-none-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:be7f478ff9f96a128b599a964fc60a6a87b9fa332ee1bd44fa243ac88d50291c", size = 187834, upload-time = "2025-08-11T15:09:54.115Z" },
+ { url = "https://files.pythonhosted.org/packages/56/c7/ba22748fb59f7f896b609cd3e568d28a0a367a6d953c24c461fe04fc4433/ninja-1.13.0-py3-none-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:60056592cf495e9a6a4bea3cd178903056ecb0943e4de45a2ea825edb6dc8d3e", size = 202736, upload-time = "2025-08-11T15:09:55.745Z" },
+ { url = "https://files.pythonhosted.org/packages/79/22/d1de07632b78ac8e6b785f41fa9aad7a978ec8c0a1bf15772def36d77aac/ninja-1.13.0-py3-none-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:1c97223cdda0417f414bf864cfb73b72d8777e57ebb279c5f6de368de0062988", size = 179034, upload-time = "2025-08-11T15:09:57.394Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/de/0e6edf44d6a04dabd0318a519125ed0415ce437ad5a1ec9b9be03d9048cf/ninja-1.13.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fb46acf6b93b8dd0322adc3a4945452a4e774b75b91293bafcc7b7f8e6517dfa", size = 180716, upload-time = "2025-08-11T15:09:58.696Z" },
+ { url = "https://files.pythonhosted.org/packages/54/28/938b562f9057aaa4d6bfbeaa05e81899a47aebb3ba6751e36c027a7f5ff7/ninja-1.13.0-py3-none-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4be9c1b082d244b1ad7ef41eb8ab088aae8c109a9f3f0b3e56a252d3e00f42c1", size = 146843, upload-time = "2025-08-11T15:10:00.046Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/fb/d06a3838de4f8ab866e44ee52a797b5491df823901c54943b2adb0389fbb/ninja-1.13.0-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:6739d3352073341ad284246f81339a384eec091d9851a886dfa5b00a6d48b3e2", size = 154402, upload-time = "2025-08-11T15:10:01.657Z" },
+ { url = "https://files.pythonhosted.org/packages/31/bf/0d7808af695ceddc763cf251b84a9892cd7f51622dc8b4c89d5012779f06/ninja-1.13.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:11be2d22027bde06f14c343f01d31446747dbb51e72d00decca2eb99be911e2f", size = 552388, upload-time = "2025-08-11T15:10:03.349Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/70/c99d0c2c809f992752453cce312848abb3b1607e56d4cd1b6cded317351a/ninja-1.13.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:aa45b4037b313c2f698bc13306239b8b93b4680eb47e287773156ac9e9304714", size = 472501, upload-time = "2025-08-11T15:10:04.735Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/43/c217b1153f0e499652f5e0766da8523ce3480f0a951039c7af115e224d55/ninja-1.13.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:5f8e1e8a1a30835eeb51db05cf5a67151ad37542f5a4af2a438e9490915e5b72", size = 638280, upload-time = "2025-08-11T15:10:06.512Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/45/9151bba2c8d0ae2b6260f71696330590de5850e5574b7b5694dce6023e20/ninja-1.13.0-py3-none-musllinux_1_2_ppc64le.whl", hash = "sha256:3d7d7779d12cb20c6d054c61b702139fd23a7a964ec8f2c823f1ab1b084150db", size = 642420, upload-time = "2025-08-11T15:10:08.35Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/fb/95752eb635bb8ad27d101d71bef15bc63049de23f299e312878fc21cb2da/ninja-1.13.0-py3-none-musllinux_1_2_riscv64.whl", hash = "sha256:d741a5e6754e0bda767e3274a0f0deeef4807f1fec6c0d7921a0244018926ae5", size = 585106, upload-time = "2025-08-11T15:10:09.818Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/31/aa56a1a286703800c0cbe39fb4e82811c277772dc8cd084f442dd8e2938a/ninja-1.13.0-py3-none-musllinux_1_2_s390x.whl", hash = "sha256:e8bad11f8a00b64137e9b315b137d8bb6cbf3086fbdc43bf1f90fd33324d2e96", size = 707138, upload-time = "2025-08-11T15:10:11.366Z" },
+ { url = "https://files.pythonhosted.org/packages/34/6f/5f5a54a1041af945130abdb2b8529cbef0cdcbbf9bcf3f4195378319d29a/ninja-1.13.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b4f2a072db3c0f944c32793e91532d8948d20d9ab83da9c0c7c15b5768072200", size = 581758, upload-time = "2025-08-11T15:10:13.295Z" },
+ { url = "https://files.pythonhosted.org/packages/95/97/51359c77527d45943fe7a94d00a3843b81162e6c4244b3579fe8fc54cb9c/ninja-1.13.0-py3-none-win32.whl", hash = "sha256:8cfbb80b4a53456ae8a39f90ae3d7a2129f45ea164f43fadfa15dc38c4aef1c9", size = 267201, upload-time = "2025-08-11T15:10:15.158Z" },
+ { url = "https://files.pythonhosted.org/packages/29/45/c0adfbfb0b5895aa18cec400c535b4f7ff3e52536e0403602fc1a23f7de9/ninja-1.13.0-py3-none-win_amd64.whl", hash = "sha256:fb8ee8719f8af47fed145cced4a85f0755dd55d45b2bddaf7431fa89803c5f3e", size = 309975, upload-time = "2025-08-11T15:10:16.697Z" },
+ { url = "https://files.pythonhosted.org/packages/df/93/a7b983643d1253bb223234b5b226e69de6cda02b76cdca7770f684b795f5/ninja-1.13.0-py3-none-win_arm64.whl", hash = "sha256:3c0b40b1f0bba764644385319028650087b4c1b18cdfa6f45cb39a3669b81aa9", size = 290806, upload-time = "2025-08-11T15:10:18.018Z" },
+]
+
+[[package]]
+name = "nltk"
+version = "3.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "joblib" },
+ { name = "regex" },
+ { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3c/87/db8be88ad32c2d042420b6fd9ffd4a149f9a0d7f0e86b3f543be2eeeedd2/nltk-3.9.1.tar.gz", hash = "sha256:87d127bd3de4bd89a4f81265e5fa59cb1b199b27440175370f7417d2bc7ae868", size = 2904691, upload-time = "2024-08-18T19:48:37.769Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4d/66/7d9e26593edda06e8cb531874633f7c2372279c3b0f46235539fe546df8b/nltk-3.9.1-py3-none-any.whl", hash = "sha256:4fa26829c5b00715afe3061398a8989dc643b92ce7dd93fb4585a70930d168a1", size = 1505442, upload-time = "2024-08-18T19:48:21.909Z" },
+]
+
+[[package]]
+name = "nodeenv"
+version = "1.10.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/24/bf/d1bda4f6168e0b2e9e5958945e01910052158313224ada5ce1fb2e1113b8/nodeenv-1.10.0.tar.gz", hash = "sha256:996c191ad80897d076bdfba80a41994c2b47c68e224c542b48feba42ba00f8bb", size = 55611, upload-time = "2025-12-20T14:08:54.006Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" },
+]
+
+[[package]]
+name = "notebook"
+version = "7.6.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-builder" },
+ { name = "jupyter-server" },
+ { name = "jupyterlab" },
+ { name = "jupyterlab-server" },
+ { name = "notebook-shim" },
+ { name = "tornado" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0d/44/d5c65783f490298473bb1c05722e05ee2256231389559c2c5ae0a3e5d975/notebook-7.6.0.tar.gz", hash = "sha256:ea13e79e601bf273074895fdfb17dd3f2da916d3c045e0b9c47d18b16ab62481", size = 5497344, upload-time = "2026-06-18T16:18:55.202Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/93/d1/e617c40db57ff40e75f43a7d4d1c305e3a54c053ab5cb0534a6c314664f9/notebook-7.6.0-py3-none-any.whl", hash = "sha256:98aa2811b54ac191321d5dfce12ca700f8a511a33a26e4de2fa106a357c43d6a", size = 5544575, upload-time = "2026-06-18T16:18:52.551Z" },
+]
+
+[[package]]
+name = "notebook-shim"
+version = "0.2.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-server" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/54/d2/92fa3243712b9a3e8bafaf60aac366da1cada3639ca767ff4b5b3654ec28/notebook_shim-0.2.4.tar.gz", hash = "sha256:b4b2cfa1b65d98307ca24361f5b30fe785b53c3fd07b7a47e89acb5e6ac638cb", size = 13167, upload-time = "2024-02-14T23:35:18.353Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f9/33/bd5b9137445ea4b680023eb0469b2bb969d61303dedb2aac6560ff3d14a1/notebook_shim-0.2.4-py3-none-any.whl", hash = "sha256:411a5be4e9dc882a074ccbcae671eda64cceb068767e9a3419096986560e1cef", size = 13307, upload-time = "2024-02-14T23:35:16.286Z" },
+]
+
+[[package]]
+name = "numba"
+version = "0.65.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "llvmlite" },
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/49/61/7299643b9c18d669e04be7c5bcb64d985070d07553274817b45b049e7bfe/numba-0.65.0.tar.gz", hash = "sha256:edad0d9f6682e93624c00125a471ae4df186175d71fd604c983c377cdc03e68b", size = 2764131, upload-time = "2026-04-01T03:52:01.946Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8b/f8/eee0f1ff456218db036bfc9023995ec1f85a9dc8f2422f1594f6a87829e0/numba-0.65.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:c6334094563a456a695c812e6846288376ca02327cf246cdcc83e1bb27862367", size = 2680679, upload-time = "2026-04-01T03:51:39.491Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/8f/3d116e4b8e92f6abace431afa4b2b944f4d65bdee83af886f5c4b263df95/numba-0.65.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b8a9008411615c69d083d1dcf477f75a5aa727b30beb16e139799e2be945cdfd", size = 3809537, upload-time = "2026-04-01T03:51:41.42Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/2c/6a3ca4128e253cb67affe06deb47688f51ce968f5111e2a06d010e6f1fa6/numba-0.65.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:af96c0cba53664efcb361528b8c75e011a6556c859c7e08424c2715201c6cf7a", size = 3508615, upload-time = "2026-04-01T03:51:43.444Z" },
+ { url = "https://files.pythonhosted.org/packages/96/0e/267f9a36fb282c104a971d7eecb685b411c47dce2a740fe69cf5fc2945d9/numba-0.65.0-cp313-cp313-win_amd64.whl", hash = "sha256:6254e73b9c929dc736a1fbd3d6f5680789709a5067cae1fa7198707385129c04", size = 2749938, upload-time = "2026-04-01T03:51:45.218Z" },
+]
+
+[[package]]
+name = "numpy"
+version = "2.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/ad/fed0499ce6a338d2a03ebae59cd15093910c8875328855781952abf6c2fe/numpy-2.4.6.tar.gz", hash = "sha256:f3a3570c4a2a16746ac2c31a7c7c7b0c186b95ce902e33db6f28094ed7387dda", size = 20735807, upload-time = "2026-05-18T23:37:14.07Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/82/bdab26d7438c6791ca31b7c024ca37c1eab8b726ba236129005cd4a06e45/numpy-2.4.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:511dbaf848decaaaf4b4ca48032619fb3138710c4bf7da7617765edad1ef96b0", size = 16684648, upload-time = "2026-05-18T23:34:29.41Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/30/a80189bcc7f5e4258b3fbc3968d909d1756f54d023299ecc39ad6fdb9ef8/numpy-2.4.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bf162abab1c1a736333192707cef898e735a5ca00f38f27eeedf44b39d9e85eb", size = 14693902, upload-time = "2026-05-18T23:34:33.013Z" },
+ { url = "https://files.pythonhosted.org/packages/97/12/70b5d0d7c15e1ebb8a6a84a8caa1d19e181d84fb58bb6d70aca29099dec1/numpy-2.4.6-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:043191bfa8eab18c776647b62723ac9dddece59743b13f49b2016094129c2b3f", size = 5198992, upload-time = "2026-05-18T23:34:36.132Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/8c/ebd2a8f8a83541f8d38cc5667e8c2b69cecfd30da6e45693e8158857d44b/numpy-2.4.6-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:6180d8b35af935aed8ece3a85e0a43f87393ae0ac87c8d2c8bd2c993f7270ef3", size = 6546944, upload-time = "2026-05-18T23:34:38.484Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/c5/7b863a97a91671a0338f4253bd3b5a3d3852f0692dae91711c9f4a10e787/numpy-2.4.6-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:72fbe16c6fac95aedf5937fa873445cec2110be35d8a4e9433d7501fd98dae6b", size = 15669392, upload-time = "2026-05-18T23:34:41.257Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/9d/3584b9984ca4c047aea75214ce1a4c4c73d849bd71b604264b7f5653f8a8/numpy-2.4.6-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a7830bab239b79cda9c08c2da014761cafb48da6150e1da17ac06283f43b6089", size = 16633220, upload-time = "2026-05-18T23:34:45.075Z" },
+ { url = "https://files.pythonhosted.org/packages/05/ae/7c67fba23bd98caec7c99261f3a16072ade14813486b0282cb29846de832/numpy-2.4.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ef4aea96ce4d3b074422cb4f2f64e216bf9e213004bb58ecfdf50ea02ea8eb9a", size = 17020800, upload-time = "2026-05-18T23:34:49.065Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/5d/3b6725cb31d983c5e66916f5d36f6d7e5521129e4c4404d64f918292a5b6/numpy-2.4.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dfa20cc6ca228e6b155b11da03825975ce66aea520985dbbddf0f2a5a495c605", size = 18357600, upload-time = "2026-05-18T23:34:52.709Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/da/2ccc6c2fe8898dee01d90c75c5f5f914a23daf99e3e0f59516a08760c8b5/numpy-2.4.6-cp313-cp313-win32.whl", hash = "sha256:56b39e5e0622a09a25bf5baf62f4bcf0cb8a41ae6e2819cf49bbc5a74c083f91", size = 5961134, upload-time = "2026-05-18T23:34:55.618Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/cd/9cc4dc876fb065d5c220aae4d5e14826b2715331bb7618ce1fb07a679d99/numpy-2.4.6-cp313-cp313-win_amd64.whl", hash = "sha256:c4fc99836233ea196540b17ab0983aff60ed07941751930f5f4d05bc3b3b7359", size = 12318598, upload-time = "2026-05-18T23:34:58.928Z" },
+ { url = "https://files.pythonhosted.org/packages/39/1e/c0bcba1f8694116485fe28fd1be698c278fcda4141c5b0e53a2aed8b12a8/numpy-2.4.6-cp313-cp313-win_arm64.whl", hash = "sha256:a7c711e21628b52034bb5ab8d1bce291f752fcc5e92accc615778acee1ff4778", size = 10222272, upload-time = "2026-05-18T23:35:02.167Z" },
+ { url = "https://files.pythonhosted.org/packages/63/6d/cc5619247c8f4204e507f5883528372e4ac4bb189e579fb859a12e480b1f/numpy-2.4.6-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:112b06a867b235ef466ed3508ddf0238050df9c727cafb5301ac385b899189a1", size = 14821197, upload-time = "2026-05-18T23:35:05.468Z" },
+ { url = "https://files.pythonhosted.org/packages/00/58/f1c39161c87d9e9bed660f1ed4bafc0e403d5ec9650b6dd77aead07d489b/numpy-2.4.6-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:eaf7fa2de5c0be8ae6ff8e9bea2ccd725e980541244521d8d4b5f3354a27babe", size = 5326287, upload-time = "2026-05-18T23:35:08.693Z" },
+ { url = "https://files.pythonhosted.org/packages/af/57/3917ab0fd97f271a8694513581b8a36c655f111c446852c302f04ccdb6fc/numpy-2.4.6-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:7265a2f3d436e54ef9f2b52b5c937e6be778781bd97a590319d7348f1c1ca997", size = 6646763, upload-time = "2026-05-18T23:35:11.459Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/0f/037e64c494b67581ae18193d770adef354c41f3f2c8ebf865602d949bf8f/numpy-2.4.6-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f74a575920ab21fe304421a3fc28793d82e299cae9eccb37084e9fc7f3617c20", size = 15728070, upload-time = "2026-05-18T23:35:14.79Z" },
+ { url = "https://files.pythonhosted.org/packages/21/a6/5d2bae9c9542eb4df16dc9c46dc79c186e9bad53805dfa5399a6023c6db0/numpy-2.4.6-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ede83e07a75dd06bc501566c1eca2afc0d61677c1472ac9ad93fdee6e638a48d", size = 16681752, upload-time = "2026-05-18T23:35:18.836Z" },
+ { url = "https://files.pythonhosted.org/packages/92/14/23d1dfb410ae362cd59ce53e936b1513d545eb40db3949ced632e19a459e/numpy-2.4.6-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:68bb27509ac1b9a3443094260f6326150663b06abe40b73a2f81160623da5b67", size = 17086024, upload-time = "2026-05-18T23:35:22.52Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/6e/23595a2c642cdf3bc567877064bdd7f91c8b0038a4453cf2daf7248eafe9/numpy-2.4.6-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:a0df0043bdb289bde1f62da130d20df23d58b45429f752bc7a8fc5325a225ecd", size = 18403398, upload-time = "2026-05-18T23:35:26.398Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/90/0ac3bc947217e66dec77e7cbc6a1979d1af70b6461b82f620d3bccd5e4c8/numpy-2.4.6-cp313-cp313t-win32.whl", hash = "sha256:29a287e0cf63ff528da061de6b9f64a4618da591ca1046aafc54062e40ca7eab", size = 6084971, upload-time = "2026-05-18T23:35:29.387Z" },
+ { url = "https://files.pythonhosted.org/packages/77/71/5673e351671a1d2bd6063b91b44f70c0affea7d1516fa7a6572941ba4aa1/numpy-2.4.6-cp313-cp313t-win_amd64.whl", hash = "sha256:25c692919ac5a01f170a3bfcd62d745b24fd095c353d50812637d6fcab442e75", size = 12458532, upload-time = "2026-05-18T23:35:32.175Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/88/19d3503c5046e688f049274b27a3ef3d771152fa80d3ba3d01a3dff61abe/numpy-2.4.6-cp313-cp313t-win_arm64.whl", hash = "sha256:1e978ec1e8bd0e0e4de6bb75de9d30cbb74db6b6a2bb727618613703ca0167dd", size = 10291881, upload-time = "2026-05-18T23:35:35.465Z" },
+]
+
+[[package]]
+name = "nvidia-cublas"
+version = "13.1.1.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cuda-nvrtc", version = "13.0.88", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/cd/154ca20c38269e05eff77c1464e6c1da89f50a6390b565e9d82e06bc11e1/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:37936a16db8fe4ac1f065c2139360608a543a09275cb1a1af612e08cfa065436", size = 423138758, upload-time = "2026-04-08T18:46:58.655Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-cccl"
+version = "13.3.3.4.1"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/96/bd/572971ffc14bd36676c821fc15d991b08fe6179cb09368250147475f954d/nvidia_cuda_cccl-13.3.3.4.1-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:067d19b4b3c9d0f2ebec9f29a311b2863db96bf98e058bbc331597d51ce818cf", size = 3454030, upload-time = "2026-06-29T16:41:49.092Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/ab/049726d90147865a3ea53bae6cb7c35b98bf1fdf96cdb967101329625f83/nvidia_cuda_cccl-13.3.3.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cc0adc188d570b09f4d606c7dc05a42aa3d8aa082e0d60f7bbfc5b6435f627c6", size = 3454034, upload-time = "2026-06-29T16:42:07.435Z" },
+ { url = "https://files.pythonhosted.org/packages/24/d3/b1afcd9c40ceca72022579215fcaf5318cd747fd896cb928d4a1de924ff8/nvidia_cuda_cccl-13.3.3.4.1-py3-none-win_amd64.whl", hash = "sha256:d7c92cc03047031fa7af30866636d35ce4af409c28fc7dd8f69cb17053741399", size = 3454014, upload-time = "2026-06-29T17:09:09.012Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-crt"
+version = "13.3.73"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fa/41/2089e411507d66458d67208bdd1bc562d492bb6458c3d2aea4603072a219/nvidia_cuda_crt-13.3.73-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:60aacc0b5e1e8b40c62abe4d1ab16440add91b99bd2f17f62dd091586b73d166", size = 157353, upload-time = "2026-06-29T16:42:38.163Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/ce/16d76f4b5b3f7460f5ebd17516685495c149c66651ffdd381f90e4d4e65c/nvidia_cuda_crt-13.3.73-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:df14a17ae1c5c3171265411212246654d780f89344ea85344466c6b955247543", size = 157352, upload-time = "2026-06-29T16:43:09.209Z" },
+ { url = "https://files.pythonhosted.org/packages/49/b3/6791ffba6f4b8e0d3ed875285aad8078ee407afa464ecd934ae298c205b1/nvidia_cuda_crt-13.3.73-py3-none-win_amd64.whl", hash = "sha256:af04e75148db1f0eea30958f33a9ec5a5a2dc2afa99ca4323f9a93b840602ca5", size = 158286, upload-time = "2026-06-29T17:09:28.621Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-cupti"
+version = "13.0.85"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/2a/80353b103fc20ce05ef51e928daed4b6015db4aaa9162ed0997090fe2250/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_aarch64.whl", hash = "sha256:796bd679890ee55fb14a94629b698b6db54bcfd833d391d5e94017dd9d7d3151", size = 10310827, upload-time = "2025-09-04T08:26:42.012Z" },
+ { url = "https://files.pythonhosted.org/packages/33/6d/737d164b4837a9bbd202f5ae3078975f0525a55730fe871d8ed4e3b952b0/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_x86_64.whl", hash = "sha256:4eb01c08e859bf924d222250d2e8f8b8ff6d3db4721288cf35d14252a4d933c8", size = 10715597, upload-time = "2025-09-04T08:26:51.312Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-nvcc"
+version = "13.2.78"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cuda-crt" },
+ { name = "nvidia-cuda-runtime", version = "13.0.96", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cuda-runtime", version = "13.3.29", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "nvidia-nvvm" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/df/faf551572ae1359290afa5cb05d2c4b7e6674b07b8283b20eab4dbad15f6/nvidia_cuda_nvcc-13.2.78-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:dfc76950c775cd00ce588f15192f08c9b858c0dcfa7da685acf39a3d0d8f588b", size = 38713559, upload-time = "2026-04-13T09:42:17.478Z" },
+ { url = "https://files.pythonhosted.org/packages/65/0f/c7c7d538c61794130e759ad74710ab5aa8cab1f700ee1754381f8c665605/nvidia_cuda_nvcc-13.2.78-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c3bd144dd9b6b25e062589acb7bbd43d93d3120c72fad71da808f9817aba1239", size = 44040318, upload-time = "2026-04-13T09:42:50.457Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/f1/533329b960fad3d800a50e89f43a2e1b8dade07457ce340d4f0858203dcc/nvidia_cuda_nvcc-13.2.78-py3-none-win_amd64.whl", hash = "sha256:6bc1047a44ff0751b0506cb6d8c7565edb0d3ff71f69d562333c9d1c540dcfd1", size = 32002789, upload-time = "2026-04-13T10:05:40.376Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-nvdisasm"
+version = "13.3.73"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/92/be/e9de501cb71b10f7654381a485fa4ebf470ea25c3dce018cccaecf8a8f9a/nvidia_cuda_nvdisasm-13.3.73-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:dd4751884f9016b9b6dbf007abdeb5681d0a2edc731dd3d2fda9d6d878e88f73", size = 4744517, upload-time = "2026-06-29T16:48:33.527Z" },
+ { url = "https://files.pythonhosted.org/packages/86/3e/88460ebd737e559e8e9843db7a63f8ced9ec7be1882344438819dd13aebc/nvidia_cuda_nvdisasm-13.3.73-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fa17084b07c0dca68a42892f771b4b1b40fbe9b91660209623e61cea611cae8c", size = 4782824, upload-time = "2026-06-29T16:49:05.109Z" },
+ { url = "https://files.pythonhosted.org/packages/32/63/00d687730b124f94345f83023363251310ccc08eeac269ec2eeff6b097f3/nvidia_cuda_nvdisasm-13.3.73-py3-none-win_amd64.whl", hash = "sha256:da2fab133c3d095d83f13587eb87149beabd199b34a3cc270d0aa99449a628c3", size = 5015368, upload-time = "2026-06-29T17:22:50.207Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-nvrtc"
+version = "13.0.88"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c3/68/483a78f5e8f31b08fb1bb671559968c0ca3a065ac7acabfc7cee55214fd6/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:ad9b6d2ead2435f11cbb6868809d2adeeee302e9bb94bcf0539c7a40d80e8575", size = 90215200, upload-time = "2025-09-04T08:28:44.204Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/dc/6bb80850e0b7edd6588d560758f17e0550893a1feaf436807d64d2da040f/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d27f20a0ca67a4bb34268a5e951033496c5b74870b868bacd046b1b8e0c3267b", size = 43015449, upload-time = "2025-09-04T08:28:20.239Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-nvrtc"
+version = "13.3.33"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform == 'darwin'",
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a1/42/edce72f2c5a0f587168109c867f25f4a9a6cd7289ecf0d68ed2b1070f273/nvidia_cuda_nvrtc-13.3.33-py3-none-win_amd64.whl", hash = "sha256:7d2af818851c0c224d5f92221e9226e51ee23c236df4b51f9194563979c888be", size = 45319163, upload-time = "2026-05-26T17:02:49.217Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-runtime"
+version = "13.0.96"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/87/4f/17d7b9b8e285199c58ce28e31b5c5bbaa4d8271af06a89b6405258245de2/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ef9bcbe90493a2b9d810e43d249adb3d02e98dd30200d86607d8d02687c43f55", size = 2261060, upload-time = "2025-10-09T08:55:15.78Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/24/d1558f3b68b1d26e706813b1d10aa1d785e4698c425af8db8edc3dced472/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7f82250d7782aa23b6cfe765ecc7db554bd3c2870c43f3d1821f1d18aebf0548", size = 2243632, upload-time = "2025-10-09T08:55:36.117Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-runtime"
+version = "13.3.29"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform == 'darwin'",
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/27/b53a5e0397842a5c11f0e1a39d4e5b2f22638a4126e83b3c4e196f62c969/nvidia_cuda_runtime-13.3.29-py3-none-win_amd64.whl", hash = "sha256:0667ec61c3d897388efa305ed4f7609ace88849a753ba9c6311d06dca55fff4f", size = 2630354, upload-time = "2026-05-26T17:00:05.389Z" },
+]
+
+[[package]]
+name = "nvidia-cudnn-cu13"
+version = "9.20.0.48"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cublas", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/56/c5/83384d846b2fd17c44bd499b36c75a45ed4f095fbbb2252294e89cea5c5c/nvidia_cudnn_cu13-9.20.0.48-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:e31454ae00094b0c55319d9d15b6fa2fc50a9e1c0f5c8c80fb75258234e731e1", size = 444574296, upload-time = "2026-03-09T19:28:27.751Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/5e/edb9c0ae051602c3ccaffe424256463636d639e27d7f302dde9975ef9e7a/nvidia_cudnn_cu13-9.20.0.48-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0c45dd8eeb50b603f07995b1b300c62ffe6a1980482b82b3bcf94a4ca9d49304", size = 366173588, upload-time = "2026-03-09T19:29:34.474Z" },
+]
+
+[[package]]
+name = "nvidia-cudnn-frontend"
+version = "1.25.0"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0e/37/ea07ff3578cb3cc847fbed4e7eb84bf02afa81bed437dc0a5e8b7f040c9f/nvidia_cudnn_frontend-1.25.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e8a223ef2e821bbe89fdc6a461cbce36b509e2daaa0a4425eb387060e0cc6ec7", size = 3263340, upload-time = "2026-06-10T21:08:53.528Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/ff/072bc1ed5e6d7efaf785498e6610a0b2152603c1e020faa6e6b8761d4505/nvidia_cudnn_frontend-1.25.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:95ab349dfc871a4e7e5b7ceaed649a22b968ce02bca9be1d0a0a577e80fa6832", size = 3414303, upload-time = "2026-06-10T21:09:15.951Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/7f/098f9d195f31e2d5ed5da91aa5dea693b582c70a8db0829ba9128a5f0cb2/nvidia_cudnn_frontend-1.25.0-cp313-cp313-win_amd64.whl", hash = "sha256:18cd90ff8429bc65888d7aeb0f9763a237667be652d286be7878ed0dc83216dd", size = 2798191, upload-time = "2026-06-10T21:09:39.559Z" },
+]
+
+[[package]]
+name = "nvidia-cufft"
+version = "12.0.0.61"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/2f/7b57e29836ea8714f81e9898409196f47d772d5ddedddf1592eadb8ab743/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6c44f692dce8fd5ffd3e3df134b6cdb9c2f72d99cf40b62c32dde45eea9ddad3", size = 214085489, upload-time = "2025-09-04T08:31:56.044Z" },
+]
+
+[[package]]
+name = "nvidia-cufile"
+version = "1.15.1.6"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/70/4f193de89a48b71714e74602ee14d04e4019ad36a5a9f20c425776e72cd6/nvidia_cufile-1.15.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08a3ecefae5a01c7f5117351c64f17c7c62efa5fffdbe24fc7d298da19cd0b44", size = 1223672, upload-time = "2025-09-04T08:32:22.779Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/73/cc4a14c9813a8a0d509417cf5f4bdaba76e924d58beb9864f5a7baceefbf/nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:bdc0deedc61f548bddf7733bdc216456c2fdb101d020e1ab4b88d232d5e2f6d1", size = 1136992, upload-time = "2025-09-04T08:32:14.119Z" },
+]
+
+[[package]]
+name = "nvidia-curand"
+version = "10.4.0.35"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/72/7c2ae24fb6b63a32e6ae5d241cc65263ea18d08802aaae087d9f013335a2/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:133df5a7509c3e292aaa2b477afd0194f06ce4ea24d714d616ff36439cee349a", size = 61962106, upload-time = "2025-08-04T10:21:41.128Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/9f/be0a41ca4a4917abf5cb9ae0daff1a6060cc5de950aec0396de9f3b52bc5/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:1aee33a5da6e1db083fe2b90082def8915f30f3248d5896bcec36a579d941bfc", size = 59544258, upload-time = "2025-08-04T10:22:03.992Z" },
+]
+
+[[package]]
+name = "nvidia-cusolver"
+version = "12.0.4.66"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cublas", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cusparse", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/67/cba3777620cdacb99102da4042883709c41c709f4b6323c10781a9c3aa34/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0a759da5dea5c0ea10fd307de75cdeb59e7ea4fcb8add0924859b944babf1112", size = 200941980, upload-time = "2025-09-04T08:33:22.767Z" },
+]
+
+[[package]]
+name = "nvidia-cusparse"
+version = "12.6.3.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/18/623c77619c31d62efd55302939756966f3ecc8d724a14dab2b75f1508850/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2b3c89c88d01ee0e477cb7f82ef60a11a4bcd57b6b87c33f789350b59759360b", size = 145942937, upload-time = "2025-09-04T08:33:58.029Z" },
+]
+
+[[package]]
+name = "nvidia-cusparselt-cu13"
+version = "0.8.1"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/46/e1/cdc1797eadf82d3a9a575a19b33fdc871a97edbec42c00b5b5e914f4aff4/nvidia_cusparselt_cu13-0.8.1-py3-none-manylinux2014_aarch64.whl", hash = "sha256:4dca476c50bf4780d46cd0bfbd82e2bc10a08e4fef7950917ce8d7578d22a23f", size = 221051344, upload-time = "2025-09-05T18:49:51.289Z" },
+ { url = "https://files.pythonhosted.org/packages/34/7d/2661f2fb3ac4302f3a246f5fc030213ac60c1fe0bce84f9783dbd831dbb7/nvidia_cusparselt_cu13-0.8.1-py3-none-manylinux2014_x86_64.whl", hash = "sha256:786ce87568c303fadb5afcc7102d454cd3040d75f6f8626f5db460d1871f4dd0", size = 170148586, upload-time = "2025-09-05T18:50:50.248Z" },
+]
+
+[[package]]
+name = "nvidia-cutlass-dsl"
+version = "4.6.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cutlass-dsl-libs-base" },
+ { name = "nvidia-cutlass-dsl-libs-cu12" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b2/ec/14e6ecbfed31ec35bbd1bb6965ae61f879370906a8f9c2e851e292704ba4/nvidia_cutlass_dsl-4.6.2-py3-none-any.whl", hash = "sha256:06ac62deb182a852dfb053032cf945bf02b9aa1bf502a18b129d280d7babb26c", size = 10460, upload-time = "2026-08-05T14:39:41.657Z" },
+]
+
+[package.optional-dependencies]
+cu13 = [
+ { name = "nvidia-cutlass-dsl-libs-cu13" },
+]
+
+[[package]]
+name = "nvidia-cutlass-dsl-libs-base"
+version = "4.6.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-python", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-python", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "numpy" },
+ { name = "nvidia-cuda-nvdisasm" },
+ { name = "nvidia-cutlass-dsl-libs-core" },
+ { name = "protobuf" },
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/e9/a402e079e926f1fbcf82e30dfcd87aa924aaf91b02db08d57e83b158dbb2/nvidia_cutlass_dsl_libs_base-4.6.2-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:334a9be3c5054286666b14f81c9278075c2007b947ba75f7ee44b94aadf26415", size = 3321794, upload-time = "2026-08-05T14:13:12.501Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/4c/0a6c6e785e8fb2bfc6a0941559d30fbaf6264ae77ee1a26a42e37fd6efdc/nvidia_cutlass_dsl_libs_base-4.6.2-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:896f8a87d0815e59066b0607930c9905a5cf8f5c7a01a423b0093bc876bac4b4", size = 2825075, upload-time = "2026-08-05T14:13:34.746Z" },
+]
+
+[[package]]
+name = "nvidia-cutlass-dsl-libs-core"
+version = "4.6.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-python", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-python", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "numpy" },
+ { name = "nvidia-cuda-nvdisasm" },
+ { name = "protobuf" },
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7c/ef/10c0089234a32f1379e1f213f28f8c7521a9ff0154e3acc58f613d5ef3be/nvidia_cutlass_dsl_libs_core-4.6.2-py3-none-any.whl", hash = "sha256:571d4b46fca1bfbb123d0dc348eaa7fd80af0014ddffc07b849e8195b2364333", size = 772393, upload-time = "2026-08-05T14:09:50.76Z" },
+]
+
+[[package]]
+name = "nvidia-cutlass-dsl-libs-cu12"
+version = "4.6.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-python", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-python", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "numpy" },
+ { name = "nvidia-cuda-nvdisasm" },
+ { name = "nvidia-cutlass-dsl-libs-base" },
+ { name = "protobuf" },
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e1/54/6006a2b4fcd05f32be451fe8881968dcbed526438b4eae9a527534062c79/nvidia_cutlass_dsl_libs_cu12-4.6.2-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:2f3b163060325777a3847954b3a599556b2bff500eca19806a81d1d635bb4149", size = 87012255, upload-time = "2026-08-05T14:21:44.507Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/7a/a08ac0ecdd88bb129a56d95b7c58ea6826e6f5fa00191ef7ecdff8085836/nvidia_cutlass_dsl_libs_cu12-4.6.2-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:84243743382948c83ca627c2dcb106db3844c31d45b8cc2af3d14bc9df0a535e", size = 88453568, upload-time = "2026-08-05T14:22:49.158Z" },
+]
+
+[[package]]
+name = "nvidia-cutlass-dsl-libs-cu13"
+version = "4.6.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-python", version = "12.9.7", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin' or sys_platform == 'emscripten' or sys_platform == 'win32'" },
+ { name = "cuda-python", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "numpy" },
+ { name = "nvidia-cuda-nvdisasm" },
+ { name = "nvidia-cutlass-dsl-libs-base" },
+ { name = "protobuf" },
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/33/35/a38b52a8f99d1549c31680deae6a44328f3840e76a1202d5ab0af5b91f58/nvidia_cutlass_dsl_libs_cu13-4.6.2-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:5d215db842c47862232537a6737b0cbf4992318aed24ddb1d4f18c3a43022682", size = 86711155, upload-time = "2026-08-05T14:29:18.118Z" },
+ { url = "https://files.pythonhosted.org/packages/de/b7/75d9470796d4c8fb8e10165825a9cd1e95316bab45af1a6a3dd9555cfe70/nvidia_cutlass_dsl_libs_cu13-4.6.2-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:ad108ba4a6d35c0549fa3860ccbb5a90f005f240d2750ac9f8c3bdaa1f1864cc", size = 88036124, upload-time = "2026-08-05T14:31:27.349Z" },
+]
+
+[[package]]
+name = "nvidia-ml-py"
+version = "13.610.43"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f0/b5/a8fbc356f768fa5c9cfd646668fd7d34bf55bdd1c6e20754642a64d930d4/nvidia_ml_py-13.610.43.tar.gz", hash = "sha256:65437eb73d68d0c62c931ca4d45038472faff03bd0b8729abba4b899f70d60f2", size = 52109, upload-time = "2026-06-01T18:54:08.829Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/23/45/caa600acfab94560807a20a64b5830d2cd3c3202b7f1328644d70b7d6bd8/nvidia_ml_py-13.610.43-py3-none-any.whl", hash = "sha256:f13c72698edef492f985cc225f14faafe68ae065a2e407f45bdf6f4b9b43fde8", size = 53163, upload-time = "2026-06-01T18:54:07.704Z" },
+]
+
+[[package]]
+name = "nvidia-nccl-cu13"
+version = "2.29.7"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/72/0d/daf50d44177ee0cbc7ff0a0c91eb5ff676c82be42f9a970bc7597f440c3a/nvidia_nccl_cu13-2.29.7-py3-none-manylinux_2_18_aarch64.whl", hash = "sha256:674a12383e3c38a1bcccae7d4f3633b37852230b6047883cb2f4c2d1b36d9bf5", size = 206014712, upload-time = "2026-03-03T05:34:20.843Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f4/58e4e91b6919367c7aafb8e36fce9aad1a3047e536bf7e2fd560927d3a4c/nvidia_nccl_cu13-2.29.7-py3-none-manylinux_2_18_x86_64.whl", hash = "sha256:edd81538446786ec3b73972543e53bb43bcaf0bfc8ef76cb679fcc390ffe136d", size = 205976000, upload-time = "2026-03-03T05:36:24.472Z" },
+]
+
+[[package]]
+name = "nvidia-nvjitlink"
+version = "13.3.33"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f0/ee/580ca6f29dcab0221db8706badca1bbbb084f1975c4d4e83329c3a7e31f0/nvidia_nvjitlink-13.3.33-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:26a6de7fb4c8fdaa7703d3dad720d6d427ddfea5c48a528fd97c11733ad830e5", size = 40742423, upload-time = "2026-05-26T16:54:51.613Z" },
+ { url = "https://files.pythonhosted.org/packages/69/30/45414e35ff2eee7db3da037e5707037ccf9d2b5218ffbdb055ea4d5aa98a/nvidia_nvjitlink-13.3.33-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ce48b37dfeb3cb1eae4cf85adacb47d7a6539ea2272870c9a3628ce275c2037e", size = 39168635, upload-time = "2026-05-26T16:54:13.906Z" },
+]
+
+[[package]]
+name = "nvidia-nvshmem-cu13"
+version = "3.4.5"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dc/0f/05cc9c720236dcd2db9c1ab97fff629e96821be2e63103569da0c9b72f19/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dc2a197f38e5d0376ad52cd1a2a3617d3cdc150fd5966f4aee9bcebb1d68fe9", size = 60215947, upload-time = "2025-09-06T00:32:20.022Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/35/a9bf80a609e74e3b000fef598933235c908fcefcef9026042b8e6dfde2a9/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:290f0a2ee94c9f3687a02502f3b9299a9f9fe826e6d0287ee18482e78d495b80", size = 60412546, upload-time = "2025-09-06T00:32:41.564Z" },
+]
+
+[[package]]
+name = "nvidia-nvtx"
+version = "13.0.85"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c2/f3/d86c845465a2723ad7e1e5c36dcd75ddb82898b3f53be47ebd429fb2fa5d/nvidia_nvtx-13.0.85-py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4936d1d6780fbe68db454f5e72a42ff64d1fd6397df9f363ae786930fd5c1cd4", size = 148047, upload-time = "2025-09-04T08:29:01.761Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
+]
+
+[[package]]
+name = "nvidia-nvvm"
+version = "13.2.78"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e8/1f/930d63ccc8adcdf27bfc051a24e3e4da2cf6ef987848d6d1d642e29d704b/nvidia_nvvm-13.2.78-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:f5aa433631109bbdec81802c5b5f319bf10bc891fe2f212e4e445845211d6f77", size = 64279462, upload-time = "2026-04-13T10:02:25.719Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/fd/db44b7a662a6af75a9a0683ca4580c855a3f5fcfdf1261b0ddb9fce0ee26/nvidia_nvvm-13.2.78-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:88075f87a361a1dce95c799cabc028f7093af616a5702dcfb74eba4045dbbd5f", size = 61886055, upload-time = "2026-04-13T10:02:00.345Z" },
+ { url = "https://files.pythonhosted.org/packages/35/b9/c3862fd1073326c61233f05e816c17a28ab86a361db1b7561c7f33ac3af4/nvidia_nvvm-13.2.78-py3-none-win_amd64.whl", hash = "sha256:cf8e91654e74285e9c574b3a45b92928c0a6d135928906cf11ce470bbec6a8ec", size = 56752219, upload-time = "2026-04-13T10:15:11.102Z" },
+]
+
+[[package]]
+name = "nvtx"
+version = "0.2.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/92/dd/692765e87de30bae1522cdffaa0f2b52949658a92a0fa6d96b1a01eae9d2/nvtx-0.2.15.tar.gz", hash = "sha256:2287d3be05b85661deb386f878d1f536c2e532774aa9ec7a50c434942ed81ae5", size = 121230, upload-time = "2026-03-18T10:01:25.547Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/c9/8341224b8284f7deb6a634119939de5885adc421e64b6743693b30da2186/nvtx-0.2.15-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d28660d9c46f8ba750d781572b6aa5a1e6221abba224ab32d7fb32c2d0fd67df", size = 780787, upload-time = "2026-03-18T10:10:40.634Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/c0/4a5bb7897918de7c7e0191d9342df8ae4cb797ff07276e0f20d13e497ce7/nvtx-0.2.15-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10749686633f880ad53dcdbb2179fad41b45dcf5b7631d4a1070a577577bd386", size = 782575, upload-time = "2026-03-18T10:13:57.3Z" },
+ { url = "https://files.pythonhosted.org/packages/38/b9/6b381ac7c5a3ded331aebbf25f8959d19b51d320fb2514c76c6b6edddaaa/nvtx-0.2.15-cp313-cp313-win_amd64.whl", hash = "sha256:a6650b029263d12f8427a4dee8bd59cb9c91bccb60543bfcb20bc2b00fdcd672", size = 128764, upload-time = "2026-03-18T10:02:33.343Z" },
+ { url = "https://files.pythonhosted.org/packages/75/69/a9acb6d95d2e0e381b2956544768528dd8d7a9e827af8c2014169d838284/nvtx-0.2.15-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:25813ead4fff4d3a6e04f69a72507b096a6bdbecefa369f1100b0e584767bca8", size = 833375, upload-time = "2026-03-18T10:06:31.955Z" },
+ { url = "https://files.pythonhosted.org/packages/38/56/c7e8645061cc2fc23f3a54f33e1e340df59216f07dcfb97d46b8ae7dd26c/nvtx-0.2.15-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3741edac4678b92f03d22a3f0a2dfd469f422f85e63db71b038e02525b2404ad", size = 788639, upload-time = "2026-03-18T10:12:01.69Z" },
+ { url = "https://files.pythonhosted.org/packages/96/03/fadd82acdbca6d1c49ac517081a0c3714346f52f4c7e1d4449d77605b4aa/nvtx-0.2.15-cp313-cp313t-win_amd64.whl", hash = "sha256:8be06c3c8c267eba56a0396366b9593092e0b75ea8d3702b303d48c0a1662f0e", size = 142609, upload-time = "2026-03-18T10:01:48.832Z" },
+]
+
+[[package]]
+name = "openai"
+version = "2.44.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "distro" },
+ { name = "httpx" },
+ { name = "jiter" },
+ { name = "pydantic" },
+ { name = "sniffio" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/49/f5/7c7cb955305cb41f7f3c5fd7e0e38bf6bbf2658468863d4b7b868a5cb8df/openai-2.44.0.tar.gz", hash = "sha256:68a5a5ffad82b8ff7d451c437529fb64f7c3b8123aaf0c021966a882d9e3947d", size = 988753, upload-time = "2026-06-24T20:56:02.293Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ae/f4/561ed79fd94876160018a5e75254cfcb9b0e62d4dded9dcb20072e86d623/openai-2.44.0-py3-none-any.whl", hash = "sha256:0a2a3ab2e29aeda368700f662ff9ba0f9df17ba4c54577a64e08b8115a3cc0ad", size = 1366216, upload-time = "2026-06-24T20:55:58.882Z" },
+]
+
+[[package]]
+name = "openai-harmony"
+version = "0.0.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pydantic" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3e/92/2d038d096f29179c7c9571b431f9e739f87a487121901725e23fe338dd9d/openai_harmony-0.0.8.tar.gz", hash = "sha256:6e43f98e6c242fa2de6f8ea12eab24af63fa2ed3e89c06341fb9d92632c5cbdf", size = 284777, upload-time = "2025-11-05T19:07:06.727Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/45/c6/2502f416d46be3ec08bb66d696cccffb57781a499e3ff2e4d7c174af4e8f/openai_harmony-0.0.8-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:029ec25ca74abe48fdb58eb9fdd2a8c1618581fc33ce8e5653f8a1ffbfbd9326", size = 2627806, upload-time = "2025-11-05T19:06:57.063Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/d2/ce6953ca87db9cae3e775024184da7d1c5cb88cead19a2d75b42f00a959c/openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e4f709815924ec325b9a890e6ab2bbb0ceec8e319a4e257328eb752cf36b2efc", size = 2948463, upload-time = "2025-11-05T19:06:48.17Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/4c/b553c9651662d6ce102ca7f3629d268b23df1abe5841e24bed81e8a8e949/openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5cfcfd963b50a41fc656c84d3440ca6eecdccd6c552158ce790b8f2e33dfb5a9", size = 2704083, upload-time = "2025-11-05T19:06:50.205Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/af/4eec8f9ab9c27bcdb444460c72cf43011d176fc44c79d6e113094ca1e152/openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0a3a16972aa1cee38ea958470cd04ac9a2d5ac38fdcf77ab686611246220c158", size = 2959765, upload-time = "2025-11-05T19:06:53.62Z" },
+ { url = "https://files.pythonhosted.org/packages/11/3c/33f3374e4624e0e776f6b13b73c45a7ead7f9c4529f8369ed5bfcaa30cac/openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b4d5cfa168e74d08f8ba6d58a7e49bc7daef4d58951ec69b66b0d56f4927a68d", size = 3427031, upload-time = "2025-11-05T19:06:51.829Z" },
+ { url = "https://files.pythonhosted.org/packages/25/3f/1a192b93bb47c6b44cd98ba8cc1d3d2a9308f1bb700c3017e6352da11bda/openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c007d277218a50db8839e599ed78e0fffe5130f614c3f6d93ae257f282071a29", size = 2953260, upload-time = "2025-11-05T19:06:55.406Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/f8/93b582cad3531797c3db7c2db5400fd841538ccddfd9f5e3df61be99a630/openai_harmony-0.0.8-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:8565d4f5a0638da1bffde29832ed63c9e695c558611053add3b2dc0b56c92dbc", size = 3127044, upload-time = "2025-11-05T19:06:59.553Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/10/4327dbf87f75ae813405fd9a9b4a5cde63d506ffed0a096a440a4cabd89c/openai_harmony-0.0.8-cp38-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:cbaa3bda75ef0d8836e1f8cc84af62f971b1d756d740efc95c38c3e04c0bfde2", size = 2932931, upload-time = "2025-11-05T19:07:01.437Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/c8/1774eec4f6f360ef57618fb8f52e3d3af245b2491bd0297513aa09eec04b/openai_harmony-0.0.8-cp38-abi3-musllinux_1_2_i686.whl", hash = "sha256:772922a9bd24e133950fad71eb1550836f415a88e8c77870e12d0c3bd688ddc2", size = 2996140, upload-time = "2025-11-05T19:07:03.438Z" },
+ { url = "https://files.pythonhosted.org/packages/60/c3/3d1e01e2dba517a91760e4a03e4f20ffc75039a6fe584d0e6f9b5c78fd15/openai_harmony-0.0.8-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:007b0476a1f331f8130783f901f1da6f5a7057af1a4891f1b6a31dec364189b5", size = 3205080, upload-time = "2025-11-05T19:07:05.078Z" },
+ { url = "https://files.pythonhosted.org/packages/14/63/119de431572d7c70a7bf1037034a9be6ed0a7502a7498ba7302bca5b3242/openai_harmony-0.0.8-cp38-abi3-win32.whl", hash = "sha256:a9b5f893326b28d9e935ade14b4f655f5a840942473bc89b201c25f7a15af9cf", size = 2082457, upload-time = "2025-11-05T19:07:09.631Z" },
+ { url = "https://files.pythonhosted.org/packages/40/1f/c83cf5a206c263ee70448a5ae4264682555f4d0b5bed0d2cc6ca1108103d/openai_harmony-0.0.8-cp38-abi3-win_amd64.whl", hash = "sha256:39d44f0d8f466bd56698e7ead708bead3141e27b9b87e3ab7d5a6d0e4a869ee5", size = 2438369, upload-time = "2025-11-05T19:07:08.1Z" },
+]
+
+[[package]]
+name = "opencv-python-headless"
+version = "5.0.0.93"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1d/99/76b7c80252aa83c1af16393454aafd125a0287101afe8deb0a6821af0e30/opencv_python_headless-5.0.0.93.tar.gz", hash = "sha256:b82f9831daab90b725c7c1ee1b36cb5732c367096ac76d119e64e14eb70d5f3c", size = 81817738, upload-time = "2026-07-02T07:01:06.039Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/53/7c/8c8097891c509d98cd128493835c95631c80be6a8f37ed9d25716c2e16f1/opencv_python_headless-5.0.0.93-cp37-abi3-macosx_13_0_arm64.whl", hash = "sha256:030ca5e0837a2963ab36ef896baa9767eb8d2b83353fb28af5a521e40dd8756f", size = 48322581, upload-time = "2026-07-02T05:50:34.207Z" },
+ { url = "https://files.pythonhosted.org/packages/90/8c/eab2ad388c3cbab2a350c10c2ef19ce6bd099240afc31789032c996bab52/opencv_python_headless-5.0.0.93-cp37-abi3-macosx_14_0_x86_64.whl", hash = "sha256:1e55af3abfb462eeeabe5c775f12bdb36216d8a93a3583d69e6bd6e1d6ba7d00", size = 34782894, upload-time = "2026-07-02T05:51:39.856Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/78/afca939f40ffe2b2380bfa86f812b2f7d4acc5a27b27dc41b49cad7ce7b4/opencv_python_headless-5.0.0.93-cp37-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:10818d91510e05c04568ae12b5cd120779c70c01bf897b001a6221fe430df80f", size = 36521085, upload-time = "2026-07-02T06:55:24.429Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/97/8170e9819764c47e436c130d3ff6cfb73b58f923eae9d3a03d8982b04aec/opencv_python_headless-5.0.0.93-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:09a872a157c1376ab922a69bbf22f9a95bcc7b658a9d8b436a60212b02b2eeb4", size = 56563598, upload-time = "2026-07-02T06:55:47.355Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/98/1a28a7101e31801042b3098871a74b76c61581d328ef40774ff4edb53a56/opencv_python_headless-5.0.0.93-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:840bd717c21e5c11cadadc022a823315ea417f961213d06b4df010e019eb16f4", size = 39648433, upload-time = "2026-07-02T06:56:04.255Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/21/f6ef335f6e65724aa78b8d792b48d40a48c381715f1e62f5a5049e09d07e/opencv_python_headless-5.0.0.93-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:ed709fdf9aa0bd1f2ed8549e71d19449b03a675bb581eb292285f6861953be37", size = 61204038, upload-time = "2026-07-02T06:56:41.823Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/8f/b8756467ea991449a293797f6b3fa80fcfdd29598a0a60d1cd5715b96e61/opencv_python_headless-5.0.0.93-cp37-abi3-win32.whl", hash = "sha256:c6bcd96b185975ea240d22cfdb15a1f6d080cc95264cfbe2621f21bb144d89b9", size = 35411237, upload-time = "2026-07-02T05:50:12.901Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/88/763b967f7efd7226b82c9fae16d560cba049b1f0c036647e65c610fd636e/opencv_python_headless-5.0.0.93-cp37-abi3-win_amd64.whl", hash = "sha256:829717b6a95554f273e49e357cee3b3a2a26b6f4842fbc1bed2b45bdd8f87e0e", size = 43825962, upload-time = "2026-07-02T05:50:09.627Z" },
+]
+
+[[package]]
+name = "opentelemetry-api"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ae/cc/e4c9584181f86494df0f6bdec1a4f3280c50db44704dc2a407e994fc87bb/opentelemetry_api-1.43.0.tar.gz", hash = "sha256:107d0d03857ea8fc7c5fcbbbd83f800c281f0d560553d61c1d675fccfd1761c1", size = 73476, upload-time = "2026-06-24T15:19:55.323Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/17/83/6dba32b85f31868400440dc7ad2ca1eab94cbbf3a7b0459ed39f8311a9e2/opentelemetry_api-1.43.0-py3-none-any.whl", hash = "sha256:20acf45e9b21851926835292e4045d290acade1edd2ff3de86d2f069687ba1fd", size = 61912, upload-time = "2026-06-24T15:19:35.434Z" },
+]
+
+[[package]]
+name = "opentelemetry-exporter-otlp"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "opentelemetry-exporter-otlp-proto-grpc" },
+ { name = "opentelemetry-exporter-otlp-proto-http" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ee/47/b77366bcbe719373a8cac2b4e8ad01f8ff3c9f2c223374d77ece280aae6f/opentelemetry_exporter_otlp-1.43.0.tar.gz", hash = "sha256:65aded6c50ee7dd2b9948c9d0e59ddb4ed4eea6e8532fba95cbe6a4a64a566ba", size = 6086, upload-time = "2026-06-24T15:19:58.003Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d9/26/d28cc854d2eb779f91351216c51ed0d54886f89f2dd56db9d493ba9fd429/opentelemetry_exporter_otlp-1.43.0-py3-none-any.whl", hash = "sha256:70f3fe740a64596d4157588a2ee7e4fd37d2acc0c0f522a2882b8c29316cd0f0", size = 6726, upload-time = "2026-06-24T15:19:38.437Z" },
+]
+
+[[package]]
+name = "opentelemetry-exporter-otlp-proto-common"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "opentelemetry-proto" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/55/c1/e8098490ab15abf116dcaf9fa89ededcb35547c7d08d4b5a62f573dc1e63/opentelemetry_exporter_otlp_proto_common-1.43.0.tar.gz", hash = "sha256:c4e32ba6d6b13bdb2b8f6764c4fd28d00192826561aa04f6d14eedfce7ac076f", size = 20197, upload-time = "2026-06-24T15:20:00.247Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d0/b2/41ebc74ae1d5859901f1b69305de58724bf043381103d6ef413521cbc35a/opentelemetry_exporter_otlp_proto_common-1.43.0-py3-none-any.whl", hash = "sha256:123c3f9cc87218562490c63b36f497bf3a722faf174a515d1443f31ababa6264", size = 17048, upload-time = "2026-06-24T15:19:41.264Z" },
+]
+
+[[package]]
+name = "opentelemetry-exporter-otlp-proto-grpc"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "googleapis-common-protos" },
+ { name = "grpcio" },
+ { name = "opentelemetry-api" },
+ { name = "opentelemetry-exporter-otlp-proto-common" },
+ { name = "opentelemetry-proto" },
+ { name = "opentelemetry-sdk" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e2/1d/6336453716ca0a240d4417d19e6d5b77a5e7163e5670ec4f7ec4d3ede7bf/opentelemetry_exporter_otlp_proto_grpc-1.43.0.tar.gz", hash = "sha256:1b3e0627daa9bc21884d4a13946807c255eb558bfe5bdd543dffb6f4c9faee0d", size = 27213, upload-time = "2026-06-24T15:20:00.907Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6d/74/2700b5d5c946bf2dba87073fce3dfc198c46bc92ea3d5693f54bc51c90b1/opentelemetry_exporter_otlp_proto_grpc-1.43.0-py3-none-any.whl", hash = "sha256:6a10d1feacffffda19acacbf277b736094b1e2f4dbb98c90ccb2c6e1962e2ec6", size = 19626, upload-time = "2026-06-24T15:19:42.233Z" },
+]
+
+[[package]]
+name = "opentelemetry-exporter-otlp-proto-http"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "googleapis-common-protos" },
+ { name = "opentelemetry-api" },
+ { name = "opentelemetry-exporter-otlp-proto-common" },
+ { name = "opentelemetry-proto" },
+ { name = "opentelemetry-sdk" },
+ { name = "requests" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fc/92/0b9f56412483a8891d4843890294796c9df8ab42417bd9bad8035d840cb3/opentelemetry_exporter_otlp_proto_http-1.43.0.tar.gz", hash = "sha256:fa8a42bb7d00ee5391f4c0b04d8e6a46c03caa437903296ab73a81dc11ba118f", size = 25406, upload-time = "2026-06-24T15:20:01.515Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/20/b685ed7af2e17c29ffc8af56f1fa8bc2033258fc30fb0d2b722f49d13ba0/opentelemetry_exporter_otlp_proto_http-1.43.0-py3-none-any.whl", hash = "sha256:647f603aa8efdbdb4dbff842e0729d0406a6fff26b295a72d3d60e7d963b2610", size = 21795, upload-time = "2026-06-24T15:19:43.164Z" },
+]
+
+[[package]]
+name = "opentelemetry-proto"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "protobuf" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e0/b9/d357faefb40bda1d4799913e6af611171ff22a2dedcb93576bc92242d056/opentelemetry_proto-1.43.0.tar.gz", hash = "sha256:224778df17e1f3fafeaaa21d874236ca5f6ffc2f86e0899298ec7351aac27924", size = 46481, upload-time = "2026-06-24T15:20:07.625Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ed/a7/3e5308cf548b8f72529c7db1afdb3a404211982376a12927fd7759f77bf3/opentelemetry_proto-1.43.0-py3-none-any.whl", hash = "sha256:c58f1f7ef84bc7dc2834016c0c37fe0081dde7ca9f6339be1970fbf9cdaaa90d", size = 72489, upload-time = "2026-06-24T15:19:51.164Z" },
+]
+
+[[package]]
+name = "opentelemetry-sdk"
+version = "1.43.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "opentelemetry-api" },
+ { name = "opentelemetry-semantic-conventions" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3e/eb/5041074274ac0956b03637cc039d434569112468e875eddfcc9a0674ce06/opentelemetry_sdk-1.43.0.tar.gz", hash = "sha256:d8187c81c162df9913e4003dd6485f7390d9a24fc17026ec7387b8b8218b08e9", size = 254744, upload-time = "2026-06-24T15:20:08.467Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/49/e3/b17be23af124201c9f52eececd4cc8ddfed1597d37b4ee771895d325805c/opentelemetry_sdk-1.43.0-py3-none-any.whl", hash = "sha256:d1323a547c1ce69d6a069a17a44b7da82bb8b332051ecb074041f87642c86823", size = 178852, upload-time = "2026-06-24T15:19:52.169Z" },
+]
+
+[[package]]
+name = "opentelemetry-semantic-conventions"
+version = "0.64b0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "opentelemetry-api" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5a/30/5f26df29509eccd86b99b481ac9ffa39da49ba9577cc69071c552ae30447/opentelemetry_semantic_conventions-0.64b0.tar.gz", hash = "sha256:72f76fb2d1582d9d033dd1fcd84532e961e6ff3d90d24ba6fabc72975a83864c", size = 148340, upload-time = "2026-06-24T15:20:09.267Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f2/ca/23ba87a221b574a7c5a99d48849d80bfe8b047624681357e2b002e566187/opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl", hash = "sha256:ea77e85e354b8f604ddbe5f3d9135216f982fa4d77e5859ac30f6d8a50505aa6", size = 203713, upload-time = "2026-06-24T15:19:53.339Z" },
+]
+
+[[package]]
+name = "opentelemetry-semantic-conventions-ai"
+version = "0.5.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "opentelemetry-sdk" },
+ { name = "opentelemetry-semantic-conventions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/24/02/10aeacc37a38a3a8fa16ff67bec1ae3bf882539f6f9efb0f70acf802ca2d/opentelemetry_semantic_conventions_ai-0.5.1.tar.gz", hash = "sha256:153906200d8c1d2f8e09bd78dbef526916023de85ac3dab35912bfafb69ff04c", size = 26533, upload-time = "2026-03-26T14:20:38.73Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/55/22/41fb05f1dc5fda2c468e05a41814c20859016c85117b66c8a257cae814f6/opentelemetry_semantic_conventions_ai-0.5.1-py3-none-any.whl", hash = "sha256:25aeb22bd261543b4898a73824026d96770e5351209c7d07a0b1314762b1f6e4", size = 11250, upload-time = "2026-03-26T14:20:37.108Z" },
+]
+
+[[package]]
+name = "outlines-core"
+version = "0.2.14"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6a/04/4a0812eb27c086cfd2e66e7ec9150f33e105912a9b7f8b335e3479f03a06/outlines_core-0.2.14.tar.gz", hash = "sha256:64808deed1591ca3029ff64346ceb974cd5d780c916ea82504951fe83523039e", size = 191539, upload-time = "2026-01-09T15:59:10.016Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/13/9d/e6c81c975c123f0639d5f6909c987e510d43e07c2e1e6495b21639c4dec6/outlines_core-0.2.14-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:8b3e8d668188282a1f7666732bb8a01958ab134db35bb792e7442a40e55ff1e7", size = 2049297, upload-time = "2026-01-09T15:58:39.184Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/d1/5ce55ef724aed0915edc877b6dd610d39b3169e4341154bb53daa022065a/outlines_core-0.2.14-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:66e695b375b180725fb534d9adf298531c152ec3d881e3b9e01c82b5dd269f52", size = 2200944, upload-time = "2026-01-09T15:58:40.257Z" },
+ { url = "https://files.pythonhosted.org/packages/32/e3/60ad781251eedcf1496317ecd58eb2e4488717ba63b10494ab49dfd05e5d/outlines_core-0.2.14-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:6bd166d3b07acef2f60d4ede44592a26d3f7d8712876bfc8e22150045def5857", size = 2049607, upload-time = "2026-01-09T15:58:41.635Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/2d/662d6a76face5b4b3481f888900d00856c37aa2927341a023866457da212/outlines_core-0.2.14-cp313-cp313-macosx_15_0_x86_64.whl", hash = "sha256:9d45462d7548aa0e17176a691ae73447f3e6bed9658a0cd96fe72eadf7474475", size = 2197755, upload-time = "2026-01-09T15:58:42.861Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/9a/4b62903de006d991b58674ff033c1b6fb92be5767360376fc961f6771bdb/outlines_core-0.2.14-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6453e23f01d98ec48e3a4141d7112792ce77001dfb28d91d6fd89f47009f91ef", size = 2341051, upload-time = "2026-01-09T15:58:44.415Z" },
+ { url = "https://files.pythonhosted.org/packages/50/36/1532f7d9ab16c676812d94528e89964aa0d15f12adcb285e6ed86f86f2fe/outlines_core-0.2.14-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:7deef6df74cb247f2a3a62f03438ba967456504b0555ec7029f8db834e054448", size = 2236778, upload-time = "2026-01-09T15:58:45.437Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/5a/dfd94f15f4c04e691e7fdf30cf8b9b22bf2cbc426b3ef270af3e200596d5/outlines_core-0.2.14-cp313-cp313-win32.whl", hash = "sha256:bb008c7ecc034bcfda0ddc10a4d1f2181a4b61ec1643ee56183dd6fa64139c9d", size = 1842727, upload-time = "2026-01-09T15:58:46.723Z" },
+ { url = "https://files.pythonhosted.org/packages/34/35/e24ab5d2116812464380587435297d8ece2f0218c2ba8afc9f541e3a6911/outlines_core-0.2.14-cp313-cp313-win_amd64.whl", hash = "sha256:eb27e92204b296a063ac58f361153be4e78c8103a96e0b1c085b22d4fc3534cf", size = 2137108, upload-time = "2026-01-09T15:58:47.784Z" },
+]
+
+[[package]]
+name = "packaging"
+version = "26.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" },
+]
+
+[[package]]
+name = "pandas"
+version = "3.0.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "python-dateutil" },
+ { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f8/87/4341c6252d1c47b08768c3d25ac487362bf403f0313ddae4a2a26c9b1b4c/pandas-3.0.3.tar.gz", hash = "sha256:696a4a00a2a2a35d4e5deb3fc946641b96c944f02230e4f76137fe35d806c4fc", size = 4651414, upload-time = "2026-05-11T18:54:29.21Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c5/90/62d8302883c44308c477e222c3daf7c813a34c8e96985882fbd53d964352/pandas-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:67b3b64c11910cfa29f4e94a14d3bff9ee693b6fc76055e7cad549cee0aec5fa", size = 10331071, upload-time = "2026-05-11T18:52:58.838Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/ae/6a6493c783a101f165e4356953ba3c74d6f77f0042fa7d753da9dfbb640c/pandas-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:39436b377d56d2a2e52d0395bdbee171f01068e99af5250509aceeb929f765c7", size = 9875690, upload-time = "2026-05-11T18:53:01.431Z" },
+ { url = "https://files.pythonhosted.org/packages/62/7c/5df8e9f56c69a2769fbe9382a5ef8f2658c007e376434e1e2cbb57ad895f/pandas-3.0.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d4be06d68f9ddcfc645b87534911da79a8fbffc7573c80e0edcf42a5020624d8", size = 10381634, upload-time = "2026-05-11T18:53:04.393Z" },
+ { url = "https://files.pythonhosted.org/packages/99/68/1237369725aa617bb358263d535803e3053fdbc593513ec5ed9c9896b5b6/pandas-3.0.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a4eeb6830daf35a71cc09649bd823e2b542dac246cdee9614c6e4bd65028cd6a", size = 10891243, upload-time = "2026-05-11T18:53:07.643Z" },
+ { url = "https://files.pythonhosted.org/packages/25/93/77d108e8af7222b4a503ebde0e30215b1c2e4f8e53a526431890f22d5586/pandas-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1928e07221f82db493cd4af1e23c1bfca524a19a4699887975bff68f49a72bfb", size = 11388659, upload-time = "2026-05-11T18:53:10.634Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/bd/eff5b4399f332ac386c853f6cd2bd3fa2ca0061b9f36ecd9c4d7c4265649/pandas-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:51b1fe551acb77dac643c6fda86084d8d446c10fe64b06a9cc29c4cc8540e7f2", size = 11942880, upload-time = "2026-05-11T18:53:13.536Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/20/559ace4200982c3887d0b86bfd0d856a2143ef8ddab63cc07934951a964c/pandas-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:a82d532a3351d435432cd913edbccaf8b8e01d4dd0e5ced5a8d2e8ecd94c7e44", size = 9757091, upload-time = "2026-05-11T18:53:16.306Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/66/69055a09fe200f29f922a3eeec4804611900b95f52d932ece3393c3c0c19/pandas-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:275c14e0fce14a2ec20eee474aecd305478ea3c1e6f6a9d8fe219a165542717e", size = 9057282, upload-time = "2026-05-11T18:53:18.768Z" },
+ { url = "https://files.pythonhosted.org/packages/57/0e/efe801b0e6811e8e650cd21b7f2608e30f08a7067e2bf6e8752b0d56ee3c/pandas-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:46997386d528eb40376ecd6b033cf4a8a1e5282580f68f43de875b78cba2199d", size = 10767016, upload-time = "2026-05-11T18:53:21.227Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/dc/eb55135a1d5f0f0519f28da1f609a206d2cad1f9c35c32d51e38dd7261ae/pandas-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:261e308dfb22448384b7580cf719d2f998fe2966c92893c3e77d14008af1f066", size = 10420210, upload-time = "2026-05-11T18:53:23.982Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/3e/b1d5d955ce33ffecb407465a60bc32769d74fcf68224b7ae67ae11d4dea4/pandas-3.0.3-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dd1a5d1def6a46002e964510bdc67c368aa0951df5d1d9f8365336f5a1f490cd", size = 10336126, upload-time = "2026-05-11T18:53:26.731Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/76/a01261711ab60a22d71b862f0de20e4c504bf80457270ad8cb42110f6abc/pandas-3.0.3-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d72828c20c6d6e83e1e22a6a3b47b326b71664112fa9705dcbccfd7a39b62085", size = 10728051, upload-time = "2026-05-11T18:53:29.125Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/21/ea191195e587b18cf682e97f433f81b2d0fbe341380e80a3e0d6e4403c8e/pandas-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d26cbe1fcfc12e8fd900e2454163e466b2d3af84f7c75481df7683ffc073d870", size = 11350796, upload-time = "2026-05-11T18:53:32.056Z" },
+ { url = "https://files.pythonhosted.org/packages/64/69/f0eaaf54939f0e8c6768fd06be9af2cef9b36048b96dfb9e1b2c685a807e/pandas-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:3e91cec1879ada0624fc3dc9953c5cbd60208e59c0db28f540c5d6d47502422f", size = 11799741, upload-time = "2026-05-11T18:53:34.985Z" },
+ { url = "https://files.pythonhosted.org/packages/45/a4/865e0e510cae5fc2194de4db28be638952de942571ba9125934fd9c01d47/pandas-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:08d789b41f87e0905880e293cedf6197ce71fe67cc081358b1e148a491b9bd13", size = 10499958, upload-time = "2026-05-11T18:53:37.857Z" },
+]
+
+[[package]]
+name = "pandas-stubs"
+version = "3.0.3.260530"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3d/aa/c41a8a0ff86fd85dbb3ec0c1f3fa488ca64a8b5f82654ae1b07d84acefe5/pandas_stubs-3.0.3.260530.tar.gz", hash = "sha256:d1efe47b2e5a312c047d7feabec5cb7a55365747983420077e9fcbe9ab74f714", size = 113183, upload-time = "2026-05-30T17:47:40.34Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0b/e0/99ec5b02203c4e9ce878bc63d8caa06ac1f891e4d63bded9a5ced70fcb4f/pandas_stubs-3.0.3.260530-py3-none-any.whl", hash = "sha256:a6277eb1c8cebf48d9b2413fcd2e9a6b4ff479c934a223c29eacbc3058c4cb55", size = 173780, upload-time = "2026-05-30T17:47:39.13Z" },
+]
+
+[[package]]
+name = "pandocfilters"
+version = "1.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/70/6f/3dd4940bbe001c06a65f88e36bad298bc7a0de5036115639926b0c5c0458/pandocfilters-1.5.1.tar.gz", hash = "sha256:002b4a555ee4ebc03f8b66307e287fa492e4a77b4ea14d3f934328297bb4939e", size = 8454, upload-time = "2024-01-18T20:08:13.726Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ef/af/4fbc8cab944db5d21b7e2a5b8e9211a03a79852b1157e2c102fcc61ac440/pandocfilters-1.5.1-py2.py3-none-any.whl", hash = "sha256:93be382804a9cdb0a7267585f157e5d1731bbe5545a85b268d6f5fe6232de2bc", size = 8663, upload-time = "2024-01-18T20:08:11.28Z" },
+]
+
+[[package]]
+name = "parso"
+version = "0.8.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/30/4b/90c937815137d43ce71ba043cd3566221e9df6b9c805f24b5d138c9d40a7/parso-0.8.7.tar.gz", hash = "sha256:eaaac4c9fdd5e9e8852dc778d2d7405897ec510f2a298071453e5e3a07914bb1", size = 401824, upload-time = "2026-05-01T23:13:02.138Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/99/5d/8268b644392ee874ee82a635cd0df1773de230bde356c38de28e298392cc/parso-0.8.7-py2.py3-none-any.whl", hash = "sha256:a8926eb2a1b915486941fdbd31e86a4baf88fe8c210f25f2f35ecec5b574ca1c", size = 107025, upload-time = "2026-05-01T23:12:58.867Z" },
+]
+
+[[package]]
+name = "partial-json-parser"
+version = "0.2.1.1.post7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6a/6d/eed37d7ebc1e0bcd27b831c0cf1fe94881934316187c4b30d23f29ea0bd4/partial_json_parser-0.2.1.1.post7.tar.gz", hash = "sha256:86590e1ba6bcb6739a2dfc17d2323f028cb5884f4c6ce23db376999132c9a922", size = 10296, upload-time = "2025-11-17T07:27:41.202Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/42/32/658973117bf0fd82a24abbfb94fe73a5e86216e49342985e10acce54775a/partial_json_parser-0.2.1.1.post7-py3-none-any.whl", hash = "sha256:145119e5eabcf80cbb13844a6b50a85c68bf99d376f8ed771e2a3c3b03e653ae", size = 10877, upload-time = "2025-11-17T07:27:40.457Z" },
+]
+
+[[package]]
+name = "pathlib-abc"
+version = "0.5.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d6/cb/448649d7f25d228bf0be3a04590ab7afa77f15e056f8fa976ed05ec9a78f/pathlib_abc-0.5.2.tar.gz", hash = "sha256:fcd56f147234645e2c59c7ae22808b34c364bb231f685ddd9f96885aed78a94c", size = 33342, upload-time = "2025-10-10T18:37:20.524Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b1/29/c028a0731e202035f0e2e0bfbf1a3e46ad6c628cbb17f6f1cc9eea5d9ff1/pathlib_abc-0.5.2-py3-none-any.whl", hash = "sha256:4c9d94cf1b23af417ce7c0417b43333b06a106c01000b286c99de230d95eefbb", size = 19070, upload-time = "2025-10-10T18:37:19.437Z" },
+]
+
+[[package]]
+name = "pathspec"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" },
+]
+
+[[package]]
+name = "pathvalidate"
+version = "3.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fa/2a/52a8da6fe965dea6192eb716b357558e103aea0a1e9a8352ad575a8406ca/pathvalidate-3.3.1.tar.gz", hash = "sha256:b18c07212bfead624345bb8e1d6141cdcf15a39736994ea0b94035ad2b1ba177", size = 63262, upload-time = "2025-06-15T09:07:20.736Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9a/70/875f4a23bfc4731703a5835487d0d2fb999031bd415e7d17c0ae615c18b7/pathvalidate-3.3.1-py3-none-any.whl", hash = "sha256:5263baab691f8e1af96092fa5137ee17df5bdfbd6cff1fcac4d6ef4bc2e1735f", size = 24305, upload-time = "2025-06-15T09:07:19.117Z" },
+]
+
+[[package]]
+name = "pexpect"
+version = "4.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ptyprocess", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450, upload-time = "2023-11-25T09:07:26.339Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl", hash = "sha256:7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523", size = 63772, upload-time = "2023-11-25T06:56:14.81Z" },
+]
+
+[[package]]
+name = "pillow"
+version = "12.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1c/3d/bb7fca845737cf9d7dbde16ed1843984665ff2e0a518f5db43e77ec540b9/pillow-12.3.0.tar.gz", hash = "sha256:3b8182a766685eaa002637e28b4ec8d6b18819a0c71f579bf0dbaa5830297cce", size = 47025035, upload-time = "2026-07-01T11:56:38.965Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9d/ac/31fb64e1e7efb5a4b50cd3d92049ba89ac6e4d8d3bb6a74e15048ca3353e/pillow-12.3.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:21900ce7ba264168cd50defae43cd75d25c833ad4ad6e73ffc5596d12e25ac89", size = 4161684, upload-time = "2026-07-01T11:54:25.934Z" },
+ { url = "https://files.pythonhosted.org/packages/87/b4/9805e23d2b4d77842b468513841fda254ee42f0289d25088340e4ff46e2d/pillow-12.3.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:4e8c2a84d977f50b9daed6eeaf3baef67d00d5d74d932288f02cb94518ee3ace", size = 4255487, upload-time = "2026-07-01T11:54:27.935Z" },
+ { url = "https://files.pythonhosted.org/packages/df/39/ecf519435a200c693fe053a6ee4d835b41cf963a4dfc2551c4e637cb2a71/pillow-12.3.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:ae26d61dfa7a47befdc7572b521024e8745f3d809bd95ca9505a7bba9ef849ec", size = 3696433, upload-time = "2026-07-01T11:54:29.813Z" },
+ { url = "https://files.pythonhosted.org/packages/42/92/2fc3ffad878ae8dd5469ec1bc8eb83b71f48e13efdf68f02709003982a32/pillow-12.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7a743ff716f746fc19a9557f60dab1600d4613255f8a7aeb3cdde4db7eb15a66", size = 5345889, upload-time = "2026-07-01T11:54:31.97Z" },
+ { url = "https://files.pythonhosted.org/packages/10/76/8803c13605b763d33d156c4678fc77f8443389c0c51c8aef707bb02015f4/pillow-12.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d69141514cc30b774ceea5e3ed3a6635c8d8a96edf664689b890f4089111fb35", size = 4780109, upload-time = "2026-07-01T11:54:34.026Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/01/e18aff37cb0b4aac47ac90f016d347a49aca667ef97f190b06ac2aabc928/pillow-12.3.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f7401aebd7f581d7f83a439d87d474999317ee099218e5ad25d125290990ba65", size = 6263736, upload-time = "2026-07-01T11:54:36.131Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/62/de5bdd77d935331f4f802edc11e4d82950f642caad6cb2f949837b8560e2/pillow-12.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3", size = 6937129, upload-time = "2026-07-01T11:54:38.216Z" },
+ { url = "https://files.pythonhosted.org/packages/70/4d/105627a13300c5e0df1d174230b32fd1273062c96f7745fd552b945d1e1d/pillow-12.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:571b9fcb07b97ef3a492028fb3d2dc0993ca23a06138b0315286566d29ef718a", size = 6339562, upload-time = "2026-07-01T11:54:40.354Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/1d/f13de01a553988ab895ba1c722e06cf3144d4f57656fd5b81b6d881f1179/pillow-12.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:756c768d0c9c2955feb7a56c37ea24aea2e369f8d36a88da270b6a9f19e62b5e", size = 7049439, upload-time = "2026-07-01T11:54:42.489Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/f9/066794cca041b969964f779ee5fa66a9498bbf34248ac39c5d7954e4198f/pillow-12.3.0-cp313-cp313-win32.whl", hash = "sha256:a876864214e136f0eb367788dbd7df045f4806801518e2cfe9e13229cfe06d8f", size = 6473287, upload-time = "2026-07-01T11:54:44.9Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/9b/7a58e61d62be561da3a356fe2384d4059a6345fc130e23ef1c36a5b81d24/pillow-12.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:1cca606cd25738df4ed873d5ad46bbdb3d83b5cbca291f6b4ff13a4df6b0bbe8", size = 7239691, upload-time = "2026-07-01T11:54:47.141Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/b0/c4ed4f0ef8f8fa5ee8351537db6650bb8189f7e118842978dd6589065692/pillow-12.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:b629de27fda84b42cde7edef0d85f13b958b47f6e9bbcbba9b673c562a89bd8b", size = 2568185, upload-time = "2026-07-01T11:54:49.137Z" },
+]
+
+[[package]]
+name = "pip"
+version = "26.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/01/91/47e7d486260f618783899587af63ccf7980fb60245c3e63dd4571c6b57ad/pip-26.1.2.tar.gz", hash = "sha256:f49cd134c61cf2fd75e0ce2676db03e4054504a5a4986d00f8299ae632dc4605", size = 1840799, upload-time = "2026-05-31T17:33:58.56Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5d/95/6b5cb3461ea5673ba0995989746db58eb18b91b54dbf331e72f569540946/pip-26.1.2-py3-none-any.whl", hash = "sha256:382ff9f685ee3bc25864f820aa50505825f10f5458ffff07e30a6d96e5715cab", size = 1813144, upload-time = "2026-05-31T17:33:56.772Z" },
+]
+
+[[package]]
+name = "platformdirs"
+version = "4.10.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/47/e4501f49c178ae1d9f4a75073fda4204f52647993f075a9db4d14930e0c5/platformdirs-4.10.0.tar.gz", hash = "sha256:31e761a6a0ca04faf7353ea759bdba55652be214725111e5aac52dfa29d4bef7", size = 31224, upload-time = "2026-05-28T03:32:53.587Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" },
+]
+
+[[package]]
+name = "pluggy"
+version = "1.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
+]
+
+[[package]]
+name = "portalocker"
+version = "3.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pywin32", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5e/77/65b857a69ed876e1951e88aaba60f5ce6120c33703f7cb61a3c894b8c1b6/portalocker-3.2.0.tar.gz", hash = "sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac", size = 95644, upload-time = "2025-06-14T13:20:40.03Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4b/a6/38c8e2f318bf67d338f4d629e93b0b4b9af331f455f0390ea8ce4a099b26/portalocker-3.2.0-py3-none-any.whl", hash = "sha256:3cdc5f565312224bc570c49337bd21428bba0ef363bbcf58b9ef4a9f11779968", size = 22424, upload-time = "2025-06-14T13:20:38.083Z" },
+]
+
+[[package]]
+name = "pre-commit"
+version = "4.6.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cfgv" },
+ { name = "identify" },
+ { name = "nodeenv" },
+ { name = "pyyaml" },
+ { name = "virtualenv" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/8e/22/2de9408ac81acbb8a7d05d4cc064a152ccf33b3d480ebe0cd292153db239/pre_commit-4.6.0.tar.gz", hash = "sha256:718d2208cef53fdc38206e40524a6d4d9576d103eb16f0fec11c875e7716e9d9", size = 198525, upload-time = "2026-04-21T20:31:41.613Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/80/6e/4b28b62ecb6aae56769c34a8ff1d661473ec1e9519e2d5f8b2c150086b26/pre_commit-4.6.0-py2.py3-none-any.whl", hash = "sha256:e2cf246f7299edcabcf15f9b0571fdce06058527f0a06535068a86d38089f29b", size = 226472, upload-time = "2026-04-21T20:31:40.092Z" },
+]
+
+[[package]]
+name = "preshed"
+version = "3.0.13"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cymem" },
+ { name = "murmurhash" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/43/75/fe6b7bbd0dea530a001b0e24c331b21a0be2786e402abf3c57f5dce43d4b/preshed-3.0.13.tar.gz", hash = "sha256:d75f718bbfd97e992f7827e0fa7faf6a91bdd9c922d5baa4b50d62731396cb89", size = 18338, upload-time = "2026-03-23T08:57:31.378Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0e/2a/401158195d6dc7f6aef0b354d74d0e95c9da124499448c2b3dbb95b71204/preshed-3.0.13-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c0d0c14187dc0078d8a63bf190ec045a4d13e7748b6caeb557a7d575e411410b", size = 137289, upload-time = "2026-03-23T08:56:46.516Z" },
+ { url = "https://files.pythonhosted.org/packages/88/8f/e20e64573988528785447a6893b2e7ab287ecfd85b3888e978b28812fd20/preshed-3.0.13-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7770987c2e57497cd26124a9be5f652b5b3ccd0def89859ab0da8bca6144a3de", size = 136847, upload-time = "2026-03-23T08:56:47.572Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/72/18168f881359c4482d312f8dc196371bdd61c1583a52b34390da4c88bbea/preshed-3.0.13-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4a7bc48220de579be6bdb0a8715482cf36e2a625a6fd5ad26c9f43485a4a23b5", size = 831478, upload-time = "2026-03-23T08:56:48.769Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/3a/3543476091087102775568cea9885dde3453569e9aeee365809108de572f/preshed-3.0.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e5c8462472f790c16708306aef3a102a762bd19dfe3d2f8ee08bd5e12f51b835", size = 839913, upload-time = "2026-03-23T08:56:49.937Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/65/b13f01329decc44ef53cfb6b4601ba85382dcb2a4ec78d9250f03a418066/preshed-3.0.13-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:c046736239cc8d72670749b79b526e4111839a2fc461a58545d212797649129c", size = 1816452, upload-time = "2026-03-23T08:56:51.233Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/c7/f1a996c6832234efd4d543041b582418d41ac480ee55c557ec9e65344637/preshed-3.0.13-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7c333f18e9a81c8a6de0603fd8781e17115324b117c445ca91abdf7bfb1abe49", size = 1888978, upload-time = "2026-03-23T08:56:52.591Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/b9/96fb71499049885ce19545903fdd38877bbc2be0da47e37c04d01f3e9f66/preshed-3.0.13-cp313-cp313-win_amd64.whl", hash = "sha256:461327f8dd36520dcf1fd55a671e0c3c2c97a2d95e22fc85faa31173f4785dda", size = 122134, upload-time = "2026-03-23T08:56:54.392Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/a7/32a4903019d936a2316fdd330bedddac287ac26326107d24fb76a1fbc60a/preshed-3.0.13-cp313-cp313-win_arm64.whl", hash = "sha256:35d6c5acb3ee3b12b87a551913063f0cec784055c2af16e028c19fe875f079d0", size = 108497, upload-time = "2026-03-23T08:56:55.816Z" },
+]
+
+[[package]]
+name = "prometheus-client"
+version = "0.25.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/fb/d9aa83ffe43ce1f19e557c0971d04b90561b0cfd50762aafb01968285553/prometheus_client-0.25.0.tar.gz", hash = "sha256:5e373b75c31afb3c86f1a52fa1ad470c9aace18082d39ec0d2f918d11cc9ba28", size = 86035, upload-time = "2026-04-09T19:53:42.359Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8d/9b/d4b1e644385499c8346fa9b622a3f030dce14cd6ef8a1871c221a17a67e7/prometheus_client-0.25.0-py3-none-any.whl", hash = "sha256:d5aec89e349a6ec230805d0df882f3807f74fd6c1a2fa86864e3c2279059fed1", size = 64154, upload-time = "2026-04-09T19:53:41.324Z" },
+]
+
+[[package]]
+name = "prometheus-fastapi-instrumentator"
+version = "8.0.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "prometheus-client" },
+ { name = "starlette" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1b/e9/2065686d1dfa62296fdc158b6e8fd25b0cb3dca09b0632cabeb5ae81fe4d/prometheus_fastapi_instrumentator-8.0.2.tar.gz", hash = "sha256:3c252e748151768a7aefd66824a04a870144f71de48a67aed211749a9ca2a548", size = 21342, upload-time = "2026-06-23T09:39:31.611Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c6/c7/fa2b3f469a2e6001b829e0d6bc8680755349aa1329a87bf48731e9d5d30a/prometheus_fastapi_instrumentator-8.0.2-py3-none-any.whl", hash = "sha256:746002ec1e2c58b93f61444e1d104de959a9463a6a3f1c8909ac3757e16c3866", size = 20549, upload-time = "2026-06-23T09:39:32.616Z" },
+]
+
+[[package]]
+name = "prompt-toolkit"
+version = "3.0.52"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "wcwidth" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a1/96/06e01a7b38dce6fe1db213e061a4602dd6032a8a97ef6c1a862537732421/prompt_toolkit-3.0.52.tar.gz", hash = "sha256:28cde192929c8e7321de85de1ddbe736f1375148b02f2e17edd840042b1be855", size = 434198, upload-time = "2025-08-27T15:24:02.057Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/84/03/0d3ce49e2505ae70cf43bc5bb3033955d2fc9f932163e84dc0779cc47f48/prompt_toolkit-3.0.52-py3-none-any.whl", hash = "sha256:9aac639a3bbd33284347de5ad8d68ecc044b91a762dc39b7c21095fcd6a19955", size = 391431, upload-time = "2025-08-27T15:23:59.498Z" },
+]
+
+[[package]]
+name = "propcache"
+version = "0.5.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ec/44/c87281c333769159c50594f22610f77398a47ccbfbbf23074e744e86f87c/propcache-0.5.2.tar.gz", hash = "sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427", size = 50208, upload-time = "2026-05-08T21:02:12.199Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c5/09/f049e45385503fe67db75a6b6186a7b9f0c3930366dc960522c312a825b1/propcache-0.5.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:099aaf4b4d1a02265b92a977edf00b5c4f63b3b17ac6de39b0d637c9cac0188a", size = 94457, upload-time = "2026-05-08T21:00:36.355Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/65/83d1d05655baf63113731bd5a1008435e14f8d1e5a06cbe4ec5b23ad7a31/propcache-0.5.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:68ce1c44c7a813a7f71ea04315a8c7b330b63db99d059a797a4651bb6f69f117", size = 53835, upload-time = "2026-05-08T21:00:38.072Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/12/a6ba6482bb5ea3260c000c9b20881c95fa11c6b30173715668259f844ed7/propcache-0.5.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:fc299c129490f55f254cd90be0deca4764e36e9a7c08b4aa588479a3bbed3098", size = 54545, upload-time = "2026-05-08T21:00:39.319Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/19/7fa086f5764c59ec8a8e157cd93aa8497acc00aba9dcdec56bfffb32602d/propcache-0.5.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a6ae2198be502c10f09b2516e7b5d019816924bc3183a43ce792a7bd6625e6f4", size = 59886, upload-time = "2026-05-08T21:00:40.621Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/e4/5d7663dc8235956c8f5281698a3af1d351d8820341ddd890f59d9a9127f2/propcache-0.5.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6041d31504dc1779d700e1edcfb08eea334b357620b06681a4eabb57a74e574e", size = 63261, upload-time = "2026-05-08T21:00:41.775Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/4a/15a03adee24d6350da4292caeac44c34c033d2afe5e87eb370f38854560f/propcache-0.5.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f7eabc04151c78a9f4d5bbb5f1faf571e4defeb4b585e0fe95b60ff2dbe4d3d7", size = 64184, upload-time = "2026-05-08T21:00:43.018Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/c6/979176efdaa3d239e36d503d5af63a0a773b36662ed8f52e5b6a6d9fd40e/propcache-0.5.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4db0ba63d693afd40d249bd93f842b5f144f8fcbb83de05660373bcf30517b1d", size = 61534, upload-time = "2026-05-08T21:00:44.507Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/22/63e8cd1bae4c2d2be6493b6b7d10566ddafad88137cfbc99964a1119853c/propcache-0.5.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1dbcf7675229b35d31abb6547d8ebc8c27a830ac3f9a794edff6254873ec7c0a", size = 61500, upload-time = "2026-05-08T21:00:45.796Z" },
+ { url = "https://files.pythonhosted.org/packages/60/5a/28e5d9acbac1cc9ccb67045e8c1b943aa8d79fdf39c93bd73cacd68008ea/propcache-0.5.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d310c013aad2c72f1c3f2f8dd3279d460a858c551f97aeb8c63e4693cca7b4d2", size = 59994, upload-time = "2026-05-08T21:00:47.093Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/40/db650677f554a95b9c01a7c9d93d629e93a15562f5deb4573c9ee136fed2/propcache-0.5.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:06187263ddad280d05b4d8a8b3bb7d164cbebd469236544a42e6d9b28ac6a4fa", size = 56884, upload-time = "2026-05-08T21:00:48.376Z" },
+ { url = "https://files.pythonhosted.org/packages/80/45/70b39b89516ff8b96bf732fa6fded8cef20f293cb1508690101c3c07ec51/propcache-0.5.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3115559b8effafd63b142ea5ed53d63a16ea6469cbc63dce4ee194b42db5d853", size = 63464, upload-time = "2026-05-08T21:00:49.954Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/e2/fa59d3a89eac5534293124af4f1d0d0ada091ce4a0ab4610ce03fd2bdd8d/propcache-0.5.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c60462af8e6dc30c35407c7237ea908d777b22862bbee27bc4699c0d8bcdc45a", size = 61588, upload-time = "2026-05-08T21:00:51.281Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/97/efb547a55c4bc7381cfb202d6a2239ac621045277bc1ea5dfd3a7f0516c0/propcache-0.5.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:40314bca9ac559716fe374094fc81c11dcc34b64fd6c585360f5775690505704", size = 64667, upload-time = "2026-05-08T21:00:52.602Z" },
+ { url = "https://files.pythonhosted.org/packages/92/56/f5c7d9b4b7595d5127da38974d791b2153f3d1eae6c674af3583ace92ad3/propcache-0.5.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cfa21e036ce1e1db2be04ba3b85d2df1bb1702fa01932d984c5464c665228ff4", size = 62463, upload-time = "2026-05-08T21:00:54.303Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/3b/484a3a65fc9f9f60c41dcd17b428bace5389544e2c680994534a20755066/propcache-0.5.2-cp313-cp313-win32.whl", hash = "sha256:f156a3529f38063b6dbaf356e15602a7f95f8055b1295a438433a6386f10463d", size = 38621, upload-time = "2026-05-08T21:00:55.808Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/fd/3f0f10dba4dabad3bf53102be007abf55481067952bde0fdddff439e7c61/propcache-0.5.2-cp313-cp313-win_amd64.whl", hash = "sha256:dfed59d0a5aeb01e242e66ff0300bc4a265a7c05f612d30016f0b60b1017d757", size = 41649, upload-time = "2026-05-08T21:00:57.061Z" },
+ { url = "https://files.pythonhosted.org/packages/90/ec/6ce619cc32bb500a482f811f9cd509368b4e58e638d13f2c68f370d6b475/propcache-0.5.2-cp313-cp313-win_arm64.whl", hash = "sha256:ba338430e87ceb9c8f0cf754de38a9860560261e56c00376debd628698a7364f", size = 37636, upload-time = "2026-05-08T21:00:58.646Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/82/c1d268bbbf2ef981c5bf0fbbe746db617c66e3bcefe431a1aa8943fbe23a/propcache-0.5.2-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:a592f5f3da71c8691c788c13cb6734b6d17663d2e1cb8caddf0673d01ef8847d", size = 98872, upload-time = "2026-05-08T21:00:59.889Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/d4/52c871e73e864e6b34c0e2d58ac1ec5ccd149497ddc7ad2137ae98323a35/propcache-0.5.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:6a997d0489e9668a384fcfd5061b857aa5361de73191cac204d04b889cfbbafa", size = 56257, upload-time = "2026-05-08T21:01:01.195Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f0/9b90ca2a210b3d09bcfcd96ecd0f55545c091535abce2a45de2775cfd357/propcache-0.5.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:10734b5484ea113152ee25a91dccedf81631791805d2c9ccb054958e51842c94", size = 56696, upload-time = "2026-05-08T21:01:02.941Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/0e/6e9d4ba07c8e56e21ddec1e75f12148142b21ca83a51871babce095334f4/propcache-0.5.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cafca7e56c12bb02ae16d283742bef25a61122e9dab2b5b3f2ccbe589ce32164", size = 62378, upload-time = "2026-05-08T21:01:04.475Z" },
+ { url = "https://files.pythonhosted.org/packages/65/19/c10badaa463dde8a27ce884f8ee2ec37e6035b7c9f5ff0c8f74f06f08dac/propcache-0.5.2-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f064f8d2b59177878b7615df1735cd8fe3462ed6be8c7b217d17a276489c2b7f", size = 65283, upload-time = "2026-05-08T21:01:05.959Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/b6/93bea99ca80e19cef6512a8580e5b7857bbe09422d9daa7fd4ef5723306c/propcache-0.5.2-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f78abfa8dfc32376fd1aacf597b2f2fbbe0ea751419aee718af5d4f82537ef8c", size = 66616, upload-time = "2026-05-08T21:01:07.228Z" },
+ { url = "https://files.pythonhosted.org/packages/83/e4/5c7462e50625f051f37fb38b8224f7639f667184bbd34424ec83819bb1b7/propcache-0.5.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f7467da8a9822bf1a55336f877340c5bcbd3c482afc43a99771169f74a26dedc", size = 63773, upload-time = "2026-05-08T21:01:08.514Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/b6/99238894047b13c823be25027e736626cd414a52a5e30d2c3347c2733529/propcache-0.5.2-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a6ddc6ac9e25de626c1f129c1b467d7ecd33ce2237d3fd0c4e429feef0a7ee1f", size = 63664, upload-time = "2026-05-08T21:01:09.874Z" },
+ { url = "https://files.pythonhosted.org/packages/85/1e/a3a1a63116a2b8edb415a8bb9a6f0c34bd03830b1e18e8ce2904e1dc1cf4/propcache-0.5.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:2f22cbbac9e26a8e864c0985ff1268d5d939d53d9d9411a9824279097e03a2cb", size = 62643, upload-time = "2026-05-08T21:01:11.132Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/03/893cf147de2fc6543c5eaa07ad833170e7e2a2385725bbebe8c0503723bb/propcache-0.5.2-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:fc76378c62a0f04d0cd82fbb1a2cd2d7e28fcb40d5873f28a6c44e388aaa2751", size = 59595, upload-time = "2026-05-08T21:01:12.387Z" },
+ { url = "https://files.pythonhosted.org/packages/86/3b/04c1a2e12c57766568ba75ba72b3bf2042818d4c1425fab6fc07155c7cff/propcache-0.5.2-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:acd2c8edba48e31e58a363b8cf4e5c7db3b04b3f9e371f601df30d9b0d244836", size = 65711, upload-time = "2026-05-08T21:01:13.676Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/34/80f8d0099f8d6bacc4de1624c85672681c8cd1149ca2da0e38fd120b817f/propcache-0.5.2-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:452b5065457eb9991ec5eb38ff41d6cd4c991c9ac7c531c4d5849ae473a9a13f", size = 64247, upload-time = "2026-05-08T21:01:14.936Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/1a/8b08f3a5f1037e9e370c55883ceeeee0f6dd0416fb2d2d67b8bfc91f2a79/propcache-0.5.2-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:3430bb2bfe1331885c427745a751e774ee679fd4344f80b97bf879815fe8fa55", size = 67102, upload-time = "2026-05-08T21:01:16.281Z" },
+ { url = "https://files.pythonhosted.org/packages/34/68/8bdb7bb7756d76e005490649d10e4a8369e610c74d619f71e1aedf889e9c/propcache-0.5.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:cef6cea3922890dd6c9654971001fa797b526c16ab5e1e46c05fd6f877be7568", size = 64964, upload-time = "2026-05-08T21:01:17.57Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/aa/50fb0b5d3968b61a510926ff8b8465f1d6e976b3ab74496d7a4b9fc42515/propcache-0.5.2-cp313-cp313t-win32.whl", hash = "sha256:72d61e16dd78228b58c5d47be830ff3da7e5f139abdf0aef9d86cde1c5cf2191", size = 42546, upload-time = "2026-05-08T21:01:18.946Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/4c/0ddbae64321bd4a95bcbfc19307238016b5b1fee645c84626c8d539e5b74/propcache-0.5.2-cp313-cp313t-win_amd64.whl", hash = "sha256:0958834041a0166d343b8d2cedcd8bcbaeb4fdbe0cf08320c5379f143c3be6e7", size = 46330, upload-time = "2026-05-08T21:01:20.162Z" },
+ { url = "https://files.pythonhosted.org/packages/00/d9/9cddc8efb78d8af264c5ec9f6d10b62f57c515feda8d321595f56010fb23/propcache-0.5.2-cp313-cp313t-win_arm64.whl", hash = "sha256:6de8bd93ddde9b992cf2b2e0d796d501a19026b5b9fd87356d7d0779531a8d96", size = 40521, upload-time = "2026-05-08T21:01:21.399Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/ed/1cdcab6ba3d6ab7feca11fc14f0eeea80755bb53ef4e892079f31b10a25f/propcache-0.5.2-py3-none-any.whl", hash = "sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe", size = 14036, upload-time = "2026-05-08T21:02:10.673Z" },
+]
+
+[[package]]
+name = "protobuf"
+version = "6.33.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/66/70/e908e9c5e52ef7c3a6c7902c9dfbb34c7e29c25d2f81ade3856445fd5c94/protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135", size = 444531, upload-time = "2026-03-18T19:05:00.988Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fc/9f/2f509339e89cfa6f6a4c4ff50438db9ca488dec341f7e454adad60150b00/protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3", size = 425739, upload-time = "2026-03-18T19:04:48.373Z" },
+ { url = "https://files.pythonhosted.org/packages/76/5d/683efcd4798e0030c1bab27374fd13a89f7c2515fb1f3123efdfaa5eab57/protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326", size = 437089, upload-time = "2026-03-18T19:04:50.381Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/01/a3c3ed5cd186f39e7880f8303cc51385a198a81469d53d0fdecf1f64d929/protobuf-6.33.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a", size = 427737, upload-time = "2026-03-18T19:04:51.866Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/90/b3c01fdec7d2f627b3a6884243ba328c1217ed2d978def5c12dc50d328a3/protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2", size = 324610, upload-time = "2026-03-18T19:04:53.096Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/ca/25afc144934014700c52e05103c2421997482d561f3101ff352e1292fb81/protobuf-6.33.6-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3", size = 339381, upload-time = "2026-03-18T19:04:54.616Z" },
+ { url = "https://files.pythonhosted.org/packages/16/92/d1e32e3e0d894fe00b15ce28ad4944ab692713f2e7f0a99787405e43533a/protobuf-6.33.6-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593", size = 323436, upload-time = "2026-03-18T19:04:55.768Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/72/02445137af02769918a93807b2b7890047c32bfb9f90371cbc12688819eb/protobuf-6.33.6-py3-none-any.whl", hash = "sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901", size = 170656, upload-time = "2026-03-18T19:04:59.826Z" },
+]
+
+[[package]]
+name = "psutil"
+version = "7.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/51/08/510cbdb69c25a96f4ae523f733cdc963ae654904e8db864c07585ef99875/psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b", size = 130595, upload-time = "2026-01-28T18:14:57.293Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/f5/97baea3fe7a5a9af7436301f85490905379b1c6f2dd51fe3ecf24b4c5fbf/psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea", size = 131082, upload-time = "2026-01-28T18:14:59.732Z" },
+ { url = "https://files.pythonhosted.org/packages/37/d6/246513fbf9fa174af531f28412297dd05241d97a75911ac8febefa1a53c6/psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63", size = 181476, upload-time = "2026-01-28T18:15:01.884Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/b5/9182c9af3836cca61696dabe4fd1304e17bc56cb62f17439e1154f225dd3/psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312", size = 184062, upload-time = "2026-01-28T18:15:04.436Z" },
+ { url = "https://files.pythonhosted.org/packages/16/ba/0756dca669f5a9300d0cbcbfae9a4c30e446dfc7440ffe43ded5724bfd93/psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b", size = 139893, upload-time = "2026-01-28T18:15:06.378Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/61/8fa0e26f33623b49949346de05ec1ddaad02ed8ba64af45f40a147dbfa97/psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9", size = 135589, upload-time = "2026-01-28T18:15:08.03Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" },
+ { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" },
+ { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" },
+ { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" },
+]
+
+[[package]]
+name = "ptyprocess"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/20/e5/16ff212c1e452235a90aeb09066144d0c5a6a8c0834397e03f5224495c4e/ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220", size = 70762, upload-time = "2020-12-28T15:15:30.155Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35", size = 13993, upload-time = "2020-12-28T15:15:28.35Z" },
+]
+
+[[package]]
+name = "pure-eval"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/05/0a34433a064256a578f1783a10da6df098ceaa4a57bbeaa96a6c0352786b/pure_eval-0.2.3.tar.gz", hash = "sha256:5f4e983f40564c576c7c8635ae88db5956bb2229d7e9237d03b3c0b0190eaf42", size = 19752, upload-time = "2024-07-21T12:58:21.801Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" },
+]
+
+[[package]]
+name = "py-cpuinfo"
+version = "9.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/37/a8/d832f7293ebb21690860d2e01d8115e5ff6f2ae8bbdc953f0eb0fa4bd2c7/py-cpuinfo-9.0.0.tar.gz", hash = "sha256:3cdbbf3fac90dc6f118bfd64384f309edeadd902d7c8fb17f02ffa1fc3f49690", size = 104716, upload-time = "2022-10-25T20:38:06.303Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl", hash = "sha256:859625bc251f64e21f077d099d4162689c762b5d6a4c3c97553d56241c9674d5", size = 22335, upload-time = "2022-10-25T20:38:27.636Z" },
+]
+
+[[package]]
+name = "pyarrow"
+version = "24.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/91/13/13e1069b351bdc3881266e11147ffccf687505dbb0ea74036237f5d454a5/pyarrow-24.0.0.tar.gz", hash = "sha256:85fe721a14dd823aca09127acbb06c3ca723efbd436c004f16bca601b04dcc83", size = 1180261, upload-time = "2026-04-21T10:51:25.837Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6f/d3/a1abf004482026ddc17f4503db227787fa3cfe41ec5091ff20e4fea55e57/pyarrow-24.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:02b001b3ed4723caa44f6cd1af2d5c86aa2cf9971dacc2ffa55b21237713dfba", size = 34976759, upload-time = "2026-04-21T10:48:07.258Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/4a/34f0a36d28a2dd32225301b79daad44e243dc1a2bb77d43b60749be255c4/pyarrow-24.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:04920d6a71aabd08a0417709efce97d45ea8e6fb733d9ca9ecffb13c67839f68", size = 36658471, upload-time = "2026-04-21T10:48:13.347Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/78/543b94712ae8bb1a6023bcc1acf1a740fbff8286747c289cd9468fced2a5/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:a964266397740257f16f7bb2e4f08a0c81454004beab8ff59dd531b73610e9f2", size = 45675981, upload-time = "2026-04-21T10:48:20.201Z" },
+ { url = "https://files.pythonhosted.org/packages/84/9f/8fb7c222b100d314137fa40ec050de56cd8c6d957d1cfff685ce72f15b17/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:6f066b179d68c413374294bc1735f68475457c933258df594443bb9d88ddc2a0", size = 48859172, upload-time = "2026-04-21T10:48:27.541Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/d3/1ea72538e6c8b3b475ed78d1049a2c518e655761ea50fe1171fc855fcab7/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1183baeb14c5f587b1ec52831e665718ce632caab84b7cd6b85fd44f96114495", size = 49385733, upload-time = "2026-04-21T10:48:34.7Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/be/c3d8b06a1ba35f2260f8e1f771abbee7d5e345c0937aab90675706b1690a/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:806f24b4085453c197a5078218d1ee08783ebbba271badd153d1ae22a3ee804f", size = 51934335, upload-time = "2026-04-21T10:48:42.099Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/62/89e07a1e7329d2cde3e3c6994ba0839a24977a2beda8be6005ea3d860b99/pyarrow-24.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:e4505fc6583f7b05ab854934896bcac8253b04ac1171a77dfb73efef92076d91", size = 27271748, upload-time = "2026-04-21T10:49:42.532Z" },
+ { url = "https://files.pythonhosted.org/packages/17/1a/cff3a59f80b5b1658549d46611b67163f65e0664431c076ad728bf9d5af4/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:1a4e45017efbf115032e4475ee876d525e0e36c742214fbe405332480ecd6275", size = 35238554, upload-time = "2026-04-21T10:48:48.526Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/99/cce0f42a327bfef2c420fb6078a3eb834826e5d6697bf3009fe11d2ad051/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:7986f1fa71cee060ad00758bcc79d3a93bab8559bf978fab9e53472a2e25a17b", size = 36782301, upload-time = "2026-04-21T10:48:55.181Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/66/8e560d5ff6793ca29aca213c53eec0dd482dd46cb93b2819e5aab52e4252/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:d3e0b61e8efb24ed38898e5cdc5fffa9124be480008d401a1f8071500494ae42", size = 45721929, upload-time = "2026-04-21T10:49:03.676Z" },
+ { url = "https://files.pythonhosted.org/packages/27/0c/a26e25505d030716e078d9f16eb74973cbf0b33b672884e9f9da1c83b871/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:55a3bc1e3df3b5567b7d27ef551b2283f0c68a5e86f1cd56abc569da4f31335b", size = 48825365, upload-time = "2026-04-21T10:49:11.714Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/eb/771f9ecb0c65e73fe9dccdd1717901b9594f08c4515d000c7c62df573811/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:641f795b361874ac9da5294f8f443dfdbee355cf2bd9e3b8d97aaac2306b9b37", size = 49451819, upload-time = "2026-04-21T10:49:21.474Z" },
+ { url = "https://files.pythonhosted.org/packages/48/da/61ae89a88732f5a785646f3ec6125dbb640fa98a540eb2b9889caa561403/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8adc8e6ce5fccf5dc707046ae4914fd537def529709cc0d285d37a7f9cd442ca", size = 51909252, upload-time = "2026-04-21T10:49:31.164Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/1a/8dd5cafab7b66573fa91c03d06d213356ad4edd71813aa75e08ce2b3a844/pyarrow-24.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:9b18371ad2f44044b81a8d23bc2d8a9b6a6226dca775e8e16cfee640473d6c5d", size = 27388127, upload-time = "2026-04-21T10:49:37.334Z" },
+]
+
+[[package]]
+name = "pybase64"
+version = "1.4.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/b8/4ed5c7ad5ec15b08d35cc79ace6145d5c1ae426e46435f4987379439dfea/pybase64-1.4.3.tar.gz", hash = "sha256:c2ed274c9e0ba9c8f9c4083cfe265e66dd679126cd9c2027965d807352f3f053", size = 137272, upload-time = "2025-12-06T13:27:04.013Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3a/50/b7170cb2c631944388fe2519507fe3835a4054a6a12a43f43781dae82be1/pybase64-1.4.3-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:ea4b785b0607d11950b66ce7c328f452614aefc9c6d3c9c28bae795dc7f072e1", size = 33901, upload-time = "2025-12-06T13:23:40.951Z" },
+ { url = "https://files.pythonhosted.org/packages/48/8b/69f50578e49c25e0a26e3ee72c39884ff56363344b79fc3967f5af420ed6/pybase64-1.4.3-cp313-cp313-android_21_x86_64.whl", hash = "sha256:6a10b6330188c3026a8b9c10e6b9b3f2e445779cf16a4c453d51a072241c65a2", size = 40807, upload-time = "2025-12-06T13:23:42.006Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/8d/20b68f11adfc4c22230e034b65c71392e3e338b413bf713c8945bd2ccfb3/pybase64-1.4.3-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:27fdff227a0c0e182e0ba37a99109645188978b920dfb20d8b9c17eeee370d0d", size = 30932, upload-time = "2025-12-06T13:23:43.348Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/79/b1b550ac6bff51a4880bf6e089008b2e1ca16f2c98db5e039a08ac3ad157/pybase64-1.4.3-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:2a8204f1fdfec5aa4184249b51296c0de95445869920c88123978304aad42df1", size = 31394, upload-time = "2025-12-06T13:23:44.317Z" },
+ { url = "https://files.pythonhosted.org/packages/82/70/b5d7c5932bf64ee1ec5da859fbac981930b6a55d432a603986c7f509c838/pybase64-1.4.3-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:874fc2a3777de6baf6aa921a7aa73b3be98295794bea31bd80568a963be30767", size = 38078, upload-time = "2025-12-06T13:23:45.348Z" },
+ { url = "https://files.pythonhosted.org/packages/56/fe/e66fe373bce717c6858427670736d54297938dad61c5907517ab4106bd90/pybase64-1.4.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2dc64a94a9d936b8e3449c66afabbaa521d3cc1a563d6bbaaa6ffa4535222e4b", size = 38158, upload-time = "2025-12-06T13:23:46.872Z" },
+ { url = "https://files.pythonhosted.org/packages/80/a9/b806ed1dcc7aed2ea3dd4952286319e6f3a8b48615c8118f453948e01999/pybase64-1.4.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e48f86de1c145116ccf369a6e11720ce696c2ec02d285f440dfb57ceaa0a6cb4", size = 31672, upload-time = "2025-12-06T13:23:47.88Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/c9/24b3b905cf75e23a9a4deaf203b35ffcb9f473ac0e6d8257f91a05dfce62/pybase64-1.4.3-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:1d45c8fe8fe82b65c36b227bb4a2cf623d9ada16bed602ce2d3e18c35285b72a", size = 68244, upload-time = "2025-12-06T13:23:49.026Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/cd/d15b0c3e25e5859fab0416dc5b96d34d6bd2603c1c96a07bb2202b68ab92/pybase64-1.4.3-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:ad70c26ba091d8f5167e9d4e1e86a0483a5414805cdb598a813db635bd3be8b8", size = 71620, upload-time = "2025-12-06T13:23:50.081Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/31/4ca953cc3dcde2b3711d6bfd70a6f4ad2ca95a483c9698076ba605f1520f/pybase64-1.4.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e98310b7c43145221e7194ac9fa7fffc84763c87bfc5e2f59f9f92363475bdc1", size = 59930, upload-time = "2025-12-06T13:23:51.68Z" },
+ { url = "https://files.pythonhosted.org/packages/60/55/e7f7bdcd0fd66e61dda08db158ffda5c89a306bbdaaf5a062fbe4e48f4a1/pybase64-1.4.3-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.whl", hash = "sha256:398685a76034e91485a28aeebcb49e64cd663212fd697b2497ac6dfc1df5e671", size = 56425, upload-time = "2025-12-06T13:23:52.732Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/65/b592c7f921e51ca1aca3af5b0d201a98666d0a36b930ebb67e7c2ed27395/pybase64-1.4.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7e46400a6461187ccb52ed75b0045d937529e801a53a9cd770b350509f9e4d50", size = 59327, upload-time = "2025-12-06T13:23:53.856Z" },
+ { url = "https://files.pythonhosted.org/packages/23/95/1613d2fb82dbb1548595ad4179f04e9a8451bfa18635efce18b631eabe3f/pybase64-1.4.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:1b62b9f2f291d94f5e0b76ab499790b7dcc78a009d4ceea0b0428770267484b6", size = 60294, upload-time = "2025-12-06T13:23:54.937Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/73/40431f37f7d1b3eab4673e7946ff1e8f5d6bd425ec257e834dae8a6fc7b0/pybase64-1.4.3-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:f30ceb5fa4327809dede614be586efcbc55404406d71e1f902a6fdcf322b93b2", size = 54858, upload-time = "2025-12-06T13:23:56.031Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/84/f6368bcaf9f743732e002a9858646fd7a54f428490d427dd6847c5cfe89e/pybase64-1.4.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0d5f18ed53dfa1d4cf8b39ee542fdda8e66d365940e11f1710989b3cf4a2ed66", size = 58629, upload-time = "2025-12-06T13:23:57.12Z" },
+ { url = "https://files.pythonhosted.org/packages/43/75/359532f9adb49c6b546cafc65c46ed75e2ccc220d514ba81c686fbd83965/pybase64-1.4.3-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:119d31aa4b58b85a8ebd12b63c07681a138c08dfc2fe5383459d42238665d3eb", size = 52448, upload-time = "2025-12-06T13:23:58.298Z" },
+ { url = "https://files.pythonhosted.org/packages/92/6c/ade2ba244c3f33ed920a7ed572ad772eb0b5f14480b72d629d0c9e739a40/pybase64-1.4.3-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:3cf0218b0e2f7988cf7d738a73b6a1d14f3be6ce249d7c0f606e768366df2cce", size = 68841, upload-time = "2025-12-06T13:23:59.886Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/51/b345139cd236be382f2d4d4453c21ee6299e14d2f759b668e23080f8663f/pybase64-1.4.3-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:12f4ee5e988bc5c0c1106b0d8fc37fb0508f12dab76bac1b098cb500d148da9d", size = 57910, upload-time = "2025-12-06T13:24:00.994Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/b8/9f84bdc4f1c4f0052489396403c04be2f9266a66b70c776001eaf0d78c1f/pybase64-1.4.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:937826bc7b6b95b594a45180e81dd4d99bd4dd4814a443170e399163f7ff3fb6", size = 54335, upload-time = "2025-12-06T13:24:02.046Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/c7/be63b617d284de46578a366da77ede39c8f8e815ed0d82c7c2acca560fab/pybase64-1.4.3-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:88995d1460971ef80b13e3e007afbe4b27c62db0508bc7250a2ab0a0b4b91362", size = 56486, upload-time = "2025-12-06T13:24:03.141Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/96/f252c8f9abd6ded3ef1ccd3cdbb8393a33798007f761b23df8de1a2480e6/pybase64-1.4.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:72326fe163385ed3e1e806dd579d47fde5d8a59e51297a60fc4e6cbc1b4fc4ed", size = 70978, upload-time = "2025-12-06T13:24:04.221Z" },
+ { url = "https://files.pythonhosted.org/packages/af/51/0f5714af7aeef96e30f968e4371d75ad60558aaed3579d7c6c8f1c43c18a/pybase64-1.4.3-cp313-cp313-win32.whl", hash = "sha256:b1623730c7892cf5ed0d6355e375416be6ef8d53ab9b284f50890443175c0ac3", size = 33684, upload-time = "2025-12-06T13:24:05.29Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/ad/0cea830a654eb08563fb8214150ef57546ece1cc421c09035f0e6b0b5ea9/pybase64-1.4.3-cp313-cp313-win_amd64.whl", hash = "sha256:8369887590f1646a5182ca2fb29252509da7ae31d4923dbb55d3e09da8cc4749", size = 35832, upload-time = "2025-12-06T13:24:06.35Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/0d/eec2a8214989c751bc7b4cad1860eb2c6abf466e76b77508c0f488c96a37/pybase64-1.4.3-cp313-cp313-win_arm64.whl", hash = "sha256:860b86bca71e5f0237e2ab8b2d9c4c56681f3513b1bf3e2117290c1963488390", size = 31175, upload-time = "2025-12-06T13:24:07.419Z" },
+ { url = "https://files.pythonhosted.org/packages/db/c9/e23463c1a2913686803ef76b1a5ae7e6fac868249a66e48253d17ad7232c/pybase64-1.4.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:eb51db4a9c93215135dccd1895dca078e8785c357fabd983c9f9a769f08989a9", size = 38497, upload-time = "2025-12-06T13:24:08.873Z" },
+ { url = "https://files.pythonhosted.org/packages/71/83/343f446b4b7a7579bf6937d2d013d82f1a63057cf05558e391ab6039d7db/pybase64-1.4.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:a03ef3f529d85fd46b89971dfb00c634d53598d20ad8908fb7482955c710329d", size = 32076, upload-time = "2025-12-06T13:24:09.975Z" },
+ { url = "https://files.pythonhosted.org/packages/46/fc/cb64964c3b29b432f54d1bce5e7691d693e33bbf780555151969ffd95178/pybase64-1.4.3-cp313-cp313t-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:2e745f2ce760c6cf04d8a72198ef892015ddb89f6ceba489e383518ecbdb13ab", size = 72317, upload-time = "2025-12-06T13:24:11.129Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/b7/fab2240da6f4e1ad46f71fa56ec577613cf5df9dce2d5b4cfaa4edd0e365/pybase64-1.4.3-cp313-cp313t-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6fac217cd9de8581a854b0ac734c50fd1fa4b8d912396c1fc2fce7c230efe3a7", size = 75534, upload-time = "2025-12-06T13:24:12.433Z" },
+ { url = "https://files.pythonhosted.org/packages/91/3b/3e2f2b6e68e3d83ddb9fa799f3548fb7449765daec9bbd005a9fbe296d7f/pybase64-1.4.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:da1ee8fa04b283873de2d6e8fa5653e827f55b86bdf1a929c5367aaeb8d26f8a", size = 65399, upload-time = "2025-12-06T13:24:13.928Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/08/476ac5914c3b32e0274a2524fc74f01cbf4f4af4513d054e41574eb018f6/pybase64-1.4.3-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.whl", hash = "sha256:b0bf8e884ee822ca7b1448eeb97fa131628fe0ff42f60cae9962789bd562727f", size = 60487, upload-time = "2025-12-06T13:24:15.177Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/b8/618a92915330cc9cba7880299b546a1d9dab1a21fd6c0292ee44a4fe608c/pybase64-1.4.3-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:1bf749300382a6fd1f4f255b183146ef58f8e9cb2f44a077b3a9200dfb473a77", size = 63959, upload-time = "2025-12-06T13:24:16.854Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/52/af9d8d051652c3051862c442ec3861259c5cdb3fc69774bc701470bd2a59/pybase64-1.4.3-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:153a0e42329b92337664cfc356f2065248e6c9a1bd651bbcd6dcaf15145d3f06", size = 64874, upload-time = "2025-12-06T13:24:18.328Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/51/5381a7adf1f381bd184d33203692d3c57cf8ae9f250f380c3fecbdbe554b/pybase64-1.4.3-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:86ee56ac7f2184ca10217ed1c655c1a060273e233e692e9086da29d1ae1768db", size = 58572, upload-time = "2025-12-06T13:24:19.417Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/f0/578ee4ffce5818017de4fdf544e066c225bc435e73eb4793cde28a689d0b/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:0e71a4db76726bf830b47477e7d830a75c01b2e9b01842e787a0836b0ba741e3", size = 63636, upload-time = "2025-12-06T13:24:20.497Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/ad/8ae94814bf20159ea06310b742433e53d5820aa564c9fdf65bf2d79f8799/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:2ba7799ec88540acd9861b10551d24656ca3c2888ecf4dba2ee0a71544a8923f", size = 56193, upload-time = "2025-12-06T13:24:21.559Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/31/6438cfcc3d3f0fa84d229fa125c243d5094e72628e525dfefadf3bcc6761/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:2860299e4c74315f5951f0cf3e72ba0f201c3356c8a68f95a3ab4e620baf44e9", size = 72655, upload-time = "2025-12-06T13:24:22.673Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/0d/2bbc9e9c3fc12ba8a6e261482f03a544aca524f92eae0b4908c0a10ba481/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:bb06015db9151f0c66c10aae8e3603adab6b6cd7d1f7335a858161d92fc29618", size = 62471, upload-time = "2025-12-06T13:24:23.8Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/0b/34d491e7f49c1dbdb322ea8da6adecda7c7cd70b6644557c6e4ca5c6f7c7/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:242512a070817272865d37c8909059f43003b81da31f616bb0c391ceadffe067", size = 58119, upload-time = "2025-12-06T13:24:24.994Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/17/c21d0cde2a6c766923ae388fc1f78291e1564b0d38c814b5ea8a0e5e081c/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:5d8277554a12d3e3eed6180ebda62786bf9fc8d7bb1ee00244258f4a87ca8d20", size = 60791, upload-time = "2025-12-06T13:24:26.046Z" },
+ { url = "https://files.pythonhosted.org/packages/92/b2/eaa67038916a48de12b16f4c384bcc1b84b7ec731b23613cb05f27673294/pybase64-1.4.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f40b7ddd698fc1e13a4b64fbe405e4e0e1279e8197e37050e24154655f5f7c4e", size = 74701, upload-time = "2025-12-06T13:24:27.466Z" },
+ { url = "https://files.pythonhosted.org/packages/42/10/abb7757c330bb869ebb95dab0c57edf5961ffbd6c095c8209cbbf75d117d/pybase64-1.4.3-cp313-cp313t-win32.whl", hash = "sha256:46d75c9387f354c5172582a9eaae153b53a53afeb9c19fcf764ea7038be3bd8b", size = 33965, upload-time = "2025-12-06T13:24:28.548Z" },
+ { url = "https://files.pythonhosted.org/packages/63/a0/2d4e5a59188e9e6aed0903d580541aaea72dcbbab7bf50fb8b83b490b6c3/pybase64-1.4.3-cp313-cp313t-win_amd64.whl", hash = "sha256:d7344625591d281bec54e85cbfdab9e970f6219cac1570f2aa140b8c942ccb81", size = 36207, upload-time = "2025-12-06T13:24:29.646Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/05/95b902e8f567b4d4b41df768ccc438af618f8d111e54deaf57d2df46bd76/pybase64-1.4.3-cp313-cp313t-win_arm64.whl", hash = "sha256:28a3c60c55138e0028313f2eccd321fec3c4a0be75e57a8d3eb883730b1b0880", size = 31505, upload-time = "2025-12-06T13:24:30.687Z" },
+]
+
+[[package]]
+name = "pycountry"
+version = "26.2.16"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/de/1d/061b9e7a48b85cfd69f33c33d2ef784a531c359399ad764243399673c8f5/pycountry-26.2.16.tar.gz", hash = "sha256:5b6027d453fcd6060112b951dd010f01f168b51b4bf8a1f1fc8c95c8d94a0801", size = 7711342, upload-time = "2026-02-17T03:42:52.367Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9c/42/7703bd45b62fecd44cd7d3495423097e2f7d28bc2e99e7c1af68892ab157/pycountry-26.2.16-py3-none-any.whl", hash = "sha256:115c4baf7cceaa30f59a4694d79483c9167dbce7a9de4d3d571c5f3ea77c305a", size = 8044600, upload-time = "2026-02-17T03:42:49.777Z" },
+]
+
+[[package]]
+name = "pycparser"
+version = "3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
+]
+
+[[package]]
+name = "pydantic"
+version = "2.13.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "annotated-types" },
+ { name = "pydantic-core" },
+ { name = "typing-extensions" },
+ { name = "typing-inspection" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/18/a5/b60d21ac674192f8ab0ba4e9fd860690f9b4a6e51ca5df118733b487d8d6/pydantic-2.13.4.tar.gz", hash = "sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6", size = 844775, upload-time = "2026-05-06T13:43:05.343Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl", hash = "sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba", size = 472262, upload-time = "2026-05-06T13:43:02.641Z" },
+]
+
+[package.optional-dependencies]
+email = [
+ { name = "email-validator" },
+]
+
+[[package]]
+name = "pydantic-core"
+version = "2.46.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/9d/56/921726b776ace8d8f5db44c4ef961006580d91dc52b803c489fafd1aa249/pydantic_core-2.46.4.tar.gz", hash = "sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1", size = 471464, upload-time = "2026-05-06T13:37:06.98Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/51/a2/5d30b469c5267a17b39dec53208222f76a8d351dfac4af661888c5aee77d/pydantic_core-2.46.4-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008", size = 2106306, upload-time = "2026-05-06T13:37:48.029Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/81/4fa520eaffa8bd7d1525e644cd6d39e7d60b1592bc5b516693c7340b50f1/pydantic_core-2.46.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4", size = 1951906, upload-time = "2026-05-06T13:37:17.012Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d5/fd02da45b659668b05923b17ba3a0100a0a3d5541e3bd8fcc4ecb711309e/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76", size = 1976802, upload-time = "2026-05-06T13:37:35.113Z" },
+ { url = "https://files.pythonhosted.org/packages/21/f2/95727e1368be3d3ed485eaab7adbd7dda408f33f7a36e8b48e0144002b91/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3", size = 2052446, upload-time = "2026-05-06T13:37:12.313Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/86/5d99feea3f77c7234b8718075b23db11532773c1a0dbd9b9490215dc2eeb/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76", size = 2232757, upload-time = "2026-05-06T13:39:01.149Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/3a/508ac615935ef7588cf6d9e9b91309fdc2da751af865e02a9098de88258c/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4", size = 2309275, upload-time = "2026-05-06T13:37:41.406Z" },
+ { url = "https://files.pythonhosted.org/packages/07/f8/41db9de19d7987d6b04715a02b3b40aea467000275d9d758ffaa31af7d50/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a", size = 2094467, upload-time = "2026-05-06T13:39:18.847Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/e2/f35033184cb11d0052daf4416e8e10a502ea2ac006fc4f459aee872727d1/pydantic_core-2.46.4-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262", size = 2134417, upload-time = "2026-05-06T13:40:17.944Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/7b/6ceeb1cc90e193862f444ebe373d8fdf613f0a82572dde03fb10734c6c71/pydantic_core-2.46.4-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e", size = 2179782, upload-time = "2026-05-06T13:40:32.618Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/f2/c8d7773ede6af08036423a00ae0ceffce266c3c52a096c435d68c896083f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd", size = 2188782, upload-time = "2026-05-06T13:36:51.018Z" },
+ { url = "https://files.pythonhosted.org/packages/59/31/0c864784e31f09f05cdd87606f08923b9c9e7f6e51dd27f20f62f975ce9f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be", size = 2328334, upload-time = "2026-05-06T13:40:37.764Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/eb/4f6c8a41efa30baa755590f4141abf3a8c370fab610915733e74134a7270/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d", size = 2372986, upload-time = "2026-05-06T13:39:34.152Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/24/b375a480d53113860c299764bfe9f349a3dc9108b3adc0d7f0d786492ebf/pydantic_core-2.46.4-cp313-cp313-win32.whl", hash = "sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb", size = 1973693, upload-time = "2026-05-06T13:37:55.072Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/e8/cff247591966f2d22ec8c003cd7587e27b7ba7b81ab2fb888e3ab75dc285/pydantic_core-2.46.4-cp313-cp313-win_amd64.whl", hash = "sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292", size = 2071819, upload-time = "2026-05-06T13:38:49.139Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/1a/f4aee670d5670e9e148e0c82c7db98d780be566c6e6a97ee8035528ca0b3/pydantic_core-2.46.4-cp313-cp313-win_arm64.whl", hash = "sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d", size = 2027411, upload-time = "2026-05-06T13:40:45.796Z" },
+]
+
+[[package]]
+name = "pydantic-extra-types"
+version = "2.11.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pydantic" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/66/71/dba38ee2651f84f7842206adbd2233d8bbdb59fb85e9fa14232486a8c471/pydantic_extra_types-2.11.1.tar.gz", hash = "sha256:46792d2307383859e923d8fcefa82108b1a141f8a9c0198982b3832ab5ef1049", size = 172002, upload-time = "2026-03-16T08:08:03.92Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/17/c1/3226e6d7f5a4f736f38ac11a6fbb262d701889802595cdb0f53a885ac2e0/pydantic_extra_types-2.11.1-py3-none-any.whl", hash = "sha256:1722ea2bddae5628ace25f2aa685b69978ef533123e5638cfbddb999e0100ec1", size = 79526, upload-time = "2026-03-16T08:08:02.533Z" },
+]
+
+[package.optional-dependencies]
+pycountry = [
+ { name = "pycountry" },
+]
+
+[[package]]
+name = "pydantic-settings"
+version = "2.14.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pydantic" },
+ { name = "python-dotenv" },
+ { name = "typing-inspection" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5c/b5/8f48e906c3e0205276e8bd8cb7512217a87b2685304d64be27cad5b3019f/pydantic_settings-2.14.2.tar.gz", hash = "sha256:c19dd64b19097f1de80184f0cc7b0272a13ae6e170cbf240a3e27e381ed14a5f", size = 237700, upload-time = "2026-06-19T13:44:56.324Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/c1/6e422f34e569cf8e18df68d1939c81c099d2b61e4f7d9621c8a77560799c/pydantic_settings-2.14.2-py3-none-any.whl", hash = "sha256:a20c97b37910b6550d5ea50fbcc2d4187defe58cd57070b73863d069419c9440", size = 61715, upload-time = "2026-06-19T13:44:55.02Z" },
+]
+
+[[package]]
+name = "pygments"
+version = "2.20.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
+]
+
+[[package]]
+name = "pyjwt"
+version = "2.13.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/3b/81/58d0ac84e1ef3a3843791d6954d94c0b33d526c75eeb1efbce9d0a4c4077/pyjwt-2.13.0.tar.gz", hash = "sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423", size = 107515, upload-time = "2026-05-21T19:54:36.618Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a3/5e/ecf12fdb62546d64385c158514e9b2b671f7832108ef2ecd2020ce0af2d1/pyjwt-2.13.0-py3-none-any.whl", hash = "sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728", size = 31274, upload-time = "2026-05-21T19:54:35.362Z" },
+]
+
+[package.optional-dependencies]
+crypto = [
+ { name = "cryptography" },
+]
+
+[[package]]
+name = "pynvvideocodec"
+version = "2.0.4"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/46/d6/8475720d4f0f8fc9e852e0092b308643b71f24d9495ba3bd03c38b16c28d/pynvvideocodec-2.0.4-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:fcb06ef6ef24ee33b8f34950696a4e01636f2d8cdb96c407cd692931d049ac3d", size = 43176124, upload-time = "2026-05-27T04:06:26.415Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/28/33d2a4d69b48823801c0b02b1bfbeac04cd9d429ee698d248e7ba946c516/pynvvideocodec-2.0.4-cp313-cp313-manylinux_2_34_aarch64.whl", hash = "sha256:51724c6a0e3623c092cccdf93c8b09cced6881f3d0c76653f6ffbec0371f29cd", size = 35754919, upload-time = "2026-05-27T04:07:09.962Z" },
+ { url = "https://files.pythonhosted.org/packages/49/8e/8c08bbffc9021db762b6ac103ce544db4e517f9816118a35db9f0c7d2a9a/pynvvideocodec-2.0.4-cp313-cp313-win_amd64.whl", hash = "sha256:8e704a2b553a35cc2de10543a232e7feddc473f3e5478996595236e3194efc5d", size = 25692569, upload-time = "2026-05-27T04:07:38.663Z" },
+]
+
+[[package]]
+name = "pyparsing"
+version = "3.3.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f3/91/9c6ee907786a473bf81c5f53cf703ba0957b23ab84c264080fb5a450416f/pyparsing-3.3.2.tar.gz", hash = "sha256:c777f4d763f140633dcb6d8a3eda953bf7a214dc4eff598413c070bcdc117cbc", size = 6851574, upload-time = "2026-01-21T03:57:59.36Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl", hash = "sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d", size = 122781, upload-time = "2026-01-21T03:57:55.912Z" },
+]
+
+[[package]]
+name = "pytablewriter"
+version = "1.2.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dataproperty" },
+ { name = "mbstrdecoder" },
+ { name = "pathvalidate" },
+ { name = "setuptools" },
+ { name = "tabledata" },
+ { name = "tcolorpy" },
+ { name = "typepy", extra = ["datetime"] },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f6/a1/617730f290f04d347103ab40bf67d317df6691b14746f6e1ea039fb57062/pytablewriter-1.2.1.tar.gz", hash = "sha256:7bd0f4f397e070e3b8a34edcf1b9257ccbb18305493d8350a5dbc9957fced959", size = 619241, upload-time = "2025-01-01T15:37:00.04Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/21/4c/c199512f01c845dfe5a7840ab3aae6c60463b5dc2a775be72502dfd9170a/pytablewriter-1.2.1-py3-none-any.whl", hash = "sha256:e906ff7ff5151d70a5f66e0f7b75642a7f2dce8d893c265b79cc9cf6bc04ddb4", size = 91083, upload-time = "2025-01-01T15:36:55.63Z" },
+]
+
+[[package]]
+name = "pytest"
+version = "9.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "iniconfig" },
+ { name = "packaging" },
+ { name = "pluggy" },
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" },
+]
+
+[[package]]
+name = "pythainlp"
+version = "5.3.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "tzdata", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/56/47/2364543c559eacdc63633a3f29d7467a36a33089f7a8b226a247dd12de7e/pythainlp-5.3.4.tar.gz", hash = "sha256:e66fd76fb5931834fd4e32ed54337ec62350d7654f187850e4dd4f915e9f624f", size = 19306201, upload-time = "2026-04-02T18:42:46.044Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/bd/bedc1e416eb0fa2cb5eaa990207ae67dfaf02d4986f22d5ec372c0b97ae3/pythainlp-5.3.4-py3-none-any.whl", hash = "sha256:76744e51e27c895630bafd74f53a1f0aa8782cef2f7f02eebd6427fe8ce8d84d", size = 19849568, upload-time = "2026-04-02T18:42:42.384Z" },
+]
+
+[[package]]
+name = "python-crfsuite"
+version = "0.9.12"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b6/bb/946c0f96b4d3f7916f0558e19245d2248caebb3f470bcffae8fbf8d862e9/python_crfsuite-0.9.12.tar.gz", hash = "sha256:db37fccc3bd8f0c49c28a7697ca79c89d67b3fd5bf119122866169240ac4c480", size = 488298, upload-time = "2025-12-23T19:07:21.423Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ac/ea/29f9891b22903ab50ca90cc291c886803d519ee04bd62c2b3c601d3da41e/python_crfsuite-0.9.12-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2048d8768a0b4c6a6d9390e879a2b7a760bb57a7f2ba491316f5dc36f9cfd836", size = 318868, upload-time = "2025-12-23T19:06:51.119Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/74/61ccfdda6f98e8719dabf9fee18cf0660b0a33487cfad222fa5174f489a9/python_crfsuite-0.9.12-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5294008716b65606c4d416c3b2597ca14422359a4a84734ead239b29b95f2780", size = 1207620, upload-time = "2025-12-23T19:06:52.53Z" },
+ { url = "https://files.pythonhosted.org/packages/40/06/8e2219853504de660fb497a2f99e729f69b6dba76e44ffbdbb77035d3695/python_crfsuite-0.9.12-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f1641b9263c3cd1190711d0383d871b002ad325aa800fcf3c8583ef36f0bb07", size = 1246409, upload-time = "2025-12-23T19:06:54.463Z" },
+ { url = "https://files.pythonhosted.org/packages/66/dc/4c88933a6ab16ce797e8f30ff491f371bee4890ceb9b9a90192fdb278591/python_crfsuite-0.9.12-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f0e73d0a8859db0c3d1a7a3595a83810efc535d95cb79f2f675eb44ad7a7954a", size = 2166123, upload-time = "2025-12-23T19:06:56.3Z" },
+ { url = "https://files.pythonhosted.org/packages/74/52/f396a51f2089df8efdd8607ce9bdcb5228c0594dfe0e6c7b99fcbdbbc563/python_crfsuite-0.9.12-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:96f27f343ff7e7cb1e29a785d8ed4626a3470f8d42c41cda734dcbaede566722", size = 2267988, upload-time = "2025-12-23T19:06:57.591Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/2e/abcb7d53eda1355c5ef9aff6a620ecc21f527c593d0a7a7aace51a2fd2cd/python_crfsuite-0.9.12-cp313-cp313-win32.whl", hash = "sha256:385fda7f407be778f6a9440dffdeed3cdafc6f6923065a856e45997626283589", size = 282157, upload-time = "2025-12-23T19:06:59.048Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/da/282120f8d88def791610ba8310413c6ae3002f945f008979c3e440a6ad88/python_crfsuite-0.9.12-cp313-cp313-win_amd64.whl", hash = "sha256:21334c298318d4de057eacaad2ed179b7f63640e9cbd0c8141d656f58e7bd3f1", size = 303082, upload-time = "2025-12-23T19:07:00.14Z" },
+]
+
+[[package]]
+name = "python-dateutil"
+version = "2.9.0.post0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" },
+]
+
+[[package]]
+name = "python-discovery"
+version = "1.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "filelock" },
+ { name = "platformdirs" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0b/1a/cbbaf13b730abb0a16b964d984e19f2fe520c21a4dc664051359a3f5a9e7/python_discovery-1.4.2.tar.gz", hash = "sha256:8f3746c4b4968d22afbb97d36e1a0e5b66e6c0f297290f2e95f05b9b8bf18690", size = 70277, upload-time = "2026-06-11T16:10:42.383Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1a/82/a70006589557f267f15bd384c0642ad49f0d97b690c3a05b166b9dcbad3b/python_discovery-1.4.2-py3-none-any.whl", hash = "sha256:475803f53b7b2ed6e490e27373f9d8340f7d2eebf9acdaf645d7d714c97bb500", size = 33886, upload-time = "2026-06-11T16:10:41.192Z" },
+]
+
+[[package]]
+name = "python-dotenv"
+version = "1.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/82/ed/0301aeeac3e5353ef3d94b6ec08bbcabd04a72018415dcb29e588514bba8/python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3", size = 50135, upload-time = "2026-03-01T16:00:26.196Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" },
+]
+
+[[package]]
+name = "python-json-logger"
+version = "4.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f7/ff/3cc9165fd44106973cd7ac9facb674a65ed853494592541d339bdc9a30eb/python_json_logger-4.1.0.tar.gz", hash = "sha256:b396b9e3ed782b09ff9d6e4f1683d46c83ad0d35d2e407c09a9ebbf038f88195", size = 17573, upload-time = "2026-03-29T04:39:56.805Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/27/be/0631a861af4d1c875f096c07d34e9a63639560a717130e7a87cbc82b7e3f/python_json_logger-4.1.0-py3-none-any.whl", hash = "sha256:132994765cf75bf44554be9aa49b06ef2345d23661a96720262716438141b6b2", size = 15021, upload-time = "2026-03-29T04:39:55.266Z" },
+]
+
+[[package]]
+name = "python-multipart"
+version = "0.0.32"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/5b/42/55c32bb9b12693c092ad250a0e82edb5b31ddeda6eb772de5f308b3804ad/python_multipart-0.0.32.tar.gz", hash = "sha256:be54b7f3fa167bb83e4fcd936b887b708f4e57fe75911c02aebf53efaf8d938e", size = 46881, upload-time = "2026-06-04T16:18:58.647Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" },
+]
+
+[[package]]
+name = "pytz"
+version = "2026.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ff/46/dd499ec9038423421951e4fad73051febaa13d2df82b4064f87af8b8c0c3/pytz-2026.2.tar.gz", hash = "sha256:0e60b47b29f21574376f218fe21abc009894a2321ea16c6754f3cad6eb7cdd6a", size = 320861, upload-time = "2026-05-04T01:35:29.667Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/dd/96da98f892250475bdf2328112d7468abdd4acc7b902b6af23f4ed958ea0/pytz-2026.2-py2.py3-none-any.whl", hash = "sha256:04156e608bee23d3792fd45c94ae47fae1036688e75032eea2e3bf0323d1f126", size = 510141, upload-time = "2026-05-04T01:35:27.408Z" },
+]
+
+[[package]]
+name = "pyvi"
+version = "0.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "scikit-learn" },
+ { name = "sklearn-crfsuite" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/df/af/ce6ccb5458a7fed50127de2cac4ee47f88283062c859279e50de1e7cb7a9/pyvi-0.1.1.tar.gz", hash = "sha256:08abab45a83ad674cf6549725096b8530d49be51418aa2613ce1a0b45d17db2d", size = 8362510, upload-time = "2021-06-30T04:20:06.722Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2c/27/27ffee2663f42430cf3434da963f04224fec157b90799fe9e92a3564c1a6/pyvi-0.1.1-py2.py3-none-any.whl", hash = "sha256:e5e9e0e40ea1a556af12646c6b754f065342c86f1dfe84785ba926e1190c0bb7", size = 8455088, upload-time = "2021-06-30T04:20:01.957Z" },
+]
+
+[[package]]
+name = "pywin32"
+version = "312"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2d/41/12fbfd7f36ed2146d8bc9de96c2741296bf0d490b98508496cff322e274c/pywin32-312-cp313-cp313-win32.whl", hash = "sha256:7a27df850933d16a8eabfbaeb73d52b273e2da667f80d70b01a89d1f6828d02c", size = 6370184, upload-time = "2026-06-04T07:49:36.253Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/db/36a78e3403099d31d9746d13fdcde5accc43c1155f375a34d15983a479a7/pywin32-312-cp313-cp313-win_amd64.whl", hash = "sha256:c53e878d15a1c44788082bfe712a905433473aa38f86375b7cf8b45e3acbaaf9", size = 6914298, upload-time = "2026-06-04T07:49:38.876Z" },
+ { url = "https://files.pythonhosted.org/packages/84/37/c1697194092b76de9ed47ca124323f02c57ffc8a45c06f88a3d5acaf01eb/pywin32-312-cp313-cp313-win_arm64.whl", hash = "sha256:59aba5d5940842075343a5ddc6b11f1cdf0d1567fe745290359dfbcc7c2eb831", size = 6727640, upload-time = "2026-06-04T07:49:41.083Z" },
+]
+
+[[package]]
+name = "pywinpty"
+version = "3.0.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a9/ef/2d27f30c59a67be7025b2d7858c8c2d282b74d66544b2384730b82de74fd/pywinpty-3.0.5.tar.gz", hash = "sha256:61db0db063de9865adbea66db294628f8577f608d9764a4c7d3384eeacc4e81b", size = 16223484, upload-time = "2026-06-11T00:11:58.93Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b9/f4/2a464b9893cceb3b3f416356e94fdc3e1bca9476993927e4e6d99fe95382/pywinpty-3.0.5-cp313-cp313-win_amd64.whl", hash = "sha256:48db1b0ad9d0a1b81dcaaa7163a99a7808deaceb0c1b2344716dc1fc090c3c4c", size = 2090471, upload-time = "2026-06-10T23:42:11.071Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/2c/a138491a0afbdb50eb79395577bd326d4b0fbde7209417d1a8087ff2493a/pywinpty-3.0.5-cp313-cp313-win_arm64.whl", hash = "sha256:2c6008fb2d3774b48693b2fcb7f2cc317ade9dc581289a964ffeeaf81307c9b5", size = 815518, upload-time = "2026-06-10T23:42:02.363Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/15/54400049a380582acd1282665c70fcf11e0bd3713679aca78e24c3aae738/pywinpty-3.0.5-cp313-cp313t-win_amd64.whl", hash = "sha256:22ce1b780d89821cc52daf6eac0708af22d93d000ce9c7c07e37489db8594598", size = 2089920, upload-time = "2026-06-10T23:44:13.395Z" },
+ { url = "https://files.pythonhosted.org/packages/94/0c/6f24f3c0799f502259b24bdf841a99ad2b0d59df5c2525b4e2a286d14be2/pywinpty-3.0.5-cp313-cp313t-win_arm64.whl", hash = "sha256:9c2919a81bc5cfb09b86fc5a002112b2de95ca4304a07413cbeeb746a1307a5c", size = 814520, upload-time = "2026-06-10T23:43:28.588Z" },
+]
+
+[[package]]
+name = "pyyaml"
+version = "6.0.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" },
+ { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" },
+ { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" },
+ { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" },
+ { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" },
+ { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" },
+ { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" },
+]
+
+[[package]]
+name = "pyzmq"
+version = "27.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cffi", marker = "implementation_name == 'pypy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/04/0b/3c9baedbdf613ecaa7aa07027780b8867f57b6293b6ee50de316c9f3222b/pyzmq-27.1.0.tar.gz", hash = "sha256:ac0765e3d44455adb6ddbf4417dcce460fc40a05978c08efdf2948072f6db540", size = 281750, upload-time = "2025-09-08T23:10:18.157Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/92/e7/038aab64a946d535901103da16b953c8c9cc9c961dadcbf3609ed6428d23/pyzmq-27.1.0-cp312-abi3-macosx_10_15_universal2.whl", hash = "sha256:452631b640340c928fa343801b0d07eb0c3789a5ffa843f6e1a9cee0ba4eb4fc", size = 1306279, upload-time = "2025-09-08T23:08:03.807Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/5e/c3c49fdd0f535ef45eefcc16934648e9e59dace4a37ee88fc53f6cd8e641/pyzmq-27.1.0-cp312-abi3-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:1c179799b118e554b66da67d88ed66cd37a169f1f23b5d9f0a231b4e8d44a113", size = 895645, upload-time = "2025-09-08T23:08:05.301Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/e5/b0b2504cb4e903a74dcf1ebae157f9e20ebb6ea76095f6cfffea28c42ecd/pyzmq-27.1.0-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3837439b7f99e60312f0c926a6ad437b067356dc2bc2ec96eb395fd0fe804233", size = 652574, upload-time = "2025-09-08T23:08:06.828Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/9b/c108cdb55560eaf253f0cbdb61b29971e9fb34d9c3499b0e96e4e60ed8a5/pyzmq-27.1.0-cp312-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:43ad9a73e3da1fab5b0e7e13402f0b2fb934ae1c876c51d0afff0e7c052eca31", size = 840995, upload-time = "2025-09-08T23:08:08.396Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/bb/b79798ca177b9eb0825b4c9998c6af8cd2a7f15a6a1a4272c1d1a21d382f/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0de3028d69d4cdc475bfe47a6128eb38d8bc0e8f4d69646adfbcd840facbac28", size = 1642070, upload-time = "2025-09-08T23:08:09.989Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/80/2df2e7977c4ede24c79ae39dcef3899bfc5f34d1ca7a5b24f182c9b7a9ca/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_i686.whl", hash = "sha256:cf44a7763aea9298c0aa7dbf859f87ed7012de8bda0f3977b6fb1d96745df856", size = 2021121, upload-time = "2025-09-08T23:08:11.907Z" },
+ { url = "https://files.pythonhosted.org/packages/46/bd/2d45ad24f5f5ae7e8d01525eb76786fa7557136555cac7d929880519e33a/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:f30f395a9e6fbca195400ce833c731e7b64c3919aa481af4d88c3759e0cb7496", size = 1878550, upload-time = "2025-09-08T23:08:13.513Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/2f/104c0a3c778d7c2ab8190e9db4f62f0b6957b53c9d87db77c284b69f33ea/pyzmq-27.1.0-cp312-abi3-win32.whl", hash = "sha256:250e5436a4ba13885494412b3da5d518cd0d3a278a1ae640e113c073a5f88edd", size = 559184, upload-time = "2025-09-08T23:08:15.163Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/7f/a21b20d577e4100c6a41795842028235998a643b1ad406a6d4163ea8f53e/pyzmq-27.1.0-cp312-abi3-win_amd64.whl", hash = "sha256:9ce490cf1d2ca2ad84733aa1d69ce6855372cb5ce9223802450c9b2a7cba0ccf", size = 619480, upload-time = "2025-09-08T23:08:17.192Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c2/c012beae5f76b72f007a9e91ee9401cb88c51d0f83c6257a03e785c81cc2/pyzmq-27.1.0-cp312-abi3-win_arm64.whl", hash = "sha256:75a2f36223f0d535a0c919e23615fc85a1e23b71f40c7eb43d7b1dedb4d8f15f", size = 552993, upload-time = "2025-09-08T23:08:18.926Z" },
+ { url = "https://files.pythonhosted.org/packages/60/cb/84a13459c51da6cec1b7b1dc1a47e6db6da50b77ad7fd9c145842750a011/pyzmq-27.1.0-cp313-cp313-android_24_arm64_v8a.whl", hash = "sha256:93ad4b0855a664229559e45c8d23797ceac03183c7b6f5b4428152a6b06684a5", size = 1122436, upload-time = "2025-09-08T23:08:20.801Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/b6/94414759a69a26c3dd674570a81813c46a078767d931a6c70ad29fc585cb/pyzmq-27.1.0-cp313-cp313-android_24_x86_64.whl", hash = "sha256:fbb4f2400bfda24f12f009cba62ad5734148569ff4949b1b6ec3b519444342e6", size = 1156301, upload-time = "2025-09-08T23:08:22.47Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/ad/15906493fd40c316377fd8a8f6b1f93104f97a752667763c9b9c1b71d42d/pyzmq-27.1.0-cp313-cp313t-macosx_10_15_universal2.whl", hash = "sha256:e343d067f7b151cfe4eb3bb796a7752c9d369eed007b91231e817071d2c2fec7", size = 1341197, upload-time = "2025-09-08T23:08:24.286Z" },
+ { url = "https://files.pythonhosted.org/packages/14/1d/d343f3ce13db53a54cb8946594e567410b2125394dafcc0268d8dda027e0/pyzmq-27.1.0-cp313-cp313t-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:08363b2011dec81c354d694bdecaef4770e0ae96b9afea70b3f47b973655cc05", size = 897275, upload-time = "2025-09-08T23:08:26.063Z" },
+ { url = "https://files.pythonhosted.org/packages/69/2d/d83dd6d7ca929a2fc67d2c3005415cdf322af7751d773524809f9e585129/pyzmq-27.1.0-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d54530c8c8b5b8ddb3318f481297441af102517602b569146185fa10b63f4fa9", size = 660469, upload-time = "2025-09-08T23:08:27.623Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/cd/9822a7af117f4bc0f1952dbe9ef8358eb50a24928efd5edf54210b850259/pyzmq-27.1.0-cp313-cp313t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6f3afa12c392f0a44a2414056d730eebc33ec0926aae92b5ad5cf26ebb6cc128", size = 847961, upload-time = "2025-09-08T23:08:29.672Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/12/f003e824a19ed73be15542f172fd0ec4ad0b60cf37436652c93b9df7c585/pyzmq-27.1.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c65047adafe573ff023b3187bb93faa583151627bc9c51fc4fb2c561ed689d39", size = 1650282, upload-time = "2025-09-08T23:08:31.349Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/4a/e82d788ed58e9a23995cee70dbc20c9aded3d13a92d30d57ec2291f1e8a3/pyzmq-27.1.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:90e6e9441c946a8b0a667356f7078d96411391a3b8f80980315455574177ec97", size = 2024468, upload-time = "2025-09-08T23:08:33.543Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/94/2da0a60841f757481e402b34bf4c8bf57fa54a5466b965de791b1e6f747d/pyzmq-27.1.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:add071b2d25f84e8189aaf0882d39a285b42fa3853016ebab234a5e78c7a43db", size = 1885394, upload-time = "2025-09-08T23:08:35.51Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/6f/55c10e2e49ad52d080dc24e37adb215e5b0d64990b57598abc2e3f01725b/pyzmq-27.1.0-cp313-cp313t-win32.whl", hash = "sha256:7ccc0700cfdf7bd487bea8d850ec38f204478681ea02a582a8da8171b7f90a1c", size = 574964, upload-time = "2025-09-08T23:08:37.178Z" },
+ { url = "https://files.pythonhosted.org/packages/87/4d/2534970ba63dd7c522d8ca80fb92777f362c0f321900667c615e2067cb29/pyzmq-27.1.0-cp313-cp313t-win_amd64.whl", hash = "sha256:8085a9fba668216b9b4323be338ee5437a235fe275b9d1610e422ccc279733e2", size = 641029, upload-time = "2025-09-08T23:08:40.595Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/fa/f8aea7a28b0641f31d40dea42d7ef003fded31e184ef47db696bc74cd610/pyzmq-27.1.0-cp313-cp313t-win_arm64.whl", hash = "sha256:6bb54ca21bcfe361e445256c15eedf083f153811c37be87e0514934d6913061e", size = 561541, upload-time = "2025-09-08T23:08:42.668Z" },
+]
+
+[[package]]
+name = "quack-kernels"
+version = "0.6.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "apache-tvm-ffi" },
+ { name = "einops" },
+ { name = "nvidia-cutlass-dsl" },
+ { name = "torch" },
+ { name = "torch-c-dlpack-ext" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/75/5c/67c4a0d54cbb8d4d05af73132d620d1900a193b33e0d5ea6a25829b941d2/quack_kernels-0.6.4.tar.gz", hash = "sha256:8bf08a0e1a85aecc892ce84f4b6ed7abb6a44ec7c68b83040abe174bc8c2b936", size = 845485, upload-time = "2026-08-07T23:25:31.94Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/28/f5/36ec7e075e90bd92bdf30fee0606f189348c2bbb0af6aebaf7382927c622/quack_kernels-0.6.4-py3-none-any.whl", hash = "sha256:e77c5d1f1299b0b38487fe8737df6c6975daa16bca7f7eb883bd1a74d09e7e78", size = 728458, upload-time = "2026-08-07T23:25:30.549Z" },
+]
+
+[[package]]
+name = "ray"
+version = "2.56.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "filelock" },
+ { name = "jsonschema" },
+ { name = "msgpack" },
+ { name = "packaging" },
+ { name = "protobuf" },
+ { name = "pyyaml" },
+ { name = "requests" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/65/86/29c9444c9f11f0bf3d4da75d3dc7a27c0fb86f8604beea6513968dc2df26/ray-2.56.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:992047f50473b5bfea74c8f528f999968e0b4bc735af23ad476a0f4e04741aea", size = 66289930, upload-time = "2026-06-29T20:51:20.79Z" },
+ { url = "https://files.pythonhosted.org/packages/59/59/3c4d533a92e99b12b9d2e8690e41d7ace269d77deecb0aef753df7924b9e/ray-2.56.0-cp313-cp313-manylinux2014_aarch64.whl", hash = "sha256:a0e9cfe92c88ab74abca23923c15a592f49bc7617ffdda4190daca7785a7c4f6", size = 73252460, upload-time = "2026-06-29T20:51:26.216Z" },
+ { url = "https://files.pythonhosted.org/packages/59/ba/285d409253b332af91884daa4b0a3be3ce799682aa2c1e0f8dbbafaf1da4/ray-2.56.0-cp313-cp313-manylinux2014_x86_64.whl", hash = "sha256:54d2725f8b65d9615c933fec5ec62e54a67b8f2e14286026fb014606253670ef", size = 74130685, upload-time = "2026-06-29T20:51:31.59Z" },
+]
+
+[[package]]
+name = "referencing"
+version = "0.37.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "rpds-py" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" },
+]
+
+[[package]]
+name = "regex"
+version = "2026.6.28"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f1/05/e4f219230e11e774a6c9987d2ab0d0c6b8573e13a17e143d0015bee710ef/regex-2026.6.28.tar.gz", hash = "sha256:3cb4b6c5cb3060cc31efdc1fbb27c25fb9b29044afd87e40601a1c4d9db54342", size = 416101, upload-time = "2026-06-28T19:56:55.302Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b6/53/d5c1b3cc0b5a0c985563ad6fac93d73ff2b300cb84342d89f044625d6bc7/regex-2026.6.28-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:b295a83426e0e44e9e60fde99789e181bd26788a1890ae7fe2a24c69bb6246ca", size = 490329, upload-time = "2026-06-28T19:54:35.775Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/9f/0c3503e819e91ca0e7a901a8e989ebf840ac7c7aea20b1fc7f31b6759f77/regex-2026.6.28-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0c31665c0deb5c111557a1cac8c27bd5629e2f9e7fd5058900a03576c33b601c", size = 292039, upload-time = "2026-06-28T19:54:37.977Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/7f/cd004e13fcad23b3794a82307dfd222e6365eb7f598bd3caab148a830bff/regex-2026.6.28-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6bf295f2c59de77d1ea7de053607ae4dc9ceb3d57bbb6c7ec51ef4acc4ccff94", size = 289488, upload-time = "2026-06-28T19:54:39.545Z" },
+ { url = "https://files.pythonhosted.org/packages/73/4c/293fb34586fbcdc47eac436069e9c11f71fae5dadfd4889b475d7d2e5f7a/regex-2026.6.28-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:17c077586770f67e05bbffeba07fbee6b2b22244f4d4caf8d94e59d574befe04", size = 796772, upload-time = "2026-06-28T19:54:41.347Z" },
+ { url = "https://files.pythonhosted.org/packages/92/fa/c0cd1a90b7d12d9dc155cfc8bdea8df9720988ea5b07e8fa1eccbd0ab2dd/regex-2026.6.28-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0e6cb5a61486f9062397d2e189573b39d38ecfaed698fd9fb6e2756a8ebb8762", size = 865467, upload-time = "2026-06-28T19:54:43.485Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/db/0b479973046d005a1eaea299d5d536aeecb9488a16d9cbb8286338102e2d/regex-2026.6.28-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e86e91a2664f44c3a4e363a7d78fb17c27d5046882e30ea5a877f5e89b28d2ba", size = 912345, upload-time = "2026-06-28T19:54:46.091Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/5b/d65adfbd02f32212431bca1f06d1e2eb763a20b12978b454bafaf23dacb7/regex-2026.6.28-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4dfd1331c49233998d84fc5f1f4436cf7a435a7655f6cf0f490229bb5c7254e5", size = 801291, upload-time = "2026-06-28T19:54:48.3Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/09/2103686defaf9a0a31c1663782359d5b45f42524c64cca681f5481e44a5e/regex-2026.6.28-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cadea12805a1bce0b091c302b814207be26fb60a9c0e7f9ad2f9e21790a429fe", size = 777106, upload-time = "2026-06-28T19:54:50.326Z" },
+ { url = "https://files.pythonhosted.org/packages/85/5a/b57593c0aa23ed269ec332fbcf07852abcb6b746e811d9464e0d09b4e25f/regex-2026.6.28-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5f2c1682b67ad5d2376498f2a5a2a8f782fa2e4a06d0465b5e357799806e8a20", size = 785175, upload-time = "2026-06-28T19:54:52.172Z" },
+ { url = "https://files.pythonhosted.org/packages/79/59/c36e756ad29bf14d7b6c6d7138952476b21f6160286cedb98ac13481c993/regex-2026.6.28-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:64e142eb55e84868087da1375d7c36ff97d55010951849f515322a91d5fef1b4", size = 860186, upload-time = "2026-06-28T19:54:54.11Z" },
+ { url = "https://files.pythonhosted.org/packages/61/66/49808aea0da9649c300139360708fb91b7144be1f962fcebf96755fde948/regex-2026.6.28-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:abb4daabe7be63273787a62dfd6164dadf8f7a63fbec3d2730e5e5e7126d858c", size = 765754, upload-time = "2026-06-28T19:54:56.04Z" },
+ { url = "https://files.pythonhosted.org/packages/be/c5/52bbd436cf2200decdf48825fa38363eaaeebb77011ea9928a1ef9e0b9f2/regex-2026.6.28-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:ec2b2ad00ab8c16a2798cc8db80c53c4d5b8b3a2441f6cbaef06625f5ca25854", size = 852085, upload-time = "2026-06-28T19:54:57.988Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/c3/0390b66e3019497143fe768b3ba567b64d8b24f3812d09506deb86f4a0f0/regex-2026.6.28-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:bfc9677982c914d9085b8e1c3b3ae6e88f139fb56531c2416d6c8f338093c22b", size = 789600, upload-time = "2026-06-28T19:54:59.977Z" },
+ { url = "https://files.pythonhosted.org/packages/88/fd/ab5b03653a244975069fed93d73f4f5f7484c03a84cedb238292510d7182/regex-2026.6.28-cp313-cp313-win32.whl", hash = "sha256:bf54bc693fc4e0530e666ba5ec4bcba14dbe8f66b7cfc15c27317d1a6e40b9a5", size = 267088, upload-time = "2026-06-28T19:55:02.159Z" },
+ { url = "https://files.pythonhosted.org/packages/68/55/21022f7d3143210ae8d4ff905c45306237b657375cc0b97883f49db3d423/regex-2026.6.28-cp313-cp313-win_amd64.whl", hash = "sha256:e128feaf65bf3d9eb91bec92322a8f7e4835e9c798f3e9ea4b69f4def85620e3", size = 277680, upload-time = "2026-06-28T19:55:04.185Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/99/7f664804f1aef924542b0b233996b78b3e4d0a52d9951358aac99f129f51/regex-2026.6.28-cp313-cp313-win_arm64.whl", hash = "sha256:695873e0ea8d3815ea9e92e2c68faf039cc450e2c0a62a31afe2049eb11be767", size = 277017, upload-time = "2026-06-28T19:55:06.29Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/e1/9eb83518e159d719fd681c4932dc2aaff855ce72451e1d05d69466f25a96/regex-2026.6.28-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:189dbf9fc4252d9f1352bf4bd1bef885edb6cc4b7341df202a65f821aaa3891c", size = 494195, upload-time = "2026-06-28T19:55:08.292Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/e2/e259c5f2f7be269d0e2fb54275c1fa6a13fb47019f389c3f3ae457447825/regex-2026.6.28-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:9277a4c6503390aa39cb4483b87ec0384faee0850a23b5cea33d008b5d8d83f1", size = 293976, upload-time = "2026-06-28T19:55:10.014Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/4e/9bdf444014d22b045d0c82ca114fac7e07a597b5b5331b7c4ce6328426e2/regex-2026.6.28-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:17eddca4e8ea9af0b5739314776cdf0172a49731ab61f2e1ea66e066ddd46c97", size = 292340, upload-time = "2026-06-28T19:55:11.88Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/3a/f49b11e59cbfe187ace0053a460bd72a0169b8cd52e7db9421a074ce7a43/regex-2026.6.28-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e4466b8641e00c697aab5a73150150d2b2ea96b131c595691f42031abafd9f4d", size = 811704, upload-time = "2026-06-28T19:55:13.612Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/fb/ad04c39e149bf8b6cf357df5fff78341733ec366780a00c803a36735818c/regex-2026.6.28-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9cfcd4b0bdcf768c498415c170d1ed2a25a99bf0b65fa253bbd02f68ceba6475", size = 871157, upload-time = "2026-06-28T19:55:15.797Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/64/0e5ba31c11eb8ef7aac19a690c1211fc9aa9990caf09565785ebb0081b9a/regex-2026.6.28-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:80c7adf1ef647f6b1e8aa2ca280e517174cd08bdf7a2e412cdfb68bd6a0917cb", size = 917287, upload-time = "2026-06-28T19:55:18.692Z" },
+ { url = "https://files.pythonhosted.org/packages/11/75/6b78df2b858c2fcbbc4858fdc3f2975cf2703be374b2842db7d2c32591a7/regex-2026.6.28-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a043f5770e82283a22aed4cefef1a4e0f9dd8fd7184cb6ce0ad2e579e2134a9e", size = 816333, upload-time = "2026-06-28T19:55:20.973Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/01/ecfe665a3694d5eda9f3ec686c856438ada0943947b6005e90556a1e2cdf/regex-2026.6.28-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3bd630a8dba06b55254ea5ee862194edab52ec783100d2ef1cd15a9c512fee27", size = 785518, upload-time = "2026-06-28T19:55:23.003Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/0a/88f9cd88ff1e82881605c4ffd62d77ee67d051232cfe6f8e9a64b86cf0e8/regex-2026.6.28-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:b77207e3cee13086f1906a6a2a12b41244c577e8ad9370d4b35ae1d548d354f3", size = 801371, upload-time = "2026-06-28T19:55:24.888Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/97/601483732f93275482ceb9fed57813dfed7c47d3a019db6ec4a3bb6e23e0/regex-2026.6.28-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:6de82c268e5d101ee9e3ffd869924aa9a371e3a21e752cf4fa17b6ce50d219f7", size = 866517, upload-time = "2026-06-28T19:55:27.232Z" },
+ { url = "https://files.pythonhosted.org/packages/81/ed/385c2a0351b994a693453c1d1a6e9af9eb35db3c9460d76b5078acd70c62/regex-2026.6.28-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:b15859e3908544fb99cf47341dcf0bfd089147d258c4c4d8a29e5b087f8085cb", size = 772834, upload-time = "2026-06-28T19:55:29.154Z" },
+ { url = "https://files.pythonhosted.org/packages/06/bc/bbf4a5b3b29770d7f307d3c28b5b1bca0105b0cb424be0a4eb1339bc92cf/regex-2026.6.28-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:c91487a917edd48a1ea646fdf60d7936d304f0e686fa7ea8326e47efca51d816", size = 856606, upload-time = "2026-06-28T19:55:32.186Z" },
+ { url = "https://files.pythonhosted.org/packages/28/26/51d74fff82f682819979249f8d700267108ba5dc4eb284b0e11b9c85e4b3/regex-2026.6.28-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c4ac65f3e3a99fd8f3a4a74e7a6610acd1ce9dfe9b8a03d346a4922380d68aeb", size = 803475, upload-time = "2026-06-28T19:55:34.328Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/3e/6be10cefdc813533fe604dbf5d3c77d2638e7ee658b2749ebadc113b6b2e/regex-2026.6.28-cp313-cp313t-win32.whl", hash = "sha256:3f6316f258bc7e6c9c2acbe9954947bbd397a81be3742a637a555f1855d6618d", size = 269126, upload-time = "2026-06-28T19:55:36.565Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/3c/32cda905ea1a6eeeb798291c294d8ec66ee0efe0cdba28b061e248b1d396/regex-2026.6.28-cp313-cp313t-win_amd64.whl", hash = "sha256:1484bdd6fba28422df9b5ebb04055b2e1b680e8e4f08490bb21ff0f3cc50d0ab", size = 279961, upload-time = "2026-06-28T19:55:38.456Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/b9/69f4e5cd6fbe0bb420cb2dbae441ca118f2495bdda522a74da75aa9829e7/regex-2026.6.28-cp313-cp313t-win_arm64.whl", hash = "sha256:3f15020f0b69cafe57baa067ff65b29acef68ff6b1670a53bef1ca11d708e02d", size = 279266, upload-time = "2026-06-28T19:55:40.62Z" },
+]
+
+[[package]]
+name = "requests"
+version = "2.34.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "charset-normalizer" },
+ { name = "idna" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" },
+]
+
+[[package]]
+name = "rfc3339-validator"
+version = "0.1.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/ea/a9387748e2d111c3c2b275ba970b735e04e15cdb1eb30693b6b5708c4dbd/rfc3339_validator-0.1.4.tar.gz", hash = "sha256:138a2abdf93304ad60530167e51d2dfb9549521a836871b88d7f4695d0022f6b", size = 5513, upload-time = "2021-05-12T16:37:54.178Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7b/44/4e421b96b67b2daff264473f7465db72fbdf36a07e05494f50300cc7b0c6/rfc3339_validator-0.1.4-py2.py3-none-any.whl", hash = "sha256:24f6ec1eda14ef823da9e36ec7113124b39c04d50a4d3d3a3c2859577e7791fa", size = 3490, upload-time = "2021-05-12T16:37:52.536Z" },
+]
+
+[[package]]
+name = "rfc3986-validator"
+version = "0.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/da/88/f270de456dd7d11dcc808abfa291ecdd3f45ff44e3b549ffa01b126464d0/rfc3986_validator-0.1.1.tar.gz", hash = "sha256:3d44bde7921b3b9ec3ae4e3adca370438eccebc676456449b145d533b240d055", size = 6760, upload-time = "2019-10-28T16:00:19.144Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/51/17023c0f8f1869d8806b979a2bffa3f861f26a3f1a66b094288323fba52f/rfc3986_validator-0.1.1-py2.py3-none-any.whl", hash = "sha256:2f235c432ef459970b4306369336b9d5dbdda31b510ca1e327636e01f528bfa9", size = 4242, upload-time = "2019-10-28T16:00:13.976Z" },
+]
+
+[[package]]
+name = "rfc3987-syntax"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "lark" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2c/06/37c1a5557acf449e8e406a830a05bf885ac47d33270aec454ef78675008d/rfc3987_syntax-1.1.0.tar.gz", hash = "sha256:717a62cbf33cffdd16dfa3a497d81ce48a660ea691b1ddd7be710c22f00b4a0d", size = 14239, upload-time = "2025-07-18T01:05:05.015Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/71/44ce230e1b7fadd372515a97e32a83011f906ddded8d03e3c6aafbdedbb7/rfc3987_syntax-1.1.0-py3-none-any.whl", hash = "sha256:6c3d97604e4c5ce9f714898e05401a0445a641cfa276432b0a648c80856f6a3f", size = 8046, upload-time = "2025-07-18T01:05:03.843Z" },
+]
+
+[[package]]
+name = "rich"
+version = "15.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown-it-py" },
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" },
+]
+
+[[package]]
+name = "rich-toolkit"
+version = "0.20.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "rich" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/29/63/3e427c62f1992945c997d4ec31e2fcb37d26aadbe5aa44ae5b29f7f64d26/rich_toolkit-0.20.1.tar.gz", hash = "sha256:c7336ae281f435c785acecaedc4b71d4b663dc73d9c8079fea96372527e822a4", size = 203473, upload-time = "2026-06-05T08:56:57.679Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/00/88/309f07d08155da2ba1d5ceb42d270fb42fbe34a807684543e3ffc10fe713/rich_toolkit-0.20.1-py3-none-any.whl", hash = "sha256:2a6d5f8e15759b9eba5a9ee63da10b275359ead20e5a0fc92bd5b4dbae8ce4bf", size = 35525, upload-time = "2026-06-05T08:56:58.586Z" },
+]
+
+[[package]]
+name = "rignore"
+version = "0.7.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e5/f5/8bed2310abe4ae04b67a38374a4d311dd85220f5d8da56f47ae9361be0b0/rignore-0.7.6.tar.gz", hash = "sha256:00d3546cd793c30cb17921ce674d2c8f3a4b00501cb0e3dd0e82217dbeba2671", size = 57140, upload-time = "2025-11-05T21:41:21.968Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b7/8a/a4078f6e14932ac7edb171149c481de29969d96ddee3ece5dc4c26f9e0c3/rignore-0.7.6-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:2bdab1d31ec9b4fb1331980ee49ea051c0d7f7bb6baa28b3125ef03cdc48fdaf", size = 883057, upload-time = "2025-11-05T20:42:42.741Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/8f/f8daacd177db4bf7c2223bab41e630c52711f8af9ed279be2058d2fe4982/rignore-0.7.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:90f0a00ce0c866c275bf888271f1dc0d2140f29b82fcf33cdbda1e1a6af01010", size = 820150, upload-time = "2025-11-05T20:42:26.545Z" },
+ { url = "https://files.pythonhosted.org/packages/36/31/b65b837e39c3f7064c426754714ac633b66b8c2290978af9d7f513e14aa9/rignore-0.7.6-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c1ad295537041dc2ed4b540fb1a3906bd9ede6ccdad3fe79770cd89e04e3c73c", size = 897406, upload-time = "2025-11-05T20:40:53.854Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/58/1970ce006c427e202ac7c081435719a076c478f07b3a23f469227788dc23/rignore-0.7.6-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f782dbd3a65a5ac85adfff69e5c6b101285ef3f845c3a3cae56a54bebf9fe116", size = 874050, upload-time = "2025-11-05T20:41:08.922Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/00/eb45db9f90137329072a732273be0d383cb7d7f50ddc8e0bceea34c1dfdf/rignore-0.7.6-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:65cece3b36e5b0826d946494734c0e6aaf5a0337e18ff55b071438efe13d559e", size = 1167835, upload-time = "2025-11-05T20:41:24.997Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/f1/6f1d72ddca41a64eed569680587a1236633587cc9f78136477ae69e2c88a/rignore-0.7.6-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d7e4bb66c13cd7602dc8931822c02dfbbd5252015c750ac5d6152b186f0a8be0", size = 941945, upload-time = "2025-11-05T20:41:40.628Z" },
+ { url = "https://files.pythonhosted.org/packages/48/6f/2f178af1c1a276a065f563ec1e11e7a9e23d4996fd0465516afce4b5c636/rignore-0.7.6-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:297e500c15766e196f68aaaa70e8b6db85fa23fdc075b880d8231fdfba738cd7", size = 959067, upload-time = "2025-11-05T20:42:11.09Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/db/423a81c4c1e173877c7f9b5767dcaf1ab50484a94f60a0b2ed78be3fa765/rignore-0.7.6-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a07084211a8d35e1a5b1d32b9661a5ed20669970b369df0cf77da3adea3405de", size = 984438, upload-time = "2025-11-05T20:41:55.443Z" },
+ { url = "https://files.pythonhosted.org/packages/31/eb/c4f92cc3f2825d501d3c46a244a671eb737fc1bcf7b05a3ecd34abb3e0d7/rignore-0.7.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:181eb2a975a22256a1441a9d2f15eb1292839ea3f05606620bd9e1938302cf79", size = 1078365, upload-time = "2025-11-05T21:40:15.148Z" },
+ { url = "https://files.pythonhosted.org/packages/26/09/99442f02794bd7441bfc8ed1c7319e890449b816a7493b2db0e30af39095/rignore-0.7.6-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:7bbcdc52b5bf9f054b34ce4af5269df5d863d9c2456243338bc193c28022bd7b", size = 1139066, upload-time = "2025-11-05T21:40:32.771Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/88/bcfc21e520bba975410e9419450f4b90a2ac8236b9a80fd8130e87d098af/rignore-0.7.6-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:f2e027a6da21a7c8c0d87553c24ca5cc4364def18d146057862c23a96546238e", size = 1118036, upload-time = "2025-11-05T21:40:49.646Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/25/d37215e4562cda5c13312636393aea0bafe38d54d4e0517520a4cc0753ec/rignore-0.7.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ee4a18b82cbbc648e4aac1510066682fe62beb5dc88e2c67c53a83954e541360", size = 1127550, upload-time = "2025-11-05T21:41:07.648Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/76/a264ab38bfa1620ec12a8ff1c07778da89e16d8c0f3450b0333020d3d6dc/rignore-0.7.6-cp313-cp313-win32.whl", hash = "sha256:a7d7148b6e5e95035d4390396895adc384d37ff4e06781a36fe573bba7c283e5", size = 646097, upload-time = "2025-11-05T21:41:53.201Z" },
+ { url = "https://files.pythonhosted.org/packages/62/44/3c31b8983c29ea8832b6082ddb1d07b90379c2d993bd20fce4487b71b4f4/rignore-0.7.6-cp313-cp313-win_amd64.whl", hash = "sha256:b037c4b15a64dced08fc12310ee844ec2284c4c5c1ca77bc37d0a04f7bff386e", size = 726170, upload-time = "2025-11-05T21:41:38.131Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/41/e26a075cab83debe41a42661262f606166157df84e0e02e2d904d134c0d8/rignore-0.7.6-cp313-cp313-win_arm64.whl", hash = "sha256:e47443de9b12fe569889bdbe020abe0e0b667516ee2ab435443f6d0869bd2804", size = 656184, upload-time = "2025-11-05T21:41:27.396Z" },
+]
+
+[[package]]
+name = "rouge-score"
+version = "0.1.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "absl-py" },
+ { name = "nltk" },
+ { name = "numpy" },
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e2/c5/9136736c37022a6ad27fea38f3111eb8f02fe75d067f9a985cc358653102/rouge_score-0.1.2.tar.gz", hash = "sha256:c7d4da2683e68c9abf0135ef915d63a46643666f848e558a1b9f7ead17ff0f04", size = 17400, upload-time = "2022-07-22T22:46:22.909Z" }
+
+[[package]]
+name = "rpds-py"
+version = "2026.6.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/2a/9618a122aeb2a169a28b03889a2995fe297588964333d4a7d67bdf46e147/rpds_py-2026.6.3.tar.gz", hash = "sha256:1cebd1337c242e4ec2293e541f712b2da849b29f48f0c293684b71c0632625d4", size = 64051, upload-time = "2026-06-30T07:17:53.009Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a4/9e/b818ee580026ec578138e961027a68820c40afeb1ec8f6819b54fb99e196/rpds_py-2026.6.3-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:3cfe765c1da0072636ca06628261e0ea05688e160d5c8a03e0217c3854037223", size = 343012, upload-time = "2026-06-30T07:15:36.005Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/6b/686d9dc4359a8f163cfbbf89ee0b4e586431de22fe8248edb63a8cf50d49/rpds_py-2026.6.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f4d78253f6996be4901669ad25319f842f740eccf4d58e3c7f3dd39e6dde1d8f", size = 338203, upload-time = "2026-06-30T07:15:37.462Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/9b/069aa329940f8207615e091f5eedbbd40e1e15eac68a0790fd05ccdf796c/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:54f45a148e28767bf343d33a684693c70e451c6f4c0e9904709a723fafbdfc1f", size = 367984, upload-time = "2026-06-30T07:15:39.008Z" },
+ { url = "https://files.pythonhosted.org/packages/14/db/34c203e4becff3703e4d3bc121842c00b8689197f398161203a880052f4e/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:842e7b070435622248c7a2c44ae53fa1440e073cc3023bc919fed570884097a7", size = 374815, upload-time = "2026-06-30T07:15:40.253Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/7d/8071067d2cc453d916ad836e828c943f575e8a44612537759002a1e07381/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8020133a74bd81b4572dd8e4be028a6b1ebcd70e6726edc3918008c08bee6ee6", size = 490545, upload-time = "2026-06-30T07:15:41.729Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/42/da06c5aa8f0484ff07f270787434204d9f4535e2f8c3b51ed402267e63c3/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cdc7e35386f3847df728fbcb5e887e2d79c19e2fa1eba9e51b6621d23e3243af", size = 382828, upload-time = "2026-06-30T07:15:43.327Z" },
+ { url = "https://files.pythonhosted.org/packages/57/d7/fe978efc2ae50abe48eb7464668ea99f53c010c60aeebb7b35ad27f23661/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:acac386b453c2516111b50985d60ce46e7fadb5ea71ae7b25f4c946935bf27cf", size = 365678, upload-time = "2026-06-30T07:15:44.992Z" },
+ { url = "https://files.pythonhosted.org/packages/69/9d/1d8922e1990b2a6eb532b6ff53d3e73d2b3bbffc84116c75826bee73dfc6/rpds_py-2026.6.3-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:425560c6fa0415f27261727bb20bd097568485e5eb0c121f1949417d1c516885", size = 377811, upload-time = "2026-06-30T07:15:46.523Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/3d/198dceafb4fb034a6a47347e1b0735d34e0bd4a50be4e898d408ee66cb14/rpds_py-2026.6.3-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a550fb4950a06dde3beb4721f5ad4b25bf4513784665b0a8522c792e2bd822a4", size = 395382, upload-time = "2026-06-30T07:15:47.955Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/f1/13968e49655d40b6b19d8b9140296bbc6f1d86b3f0f6c346cf9f1adddf4b/rpds_py-2026.6.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4f4bca01b63096f606e095734dd56e74e175f94cfbf24ff3d63281cec61f7bb7", size = 543832, upload-time = "2026-06-30T07:15:49.33Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/ab/289bcb1b90bd3e40a2900c561fa0e2087345ecbb094f0b870f2345142b7c/rpds_py-2026.6.3-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ccffae9a092a00deb7efd545fe5e2c33c33b88e7c054337e9a74c179347d0b7d", size = 611011, upload-time = "2026-06-30T07:15:50.847Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/16/5043105e679436ccfbc8e5e0dd2d663ed18a8b8113515fd06a5e5d77c83e/rpds_py-2026.6.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1cf01971c4f2c5553b772a542e4aaf191789cd331bc2cd4ff0e6e65ba49e1e97", size = 572431, upload-time = "2026-06-30T07:15:52.394Z" },
+ { url = "https://files.pythonhosted.org/packages/85/ed/adab103321c0a6565d5ae1c2998349bc3ee175b82ccc5ae8fc04cc413075/rpds_py-2026.6.3-cp313-cp313-win32.whl", hash = "sha256:8c3d1e9c15b9d51ca0391e13da1a25a0a4df3c58a37c9dc368e0736cf7f69df0", size = 201710, upload-time = "2026-06-30T07:15:53.894Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/ed/a03b09668e74e5dabbf2e211f6468e1820c0552f7b0500082da31841bf7b/rpds_py-2026.6.3-cp313-cp313-win_amd64.whl", hash = "sha256:9250a9a0a6fd4648b3f868da8d91a4c52b5811a62df58e753d50ae4454a36f80", size = 219454, upload-time = "2026-06-30T07:15:55.25Z" },
+ { url = "https://files.pythonhosted.org/packages/27/17/b8642c12930b71bc2b25831f6708ccf0f75abcd11883932ec9ce54ba3a78/rpds_py-2026.6.3-cp313-cp313-win_arm64.whl", hash = "sha256:900a67df3fd1660b035a4761c4ce73c382ea6b35f90f9863c36c6fd8bf8b09bb", size = 215063, upload-time = "2026-06-30T07:15:56.573Z" },
+]
+
+[[package]]
+name = "ruff"
+version = "0.15.20"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/43/dc/35b341fc554ba02f217fc10da57d1a75168cfbcf75b0ef2202176d4c4f2d/ruff-0.15.20.tar.gz", hash = "sha256:1416eb04349192646b54de98f146c4f59afe37d0decfc02c3cbbf396f3a28566", size = 4755489, upload-time = "2026-06-25T17:20:37.578Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/94/d9/2d5014f0253ba541d2061d9fa7193f48e941c8b21bb88a7ff9bbe0bd0596/ruff-0.15.20-py3-none-linux_armv6l.whl", hash = "sha256:00e188c53e499c3c1637f73c91dcf2fb56d576cab76ce1be50a27c4e80e37078", size = 10839665, upload-time = "2026-06-25T17:19:44.702Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/d3/ac1798ba64f670698867fcfc591d50e7e421bef137db564858f619a30fcf/ruff-0.15.20-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:9ebd1fd9b9c95fc0bd7b2761aebec1f030013d2e193a2901b224af68fe47251b", size = 11208649, upload-time = "2026-06-25T17:19:48.787Z" },
+ { url = "https://files.pythonhosted.org/packages/47/47/d3ac899991202095dfcf3d5176be4272642be3cf981a2f1a30f72a2afb95/ruff-0.15.20-py3-none-macosx_11_0_arm64.whl", hash = "sha256:c5b16cdd67ca108185cd36dce98c576350c03b1660a751de725fb049193a0632", size = 10622638, upload-time = "2026-06-25T17:19:51.354Z" },
+ { url = "https://files.pythonhosted.org/packages/33/13/4e043fe30aa94d4ff5213a9881fc296d12960f5971b234a5263fdc225312/ruff-0.15.20-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3413bb3c3d2ca6a8208f1f4809cd2dca3c6de6d0b491c0e70847672bde6e6efd", size = 10984227, upload-time = "2026-06-25T17:19:54.044Z" },
+ { url = "https://files.pythonhosted.org/packages/76/e6/92e7bf40388bc5800073b96564f56264f7e48bfd1a498f5ced6ae6d5a769/ruff-0.15.20-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bd7ec42b3bb3da066488db093308a69c4ac5ee6d2af333a86ba6e2eb2e7dd44b", size = 10622882, upload-time = "2026-06-25T17:19:57.037Z" },
+ { url = "https://files.pythonhosted.org/packages/13/7a/43460be3f24495a3aa46d4b16873e2c4941b3b5f0b00cf88c03b7b94b339/ruff-0.15.20-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e1a36ad0eb77fba9aabfb69ede54de6f376d04ac18ebea022847046d340a8267", size = 11474808, upload-time = "2026-06-25T17:20:00.357Z" },
+ { url = "https://files.pythonhosted.org/packages/27/a0/f37077884873221c6b33b4ab49eb18f9f88e54a16a25a5bca59bef46dd66/ruff-0.15.20-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b6df3b1e4610432f0386dba04d853b5f08cbbc903410c6fcc02f620f05aff53c", size = 12293094, upload-time = "2026-06-25T17:20:03.446Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/74/165545b60256a9704c21ac0ec4a0d07933b320812f9584836c9f4aca4292/ruff-0.15.20-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e89f198a1ea6ef0d727c1cf16088bc91a6cb0ab947dedc966715691647186eae", size = 11526176, upload-time = "2026-06-25T17:20:06.301Z" },
+ { url = "https://files.pythonhosted.org/packages/86/b1/a976a136d40ade83ce743578399865f57001003a409acadc0ecbb3051082/ruff-0.15.20-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:309809086c2acb67624950a3c8133e80f32d0d3e27106c0cd60ff26657c9f24b", size = 11520767, upload-time = "2026-06-25T17:20:09.191Z" },
+ { url = "https://files.pythonhosted.org/packages/19/0f/f032696cb01c9b54c0263fa393474d7758f1cdc021a01b04e3cbc2500999/ruff-0.15.20-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:2d2374caa2f2c2f9e2b7da0a50802cfb8b79f55a9b5e49379f564544fbf56487", size = 11500132, upload-time = "2026-06-25T17:20:13.602Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/f4/51b1a14bc69e8c224b15dab9cce8e99b425e0455d462caa2b3c9be2b6a8e/ruff-0.15.20-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:a1ed17b65293e0c2f22fc387bc13198a5de94bf4429589b0ff6946b0feaf21a3", size = 10943828, upload-time = "2026-06-25T17:20:16.635Z" },
+ { url = "https://files.pythonhosted.org/packages/71/4b/fe267640783cd02bf6c5cc290b1df1051be2ec294c678b5c15fe19e52343/ruff-0.15.20-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:f701305e66b38ea6c91882490eb73459796808e4c6362a1b765255e0cdcd4053", size = 10645418, upload-time = "2026-06-25T17:20:19.4Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/c0/a65aa4ec2f5e87a1df32dc3ec1fede434fe3dfd5cbcf3b503cafc676ab54/ruff-0.15.20-py3-none-musllinux_1_2_i686.whl", hash = "sha256:5b9c0c367ad8e5d0d5b5b8537864c469a0a0e55417aadfbeca41fa61333be9f4", size = 11211770, upload-time = "2026-06-25T17:20:22.033Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/a4/0caa331d954ae2723d729d351c989cb4ca8b6077d5c6c2cb6de75e98c041/ruff-0.15.20-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:01cc00dd58f0df339d0e902219dd53990ea99996a0344e5d9cc8d45d5307e460", size = 11618698, upload-time = "2026-06-25T17:20:25.259Z" },
+ { url = "https://files.pythonhosted.org/packages/10/9b/5f14927848d2fd4aa891fd88d883788c5a7baba561c7874732364045708c/ruff-0.15.20-py3-none-win32.whl", hash = "sha256:ed65ef510e43a137207e0f01cfcf998aeddb1aeeda5c9d35023e910284d7cf21", size = 10857322, upload-time = "2026-06-25T17:20:28.612Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/f0/fe47c501f9dea92a26d788ff98bb5d92ed4cb4c88792c5c88af6b697dc8e/ruff-0.15.20-py3-none-win_amd64.whl", hash = "sha256:a525c81c70fb0380344dd1d8745d8cc1c890b7fc94a58d5a07bd8eb9557b8415", size = 11993274, upload-time = "2026-06-25T17:20:31.871Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/2b/9555445e1201d92b3195f45cdb153a0b68f24e0a4273f6e3d5ab46e212bb/ruff-0.15.20-py3-none-win_arm64.whl", hash = "sha256:2f5b2a6d614e8700388806a14996c40fab2c47b819ef57d790a34878858ed9ca", size = 11343498, upload-time = "2026-06-25T17:20:35.03Z" },
+]
+
+[[package]]
+name = "s3fs"
+version = "2025.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiobotocore" },
+ { name = "aiohttp" },
+ { name = "fsspec" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ee/f3/8e6371436666aedfd16e63ff68a51b8a8fcf5f33a0eee33c35e0b2476b27/s3fs-2025.9.0.tar.gz", hash = "sha256:6d44257ef19ea64968d0720744c4af7a063a05f5c1be0e17ce943bef7302bc30", size = 77823, upload-time = "2025-09-02T19:18:21.781Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/37/b3/ca7d58ca25b1bb6df57e6cbd0ca8d6437a4b9ce1cd35adc8a6b2949c113b/s3fs-2025.9.0-py3-none-any.whl", hash = "sha256:c33c93d48f66ed440dbaf6600be149cdf8beae4b6f8f0201a209c5801aeb7e30", size = 30319, upload-time = "2025-09-02T19:18:20.563Z" },
+]
+
+[[package]]
+name = "s3transfer"
+version = "0.14.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "botocore" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/62/74/8d69dcb7a9efe8baa2046891735e5dfe433ad558ae23d9e3c14c633d1d58/s3transfer-0.14.0.tar.gz", hash = "sha256:eff12264e7c8b4985074ccce27a3b38a485bb7f7422cc8046fee9be4983e4125", size = 151547, upload-time = "2025-09-09T19:23:31.089Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/48/f0/ae7ca09223a81a1d890b2557186ea015f6e0502e9b8cb8e1813f1d8cfa4e/s3transfer-0.14.0-py3-none-any.whl", hash = "sha256:ea3b790c7077558ed1f02a3072fb3cb992bbbd253392f4b6e9e8976941c7d456", size = 85712, upload-time = "2025-09-09T19:23:30.041Z" },
+]
+
+[[package]]
+name = "sacrebleu"
+version = "2.6.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama" },
+ { name = "lxml" },
+ { name = "numpy" },
+ { name = "portalocker" },
+ { name = "regex" },
+ { name = "tabulate" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d3/ed/d7acddcff74d690c56fe26a1f7828bdde548262828d0743414ea916c40c1/sacrebleu-2.6.0.tar.gz", hash = "sha256:91499b6cd46138d95154fff1e863c2f9be57e82f0c719d8dd718d0006cf6c566", size = 1893419, upload-time = "2026-01-12T17:17:20.799Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/06/f2/6c90ccf3ad1d09a7d662a405b274f3c93b92df59c8d6a025d26aaf34d302/sacrebleu-2.6.0-py3-none-any.whl", hash = "sha256:3edc1531575cfe4ad04ce53491a9307e234af1c3f805a1f491cbec844229a8a8", size = 100785, upload-time = "2026-01-12T17:17:18.868Z" },
+]
+
+[[package]]
+name = "safetensors"
+version = "0.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/45/06/f955dbbb1859e3bd23c8ac6141af5106e7ad5fedec4a3a6e3d60f94b7001/safetensors-0.8.0.tar.gz", hash = "sha256:fabaf3e0f18a6618d9b36560682562157f77c2b71fcffc7b432be2baed9d753d", size = 325846, upload-time = "2026-06-09T07:52:25.563Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/39/a0/f718cda65b05407d228f97602cf60dca269c979867aa5beb25410de26cd3/safetensors-0.8.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c554f85858e05226d3c2828e32395e677434685d6d94594a41643361c5e837f0", size = 473568, upload-time = "2026-06-09T07:52:18.829Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/b1/fa7c600e7dceae12e9606c7578cbc9ff1e1ed55844883ee5c92205e86226/safetensors-0.8.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:c80201d22cbf405b80647a60ada77bba06c8fba2da2743ba1e89cdcc39a81f25", size = 484562, upload-time = "2026-06-09T07:52:17.518Z" },
+ { url = "https://files.pythonhosted.org/packages/09/7d/65a7de0af421317bb36a067241e4235fff194eed60b961ed6d3f59a3fc60/safetensors-0.8.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a46e5ff292c356d6991e60942ba7f79817682d3a2cef0702136448cb9c4d235", size = 502844, upload-time = "2026-06-09T07:52:07.624Z" },
+ { url = "https://files.pythonhosted.org/packages/91/4f/3175c9d75634e0e0dda0082794193521035edd7c70a6f212bf33ca06ddf4/safetensors-0.8.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4124502b78f03534117c848f87a39b8f31e577b15eff423bf8bfb95f2a8c30d0", size = 511823, upload-time = "2026-06-09T07:52:09.565Z" },
+ { url = "https://files.pythonhosted.org/packages/20/87/846c289e7aa2299eff406335717cf43ce8777194ece8aad75772e0411615/safetensors-0.8.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7bc0a787ba8a35be368ee3574edfa2b1ad389eebd0a72e482ae275490e3f6c98", size = 633461, upload-time = "2026-06-09T07:52:11.128Z" },
+ { url = "https://files.pythonhosted.org/packages/76/22/8d64d9df2c45d5ded401df889d0ad90882804ca172d79ec4f0df8f727fe0/safetensors-0.8.0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:040070828e36dc8e122178bbbd5830ff9e97920affb84cbe0f46442497bed358", size = 545148, upload-time = "2026-06-09T07:52:13.603Z" },
+ { url = "https://files.pythonhosted.org/packages/28/50/f203ff3a3ddfe19308efc83c5a3a29ed02bf786732ec35e68bf9162f3365/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", size = 516040, upload-time = "2026-06-09T07:52:16.29Z" },
+ { url = "https://files.pythonhosted.org/packages/46/fb/cdaed17ceb2948784fd9c36b6fd3e951b608547cea81a48e8ee6f8cfdfcb/safetensors-0.8.0-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:fcdd41ec4628fee5799f807c73c353629130fbd942aa23d83c623dd6c9d52d78", size = 513832, upload-time = "2026-06-09T07:52:12.37Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/49/1e15de264dcc3b77943d2d0c56a95809956883b1c2d6d585c792523f180b/safetensors-0.8.0-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8e9f537aa183a38ace122d27303dcd986b26bd2a7591f9181d7f0c396f4677ca", size = 559930, upload-time = "2026-06-09T07:52:14.743Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/43/bf38443278eab4b1be1fce2931e2b012ad9cb7df52ada751d0aab8f7659a/safetensors-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:87eec7ffed2b809f05a398a8becb7d013f19f7837cd15d9748580d6cf30dbaf4", size = 678670, upload-time = "2026-06-09T07:52:20.032Z" },
+ { url = "https://files.pythonhosted.org/packages/72/e3/68cd3fa5b48488e84add63e04cb12f3bc28ae4638c06d4508c6e88823d0e/safetensors-0.8.0-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4a95ae2b05d7726d751da4ebf626a2ca782b706e101bd894c95bc2450b1cffcc", size = 786679, upload-time = "2026-06-09T07:52:21.322Z" },
+ { url = "https://files.pythonhosted.org/packages/29/4b/1c19c509d56e01f4fbb3d0a2e597450f6cc04d1d56cf52defb0a62dfd715/safetensors-0.8.0-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae091f16662658bdc019a4ff6cb4c085bb7d725eb5978b183ffd265863b6d2d", size = 765683, upload-time = "2026-06-09T07:52:22.594Z" },
+ { url = "https://files.pythonhosted.org/packages/27/43/41c1621732edd934d868a00d1b891584c892a7b62a9aab82ea5a0a5623ee/safetensors-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8e080062fcde23be189565e1c3305d16751a218ecf9412c8601e64204eb6f846", size = 722361, upload-time = "2026-06-09T07:52:23.924Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/3f/73ccf82579412b4a71c4ca673f10b5f1f888d7cf5af7fe24f27d30307be4/safetensors-0.8.0-cp310-abi3-win32.whl", hash = "sha256:2ddf52eac562eda224f99acfa7889d02968c1fd59a5b011ae7d8137c37e9c02d", size = 342401, upload-time = "2026-06-09T07:52:28.895Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/6d/3fba214c1e5e0f69991677ec3bc17023f0421776975e1de0c682dca475e2/safetensors-0.8.0-cp310-abi3-win_amd64.whl", hash = "sha256:096ec1a98435df7beb08853bb5aa9081a84f23d0adc67ed1a0a10550f608373f", size = 355540, upload-time = "2026-06-09T07:52:27.832Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/fc/7eedc3510d97878876e32774eebbeb61c43f148a96e915c84229a3e967aa/safetensors-0.8.0-cp310-abi3-win_arm64.whl", hash = "sha256:f7838e5135a406ad3e02efdcb8cf2e5397d368b0154537c4fec682dbc544d452", size = 340500, upload-time = "2026-06-09T07:52:26.745Z" },
+]
+
+[[package]]
+name = "scikit-learn"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "joblib" },
+ { name = "narwhals" },
+ { name = "numpy" },
+ { name = "scipy" },
+ { name = "threadpoolctl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fa/6f/37092bdb25f712817231799fc5674d8e704066a8a70c1d2d40517e18b4ab/scikit_learn-1.9.0.tar.gz", hash = "sha256:8833266989d3a5110178a9fae30783675460724d0e1efb13b14901d2c660c557", size = 7750767, upload-time = "2026-06-02T11:54:32.706Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3c/01/cf3310626b6d48d3e9be69a1223f9180360b5e6edb045f50fade723ce494/scikit_learn-1.9.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:80746d63bd4b6eaca54d36fe5feaf4d28bb38dc6f9470f81c7cad7c40155f119", size = 8705188, upload-time = "2026-06-02T11:53:41.964Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/04/5acd7ae280c5f93b6ac5ef6cdec14eef4c8d1cd91d85b3292989c94d96b1/scikit_learn-1.9.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5b934c45c252844a91d69fda3a34cff5e7307e1db10d77cb10a3980312c74713", size = 8228299, upload-time = "2026-06-02T11:53:44.817Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/39/ffe829a5b8ecb40a518724a997794657fdc354ada5e8fe8e64d998c0bac9/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:38c3dcb9a1ffb85505ec53d54c7b4aea0cff70050425a7760c2af661ac85df05", size = 8789690, upload-time = "2026-06-02T11:53:47.461Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/88/8dab5de10c638c083772a6be83a3d8106ced492f74a928c8693638e5bb50/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:da76d09304a4706db7cc1e3ebaa3b6b98a67365cc11d2996c4f1e58ba47df714", size = 9087723, upload-time = "2026-06-02T11:53:50.702Z" },
+ { url = "https://files.pythonhosted.org/packages/20/3f/7917ca72464038f6240ec70c29f94862d08a34a74291ae4d4ec5eb8186a0/scikit_learn-1.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:5808d98f15c6bf6d9d96d2348c1997392a5888ce7097e664105f930c4bca1277", size = 8184330, upload-time = "2026-06-02T11:53:53.396Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c7/15739eb2f61fda3c54639e9942414e5a19ad8a8d1f5a3266afad7cb7df80/scikit_learn-1.9.0-cp313-cp313-win_arm64.whl", hash = "sha256:d77f54c017633791bc0225a43e2f8d03745fdcfe4880268fcc4df15f505dec2e", size = 7840653, upload-time = "2026-06-02T11:53:56.035Z" },
+]
+
+[[package]]
+name = "scipy"
+version = "1.18.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a7/25/c2700dfaf6442b4effaa91af24ebce5dc9d31bb4a69706313aae70d72cd0/scipy-1.18.0.tar.gz", hash = "sha256:67b2ad2ad54c72ca6d04975a9b2df8c3638c34ddd5b28738e94fc2b57929d378", size = 30774447, upload-time = "2026-06-19T15:01:43.456Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/52/9c0136c2de7ae0779b7b366447766cec6d9f0702c56bb8ffeb04c8fd3af4/scipy-1.18.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:09143f676d157d9f546d663504ef9c1becb819824f1afc018814176411942446", size = 31036107, upload-time = "2026-06-19T15:00:14.03Z" },
+ { url = "https://files.pythonhosted.org/packages/02/73/0291a64843270f4efb86cdcf2ee0f2048631b65ec6b405398b2b4dbf11bf/scipy-1.18.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5efe260f69417b97ddae455bfb5a95e8359f7f66ad7fa9522a60feb66f169520", size = 28663303, upload-time = "2026-06-19T15:00:16.819Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/0f/10ffa0b697a572f4e0d48b92a88895d366422f019f723e7e14a84c050dac/scipy-1.18.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:68363b7eaacd8b5dd426df56d782cc156468ac79a127a1b87ca597d6e2e82197", size = 20404960, upload-time = "2026-06-19T15:00:19.635Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/d2/e896cea21ba8edd6c81d4c55b1ffcc717e79698dcbebf9641b4cfb4c6622/scipy-1.18.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:c5557d8be5da8e41353fcd4d21491fdbab83b062fc579e94dc09a7c8ab4f669b", size = 23034074, upload-time = "2026-06-19T15:00:22.107Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/b2/e83ea34279a52c03374477c74006256ec78df65fc877baa4617d6de1d202/scipy-1.18.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0d13bca67c096d89fb95ced0d8921807300fce0275643aef9533cc63a0773468", size = 33942038, upload-time = "2026-06-19T15:00:24.964Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/af/e8fe5fb136f51e2b01678b92cb4106d10d8cd68ec147ead2e7cb0ac75398/scipy-1.18.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a46f9273dbd0eb1cefba61c9b8648b4dfe3cbc14a080176f9a73e44b8336dc7f", size = 35266390, upload-time = "2026-06-19T15:00:28.059Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/49/2c5cbb907b56695fc67517811d1db234dfd83381a84814ec220aded2794d/scipy-1.18.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5aba46108853ddfc77906b6557aac839d2b52e900c1d72a1180adaaab58d265f", size = 35551324, upload-time = "2026-06-19T15:00:31.014Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/73/eda39f7a2d306ff0ffc574afd13c0bbb6d10a603d9a413998ee269487a80/scipy-1.18.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b6f758e35f12757b5d95c00bc6de2438e229c2664b7a92e96f205959d9f2dfa4", size = 37404785, upload-time = "2026-06-19T15:00:34.072Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/d2/ae881ee28d014f38e0ccbfd974a06a919ba9af34f1f74bf42b5301891d63/scipy-1.18.0-cp313-cp313-win_amd64.whl", hash = "sha256:1afac4a847207c7ff8efd321734a50b06d0280b3b2a2c0fc2f413101747ad7c7", size = 36554943, upload-time = "2026-06-19T15:00:36.903Z" },
+ { url = "https://files.pythonhosted.org/packages/70/3a/21154e2d54eb3639c6bf4dbae2e531c68356bfe95990daa30df33b30d556/scipy-1.18.0-cp313-cp313-win_arm64.whl", hash = "sha256:c5dbddf60e58c2312316d097271a8e73d40eaf2eabfa4d95ed7d3695bbf2ce7b", size = 24350911, upload-time = "2026-06-19T15:00:40.062Z" },
+]
+
+[[package]]
+name = "semver"
+version = "3.0.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/72/d1/d3159231aec234a59dd7d601e9dd9fe96f3afff15efd33c1070019b26132/semver-3.0.4.tar.gz", hash = "sha256:afc7d8c584a5ed0a11033af086e8af226a9c0b206f313e0301f8dd7b6b589602", size = 269730, upload-time = "2025-01-24T13:19:27.617Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a6/24/4d91e05817e92e3a61c8a21e08fd0f390f5301f1c448b137c57c4bc6e543/semver-3.0.4-py3-none-any.whl", hash = "sha256:9c824d87ba7f7ab4a1890799cec8596f15c1241cb473404ea1cb0c55e4b04746", size = 17912, upload-time = "2025-01-24T13:19:24.949Z" },
+]
+
+[[package]]
+name = "send2trash"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c5/f0/184b4b5f8d00f2a92cf96eec8967a3d550b52cf94362dad1100df9e48d57/send2trash-2.1.0.tar.gz", hash = "sha256:1c72b39f09457db3c05ce1d19158c2cbef4c32b8bedd02c155e49282b7ea7459", size = 17255, upload-time = "2026-01-14T06:27:36.056Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1c/78/504fdd027da3b84ff1aecd9f6957e65f35134534ccc6da8628eb71e76d3f/send2trash-2.1.0-py3-none-any.whl", hash = "sha256:0da2f112e6d6bb22de6aa6daa7e144831a4febf2a87261451c4ad849fe9a873c", size = 17610, upload-time = "2026-01-14T06:27:35.218Z" },
+]
+
+[[package]]
+name = "sentencepiece"
+version = "0.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/15/15/2e7a025fc62d764b151ae6d0f2a92f8081755ebe8d4a64099accc6f77ba6/sentencepiece-0.2.1.tar.gz", hash = "sha256:8138cec27c2f2282f4a34d9a016e3374cd40e5c6e9cb335063db66a0a3b71fad", size = 3228515, upload-time = "2025-08-12T07:00:51.718Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ba/4a/85fbe1706d4d04a7e826b53f327c4b80f849cf1c7b7c5e31a20a97d8f28b/sentencepiece-0.2.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:dcd8161eee7b41aae57ded06272905dbd680a0a04b91edd0f64790c796b2f706", size = 1943150, upload-time = "2025-08-12T06:59:53.588Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/83/4cfb393e287509fc2155480b9d184706ef8d9fa8cbf5505d02a5792bf220/sentencepiece-0.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c6c8f42949f419ff8c7e9960dbadcfbc982d7b5efc2f6748210d3dd53a7de062", size = 1325651, upload-time = "2025-08-12T06:59:55.073Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/de/5a007fb53b1ab0aafc69d11a5a3dd72a289d5a3e78dcf2c3a3d9b14ffe93/sentencepiece-0.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:097f3394e99456e9e4efba1737c3749d7e23563dd1588ce71a3d007f25475fff", size = 1253641, upload-time = "2025-08-12T06:59:56.562Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/d2/f552be5928105588f4f4d66ee37dd4c61460d8097e62d0e2e0eec41bc61d/sentencepiece-0.2.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d7b670879c370d350557edabadbad1f6561a9e6968126e6debca4029e5547820", size = 1316271, upload-time = "2025-08-12T06:59:58.109Z" },
+ { url = "https://files.pythonhosted.org/packages/96/df/0cfe748ace5485be740fed9476dee7877f109da32ed0d280312c94ec259f/sentencepiece-0.2.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c7f0fd2f2693309e6628aeeb2e2faf6edd221134dfccac3308ca0de01f8dab47", size = 1387882, upload-time = "2025-08-12T07:00:00.701Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/dd/f7774d42a881ced8e1739f393ab1e82ece39fc9abd4779e28050c2e975b5/sentencepiece-0.2.1-cp313-cp313-win32.whl", hash = "sha256:92b3816aa2339355fda2c8c4e021a5de92180b00aaccaf5e2808972e77a4b22f", size = 999541, upload-time = "2025-08-12T07:00:02.709Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/e9/932b9eae6fd7019548321eee1ab8d5e3b3d1294df9d9a0c9ac517c7b636d/sentencepiece-0.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:10ed3dab2044c47f7a2e7b4969b0c430420cdd45735d78c8f853191fa0e3148b", size = 1054669, upload-time = "2025-08-12T07:00:04.915Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/3a/76488a00ea7d6931689cda28726a1447d66bf1a4837943489314593d5596/sentencepiece-0.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:ac650534e2251083c5f75dde4ff28896ce7c8904133dc8fef42780f4d5588fcd", size = 1033922, upload-time = "2025-08-12T07:00:06.496Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/b6/08fe2ce819e02ccb0296f4843e3f195764ce9829cbda61b7513f29b95718/sentencepiece-0.2.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:8dd4b477a7b069648d19363aad0cab9bad2f4e83b2d179be668efa672500dc94", size = 1946052, upload-time = "2025-08-12T07:00:08.136Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/d9/1ea0e740591ff4c6fc2b6eb1d7510d02f3fb885093f19b2f3abd1363b402/sentencepiece-0.2.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0c0f672da370cc490e4c59d89e12289778310a0e71d176c541e4834759e1ae07", size = 1327408, upload-time = "2025-08-12T07:00:09.572Z" },
+ { url = "https://files.pythonhosted.org/packages/99/7e/1fb26e8a21613f6200e1ab88824d5d203714162cf2883248b517deb500b7/sentencepiece-0.2.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:ad8493bea8432dae8d6830365352350f3b4144415a1d09c4c8cb8d30cf3b6c3c", size = 1254857, upload-time = "2025-08-12T07:00:11.021Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/85/c72fd1f3c7a6010544d6ae07f8ddb38b5e2a7e33bd4318f87266c0bbafbf/sentencepiece-0.2.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b81a24733726e3678d2db63619acc5a8dccd074f7aa7a54ecd5ca33ca6d2d596", size = 1315722, upload-time = "2025-08-12T07:00:12.989Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/e8/661e5bd82a8aa641fd6c1020bd0e890ef73230a2b7215ddf9c8cd8e941c2/sentencepiece-0.2.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0a81799d0a68d618e89063fb423c3001a034c893069135ffe51fee439ae474d6", size = 1387452, upload-time = "2025-08-12T07:00:15.088Z" },
+ { url = "https://files.pythonhosted.org/packages/99/5e/ae66c361023a470afcbc1fbb8da722c72ea678a2fcd9a18f1a12598c7501/sentencepiece-0.2.1-cp313-cp313t-win32.whl", hash = "sha256:89a3ea015517c42c0341d0d962f3e6aaf2cf10d71b1932d475c44ba48d00aa2b", size = 1002501, upload-time = "2025-08-12T07:00:16.966Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/03/d332828c4ff764e16c1b56c2c8f9a33488bbe796b53fb6b9c4205ddbf167/sentencepiece-0.2.1-cp313-cp313t-win_amd64.whl", hash = "sha256:33f068c9382dc2e7c228eedfd8163b52baa86bb92f50d0488bf2b7da7032e484", size = 1057555, upload-time = "2025-08-12T07:00:18.573Z" },
+ { url = "https://files.pythonhosted.org/packages/88/14/5aee0bf0864df9bd82bd59e7711362908e4935e3f9cdc1f57246b5d5c9b9/sentencepiece-0.2.1-cp313-cp313t-win_arm64.whl", hash = "sha256:b3616ad246f360e52c85781e47682d31abfb6554c779e42b65333d4b5f44ecc0", size = 1036042, upload-time = "2025-08-12T07:00:20.209Z" },
+]
+
+[[package]]
+name = "sentry-sdk"
+version = "2.64.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/60/31/b7341f156a5f6f36f0b4845d6f1c28a2ae4799171dba7007f3a1e9b234b4/sentry_sdk-2.64.0.tar.gz", hash = "sha256:68be2c29e14ae310f8a39e1a79916b6d85c6cb41dcce789d14ff05fe293e4c55", size = 921020, upload-time = "2026-06-30T08:13:47.682Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/36/a8/3fb9a4319efa3b26f5be0e90e6d8918df43fa7c7e977d26390f589501d82/sentry_sdk-2.64.0-py3-none-any.whl", hash = "sha256:715ea91ca860a819e8d8a50a7bde3a80d0df3b4ed7b6660a20fb9a2d084188f1", size = 498901, upload-time = "2026-06-30T08:13:45.566Z" },
+]
+
+[[package]]
+name = "setproctitle"
+version = "1.3.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8d/48/49393a96a2eef1ab418b17475fb92b8fcfad83d099e678751b05472e69de/setproctitle-1.3.7.tar.gz", hash = "sha256:bc2bc917691c1537d5b9bca1468437176809c7e11e5694ca79a9ca12345dcb9e", size = 27002, upload-time = "2025-09-05T12:51:25.278Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5d/2f/fcedcade3b307a391b6e17c774c6261a7166aed641aee00ed2aad96c63ce/setproctitle-1.3.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c3736b2a423146b5e62230502e47e08e68282ff3b69bcfe08a322bee73407922", size = 18047, upload-time = "2025-09-05T12:49:50.271Z" },
+ { url = "https://files.pythonhosted.org/packages/23/ae/afc141ca9631350d0a80b8f287aac79a76f26b6af28fd8bf92dae70dc2c5/setproctitle-1.3.7-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3384e682b158d569e85a51cfbde2afd1ab57ecf93ea6651fe198d0ba451196ee", size = 13073, upload-time = "2025-09-05T12:49:51.46Z" },
+ { url = "https://files.pythonhosted.org/packages/87/ed/0a4f00315bc02510395b95eec3d4aa77c07192ee79f0baae77ea7b9603d8/setproctitle-1.3.7-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0564a936ea687cd24dffcea35903e2a20962aa6ac20e61dd3a207652401492dd", size = 33284, upload-time = "2025-09-05T12:49:52.741Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/e4/adf3c4c0a2173cb7920dc9df710bcc67e9bcdbf377e243b7a962dc31a51a/setproctitle-1.3.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a5d1cb3f81531f0eb40e13246b679a1bdb58762b170303463cb06ecc296f26d0", size = 34104, upload-time = "2025-09-05T12:49:54.416Z" },
+ { url = "https://files.pythonhosted.org/packages/52/4f/6daf66394152756664257180439d37047aa9a1cfaa5e4f5ed35e93d1dc06/setproctitle-1.3.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a7d159e7345f343b44330cbba9194169b8590cb13dae940da47aa36a72aa9929", size = 35982, upload-time = "2025-09-05T12:49:56.295Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/62/f2c0595403cf915db031f346b0e3b2c0096050e90e0be658a64f44f4278a/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0b5074649797fd07c72ca1f6bff0406f4a42e1194faac03ecaab765ce605866f", size = 33150, upload-time = "2025-09-05T12:49:58.025Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/29/10dd41cde849fb2f9b626c846b7ea30c99c81a18a5037a45cc4ba33c19a7/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:61e96febced3f61b766115381d97a21a6265a0f29188a791f6df7ed777aef698", size = 34463, upload-time = "2025-09-05T12:49:59.424Z" },
+ { url = "https://files.pythonhosted.org/packages/71/3c/cedd8eccfaf15fb73a2c20525b68c9477518917c9437737fa0fda91e378f/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:047138279f9463f06b858e579cc79580fbf7a04554d24e6bddf8fe5dddbe3d4c", size = 32848, upload-time = "2025-09-05T12:50:01.107Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/3e/0a0e27d1c9926fecccfd1f91796c244416c70bf6bca448d988638faea81d/setproctitle-1.3.7-cp313-cp313-win32.whl", hash = "sha256:7f47accafac7fe6535ba8ba9efd59df9d84a6214565108d0ebb1199119c9cbbd", size = 12544, upload-time = "2025-09-05T12:50:15.81Z" },
+ { url = "https://files.pythonhosted.org/packages/36/1b/6bf4cb7acbbd5c846ede1c3f4d6b4ee52744d402e43546826da065ff2ab7/setproctitle-1.3.7-cp313-cp313-win_amd64.whl", hash = "sha256:fe5ca35aeec6dc50cabab9bf2d12fbc9067eede7ff4fe92b8f5b99d92e21263f", size = 13235, upload-time = "2025-09-05T12:50:16.89Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/a4/d588d3497d4714750e3eaf269e9e8985449203d82b16b933c39bd3fc52a1/setproctitle-1.3.7-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:10e92915c4b3086b1586933a36faf4f92f903c5554f3c34102d18c7d3f5378e9", size = 18058, upload-time = "2025-09-05T12:50:02.501Z" },
+ { url = "https://files.pythonhosted.org/packages/05/77/7637f7682322a7244e07c373881c7e982567e2cb1dd2f31bd31481e45500/setproctitle-1.3.7-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:de879e9c2eab637f34b1a14c4da1e030c12658cdc69ee1b3e5be81b380163ce5", size = 13072, upload-time = "2025-09-05T12:50:03.601Z" },
+ { url = "https://files.pythonhosted.org/packages/52/09/f366eca0973cfbac1470068d1313fa3fe3de4a594683385204ec7f1c4101/setproctitle-1.3.7-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:c18246d88e227a5b16248687514f95642505000442165f4b7db354d39d0e4c29", size = 34490, upload-time = "2025-09-05T12:50:04.948Z" },
+ { url = "https://files.pythonhosted.org/packages/71/36/611fc2ed149fdea17c3677e1d0df30d8186eef9562acc248682b91312706/setproctitle-1.3.7-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7081f193dab22df2c36f9fc6d113f3793f83c27891af8fe30c64d89d9a37e152", size = 35267, upload-time = "2025-09-05T12:50:06.015Z" },
+ { url = "https://files.pythonhosted.org/packages/88/a4/64e77d0671446bd5a5554387b69e1efd915274686844bea733714c828813/setproctitle-1.3.7-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9cc9b901ce129350637426a89cfd650066a4adc6899e47822e2478a74023ff7c", size = 37376, upload-time = "2025-09-05T12:50:07.484Z" },
+ { url = "https://files.pythonhosted.org/packages/89/bc/ad9c664fe524fb4a4b2d3663661a5c63453ce851736171e454fa2cdec35c/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:80e177eff2d1ec172188d0d7fd9694f8e43d3aab76a6f5f929bee7bf7894e98b", size = 33963, upload-time = "2025-09-05T12:50:09.056Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/01/a36de7caf2d90c4c28678da1466b47495cbbad43badb4e982d8db8167ed4/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:23e520776c445478a67ee71b2a3c1ffdafbe1f9f677239e03d7e2cc635954e18", size = 35550, upload-time = "2025-09-05T12:50:10.791Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/68/17e8aea0ed5ebc17fbf03ed2562bfab277c280e3625850c38d92a7b5fcd9/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5fa1953126a3b9bd47049d58c51b9dac72e78ed120459bd3aceb1bacee72357c", size = 33727, upload-time = "2025-09-05T12:50:12.032Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/33/90a3bf43fe3a2242b4618aa799c672270250b5780667898f30663fd94993/setproctitle-1.3.7-cp313-cp313t-win32.whl", hash = "sha256:4a5e212bf438a4dbeece763f4962ad472c6008ff6702e230b4f16a037e2f6f29", size = 12549, upload-time = "2025-09-05T12:50:13.074Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/0e/50d1f07f3032e1f23d814ad6462bc0a138f369967c72494286b8a5228e40/setproctitle-1.3.7-cp313-cp313t-win_amd64.whl", hash = "sha256:cf2727b733e90b4f874bac53e3092aa0413fe1ea6d4f153f01207e6ce65034d9", size = 13243, upload-time = "2025-09-05T12:50:14.146Z" },
+]
+
+[[package]]
+name = "setuptools"
+version = "80.10.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/76/95/faf61eb8363f26aa7e1d762267a8d602a1b26d4f3a1e758e92cb3cb8b054/setuptools-80.10.2.tar.gz", hash = "sha256:8b0e9d10c784bf7d262c4e5ec5d4ec94127ce206e8738f29a437945fbc219b70", size = 1200343, upload-time = "2026-01-25T22:38:17.252Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/94/b8/f1f62a5e3c0ad2ff1d189590bfa4c46b4f3b6e49cef6f26c6ee4e575394d/setuptools-80.10.2-py3-none-any.whl", hash = "sha256:95b30ddfb717250edb492926c92b5221f7ef3fbcc2b07579bcd4a27da21d0173", size = 1064234, upload-time = "2026-01-25T22:38:15.216Z" },
+]
+
+[[package]]
+name = "shellingham"
+version = "1.5.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/58/15/8b3609fd3830ef7b27b655beb4b4e9c62313a4e8da8c676e142cc210d58e/shellingham-1.5.4.tar.gz", hash = "sha256:8dbca0739d487e5bd35ab3ca4b36e11c4078f3a234bfce294b0a0291363404de", size = 10310, upload-time = "2023-10-24T04:13:40.426Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" },
+]
+
+[[package]]
+name = "shortuuid"
+version = "1.0.13"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8c/e2/bcf761f3bff95856203f9559baf3741c416071dd200c0fc19fad7f078f86/shortuuid-1.0.13.tar.gz", hash = "sha256:3bb9cf07f606260584b1df46399c0b87dd84773e7b25912b7e391e30797c5e72", size = 9662, upload-time = "2024-03-11T20:11:06.879Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c0/44/21d6bf170bf40b41396480d8d49ad640bca3f2b02139cd52aa1e272830a5/shortuuid-1.0.13-py3-none-any.whl", hash = "sha256:a482a497300b49b4953e15108a7913244e1bb0d41f9d332f5e9925dba33a3c5a", size = 10529, upload-time = "2024-03-11T20:11:04.807Z" },
+]
+
+[[package]]
+name = "six"
+version = "1.17.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" },
+]
+
+[[package]]
+name = "sk-bench"
+version = "0.1.0"
+source = { virtual = "." }
+dependencies = [
+ { name = "hf" },
+ { name = "immutabledict" },
+ { name = "langdetect" },
+ { name = "lighteval", extra = ["multilingual", "vllm"] },
+ { name = "litellm", extra = ["caching"] },
+ { name = "more-itertools" },
+ { name = "pip" },
+ { name = "pre-commit" },
+ { name = "pyyaml" },
+ { name = "torch" },
+ { name = "transformers" },
+ { name = "vllm" },
+ { name = "wandb" },
+]
+
+[package.optional-dependencies]
+vllm = [
+ { name = "torch" },
+ { name = "vllm" },
+]
+
+[package.dev-dependencies]
+dev = [
+ { name = "jupyter" },
+ { name = "matplotlib" },
+ { name = "mypy" },
+ { name = "pandas" },
+ { name = "pandas-stubs" },
+ { name = "pre-commit" },
+ { name = "pyarrow" },
+ { name = "pytest" },
+ { name = "ruff" },
+ { name = "typos" },
+]
+hpc = [
+ { name = "deep-gemm" },
+]
+lint = [
+ { name = "pre-commit" },
+ { name = "ruff" },
+ { name = "typos" },
+]
+notebooks = [
+ { name = "jupyter" },
+ { name = "matplotlib" },
+ { name = "pandas" },
+ { name = "pyarrow" },
+]
+test = [
+ { name = "pytest" },
+]
+typing = [
+ { name = "mypy" },
+ { name = "pandas-stubs" },
+]
+
+[package.metadata]
+requires-dist = [
+ { name = "hf", specifier = ">=1.1.0" },
+ { name = "immutabledict", specifier = ">=4.3.1" },
+ { name = "langdetect", specifier = ">=1.0.9" },
+ { name = "lighteval", extras = ["multilingual", "vllm"], git = "https://github.com/huggingface/lighteval.git?rev=10b9104eb16778b174e9e240140974a039018981" },
+ { name = "litellm", extras = ["caching"], specifier = ">=1.82.0" },
+ { name = "more-itertools", specifier = ">=10.8.0" },
+ { name = "pip", specifier = ">=25.3" },
+ { name = "pre-commit", specifier = ">=4.5.1" },
+ { name = "pyyaml", specifier = ">=6.0.3" },
+ { name = "torch", specifier = "==2.13.0" },
+ { name = "torch", marker = "extra == 'vllm'", specifier = "==2.13.0" },
+ { name = "transformers", specifier = "==5.12.1" },
+ { name = "vllm", specifier = "==0.28.0" },
+ { name = "vllm", marker = "extra == 'vllm'", specifier = "==0.28.0" },
+ { name = "wandb", specifier = ">=0.26.1" },
+]
+provides-extras = ["vllm"]
+
+[package.metadata.requires-dev]
+dev = [
+ { name = "jupyter" },
+ { name = "matplotlib" },
+ { name = "mypy", specifier = ">=1.14.0" },
+ { name = "pandas" },
+ { name = "pandas-stubs", specifier = ">=2.3.3.260113" },
+ { name = "pre-commit", specifier = ">=4.1.0" },
+ { name = "pyarrow" },
+ { name = "pytest", specifier = ">=8.0.0" },
+ { name = "ruff", specifier = ">=0.9.0" },
+ { name = "typos", specifier = ">=1.29.0" },
+]
+hpc = [{ name = "deep-gemm", git = "https://github.com/deepseek-ai/DeepGEMM.git?rev=v2.1.1.post3" }]
+lint = [
+ { name = "pre-commit", specifier = ">=4.1.0" },
+ { name = "ruff", specifier = ">=0.9.0" },
+ { name = "typos", specifier = ">=1.29.0" },
+]
+notebooks = [
+ { name = "jupyter" },
+ { name = "matplotlib" },
+ { name = "pandas" },
+ { name = "pyarrow" },
+]
+test = [{ name = "pytest", specifier = ">=8.0.0" }]
+typing = [
+ { name = "mypy", specifier = ">=1.14.0" },
+ { name = "pandas-stubs", specifier = ">=2.3.3.260113" },
+]
+
+[[package]]
+name = "sklearn-crfsuite"
+version = "0.5.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "python-crfsuite" },
+ { name = "scikit-learn" },
+ { name = "tabulate" },
+ { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d7/d4/26211eb22127f8476589e92448273c698f4ac39fce2e300cd0a1f748de98/sklearn_crfsuite-0.5.0.tar.gz", hash = "sha256:136cb941ff5f7ce62a2c378988d1123ac1037241e73999ae404509a2135078e9", size = 26187, upload-time = "2024-06-18T11:08:30.42Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b2/11/a8370dd6fce65f8f4e74a0adffae72be9db5799d8ed8ddbf84415356a764/sklearn_crfsuite-0.5.0-py2.py3-none-any.whl", hash = "sha256:9cb7a881969b6426d28ada69ada7eff4c789a3c508eafc32d39c0d078ce3bebc", size = 10839, upload-time = "2024-06-18T11:08:22.743Z" },
+]
+
+[[package]]
+name = "smart-open"
+version = "8.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "wrapt" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/29/3e/79fd5fd2375a8a500b9ec2f6a0762fc1ac33e35582d4a87483a78d19408f/smart_open-8.0.0.tar.gz", hash = "sha256:5a2008d60688bd3b33c52e2ef666d3c60cf956e73e215de8c7b242cf56fdd1b2", size = 61520, upload-time = "2026-06-27T16:28:11.894Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b4/bd/1c92e69a1daff70118f21e18ef3a100c114f00f08b64a1074484f12d9020/smart_open-8.0.0-py3-none-any.whl", hash = "sha256:ff4f395c9e86f23e27771dc4ba756ad4bd145f181859a782c50d64168485761b", size = 73029, upload-time = "2026-06-27T16:28:10.589Z" },
+]
+
+[[package]]
+name = "smmap"
+version = "5.0.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" },
+]
+
+[[package]]
+name = "sniffio"
+version = "1.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" },
+]
+
+[[package]]
+name = "soupsieve"
+version = "2.8.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/47/2c/0a5f6f8ee0d5589e48c7640213ed5175d52cf540a06725b628cc1a45d6ce/soupsieve-2.8.4.tar.gz", hash = "sha256:e121fd02e975c695e4e9e8774a5ee35d74714b59307868dcc5319ad2d9e3328e", size = 121110, upload-time = "2026-05-24T13:55:57.154Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5e/f5/0c41cb68dcae6b7de4fac4188a3a9589e21fb31df21ea3a2e888db95e6c9/soupsieve-2.8.4-py3-none-any.whl", hash = "sha256:e7e6b0769c8f51ed59acab6e994b00621096cfb1c640a7509295987388fbaf65", size = 37304, upload-time = "2026-05-24T13:55:55.406Z" },
+]
+
+[[package]]
+name = "spacy"
+version = "3.8.14"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "catalogue" },
+ { name = "confection" },
+ { name = "cymem" },
+ { name = "jinja2" },
+ { name = "murmurhash" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "preshed" },
+ { name = "pydantic" },
+ { name = "requests" },
+ { name = "setuptools" },
+ { name = "spacy-legacy" },
+ { name = "spacy-loggers" },
+ { name = "srsly" },
+ { name = "thinc" },
+ { name = "tqdm" },
+ { name = "typer" },
+ { name = "wasabi" },
+ { name = "weasel" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1b/e5/822bbdfa459fee863ef2e9879a34b0ae5db7cd1e3eb76d32c766f19222e9/spacy-3.8.14-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2b4f60fa8b9641a5e93e7a96db0cdd106d05d61756bf1d0ddcd1705ad347909a", size = 6202114, upload-time = "2026-03-29T10:41:06.119Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/de/0e512154113e1f341567f2b9341835775e4180c180221e60faedaebb2f65/spacy-3.8.14-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0860c57220c633ccb20468bcd64bfb0d28908990c371a8857951d093a148dc8e", size = 6015458, upload-time = "2026-03-29T10:41:07.79Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/4f/29c7e56afc7db07348a9e0efe0243b5eef465d5dc3d56433f164378c3fa6/spacy-3.8.14-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c24620b7dba879c69cebc51ef3b1107d4d4e44a1e0d4baa439372887d00c3fd9", size = 32510659, upload-time = "2026-03-29T10:41:09.88Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/ce/cae678f664d5467016819253f5d6e52f8e68a12d8e799b651d73ec2a9a4b/spacy-3.8.14-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9699c1248d115d5825987c287a6f6acd66386ef3ebee7994ee67ba093e932c59", size = 32841057, upload-time = "2026-03-29T10:41:12.585Z" },
+ { url = "https://files.pythonhosted.org/packages/04/d4/419868afd449bdd367df005932537eea66c71e97c899ba278f3124933f3c/spacy-3.8.14-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:042d799e342fdb6bb5b02a4213a95acc9116c40ed3c849bb0a8296fbe648ec22", size = 31763252, upload-time = "2026-03-29T10:41:15.569Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/53/df5c1fee45f200b749ba72eeb536fbb2c545fc56230324954263b2f3be00/spacy-3.8.14-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:69b2264294097336e86832e8663f1ab3a7215621184863c96c082ab17ee11937", size = 32717872, upload-time = "2026-03-29T10:41:18.193Z" },
+ { url = "https://files.pythonhosted.org/packages/12/c2/f1882ec2f5cc9c4e73cf2132997a03c397d7ceeb5ee7f7bb878b51a16365/spacy-3.8.14-cp313-cp313-win_amd64.whl", hash = "sha256:4b6d4f20e291a7c70e37de2f246622b44a0ce82efaa710c9801c6bd599e75177", size = 14220335, upload-time = "2026-03-29T10:41:20.89Z" },
+]
+
+[package.optional-dependencies]
+ja = [
+ { name = "sudachidict-core" },
+ { name = "sudachipy" },
+]
+ko = [
+ { name = "natto-py" },
+]
+th = [
+ { name = "pythainlp" },
+]
+
+[[package]]
+name = "spacy-legacy"
+version = "3.0.12"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d9/79/91f9d7cc8db5642acad830dcc4b49ba65a7790152832c4eceb305e46d681/spacy-legacy-3.0.12.tar.gz", hash = "sha256:b37d6e0c9b6e1d7ca1cf5bc7152ab64a4c4671f59c85adaf7a3fcb870357a774", size = 23806, upload-time = "2023-01-23T09:04:15.104Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c3/55/12e842c70ff8828e34e543a2c7176dac4da006ca6901c9e8b43efab8bc6b/spacy_legacy-3.0.12-py2.py3-none-any.whl", hash = "sha256:476e3bd0d05f8c339ed60f40986c07387c0a71479245d6d0f4298dbd52cda55f", size = 29971, upload-time = "2023-01-23T09:04:13.45Z" },
+]
+
+[[package]]
+name = "spacy-loggers"
+version = "1.0.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/67/3d/926db774c9c98acf66cb4ed7faf6c377746f3e00b84b700d0868b95d0712/spacy-loggers-1.0.5.tar.gz", hash = "sha256:d60b0bdbf915a60e516cc2e653baeff946f0cfc461b452d11a4d5458c6fe5f24", size = 20811, upload-time = "2023-09-11T12:26:52.323Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/33/78/d1a1a026ef3af911159398c939b1509d5c36fe524c7b644f34a5146c4e16/spacy_loggers-1.0.5-py3-none-any.whl", hash = "sha256:196284c9c446cc0cdb944005384270d775fdeaf4f494d8e269466cfa497ef645", size = 22343, upload-time = "2023-09-11T12:26:50.586Z" },
+]
+
+[[package]]
+name = "srsly"
+version = "2.5.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "catalogue" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2b/db/f794f219a6c788b881252d2536a8c4a97d2bdaadc690391e1cb53d123d71/srsly-2.5.3.tar.gz", hash = "sha256:08f98dbecbff3a31466c4ae7c833131f59d3655a0ad8ac749e6e2c149e2b0680", size = 490881, upload-time = "2026-03-23T11:56:59.865Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9d/5c/12901e3794f4158abc6da750725aad6c2afddb1e4227b300fe7c71f66957/srsly-2.5.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e67b6bbacbfadea5e100266d2797f2d4cec9883ea4dc84a5537673850036a8d8", size = 656750, upload-time = "2026-03-23T11:56:10.708Z" },
+ { url = "https://files.pythonhosted.org/packages/04/61/181c26370995f96f56f1b64b801e3ca1e0d703fc36506ae28606d62369fb/srsly-2.5.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:348c231b4477d8fe86603131d0f166d2feac9c372704dfc4398be71cc5b6fb07", size = 656746, upload-time = "2026-03-23T11:56:12.28Z" },
+ { url = "https://files.pythonhosted.org/packages/77/c6/35876c78889f8ffe11ed3521644e666c3aef20ea31527b70f47456cf35c2/srsly-2.5.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b0938c2978c91ae1ef9c1f2ba35abb86330e198fb23469e356eba311e02233ee", size = 1155762, upload-time = "2026-03-23T11:56:14.075Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/da/40b71ca9906c8eb8f8feb6ac11d33dad458c85a56e1de764b96d402168a0/srsly-2.5.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5f6a837954429ecbe6dcdd27390d2fb4c7d01a3f99c9ffcf9ce66b2a6dd1b738", size = 1161092, upload-time = "2026-03-23T11:56:15.778Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/14/c0dd30cc8b93ce8137ff4766f743c882440ce49195fffc5d50eaeef311a6/srsly-2.5.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3576c125c486ce2958c2047e8858fe3cfc9ea877adfa05203b0986f9badee355", size = 1109984, upload-time = "2026-03-23T11:56:17.056Z" },
+ { url = "https://files.pythonhosted.org/packages/08/f3/34354f183d8faafc631585571224b54d1b4b67e796972c36519c074ca355/srsly-2.5.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5fb59c42922e095d1ea36085c55bc16e2adb06a7bfe57b24d381e0194ae699f2", size = 1128409, upload-time = "2026-03-23T11:56:18.761Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/d9/5531f8a19492060b4e76e4ab06aca6f096fb5128fe18cc813d1772daf653/srsly-2.5.3-cp313-cp313-win_amd64.whl", hash = "sha256:111805927f05f5db440aeeacb85ce43da0b19ce7b2a09567a9ef8d30f3cc4d83", size = 650820, upload-time = "2026-03-23T11:56:20.096Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/8a/62fb7a971eca29e12f03fb9ddacb058548c14d33e5b5675ff0f85839cc7b/srsly-2.5.3-cp313-cp313-win_arm64.whl", hash = "sha256:0f106b0a700ab56e4a7c431b0f1444009ab6cb332edc7bbf6811c2a43f4722cb", size = 637278, upload-time = "2026-03-23T11:56:21.439Z" },
+]
+
+[[package]]
+name = "sse-starlette"
+version = "3.4.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "starlette" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d2/1b/bc9e3e7a72dcdad7dc7888758f5d00f56f8909ed5cfdff822bd72bb4c520/sse_starlette-3.4.5.tar.gz", hash = "sha256:83072538bc211a2f68b7b0422226c4af3e9b62e106e07034664b832ca019842a", size = 35249, upload-time = "2026-06-20T17:36:58.322Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/78/75/c88d3f5dafd59c791da1ce27650d30bf5b70cbf1cbf01cd00e5f9e360915/sse_starlette-3.4.5-py3-none-any.whl", hash = "sha256:e71bad53323f65573c3864a6c3bd0c1eb6e5f092b2e48082b0c35927d19ca296", size = 16518, upload-time = "2026-06-20T17:36:56.729Z" },
+]
+
+[[package]]
+name = "stack-data"
+version = "0.6.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "asttokens" },
+ { name = "executing" },
+ { name = "pure-eval" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/e3/55dcc2cfbc3ca9c29519eb6884dd1415ecb53b0e934862d3559ddcb7e20b/stack_data-0.6.3.tar.gz", hash = "sha256:836a778de4fec4dcd1dcd89ed8abff8a221f58308462e1c4aa2a3cf30148f0b9", size = 44707, upload-time = "2023-09-30T13:58:05.479Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl", hash = "sha256:d5558e0c25a4cb0853cddad3d77da9891a08cb85dd9f9f91b9f8cd66e511e695", size = 24521, upload-time = "2023-09-30T13:58:03.53Z" },
+]
+
+[[package]]
+name = "stanza"
+version = "1.13.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "emoji" },
+ { name = "huggingface-hub" },
+ { name = "networkx" },
+ { name = "numpy" },
+ { name = "platformdirs" },
+ { name = "protobuf" },
+ { name = "requests" },
+ { name = "torch" },
+ { name = "tqdm" },
+ { name = "udtools" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/60/69/68561cc2877afa9105093fba3ebd472f64d4b52d02964c4b16a70d8bbe5b/stanza-1.13.0.tar.gz", hash = "sha256:0f5e80481b7200871149f919529e6d5cdf1e7362de268c63d9e8bc735a4b336c", size = 1643678, upload-time = "2026-06-18T19:14:16.817Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/14/7a4487f20096583e8ec0cfa966e016acc62bf8ef16f866c5351e920d58b4/stanza-1.13.0-py3-none-any.whl", hash = "sha256:7a4c06659e1fd96a67ea4681fda2288c1a36b6be2636d6f11585da71b8c3f6eb", size = 1883208, upload-time = "2026-06-18T19:14:12.949Z" },
+]
+
+[[package]]
+name = "starlette"
+version = "1.3.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/eb/e3/7c1dc7381d9f8ab7d854328ebfa884e62cb3f3d8549ddfd37c7814f42afa/starlette-1.3.1.tar.gz", hash = "sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0", size = 2703240, upload-time = "2026-06-12T09:23:11.602Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/bb/2799cc2ede3ed41131f8975621e7213dfc7ef4acbbaadfa440f32500c370/starlette-1.3.1-py3-none-any.whl", hash = "sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6", size = 73632, upload-time = "2026-06-12T09:23:10.017Z" },
+]
+
+[[package]]
+name = "sudachidict-core"
+version = "20260428"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "sudachipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f4/07/8c3f16746e6eb8766d7f2e6af11867d75e80dfca91db3ca4a5008848d0ea/sudachidict_core-20260428.tar.gz", hash = "sha256:5492112a4a0999116cb8e4327b5ad3263167e9451fc8b50e8e7b83fc00bd2d98", size = 9077, upload-time = "2026-04-30T01:51:42.688Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/49/14/74ee9e3b69f1bc18a1ab05b262062d10dc3d282cedbe7b3ebd6f56fb0936/sudachidict_core-20260428-py3-none-any.whl", hash = "sha256:9cbd237eac62cb6e3347aa3ccff9e78b18298d629c4fc4295ed46817ff9df193", size = 72237533, upload-time = "2026-04-30T01:52:23.661Z" },
+]
+
+[[package]]
+name = "sudachipy"
+version = "0.6.11"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/13/34/a52dc020e1ac67ffe2469d22771a850589a1184e0d91d48addd364ee02ba/sudachipy-0.6.11.tar.gz", hash = "sha256:4f03310fd3fc779b3000f49395c939d18a30081632d3cb14488426f7c07cc526", size = 180616, upload-time = "2026-04-13T05:27:26.857Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/09/71/4d4a6b771fef1571d2f6e73d25ffcb3e8de462af674464c1fd41c9787899/sudachipy-0.6.11-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f38624fc01c2bb087c875a32750e3735b67b6b66487ac30a8e3f2d3599c4e3ea", size = 2964705, upload-time = "2026-04-13T05:26:46.895Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/6f/cc4b530725321c098dbaf57fdc74306e22224b8b5c5fd88feaf8a501aa20/sudachipy-0.6.11-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7a07a9a9232137353bb96d1ae2ce7e33def1ac48b571e338b3433b90581a9bbb", size = 1526522, upload-time = "2026-04-13T05:26:48.038Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/a8/94cbedeef4179536ee716504bdf5772060944dce7fa6cf0d3648c50a46de/sudachipy-0.6.11-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d074154c652ed6637a6a3581c67346ce642e47173e849267deb8c711b74157c4", size = 1461288, upload-time = "2026-04-13T05:26:49.388Z" },
+ { url = "https://files.pythonhosted.org/packages/98/52/447115dc0a638081fd3f48a54f4ac8f455847366504fc68029ab1b7bd59b/sudachipy-0.6.11-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b87831c5f98d634da04629bb6e3cace2a9aee275e83c1b62c8fdd7da4aca2876", size = 1582788, upload-time = "2026-04-13T05:26:50.664Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/7e/1b2cbdc8f1a889b76db9aad5599a86ed08d894a197b41dcdab265b1f8440/sudachipy-0.6.11-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cc86a6832448a56bcc6200b078a9bebd6059d925328214b96ee6b8830b8d34ab", size = 1627623, upload-time = "2026-04-13T05:26:52.701Z" },
+ { url = "https://files.pythonhosted.org/packages/69/4a/d5c1021335edfe8d0e78befb591fb11f4f8543bc0db1acf0ffc4ba135650/sudachipy-0.6.11-cp313-cp313-win_amd64.whl", hash = "sha256:c56d8308e799bff8f0edc0a9921f0bacf9b0b65c5c5df031022a048402f1d9ac", size = 1326405, upload-time = "2026-04-13T05:26:54.709Z" },
+ { url = "https://files.pythonhosted.org/packages/db/20/c1ed0b8ab7a9f21d8954659de8113c5ae2f3a1667f21260b408038a94d78/sudachipy-0.6.11-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:ca04ecca9530d7b663f34f32309ad0961b3f85a8921908ecf39274e00522b598", size = 2963252, upload-time = "2026-04-13T05:26:56.263Z" },
+ { url = "https://files.pythonhosted.org/packages/67/a3/064353cbc163866aaf42f7735c9b85ea1ff8376ae1ad1b0a60f5a4e97a35/sudachipy-0.6.11-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:445cd4f8596d308582743cc48425b2c0aebcda04b37b7f18e00ad927d86194b1", size = 1525951, upload-time = "2026-04-13T05:26:57.48Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/2c/022b1c2e491bddfd2daa488314973f864941992e20d602e8713672b37765/sudachipy-0.6.11-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b49d244423c39acb6ffa44a32c0296464f7454213f872d8738e9c43db4bc33d4", size = 1460367, upload-time = "2026-04-13T05:26:58.719Z" },
+ { url = "https://files.pythonhosted.org/packages/02/51/42de4a2cb82e428b3f2d04cc5f40370477724429f6dcb5acadb3af488128/sudachipy-0.6.11-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5f7d234e77577ac62fb207e65bb925bdec08fb8a3ac8df52302d3903fd581e26", size = 1581432, upload-time = "2026-04-13T05:26:59.836Z" },
+ { url = "https://files.pythonhosted.org/packages/24/e9/b6fb38b6788fdbdeb5e08c26200468a636df4e0ed8fb85b44572e184f6e5/sudachipy-0.6.11-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2e5316b6279c93eca1b325a0f6eda16c284d647daaa1215ee994dd9c543decd4", size = 1627693, upload-time = "2026-04-13T05:27:00.929Z" },
+]
+
+[[package]]
+name = "supervisor"
+version = "4.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a9/b5/37e7a3706de436a8a2d75334711dad1afb4ddffab09f25e31d89e467542f/supervisor-4.3.0.tar.gz", hash = "sha256:4a2bf149adf42997e1bb44b70c43b613275ec9852c3edacca86a9166b27e945e", size = 468912, upload-time = "2025-08-23T18:25:02.418Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0e/65/5e726c372da8a5e35022a94388b12252710aad0c2351699c3d76ae8dba78/supervisor-4.3.0-py2.py3-none-any.whl", hash = "sha256:0bcb763fddafba410f35cbde226aa7f8514b9fb82eb05a0c85f6588d1c13f8db", size = 320736, upload-time = "2025-08-23T18:25:00.767Z" },
+]
+
+[[package]]
+name = "sympy"
+version = "1.14.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mpmath" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" },
+]
+
+[[package]]
+name = "tabledata"
+version = "1.3.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dataproperty" },
+ { name = "typepy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/75/65/2f54f0dedd775dde48e300023d20e13ad329a51e33dcadb6d47b4dc95768/tabledata-1.3.5.tar.gz", hash = "sha256:98c64d0ad6b520846b41000fb3f5b2f42fa7ca2675c2c669e5ccab6b93082a36", size = 25396, upload-time = "2026-05-11T12:03:26.367Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c7/86/37fa0e1437089f08b8b1b8c8ad93f6b57e9427753f002914299323300a9e/tabledata-1.3.5-py3-none-any.whl", hash = "sha256:a1e57afc4767b51bef551114c0df31f205d712dbb75e3caf9be7834a79f23136", size = 11919, upload-time = "2026-05-11T12:03:24.907Z" },
+]
+
+[[package]]
+name = "tabulate"
+version = "0.10.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/46/58/8c37dea7bbf769b20d58e7ace7e5edfe65b849442b00ffcdd56be88697c6/tabulate-0.10.0.tar.gz", hash = "sha256:e2cfde8f79420f6deeffdeda9aaec3b6bc5abce947655d17ac662b126e48a60d", size = 91754, upload-time = "2026-03-04T18:55:34.402Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" },
+]
+
+[[package]]
+name = "tcolorpy"
+version = "0.1.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/80/cc/44f2d81d8f9093aad81c3467a5bf5718d2b5f786e887b6e4adcfc17ec6b9/tcolorpy-0.1.7.tar.gz", hash = "sha256:0fbf6bf238890bbc2e32662aa25736769a29bf6d880328f310c910a327632614", size = 299437, upload-time = "2024-12-29T15:24:23.847Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/a2/ed023f2edd1e011b4d99b6727bce8253842d66c3fbf9ed0a26fc09a92571/tcolorpy-0.1.7-py3-none-any.whl", hash = "sha256:26a59d52027e175a37e0aba72efc99dda43f074db71f55b316d3de37d3251378", size = 8096, upload-time = "2024-12-29T15:24:21.33Z" },
+]
+
+[[package]]
+name = "tenacity"
+version = "9.1.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/47/c6/ee486fd809e357697ee8a44d3d69222b344920433d3b6666ccd9b374630c/tenacity-9.1.4.tar.gz", hash = "sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a", size = 49413, upload-time = "2026-02-07T10:45:33.841Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" },
+]
+
+[[package]]
+name = "termcolor"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b8/85/147a0529b4e80b6b9d021ca8db3a820fcac53ec7374b87073d004aaf444c/termcolor-2.3.0.tar.gz", hash = "sha256:b5b08f68937f138fe92f6c089b99f1e2da0ae56c52b78bf7075fd95420fd9a5a", size = 12163, upload-time = "2023-04-23T19:45:24.004Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/67/e1/434566ffce04448192369c1a282931cf4ae593e91907558eaecd2e9f2801/termcolor-2.3.0-py3-none-any.whl", hash = "sha256:3afb05607b89aed0ffe25202399ee0867ad4d3cb4180d98aaf8eefa6a5f7d475", size = 6872, upload-time = "2023-04-23T19:45:22.671Z" },
+]
+
+[[package]]
+name = "terminado"
+version = "0.18.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ptyprocess", marker = "os_name != 'nt'" },
+ { name = "pywinpty", marker = "os_name == 'nt' and sys_platform != 'darwin'" },
+ { name = "tornado" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/8a/11/965c6fd8e5cc254f1fe142d547387da17a8ebfd75a3455f637c663fb38a0/terminado-0.18.1.tar.gz", hash = "sha256:de09f2c4b85de4765f7714688fff57d3e75bad1f909b589fde880460c753fd2e", size = 32701, upload-time = "2024-03-12T14:34:39.026Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6a/9e/2064975477fdc887e47ad42157e214526dcad8f317a948dee17e1659a62f/terminado-0.18.1-py3-none-any.whl", hash = "sha256:a4468e1b37bb318f8a86514f65814e1afc977cf29b3992a4500d9dd305dcceb0", size = 14154, upload-time = "2024-03-12T14:34:36.569Z" },
+]
+
+[[package]]
+name = "textual"
+version = "8.2.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown-it-py", extra = ["linkify"] },
+ { name = "mdit-py-plugins" },
+ { name = "platformdirs" },
+ { name = "pygments" },
+ { name = "rich" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/00/21/39a76b01bd5eea82a04baaca7580e105d8c59450df03998345bb2cfb307b/textual-8.2.8.tar.gz", hash = "sha256:3f106a9fbc73e39dd266c9712432087de78a6d644084c7c241d6a25c3169115b", size = 1860502, upload-time = "2026-06-30T06:51:24.495Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/be/35261223d9416a0751cdff1c7b4a6f881387218a12d439fe22fefebc8c04/textual-8.2.8-py3-none-any.whl", hash = "sha256:267375fd402dc8d981457212efa71f0e3365fd17bba144ba9bb3ed7563cb374a", size = 731418, upload-time = "2026-06-30T06:51:26.364Z" },
+]
+
+[[package]]
+name = "thinc"
+version = "8.3.13"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "blis" },
+ { name = "catalogue" },
+ { name = "confection" },
+ { name = "cymem" },
+ { name = "murmurhash" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "preshed" },
+ { name = "pydantic" },
+ { name = "setuptools" },
+ { name = "srsly" },
+ { name = "wasabi" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/13/46/76df95f2c327f9a9cef30c1523bf285627897097163584dcf5f77b2ebce2/thinc-8.3.13.tar.gz", hash = "sha256:68e658549fc1eb3ff92aed5147fcbb9c15d6e9cc0e623b4d0998d16522ffb4f9", size = 194640, upload-time = "2026-03-23T07:22:36.41Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/af/b9/7b46942176df459d1804a9e77b0976f7c56f3abf3ec7485d0e5f836a0382/thinc-8.3.13-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c2811dfd8d46d8b5d3b39051b23e64006b2994a5143b1978b436938018792af8", size = 817337, upload-time = "2026-03-23T07:22:09.538Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/79/53085a72cd8f4fc4e6e313d05ea5aa98e870684f4a0fb318a9875fc0a964/thinc-8.3.13-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:5593e6300cb1ebe0c0e546e9c9fb49e7c2627a0aa688795cd4f995a8b820d2ec", size = 788120, upload-time = "2026-03-23T07:22:11.215Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/3e/d61b462b16da95ac6885f95bb395e672040ee594833e571a6edcffd234f5/thinc-8.3.13-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f697174d3fb474966ce50b430bbafa101a6d2f7ffb559dac4b5c59389ef72d22", size = 3844666, upload-time = "2026-03-23T07:22:12.67Z" },
+ { url = "https://files.pythonhosted.org/packages/78/4c/898cc654bb123734c71ec5a425c02ca34439517d01ce1c95a6563295580e/thinc-8.3.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e9c7c5c104737b414c8c4ec578e67d78b6c859afe25cbc0684402e721415bd7f", size = 3890658, upload-time = "2026-03-23T07:22:14.668Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/56/1abdbf0a4ad628e8a05d6516fe0745969649d805367a3dccad8ee872981b/thinc-8.3.13-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7a99d0e242d1ccd23f9ae6bea7cd502f8626efa65c156b91d84581d0356696c3", size = 4819933, upload-time = "2026-03-23T07:22:16.85Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/22/b84dbdc6be5055bbdb2a7352e2c393f67e8593c137f1b83c82bf1e062b6e/thinc-8.3.13-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e676edd21a747afbe3e6b9f3fca8b962e36d146ded03b070cb0c28e2dfbe9499", size = 5018099, upload-time = "2026-03-23T07:22:18.356Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/a8/763cd7ba949334c9d2cddc92dadb68b344cb9546dc01b8d4a733dcaa16c1/thinc-8.3.13-cp313-cp313-win_amd64.whl", hash = "sha256:8ad40307f20e83f77af28ff5c6be0b86af7a8b251d1231c545508d2763157d8f", size = 1720309, upload-time = "2026-03-23T07:22:19.81Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/15/a11f7bb3cbc97dfecf32a90552f5a8f8a5c99316a99c6c17bdabf5baf256/thinc-8.3.13-cp313-cp313-win_arm64.whl", hash = "sha256:723949cab11d1925c15447928513a718276316cec6e0de28337cca0a62be0521", size = 1644606, upload-time = "2026-03-23T07:22:21.339Z" },
+]
+
+[[package]]
+name = "threadpoolctl"
+version = "3.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b7/4d/08c89e34946fce2aec4fbb45c9016efd5f4d7f24af8e5d93296e935631d8/threadpoolctl-3.6.0.tar.gz", hash = "sha256:8ab8b4aa3491d812b623328249fab5302a68d2d71745c8a4c719a2fcaba9f44e", size = 21274, upload-time = "2025-03-13T13:49:23.031Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" },
+]
+
+[[package]]
+name = "tiktoken"
+version = "0.13.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "regex" },
+ { name = "requests" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e4/e5/5f3cb2159769d0f4324c0e9e87f9de3c4b1cd45848a96b2eb3566ad5ca77/tiktoken-0.13.0.tar.gz", hash = "sha256:c9435714c3a84c2319499de9a300c0e604449dd0799ff246458b3bb6a7f433c1", size = 38986, upload-time = "2026-05-15T04:51:27.153Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9c/83/b096c859c2a47c11731bf2f5885f4028b809dfe2396582883eed9cae372f/tiktoken-0.13.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5df5d1507bd245f1ccad4a074698240021239e455eb0bb4ced4e3d7181872154", size = 1034228, upload-time = "2026-05-15T04:50:40.988Z" },
+ { url = "https://files.pythonhosted.org/packages/53/61/c68e123b6d753e3fc2751e9b18e732c9d8bf1e1926762e736eee935d931c/tiktoken-0.13.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8fe806a50664e83a6ffd56cbd1e4f5dcc6cd32a3e7538f70dc38b1a271384545", size = 982978, upload-time = "2026-05-15T04:50:42.195Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/8b/96cc178cc584e65d363134500f297790b06cd48cdeb1e8fcf7bbe60f4715/tiktoken-0.13.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:125bc05005e747f993a83dc67934249932d6e4209854452cd4c0b1d53fba3ba2", size = 1116355, upload-time = "2026-05-15T04:50:43.564Z" },
+ { url = "https://files.pythonhosted.org/packages/86/f5/bab735d2c72ea55404b295d02d092644eb5f7cc6205e34d35eb9abfb9ab2/tiktoken-0.13.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:5e6358911cab4adee6712da27d65573496a4f68cf8a2b5fca6a4ad10fc5748cf", size = 1135772, upload-time = "2026-05-15T04:50:44.782Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/b9/6de04ebdf904edfaad87788011b3735087a0c9ea671b9027e1e4e965e8c8/tiktoken-0.13.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:975cbd78d085d75d26b59660e262736dcaed1e35f8f142cd6291025c01d25486", size = 1182415, upload-time = "2026-05-15T04:50:46.422Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/9c/470a05f3b1caf038f44880e334d47ab674e0c80d514c66b375d14d5afa10/tiktoken-0.13.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:75ab9bc99fa020a4c283424590ecd7f3afd70c1c281cb3fa3192a6c3af9f9615", size = 1239879, upload-time = "2026-05-15T04:50:48.052Z" },
+ { url = "https://files.pythonhosted.org/packages/42/a6/c1936d16055436cb32e6c6128d68629622e00f4768562f55653752d34768/tiktoken-0.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:6b1615f0ff71953d19729ceb18865429c185b0a23c5353f1bbca34a394bf60f7", size = 874829, upload-time = "2026-05-15T04:50:49.202Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/07/acb5992c3772b5a36284f742cfb7a5895aa4471d1848ac31464ad50d7fdf/tiktoken-0.13.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:6eb4a5bfbc6426938026b1a334e898ac53541360d62d8c689870160cc80abd67", size = 1033600, upload-time = "2026-05-15T04:50:50.4Z" },
+ { url = "https://files.pythonhosted.org/packages/14/e9/742e9aec30f59b9f161f7ff7cd072e02ea836c9e1c0854a8076dfcd40d5c/tiktoken-0.13.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:43cee3e5400573b2046fbf092cc7a5bc30164f9e4c95ce20714da929df48737a", size = 982516, upload-time = "2026-05-15T04:50:52.03Z" },
+ { url = "https://files.pythonhosted.org/packages/72/74/ca1541b053e7648254d2e4b42a253e1bb4359f2c91a0a8d49228c794e1a0/tiktoken-0.13.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:7de52e3f566d19b3b11bd37eea552c6c305ad74081f736882bd44d148ed4c48d", size = 1115518, upload-time = "2026-05-15T04:50:53.543Z" },
+ { url = "https://files.pythonhosted.org/packages/46/e3/93825eaf5a4a504795b787e5d5dea07fbeb3dabf97aa7b450be8bde59c89/tiktoken-0.13.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:51384448aa508e4df84c0f7c1dc3211c7f7b8096325660ee5fc82f3e11b381ce", size = 1136867, upload-time = "2026-05-15T04:50:55.191Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/46/002b68de6827091d5ae90b048f326e8aad8d953520950e5ce1508879414f/tiktoken-0.13.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:e28157350f7ebf35008dd8e9e0fdb621f976e4230c881099c85e8cf07eaa50e2", size = 1181826, upload-time = "2026-05-15T04:50:56.296Z" },
+ { url = "https://files.pythonhosted.org/packages/db/c6/d393e3185a276505182f7abd93fe714f3c444a2be9180798fa052347504e/tiktoken-0.13.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:165cf1820ea4a354985c2490a5205d4cc74661c934aca79dd0368232fff94e0f", size = 1239489, upload-time = "2026-05-15T04:50:57.918Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/4d/bc07d1f1635d4897a202acc0ae11c2886eaa7325c359ba4741b47bf8e225/tiktoken-0.13.0-cp313-cp313t-win_amd64.whl", hash = "sha256:6c43a675ca14f6f2749ba7f12075d37456015a24b859f2517b9beb4ef30807ec", size = 873820, upload-time = "2026-05-15T04:50:59.528Z" },
+]
+
+[[package]]
+name = "tilelang"
+version = "0.1.12"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "apache-tvm-ffi" },
+ { name = "cloudpickle" },
+ { name = "ml-dtypes" },
+ { name = "numpy" },
+ { name = "psutil" },
+ { name = "setuptools", marker = "sys_platform == 'darwin'" },
+ { name = "torch" },
+ { name = "torch-c-dlpack-ext" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+ { name = "z3-solver" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6d/9a/5776adb5b6e03e141c8cff020e16c45b87c9b7a7a5bdfe83416d6cc23933/tilelang-0.1.12.tar.gz", hash = "sha256:595b39581d099a53f875bd8d553863e7e4f58998052b58764f5472cbfbb87043", size = 93565090, upload-time = "2026-07-08T04:22:52.901Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/13/51/4ae1fb532406082a4ebf16c890f5f98db14b8588f2c486334ba23578b4bc/tilelang-0.1.12-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:8741d6cd519782645d55c17abd8986402c62948bc2122d3417c445d11de5dd16", size = 38347295, upload-time = "2026-07-08T04:22:35.691Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/53/f281a0bd9ee7e03d6a97828fc0e443321ed26ea2b0bd74bf9f1d9451d30f/tilelang-0.1.12-cp38-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bbeb5573cbe2544a51a5c9a6cc737c5e3b5c2d95411caa3687fd9b08eb9d5f97", size = 50501074, upload-time = "2026-07-08T04:22:39.973Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/a4/ff0b14d00d29fe418063cd97490f55aa9cad6f981a6fbbf29124b024637d/tilelang-0.1.12-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:a1701eba4ad603d73f1b128b88f6a2350f1b1161729e5b2d1e540a1ec7a82d52", size = 46281933, upload-time = "2026-07-08T04:22:44.244Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/15/1a8a5b5df0a23ece80b4da8e49c43418244e5ab165ed1938a1cc2656cda2/tilelang-0.1.12-cp38-abi3-win_amd64.whl", hash = "sha256:daeb3d529ab9e7665adb94c4bf0d47732294709aa9b8b9bd673fa48001385422", size = 34020025, upload-time = "2026-07-08T04:22:48.066Z" },
+]
+
+[[package]]
+name = "tinycss2"
+version = "1.5.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "webencodings" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a3/ae/2ca4913e5c0f09781d75482874c3a95db9105462a92ddd303c7d285d3df2/tinycss2-1.5.1.tar.gz", hash = "sha256:d339d2b616ba90ccce58da8495a78f46e55d4d25f9fd71dfd526f07e7d53f957", size = 88195, upload-time = "2025-11-23T10:29:10.082Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/60/45/c7b5c3168458db837e8ceab06dc77824e18202679d0463f0e8f002143a97/tinycss2-1.5.1-py3-none-any.whl", hash = "sha256:3415ba0f5839c062696996998176c4a3751d18b7edaaeeb658c9ce21ec150661", size = 28404, upload-time = "2025-11-23T10:29:08.676Z" },
+]
+
+[[package]]
+name = "tokenizers"
+version = "0.22.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115, upload-time = "2026-01-05T10:45:15.988Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275, upload-time = "2026-01-05T10:41:02.158Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472, upload-time = "2026-01-05T10:41:00.276Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736, upload-time = "2026-01-05T10:40:32.165Z" },
+ { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835, upload-time = "2026-01-05T10:40:38.847Z" },
+ { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673, upload-time = "2026-01-05T10:40:56.614Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818, upload-time = "2026-01-05T10:40:44.507Z" },
+ { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195, upload-time = "2026-01-05T10:40:51.139Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982, upload-time = "2026-01-05T10:40:58.331Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245, upload-time = "2026-01-05T10:41:04.053Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069, upload-time = "2026-01-05T10:45:10.673Z" },
+ { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263, upload-time = "2026-01-05T10:45:12.559Z" },
+ { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429, upload-time = "2026-01-05T10:45:14.333Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" },
+ { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" },
+ { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" },
+]
+
+[[package]]
+name = "tokenspeed-mla"
+version = "0.1.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "apache-tvm-ffi", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cutlass-dsl", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "tokenspeed-triton", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "torch", marker = "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/65/81d7e9f14472bc4c6abb576c9b1edd8e40ab01832027c4e647bfe2890749/tokenspeed_mla-0.1.8-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:952209cf4b29a54e6b6e7e088be9d4a40f24792b06fdfa330de8077b9926a7d9", size = 752341, upload-time = "2026-06-24T03:36:28.619Z" },
+ { url = "https://files.pythonhosted.org/packages/27/df/0037ade72b165ac97859040919e006aa3d80cb8cc3a79420fb6c03eb16a0/tokenspeed_mla-0.1.8-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:6a7526d7327746893f8c20d24aa63ba5b8a123d0dfd6e66388e13b768b6452c6", size = 755827, upload-time = "2026-06-24T03:36:29.96Z" },
+]
+
+[[package]]
+name = "tokenspeed-triton"
+version = "3.7.10.post20260531"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/84/58/fdb5fb70d99c1f18f01c2198420fa2a0f7e5301bd7dd5b5f34b22a3cb87b/tokenspeed_triton-3.7.10.post20260531-cp312-abi3-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16cd0a3fc1cffeb458a7e03e8688714f49fdf0b5a108bfca999f46597c3faabb", size = 81636010, upload-time = "2026-05-31T01:29:17.699Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/49/7bae94729bfd7a3f331795251302f0b0c8e54a7ec25b3af5d5bfe133367c/tokenspeed_triton-3.7.10.post20260531-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b90ac41e7f15933797545ff1a9e803a9d8beb4ca9ba70f6d41a9e0fc26484f5c", size = 85888791, upload-time = "2026-05-31T01:29:25.584Z" },
+]
+
+[[package]]
+name = "torch"
+version = "2.13.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-bindings", version = "13.3.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'linux'" },
+ { name = "cuda-toolkit", extra = ["cublas", "cudart", "cufft", "cufile", "cupti", "curand", "cusolver", "cusparse", "nvjitlink", "nvrtc", "nvtx"], marker = "sys_platform == 'linux'" },
+ { name = "filelock" },
+ { name = "fsspec" },
+ { name = "jinja2" },
+ { name = "networkx" },
+ { name = "nvidia-cudnn-cu13", marker = "sys_platform == 'linux'" },
+ { name = "nvidia-cusparselt-cu13", marker = "sys_platform == 'linux'" },
+ { name = "nvidia-nccl-cu13", marker = "sys_platform == 'linux'" },
+ { name = "nvidia-nvshmem-cu13", marker = "sys_platform == 'linux'" },
+ { name = "setuptools" },
+ { name = "sympy" },
+ { name = "triton", version = "3.7.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'linux'" },
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0d/fa/c1c10b7aff4a9a3e8956d4f0a5f468fa6db7abc3208805719076772b4833/torch-2.13.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:33449899ce5496c1b84b4853179d94fd102028ae1407314d9fb956bb79e70d09", size = 111213743, upload-time = "2026-07-08T16:03:28.579Z" },
+ { url = "https://files.pythonhosted.org/packages/11/18/9ecb37b56293a0be8d80f810bf672a72fe7e02f8b475d5ef1b9bf8a0d748/torch-2.13.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:1e09d6a722504957c694faceca843acde562786df1144ebcc5a74075ec7f6005", size = 427213008, upload-time = "2026-07-08T16:03:44.106Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/5a/7c50ba1b7b713d71d34669c6d13dab0a11531a3eceb0307a5162dbfec0f7/torch-2.13.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:a3a9a21312872af8a26950b2c15680335a386a1f56ed03e780653d78b9607e9e", size = 526602329, upload-time = "2026-07-08T16:03:12.649Z" },
+ { url = "https://files.pythonhosted.org/packages/91/3d/e7adcc6aaf36961cd18f56cf8ad0f3058c3a5c84ccf391762176c94581b8/torch-2.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:49b58f1e2c52440abb6f17c28f0335fe6c6d01ad1a7f55b0183b81e4b34d64e6", size = 122057920, upload-time = "2026-07-08T16:03:01.808Z" },
+]
+
+[[package]]
+name = "torch-c-dlpack-ext"
+version = "0.1.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "torch" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/37/de/921b6491efce5c389a5ef9bbed3d2d6660005840dae488124173180859ab/torch_c_dlpack_ext-0.1.5.tar.gz", hash = "sha256:d06f0357d575d22a168cc77acb9020fc4bae30968ceb6718a055dcbe92bacabe", size = 12913, upload-time = "2026-01-12T11:25:08.484Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ca/c6/65346a201d921b616731311fc9941f15137672b444cebdad702cb52ccee0/torch_c_dlpack_ext-0.1.5-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:74acea2ed395cadda63342845b9e9ee7cd4537846223dacfb4431b4610109265", size = 1993243, upload-time = "2026-01-12T11:24:51.079Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/ec/faf10be09a5812b1c5ec9922b53fb5def5fc4080b81a653b9347bb169ebb/torch_c_dlpack_ext-0.1.5-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:49f1e99d13c64e22dac0a34a1560e9e5a398a49a9fa81df83053e04fde6ec5bd", size = 443798, upload-time = "2026-01-12T11:24:52.754Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/68/f434b48700f3e04f33882f54d8d3910327b935f55e14ec49da7d607bf470/torch_c_dlpack_ext-0.1.5-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:debe62e5ef93e631065d6b9f6e60d3d39bae6b89fa1b25d9523f40b3efbf8aba", size = 755004, upload-time = "2026-01-12T11:24:54.004Z" },
+ { url = "https://files.pythonhosted.org/packages/03/a8/cc64e563f05ea99bd79bdb43f71f0f46452d3acd734da4843ede5fc73a35/torch_c_dlpack_ext-0.1.5-cp313-cp313-win_amd64.whl", hash = "sha256:30e3eab616dbc81dfdb7492aca557be551a9163ba9b585f97394a42b336b113a", size = 999126, upload-time = "2026-01-12T11:24:55.44Z" },
+]
+
+[[package]]
+name = "torchaudio"
+version = "2.11.0"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/9e/f76fcd9877c8c78f258ee34e0fb8291fdb91e6218d582d9ca66b1e4bd4ae/torchaudio-2.11.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e3f9696a9ef1d49acc452159b052370c636406d072e9d8f10895fda87b591ea9", size = 679904, upload-time = "2026-03-23T18:13:28.329Z" },
+ { url = "https://files.pythonhosted.org/packages/85/70/249c1498ebdad3e7752866635ec0855fc0dcf898beccda5a9d2b9df8e4d0/torchaudio-2.11.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:b034d7672f1c415434f48ef17807f2cce47f29e8795338c751d4e596c9fbe8b5", size = 1618523, upload-time = "2026-03-23T18:13:15.703Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/98/be13fe35d9aa5c26381c0e453c828a789d15c007f8f7d08c95341d19974d/torchaudio-2.11.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:1c1101c1243ef0e4063ec63298977e2d3655c15cf88d9eb0a1bd4fe2db9f47ea", size = 1771992, upload-time = "2026-03-23T18:13:35.343Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/8b/2bbb3dca6ff28cba0de250874d5ef4fc2822c47a934b59b3974cff3219ef/torchaudio-2.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:986f4df5ed17b003dc52489468601720090e65f964f8bebccf90eb45bba75744", size = 328662, upload-time = "2026-03-23T18:13:18.308Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/ce/52c652d30af7d6e96c8f1735d26131e94708e3f38d852b8fa97958804dd8/torchaudio-2.11.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:bda09ea630ae7207384fb0f28c35e4f8c0d82dd6eba020b6b335ad0caa9fed49", size = 680814, upload-time = "2026-03-23T18:13:17.08Z" },
+ { url = "https://files.pythonhosted.org/packages/06/95/1ad1507482e7263e556709a3f5f87fecd375a0742cdaf238806c8e72eaad/torchaudio-2.11.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:9fe3083c62e035646483a14e180d33561bdc2eed436c9ab1259c137fb7120b4a", size = 1618546, upload-time = "2026-03-23T18:13:29.686Z" },
+ { url = "https://files.pythonhosted.org/packages/98/4c/480328ba07487eb9890406720304d0d460dd7a6a64098614f5aa53b662ca/torchaudio-2.11.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:13cff988697ccbad539987599f9dc672f40c417bed67570b365e4e5002bbd096", size = 1771991, upload-time = "2026-03-23T18:13:30.843Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/98/5d4790e2d6548768999acd34999d5aeefce8bcc23a07afaa5f03e723f557/torchaudio-2.11.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ed404c4399ad7f172c86a47c1b25293d322d1d58e26b10b0456a86cf67d37d84", size = 328661, upload-time = "2026-03-23T18:13:34.359Z" },
+]
+
+[[package]]
+name = "torchcodec"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/62/59ef3c1f377a08ca965cf71abd763db5cbf8f6a97901ca09dbe0809c4a29/torchcodec-0.16.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:154c5cf6c2dd2801700dcf0939081f614c50dcbf99f5ecc7f6854e98b7959339", size = 3898103, upload-time = "2026-08-13T11:56:15.303Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/83/7844821bd5d0a1d8bee0568d417d2b3f0663f379666edd22348f0f90e427/torchcodec-0.16.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d14e43ace5f9fe095287a5b4a39f1c4353c01493749e40ff4e17ebb791f741e8", size = 8368167, upload-time = "2026-08-13T11:56:17.301Z" },
+ { url = "https://files.pythonhosted.org/packages/41/1c/9b12cfd462ff2a7cbad52907de9dd1963bca7aa4bf6a4e2d618195af7275/torchcodec-0.16.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:26f4c4ee2896ddee5b38bf31ae96e721b853dba6b0b4104ac2e11391ddd7966d", size = 9471404, upload-time = "2026-08-13T11:56:19.143Z" },
+ { url = "https://files.pythonhosted.org/packages/38/76/0551e83bb23731806d98469cc8cdc887d55c3a242d7377a52430aa2cd6bf/torchcodec-0.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:158e1ed469a5300aeb843102f8d6c4354f97a835ce5dca087aa827c135bd4869", size = 6427477, upload-time = "2026-08-13T11:56:21.206Z" },
+]
+
+[[package]]
+name = "torchvision"
+version = "0.28.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "pillow" },
+ { name = "torch" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/20/55/08a726c14c67b37c8aca04b077766909f1c7ed23f76116884fe63b9bd033/torchvision-0.28.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:d483b4aa3f5237569053f749cd1a2b5bb548ca456e40461a5dd087f21149d123", size = 1856021, upload-time = "2026-07-08T16:07:45.386Z" },
+ { url = "https://files.pythonhosted.org/packages/db/8f/40beacd53809194f5259e590d1afaeaa8ad57da15f77c646e6560bcc4616/torchvision-0.28.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:bb6dd6918460ed89cc7644adcc2402991474d6933cf1ce92b390641cb233fddf", size = 7797014, upload-time = "2026-07-08T16:07:43.04Z" },
+ { url = "https://files.pythonhosted.org/packages/32/db/062cdb5a84380a60439775311fff34d89229760d2a50680393dc18699956/torchvision-0.28.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:ad7b3a439265cc3739a4ab5b4c998c0e38ea99c0ee7ca4dea35c5d0b099ec237", size = 7674669, upload-time = "2026-07-08T16:07:38.91Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/a6/b4081e2d04e1541abf82785ac9e5178a494c19330391f551356c8c18b7b3/torchvision-0.28.0-cp313-cp313-win_amd64.whl", hash = "sha256:7e9dd6f60d6e15f8dc27d4f877fdb6002fc70d70272412135f1c2ff9cfa08d3b", size = 4157380, upload-time = "2026-07-08T16:07:40.22Z" },
+]
+
+[[package]]
+name = "tornado"
+version = "6.5.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/64/24/95ec527ad67b76d59299e5465b3935d05e4294b7e0290a3924b7487df30b/tornado-6.5.7.tar.gz", hash = "sha256:66c513a76cda70d53907bc27cf1447557699c2e95aa48ba27a442ff61c3ddfc2", size = 519252, upload-time = "2026-06-08T17:34:51.232Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/02/dc/c7043cab6fed8ae159fc1923ce829ada35c4dbd797d408a43858ffaf9639/tornado-6.5.7-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:148b2eb15c2c765a50796172c1e499649b35f30d2e3c3d3e15913cfa56bfb163", size = 448543, upload-time = "2026-06-08T17:34:38.052Z" },
+ { url = "https://files.pythonhosted.org/packages/92/4f/090b1431e5a43df696feceffc268c5383cc079ecb5f08ce58f917109aafe/tornado-6.5.7-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:9da38de27f1da3b78a966f0dae12b5a1ea9afe72ca805d84ff06508272ddf100", size = 446707, upload-time = "2026-06-08T17:34:39.594Z" },
+ { url = "https://files.pythonhosted.org/packages/37/d8/ef374952fd5da67d4463122c2b8e5a96536ec10b4b339254c6dcde81d01c/tornado-6.5.7-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:8d759e71906ee783f8867b93bf26a265743da4c1e2f4a018464c1ba019862972", size = 449774, upload-time = "2026-06-08T17:34:41.204Z" },
+ { url = "https://files.pythonhosted.org/packages/35/37/d434c73f4c6e014b745b9b37085f34f40c022f007efff3d7fe65991899f3/tornado-6.5.7-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8a46347a18f23fb92b396beebe0fb78f61dda0cc302445202c16203d8a18848b", size = 450745, upload-time = "2026-06-08T17:34:42.531Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/2b/56b9aff361d7f1ab728a805ec7d7ea835f8807afa9f5cc690ea0e630efb9/tornado-6.5.7-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:7778b30bef919231265e91c69963ce0f49a1e9c07ac900bbe75b19ce2575ba92", size = 450578, upload-time = "2026-06-08T17:34:43.787Z" },
+ { url = "https://files.pythonhosted.org/packages/02/30/a7444fb23aa76860a14198fab96ac79f1866b0a6e19e26c4381b0938e50f/tornado-6.5.7-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e726f0c75da7726eec023aa62751ff8878bd2737e34fbdd33b1ae5897d2200f5", size = 449985, upload-time = "2026-06-08T17:34:45.326Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/42/5f0e56c01e8d9d36f4e23f367b85ae6cae0c1ecddd5e6977d8388ad27488/tornado-6.5.7-cp39-abi3-win32.whl", hash = "sha256:f8de3bf12d3efdd0cbe7c8887868198f8a91415e3f29fcf258d9b8eb7b1d9ae4", size = 451047, upload-time = "2026-06-08T17:34:46.784Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/a4/b393076ffb21b469eec5b328a0534cf03a3b90bfc6b1f09507cdd075d938/tornado-6.5.7-cp39-abi3-win_amd64.whl", hash = "sha256:de942f843533a039ef9fa3d9c88c7cd8a7c94553fb5ad0154270989b3d99a2c4", size = 451485, upload-time = "2026-06-08T17:34:48.248Z" },
+ { url = "https://files.pythonhosted.org/packages/71/2e/7b1c769803121b809112cf9a00681c472eae1d80e32d7ec0e0bd61d0d0e1/tornado-6.5.7-cp39-abi3-win_arm64.whl", hash = "sha256:ff934fce95643af5f11efdae618eaa73d469dc588641e5c8d19295a0c65c4796", size = 450506, upload-time = "2026-06-08T17:34:49.702Z" },
+]
+
+[[package]]
+name = "tqdm"
+version = "4.68.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/87/d7/0535a28b1f5f24f6612fb3ff1e89fb1a8d160fee0f976e0aa6803862134b/tqdm-4.68.3.tar.gz", hash = "sha256:00dfa48452b6b6cfae3dd9885636c23d3422d1ec97c66d96818cbd5e0821d482", size = 170596, upload-time = "2026-06-17T07:36:52.105Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d8/8e/bb97bb0c71802080bfc8952937d174e49cfc50de5c951dd47b2496f0dcdb/tqdm-4.68.3-py3-none-any.whl", hash = "sha256:39832cc2def2789a6f29df83f172db7416cea70052c0907a57801c5f2fdccb03", size = 78337, upload-time = "2026-06-17T07:36:50.132Z" },
+]
+
+[[package]]
+name = "traitlets"
+version = "5.15.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/57/a9/a2584b8313b89f94869ddb3c4074617a691de1812a614d2d50e32ca5a7a6/traitlets-5.15.1.tar.gz", hash = "sha256:7b1c07854fe25acb39e009bae49f11b79ff6cbb2f27999104e9110e7a6b53722", size = 163344, upload-time = "2026-06-03T12:26:06.181Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/96/8d/1080ee4c231f361b6ce4470d556c8c435b67c7e0753aaa641497ee92f88b/traitlets-5.15.1-py3-none-any.whl", hash = "sha256:770a53705f84b81ac107e83a1b3328ff2dae16094d8fc3cfc004e4b22dfd8e92", size = 85858, upload-time = "2026-06-03T12:26:04.395Z" },
+]
+
+[[package]]
+name = "transformers"
+version = "5.12.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pyyaml" },
+ { name = "regex" },
+ { name = "safetensors" },
+ { name = "tokenizers" },
+ { name = "tqdm" },
+ { name = "typer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/aa/7c/8240f612819718100a9346dc28dea6a11370c3ca9c8c6eabadd3dea4ef29/transformers-5.12.1.tar.gz", hash = "sha256:679ee731c8225347889ad4fb3b2c926a62e9da3b7d284e9d12c791da7272466b", size = 8924054, upload-time = "2026-06-15T17:27:50.604Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/df/56/bbd60dd8668055803bf8ba55a81f9b8a8b31497f620109a9671d26a2076d/transformers-5.12.1-py3-none-any.whl", hash = "sha256:2a5e109d2021265df7098ffbb738295acaf5ad256f12cbc586db2ea4dcbb1a8a", size = 11150587, upload-time = "2026-06-15T17:27:46.679Z" },
+]
+
+[[package]]
+name = "triton"
+version = "3.6.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform == 'win32'",
+ "sys_platform == 'emscripten'",
+ "sys_platform == 'darwin'",
+]
+
+[[package]]
+name = "triton"
+version = "3.7.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/00/42/c5089d4d9327fcd1e862c599cc2927f39418f84dd11a84cb2ccff9d4787a/triton-3.7.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cdbfc09d9ec58bc5e68321525653220de7515c199e7a8097a97c85e62b52cd0a", size = 184694629, upload-time = "2026-06-17T20:03:53.444Z" },
+ { url = "https://files.pythonhosted.org/packages/07/42/2c3ac59253ae8892b6f307875263dd23dc875cdf732d3aea40d6d41fb7cb/triton-3.7.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:58c0e131da05134a2a4788ccbcc0c1105cf0f54c8e98f19e34cd465396dc15eb", size = 197729241, upload-time = "2026-06-17T19:53:27.801Z" },
+]
+
+[[package]]
+name = "typepy"
+version = "1.3.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mbstrdecoder" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/8b/9f/ae119b0e0fd0fe8dcb0e1eeebfeb62f37fdc0b467267cff15cdb746ba38b/typepy-1.3.5.tar.gz", hash = "sha256:a1c5f54c41860f89bab175f512b11e8c9a57cfe7b8b3d5ae5d52d828b756b6dd", size = 39883, upload-time = "2026-05-04T14:04:32.835Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/64/71/75cf08c49b64a9419f1f2cef9be072ac34f6b784da2851489470b7c7ba15/typepy-1.3.5-py3-none-any.whl", hash = "sha256:de361b59609c7503efc2edbe9d7a4e053ae71307bf90ae1678ec4d6bcd807922", size = 31530, upload-time = "2026-05-04T14:04:31.46Z" },
+]
+
+[package.optional-dependencies]
+datetime = [
+ { name = "packaging" },
+ { name = "python-dateutil" },
+ { name = "pytz" },
+]
+
+[[package]]
+name = "typer"
+version = "0.26.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "annotated-doc" },
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "rich" },
+ { name = "shellingham" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7c/f7/68adc395201b20b872d68e975386832e8005ffeacedd43a1d837a32815be/typer-0.26.8.tar.gz", hash = "sha256:c244a6bd558886fe3f8780efb6bdd28bb9aff005a94eedebaa5cb32926fe2f7e", size = 202097, upload-time = "2026-06-26T09:22:45.705Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/80/87/b9fd69c92c6102a066e1b86a35243f53e70bd4c709f2a26d9f4fee4f4dc0/typer-0.26.8-py3-none-any.whl", hash = "sha256:3512ca79ac5c11113414b36e80281b872884477722440691c89d1112e321a49c", size = 122564, upload-time = "2026-06-26T09:22:44.72Z" },
+]
+
+[[package]]
+name = "typing-extensions"
+version = "4.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" },
+]
+
+[[package]]
+name = "typing-inspection"
+version = "0.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" },
+]
+
+[[package]]
+name = "typos"
+version = "1.48.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8f/de/056cddea7634857249aa980d4dd3696f4519ea17f29a1bac65fc53307101/typos-1.48.0.tar.gz", hash = "sha256:75fa7b982ff943c8efc67606cdbaa866791d410585aa9b642e8f9cdfe16be217", size = 1838686, upload-time = "2026-06-30T18:59:31.381Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bc/05/af202f90e61bb12f1a48fcce660796b8442a6cea9c9a2d6262fe46e6d243/typos-1.48.0-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:86ed4008416c39ff353094a1d8fb738704ec9dfa540bc046c1e6da750c4ea527", size = 3478006, upload-time = "2026-06-30T18:59:18.098Z" },
+ { url = "https://files.pythonhosted.org/packages/79/5e/1e2d4f058247375baa7f33df017700a21cd574600ba515606e8c505bea14/typos-1.48.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:db8182b546ae7067c4d6e61017c1498dca44551e27ec8036c352e46cd5e40c66", size = 3387908, upload-time = "2026-06-30T18:59:19.67Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/0d/beb9fe4df9a7af13b25ebc245b8543f07ca9163afadc24515da0dc95f88c/typos-1.48.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fa7ec06c13e9f22348d9c73ca1b99e861c47dbd0526c95a9283378040999a5de", size = 8254588, upload-time = "2026-06-30T18:59:21.1Z" },
+ { url = "https://files.pythonhosted.org/packages/95/77/1f2cf02f62dcf65548ca09d59b7f4aab2579fb33a427b1e4ec13e065da82/typos-1.48.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4e686af7a45c2fa2da3819d4620106ad2bc8e02ab274e65ed27f28dfb13ad1d6", size = 7334558, upload-time = "2026-06-30T18:59:22.638Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/de/20a2d4b8fc67376b6aad876671b840d5386b70782f7d280c94d08b945bb0/typos-1.48.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2dbe2950add1a04f5653b9f51e97107c4b229e488a26630f344286725ebe61d9", size = 7756907, upload-time = "2026-06-30T18:59:24.137Z" },
+ { url = "https://files.pythonhosted.org/packages/56/3a/1ef3bca7f932f03f43bb3101c963450ebd9281aec7adca25a99d7e010f43/typos-1.48.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:b2dcdcdeca1711014e3070ab09d0b617d1fae166a41a2819c6c81c2ad0f2e559", size = 7105528, upload-time = "2026-06-30T18:59:25.527Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/79/58920d4fe3ebfaae69f481c1d11a58c0868ac60cd84d8d7555ad8fa9c9de/typos-1.48.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:0aa61e85ab19d548b0b16d5d5ab88c034665de772689bdbb74c5b0740a458629", size = 8152031, upload-time = "2026-06-30T18:59:27.151Z" },
+ { url = "https://files.pythonhosted.org/packages/83/e6/a1bc6bc09d6ad96678576a8a76c0f57604b6d52f12b6ef97563dfd618361/typos-1.48.0-py3-none-win32.whl", hash = "sha256:67afb5bc1e0de783a341cc891240c27f292c6ec440c4b6204f1295bb3dcbbb49", size = 3145061, upload-time = "2026-06-30T18:59:28.647Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/f9/505a9fcc67d2b5ffb65f2ea0462f17185162c03c010d21343ffe9cb6a904/typos-1.48.0-py3-none-win_amd64.whl", hash = "sha256:323126648140af40eacd90d729c0296aa746a82b765fba19b9aad9778bb39732", size = 3318798, upload-time = "2026-06-30T18:59:29.997Z" },
+]
+
+[[package]]
+name = "tzdata"
+version = "2026.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ba/19/1b9b0e29f30c6d35cb345486df41110984ea67ae69dddbc0e8a100999493/tzdata-2026.2.tar.gz", hash = "sha256:9173fde7d80d9018e02a662e168e5a2d04f87c41ea174b139fbef642eda62d10", size = 198254, upload-time = "2026-04-24T15:22:08.651Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ce/e4/dccd7f47c4b64213ac01ef921a1337ee6e30e8c6466046018326977efd95/tzdata-2026.2-py2.py3-none-any.whl", hash = "sha256:bbe9af844f658da81a5f95019480da3a89415801f6cc966806612cc7169bffe7", size = 349321, upload-time = "2026-04-24T15:22:05.876Z" },
+]
+
+[[package]]
+name = "uc-micro-py"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/78/67/9a363818028526e2d4579334460df777115bdec1bb77c08f9db88f6389f2/uc_micro_py-2.0.0.tar.gz", hash = "sha256:c53691e495c8db60e16ffc4861a35469b0ba0821fe409a8a7a0a71864d33a811", size = 6611, upload-time = "2026-03-01T06:31:27.526Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/61/73/d21edf5b204d1467e06500080a50f79d49ef2b997c79123a536d4a17d97c/uc_micro_py-2.0.0-py3-none-any.whl", hash = "sha256:3603a3859af53e5a39bc7677713c78ea6589ff188d70f4fee165db88e22b242c", size = 6383, upload-time = "2026-03-01T06:31:26.257Z" },
+]
+
+[[package]]
+name = "udapi"
+version = "0.5.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama" },
+ { name = "termcolor" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/52/ce/ad2ddbd1f1adc8818709aa76e0743361fdac7d2de0d72c7724347d206d41/udapi-0.5.2.tar.gz", hash = "sha256:bea62dd380a885a485cb7810ce69d5154ba3f9345140e6198e0b39858ecb9ae8", size = 321364, upload-time = "2026-03-16T17:45:43.751Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c0/bf/7314295832b37c6c02ebafdac3a240cc557674a2a61a27a6291f32abdd50/udapi-0.5.2-py3-none-any.whl", hash = "sha256:d75939dbdb224b973acc88b9c8074fd864d7b98e05e084fef54e7f02297aa8a7", size = 418680, upload-time = "2026-03-16T17:45:42.173Z" },
+]
+
+[[package]]
+name = "udtools"
+version = "0.2.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "regex" },
+ { name = "udapi" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/51/bc/e9402ed13ee1618f87e91e1950279ec9ce15fbceb1392d9b13087f95d2df/udtools-0.2.8.tar.gz", hash = "sha256:6dd0cbbf29200c2aa3a34ff63b0ea2705fb1b268f2f9e0fdb5ae6c492a87fe42", size = 1707883, upload-time = "2026-06-18T18:49:17.279Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/e0/6d021ce8f05b55c9b506662e242af66d594818e7049fd06d3089480eebfb/udtools-0.2.8-py3-none-any.whl", hash = "sha256:2139e402a004c4563c9b9650c0bfdf7bb752ce7b71ea380617796ae31a6f5ff5", size = 794185, upload-time = "2026-06-18T18:49:15.605Z" },
+]
+
+[[package]]
+name = "universal-pathlib"
+version = "0.3.10"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "fsspec" },
+ { name = "pathlib-abc" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3d/6e/d997a70ee8f4c61f9a7e2f4f8af721cf072a3326848fc881b05187e52558/universal_pathlib-0.3.10.tar.gz", hash = "sha256:4487cbc90730a48cfb64f811d99e14b6faed6d738420cd5f93f59f48e6930bfb", size = 261110, upload-time = "2026-02-22T14:40:58.87Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dd/1a/5d9a402b39ec892d856bbdd9db502ff73ce28cdf4aff72eb1ce1d6843506/universal_pathlib-0.3.10-py3-none-any.whl", hash = "sha256:dfaf2fb35683d2eb1287a3ed7b215e4d6016aa6eaf339c607023d22f90821c66", size = 83528, upload-time = "2026-02-22T14:40:57.316Z" },
+]
+
+[[package]]
+name = "uri-template"
+version = "1.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/31/c7/0336f2bd0bcbada6ccef7aaa25e443c118a704f828a0620c6fa0207c1b64/uri-template-1.3.0.tar.gz", hash = "sha256:0e00f8eb65e18c7de20d595a14336e9f337ead580c70934141624b6d1ffdacc7", size = 21678, upload-time = "2023-06-21T01:49:05.374Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/00/3fca040d7cf8a32776d3d81a00c8ee7457e00f80c649f1e4a863c8321ae9/uri_template-1.3.0-py3-none-any.whl", hash = "sha256:a44a133ea12d44a0c0f06d7d42a52d71282e77e2f937d8abd5655b8d56fc1363", size = 11140, upload-time = "2023-06-21T01:49:03.467Z" },
+]
+
+[[package]]
+name = "urllib3"
+version = "2.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" },
+]
+
+[[package]]
+name = "uvicorn"
+version = "0.49.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "h11" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c4/1f/fa18009dea8469069cca78a4e877a008ab78f08b064bfc9ab891579077ff/uvicorn-0.49.0.tar.gz", hash = "sha256:ebf4271aa580d9de97f93192d4595176df6e91f9aae919ca73e4fc07df1e66a3", size = 91284, upload-time = "2026-06-03T22:01:30.448Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/fa/e1388bbcf24ef3274f45c0c1c7b501fd14971037c1b6ee23610553307497/uvicorn-0.49.0-py3-none-any.whl", hash = "sha256:ba3d14c3ee7e41c6c654c46c9eb489d33213cdd30aa1696eab1374337c13f68f", size = 71376, upload-time = "2026-06-03T22:01:29.037Z" },
+]
+
+[package.optional-dependencies]
+standard = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "httptools" },
+ { name = "python-dotenv" },
+ { name = "pyyaml" },
+ { name = "uvloop", marker = "platform_python_implementation != 'PyPy' and sys_platform != 'cygwin' and sys_platform != 'win32'" },
+ { name = "watchfiles" },
+ { name = "websockets" },
+]
+
+[[package]]
+name = "uvloop"
+version = "0.22.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/06/f0/18d39dbd1971d6d62c4629cc7fa67f74821b0dc1f5a77af43719de7936a7/uvloop-0.22.1.tar.gz", hash = "sha256:6c84bae345b9147082b17371e3dd5d42775bddce91f885499017f4607fdaf39f", size = 2443250, upload-time = "2025-10-16T22:17:19.342Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/89/8c/182a2a593195bfd39842ea68ebc084e20c850806117213f5a299dfc513d9/uvloop-0.22.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:561577354eb94200d75aca23fbde86ee11be36b00e52a4eaf8f50fb0c86b7705", size = 1358611, upload-time = "2025-10-16T22:16:36.833Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/14/e301ee96a6dc95224b6f1162cd3312f6d1217be3907b79173b06785f2fe7/uvloop-0.22.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:1cdf5192ab3e674ca26da2eada35b288d2fa49fdd0f357a19f0e7c4e7d5077c8", size = 751811, upload-time = "2025-10-16T22:16:38.275Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/02/654426ce265ac19e2980bfd9ea6590ca96a56f10c76e63801a2df01c0486/uvloop-0.22.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6e2ea3d6190a2968f4a14a23019d3b16870dd2190cd69c8180f7c632d21de68d", size = 4288562, upload-time = "2025-10-16T22:16:39.375Z" },
+ { url = "https://files.pythonhosted.org/packages/15/c0/0be24758891ef825f2065cd5db8741aaddabe3e248ee6acc5e8a80f04005/uvloop-0.22.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0530a5fbad9c9e4ee3f2b33b148c6a64d47bbad8000ea63704fa8260f4cf728e", size = 4366890, upload-time = "2025-10-16T22:16:40.547Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/53/8369e5219a5855869bcee5f4d317f6da0e2c669aecf0ef7d371e3d084449/uvloop-0.22.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:bc5ef13bbc10b5335792360623cc378d52d7e62c2de64660616478c32cd0598e", size = 4119472, upload-time = "2025-10-16T22:16:41.694Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/ba/d69adbe699b768f6b29a5eec7b47dd610bd17a69de51b251126a801369ea/uvloop-0.22.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1f38ec5e3f18c8a10ded09742f7fb8de0108796eb673f30ce7762ce1b8550cad", size = 4239051, upload-time = "2025-10-16T22:16:43.224Z" },
+]
+
+[[package]]
+name = "virtualenv"
+version = "21.5.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "distlib" },
+ { name = "filelock" },
+ { name = "platformdirs" },
+ { name = "python-discovery" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f1/a5/81f987504738e6defeed61ec1c47e2aefab3c35d8eeb87e1b3f38cf28254/virtualenv-21.5.1.tar.gz", hash = "sha256:dca3bf98275a59c652b69d68e73433e597d977c2da9198882479d1a7188009c8", size = 4578798, upload-time = "2026-06-16T16:23:58.603Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2c/02/3623e6169bed617ed1e2d372f7c69f92ec28d54c4dfc997055c8578ec148/virtualenv-21.5.1-py3-none-any.whl", hash = "sha256:55aa670b67bbfb991b03fda39bd3276d92c419d702376e98c5df1c9989a26783", size = 4558820, upload-time = "2026-06-16T16:23:56.963Z" },
+]
+
+[[package]]
+name = "vllm"
+version = "0.28.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "anthropic" },
+ { name = "apache-tvm-ffi" },
+ { name = "blake3" },
+ { name = "cachetools" },
+ { name = "cbor2" },
+ { name = "cloudpickle" },
+ { name = "compressed-tensors" },
+ { name = "depyf" },
+ { name = "einops" },
+ { name = "fastapi", extra = ["standard"] },
+ { name = "fastsafetensors" },
+ { name = "filelock" },
+ { name = "flashinfer-python" },
+ { name = "huggingface-hub" },
+ { name = "humming-kernels", extra = ["cu13"] },
+ { name = "ijson" },
+ { name = "jsonschema" },
+ { name = "lark" },
+ { name = "llguidance", marker = "platform_machine == 'aarch64' or platform_machine == 'arm64' or platform_machine == 'ppc64le' or platform_machine == 's390x' or platform_machine == 'x86_64'" },
+ { name = "lm-format-enforcer" },
+ { name = "mcp" },
+ { name = "mistral-common", extra = ["image"] },
+ { name = "model-hosting-container-standards" },
+ { name = "msgspec" },
+ { name = "ninja" },
+ { name = "numba" },
+ { name = "numpy" },
+ { name = "nvidia-cudnn-frontend" },
+ { name = "nvidia-cutlass-dsl", extra = ["cu13"] },
+ { name = "nvtx" },
+ { name = "openai" },
+ { name = "openai-harmony" },
+ { name = "opencv-python-headless" },
+ { name = "opentelemetry-api" },
+ { name = "opentelemetry-exporter-otlp" },
+ { name = "opentelemetry-sdk" },
+ { name = "opentelemetry-semantic-conventions-ai" },
+ { name = "outlines-core" },
+ { name = "partial-json-parser" },
+ { name = "pillow" },
+ { name = "prometheus-client" },
+ { name = "prometheus-fastapi-instrumentator" },
+ { name = "protobuf" },
+ { name = "psutil" },
+ { name = "py-cpuinfo" },
+ { name = "pybase64" },
+ { name = "pydantic" },
+ { name = "pynvvideocodec" },
+ { name = "python-json-logger" },
+ { name = "pyyaml" },
+ { name = "pyzmq" },
+ { name = "quack-kernels" },
+ { name = "regex" },
+ { name = "requests" },
+ { name = "safetensors" },
+ { name = "sentencepiece" },
+ { name = "setproctitle" },
+ { name = "setuptools" },
+ { name = "six" },
+ { name = "starlette" },
+ { name = "tiktoken" },
+ { name = "tilelang" },
+ { name = "tokenizers" },
+ { name = "tokenspeed-mla", marker = "sys_platform == 'linux'" },
+ { name = "torch" },
+ { name = "torchaudio" },
+ { name = "torchcodec" },
+ { name = "torchvision" },
+ { name = "tqdm" },
+ { name = "transformers" },
+ { name = "typing-extensions" },
+ { name = "watchfiles" },
+ { name = "xgrammar", marker = "platform_machine == 'aarch64' or platform_machine == 'arm64' or platform_machine == 'ppc64le' or platform_machine == 's390x' or platform_machine == 'x86_64'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/22/39/096d88cfe34ee43a8b6e7c957d1ad6338fa5256474d6ff9a32008c6f8cd6/vllm-0.28.0.tar.gz", hash = "sha256:ac96dd0ec5be9c13f2aa4bfb50498c4727110406f6e4980b58a4097e4d18634a", size = 39991441, upload-time = "2026-08-26T10:08:30.56Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b4/31/007ac11dc756f7e69600b90b08e39f573eed9aaee6258198f9d9a2b06355/vllm-0.28.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:817b8181f7f61b4a62dc1d5d9ab39f2bfb60a6cb86c29879a78a147b85756787", size = 308873410, upload-time = "2026-08-26T10:08:13.738Z" },
+ { url = "https://files.pythonhosted.org/packages/87/d7/97f6ecc2ae883e601e08d7cef87cb54ececeefcfe6b5e12d5d92f8d06d6b/vllm-0.28.0-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:addb0ffdaafd8155d75e9b3f5ddb3da28fdee9e8a7097ede91f7db2e9e1a3889", size = 314500103, upload-time = "2026-08-26T10:07:25.091Z" },
+]
+
+[[package]]
+name = "wandb"
+version = "0.28.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "gitpython" },
+ { name = "packaging" },
+ { name = "platformdirs" },
+ { name = "protobuf" },
+ { name = "pydantic" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "sentry-sdk" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5f/a7/683bfbd6cbade3012bc90d3e9c4cfc72dd62566195bf4c30321946d64b77/wandb-0.28.0.tar.gz", hash = "sha256:b20e5af0fe80e2e2a466b0466a1d60cedcc578dce0f036eca04f4a0adcad95b6", size = 40558332, upload-time = "2026-06-23T00:38:50.115Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c0/47/1723605f76c5d6446b6d0db65b83eda1599721bc8c1e65bd76cc1682b1a7/wandb-0.28.0-py3-none-macosx_12_0_arm64.whl", hash = "sha256:c3dab1205a5aca4abbad1eca08902cdba86add0edfa83d8d61b4429d0e79fa87", size = 24335272, upload-time = "2026-06-23T00:38:26.002Z" },
+ { url = "https://files.pythonhosted.org/packages/81/ff/42b539bc75bc48fc86981dccde89327ba9b71504b805b9ba42cba7c26de9/wandb-0.28.0-py3-none-macosx_12_0_x86_64.whl", hash = "sha256:ae255da18726ee8e731ef82cbc85035b901a28ae14cf91604c361b44b8d44ce0", size = 25557959, upload-time = "2026-06-23T00:38:28.993Z" },
+ { url = "https://files.pythonhosted.org/packages/15/55/c3db03d04aeab3726066a418b2ef6a1f8119774ee510f4fbe992f52b7472/wandb-0.28.0-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:6dbcba12ab168aa37561f2f32dcdef8713495fc25fa7d30fdc9bfb37989694dd", size = 24878557, upload-time = "2026-06-23T00:38:31.417Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/5d/1385ce3c219cb5bd30d4027687e3f8d25969c7dfd09adad1cbd5080e1a72/wandb-0.28.0-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:325b2d0bd88be6eda5db10542499bad3710927f2569c81a84dc5eeaffc76825c", size = 26764727, upload-time = "2026-06-23T00:38:33.775Z" },
+ { url = "https://files.pythonhosted.org/packages/00/58/23b6c17a6d3d5422b007707961c4496b2f6f892624d2910c9f7742fcc202/wandb-0.28.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:8954bc1c62ae43914dce2bebfd1d9957f72350f8fbb78e5cdfe2ca9b6be8a7b8", size = 25051656, upload-time = "2026-06-23T00:38:36.281Z" },
+ { url = "https://files.pythonhosted.org/packages/89/67/9be00fb2db2281063af24a148636d2dd363d337317642ab5d8e93572c794/wandb-0.28.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:9fec6c908554c2dad33110c1312bc3028cc2e430f0679f16b84f82c8ea801e3b", size = 27074113, upload-time = "2026-06-23T00:38:38.737Z" },
+ { url = "https://files.pythonhosted.org/packages/59/b1/f7a96c09cab0c5131b1e6466659b093b401e1653cbe6bb77b462fc1c361d/wandb-0.28.0-py3-none-win32.whl", hash = "sha256:8834ef3a7c8c43b701654162783caa7ad37af48a0ff06fc35d0d65a411f76ccd", size = 24525206, upload-time = "2026-06-23T00:38:42.041Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/c4/c7bed5e981679c74e9fbb22c03ff31c42e95f266199d03d8d325f4d0e6df/wandb-0.28.0-py3-none-win_amd64.whl", hash = "sha256:ac1f82292e2da4f98297b78c3a46726b3a6c5734ecb75fc39b8db2c8a4989159", size = 24525214, upload-time = "2026-06-23T00:38:44.549Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/77/b5ce9696c8cb955521a7941fbc443e78b2f504894c6ae1a2d0b1de6e12ae/wandb-0.28.0-py3-none-win_arm64.whl", hash = "sha256:c5b0faf1b84cf79ebabed77538c1940a4c6053e815f767a4004e877a1354bed1", size = 22378208, upload-time = "2026-06-23T00:38:47.148Z" },
+]
+
+[[package]]
+name = "wasabi"
+version = "1.1.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ac/f9/054e6e2f1071e963b5e746b48d1e3727470b2a490834d18ad92364929db3/wasabi-1.1.3.tar.gz", hash = "sha256:4bb3008f003809db0c3e28b4daf20906ea871a2bb43f9914197d540f4f2e0878", size = 30391, upload-time = "2024-05-31T16:56:18.99Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/06/7c/34330a89da55610daa5f245ddce5aab81244321101614751e7537f125133/wasabi-1.1.3-py3-none-any.whl", hash = "sha256:f76e16e8f7e79f8c4c8be49b4024ac725713ab10cd7f19350ad18a8e3f71728c", size = 27880, upload-time = "2024-05-31T16:56:16.699Z" },
+]
+
+[[package]]
+name = "watchfiles"
+version = "1.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/cd/41/5e1a4bb12aac5f1493fa1bdc11154eca3b258ca4eba65d39c473fe19d8e9/watchfiles-1.2.0.tar.gz", hash = "sha256:c995fba777f1ea992f090f9236e9284cf7a5d1a0130dd5a3d82c598cacd76838", size = 108252, upload-time = "2026-05-18T04:32:04.251Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/4d/70a7feced9f87e2ff26dba42667290f41694fc64646c67261fbb8cab5d5c/watchfiles-1.2.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:01ea8d66f0693b9b60a6541c8d10263091ca9a9060d242f3c1f3143f9aad2c98", size = 399730, upload-time = "2026-05-18T04:31:38.162Z" },
+ { url = "https://files.pythonhosted.org/packages/31/3a/0da302f2307aee316922806ebd5726c542cbd787c938271cf14a074c7daf/watchfiles-1.2.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7ba0480b9a74af058f43b337e937a451e109295c420916d68ad24e3dc02f5e44", size = 392842, upload-time = "2026-05-18T04:30:27.051Z" },
+ { url = "https://files.pythonhosted.org/packages/db/ef/d5bdb705c224dbc256aa0c1ec47bf4e61ec52558f2afb44a71a1fe4d7015/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4f34e26a19f91f710c08e0183429f0d1d15df734e6bc78c31e77b9ea9c433658", size = 452989, upload-time = "2026-05-18T04:31:11.945Z" },
+ { url = "https://files.pythonhosted.org/packages/71/29/5495f2c1661949ef7a35e4d71111d129cfe7606414a26887a919d0a55406/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b4e77f6a55f858504069abd35d336a637555c09bca453dde1ee1e5ada8a6a1fb", size = 458978, upload-time = "2026-05-18T04:30:52.606Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/8c/7f9c07c433811c2fffd93e13fdfb7135de9aab5f2ae41be08960fa0047dc/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0cb4d80e212f116474a545c21c912b445f16bb0cef9e6a73a498164223e14e2f", size = 490248, upload-time = "2026-05-18T04:31:36.003Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/11/d93632febc52fbc21be90231bb7c17fd5387f46c9076fd40a5f9c2ae6910/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b974946a10af379d425e2eef5b62f5c6ebeaccf91d45eaad6f5b27ecd4f91aa0", size = 571847, upload-time = "2026-05-18T04:31:10.862Z" },
+ { url = "https://files.pythonhosted.org/packages/55/b4/383173e73aabb07ad1d9c7aa859d95437ac46a6d6a1e11005facda0c9d19/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:86bc13c25a8d1fcd70b51d0ce7c9b65e90de5666fcbfd3e34957cc73ee19aeb5", size = 465974, upload-time = "2026-05-18T04:30:17.006Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/6c/89b1a230a78f57c52dd8893adb1f92f94411721b6ec12596c56d98c74356/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca148d73dea36c9763aaa351e4d7a51780ec1584217c45276f4fe8239c768b71", size = 454782, upload-time = "2026-05-18T04:30:35.656Z" },
+ { url = "https://files.pythonhosted.org/packages/24/62/1732118367cfff0a9fce3bf62ff4bfded09ef5df21d9d446b858b3f70a96/watchfiles-1.2.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:c525543d91961c6955b2636b308569e84a1d1c5f5f2932041ab9ef46422f43e3", size = 465182, upload-time = "2026-05-18T04:30:20.846Z" },
+ { url = "https://files.pythonhosted.org/packages/28/96/716f7e5f51339bf22963f3345f9f27d7f3b30e2eadc597e257c881dd3c53/watchfiles-1.2.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:a204794696ffb8f9b10fba6f7cb5216d42f3b2b71860ccac6b6e42f5f10973b0", size = 629841, upload-time = "2026-05-18T04:31:05.397Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/fe/c40783950fd771ccf66ab3ec2722d188a9af1c7f96c6e811f36e40c6e03f/watchfiles-1.2.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:10d86db20695afe7997ac9e1717637d6714a8d0220458c33f3d2061f54cec427", size = 658028, upload-time = "2026-05-18T04:31:48.22Z" },
+ { url = "https://files.pythonhosted.org/packages/71/72/4508db1856d1d87fcbb3b63f4839bab1b5682cb0e8d224d122263c09654a/watchfiles-1.2.0-cp313-cp313-win32.whl", hash = "sha256:eb283ee99e21ad6443c8cdb06ac5b34b1308c329cbdf03fa02b445363714c799", size = 275183, upload-time = "2026-05-18T04:30:59.57Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/36/14b76ca57652e5cc5fd1c11f32a261292c08a0d19a00351013c2549cbfb2/watchfiles-1.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:a0f27f01bee51861392bb6b7c4fdb290b27d1eb194e9e28788d68102a0e898d9", size = 288059, upload-time = "2026-05-18T04:32:07.937Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/8d/0a85e395398d8d20fadfe5c5d32c726eee17a519e78fb356f2cf7531bffe/watchfiles-1.2.0-cp313-cp313-win_arm64.whl", hash = "sha256:3651aa7058595e9cfb75d35dd5ada2bf9f48a5b8a0f3562821d3e210c507e077", size = 280186, upload-time = "2026-05-18T04:31:54.484Z" },
+ { url = "https://files.pythonhosted.org/packages/37/68/36db056f1fdcc5f07302f56e631774d6835bcd6fa3ace402304621d5f9e5/watchfiles-1.2.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:faea288b6f0ab1902ef08f4ca6de005dccf856c4e0c4f21b8c5fce02d90a1b08", size = 399031, upload-time = "2026-05-18T04:30:44.576Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/64/01a9d6f66a82a5c101ce939274106cc72759d62427e153f01edd2b9f87c2/watchfiles-1.2.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:01859b11fd9fbca670f4d5da00fbac282cfea9bd67a2125d8b2833a3b5617ea9", size = 391205, upload-time = "2026-05-18T04:30:25.413Z" },
+ { url = "https://files.pythonhosted.org/packages/84/2c/0a44fe058cb4bb7b8ede6b6670698bbb7c0400740e378d00022189b7b31d/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fff610d7bb2256a317bb1e96f0d7862c7aa8076733ee5df0fd41bbe76a24a4f4", size = 451892, upload-time = "2026-05-18T04:32:14.005Z" },
+ { url = "https://files.pythonhosted.org/packages/67/a1/351e0d56cd35e6488b5c8b4fb11a809a5bc923e8fe8fed9faf8920be0c89/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b141a4891c995a039cd89e9a49e62df1dc8a559a5d1a6e4c7106d16c12777a55", size = 458867, upload-time = "2026-05-18T04:31:22.279Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/7d/9d09605187f1b838998624049fcf8bf47b73c1a3b76901fcac1782f62277/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f22943b7770483f6ea0721c6b11d022947a98eb0acae14694de034f4d0d38925", size = 490217, upload-time = "2026-05-18T04:31:43.657Z" },
+ { url = "https://files.pythonhosted.org/packages/60/5d/a17a16eccb182f04188cd308ec24b1a71a9b5c4e7098269cf35d9fa56d02/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1bc6195825b7dcd217968bb1f801a60fd4c16e8eeab5bedc7fe917d7d5995ab4", size = 571458, upload-time = "2026-05-18T04:32:11.875Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/3d/4dd457062083ab1938e5dfd45032eb425cee2ac817287ca8ff4356183e5d/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d4a4b147f5dca2a5d325a06a832fb43f345751adfbc63204aec30e0d9ca965a2", size = 464707, upload-time = "2026-05-18T04:30:43.492Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/71/ea8c57b128f5383de74d0c7d2d9c57ad7c9a65a930c451bd25d524b295b7/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4543579a9bdb0c9560039b4ffddbdb39545707659fbc430ce4c10f3f68d557f9", size = 454663, upload-time = "2026-05-18T04:30:16.061Z" },
+ { url = "https://files.pythonhosted.org/packages/53/fd/2e812bf938406d7db351f0703ddd3fc6c061cf30d96153a77bc79a943a44/watchfiles-1.2.0-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:20aa0e708b920bde876a4aa82dc7dd6ebea228a63a67cda6632c2fc87b787efa", size = 463537, upload-time = "2026-05-18T04:31:44.9Z" },
+ { url = "https://files.pythonhosted.org/packages/86/56/d17a7f1dd1bc3035f1072694a551301272f1739c2d8e319c927cb9e29b38/watchfiles-1.2.0-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:d413349d565dab74297f2a63e84a097936be69bf8f3b3801f27f380e32040f44", size = 629194, upload-time = "2026-05-18T04:31:14.141Z" },
+ { url = "https://files.pythonhosted.org/packages/be/06/f1ff66bf5cae50aa4062779a0ecd0bbaf15e466195719074078947d9a17d/watchfiles-1.2.0-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:f28b2725eb8cce327b9b3ab02415c853011dc55c95832fe90de6bc56f5315f72", size = 656194, upload-time = "2026-05-18T04:31:47.14Z" },
+]
+
+[[package]]
+name = "wcwidth"
+version = "0.8.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/34/74/c6428f875774288bec1396f5bfcbc2d925700a4dad61727fd5f2b12f249d/wcwidth-0.8.2.tar.gz", hash = "sha256:91fbef97204b96a3d4d421609b80340b760cf33e26da123ff243d76b1fda8dda", size = 1466253, upload-time = "2026-06-29T18:11:11.601Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/96/42/3e5985a0a7e57de470b320c6d6a1a67c844f6737a587f3d44dd13d1819e7/wcwidth-0.8.2-py3-none-any.whl", hash = "sha256:d63947694a0539a1d51e01eda7caf800c291020e6cdd7e28ad7b14dd33ad4f85", size = 323166, upload-time = "2026-06-29T18:11:09.888Z" },
+]
+
+[[package]]
+name = "weasel"
+version = "1.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cloudpathlib" },
+ { name = "confection" },
+ { name = "httpx" },
+ { name = "packaging" },
+ { name = "pydantic" },
+ { name = "smart-open" },
+ { name = "srsly" },
+ { name = "typer" },
+ { name = "wasabi" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ce/e5/e272bb9a045105a1fdf4b798d8086f5932a178f4d738f17a74f5c9e0ae9a/weasel-1.0.0.tar.gz", hash = "sha256:7b129b44c90cc543b760532974ca1e4eb30dad2aa2026f57bdce66354ae610fc", size = 38682, upload-time = "2026-03-20T08:10:25.266Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0a/07/57ebf7a6798b016c064bd0ca81b4c6a99daa4dc377b898bc7b41eb6b5af0/weasel-1.0.0-py3-none-any.whl", hash = "sha256:89518acee027f49d743126c3502d35e6dd14f5768be5c37c9af47c171b6005cc", size = 50713, upload-time = "2026-03-20T08:10:23.637Z" },
+]
+
+[[package]]
+name = "webcolors"
+version = "25.10.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1d/7a/eb316761ec35664ea5174709a68bbd3389de60d4a1ebab8808bfc264ed67/webcolors-25.10.0.tar.gz", hash = "sha256:62abae86504f66d0f6364c2a8520de4a0c47b80c03fc3a5f1815fedbef7c19bf", size = 53491, upload-time = "2025-10-31T07:51:03.977Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e2/cc/e097523dd85c9cf5d354f78310927f1656c422bd7b2613b2db3e3f9a0f2c/webcolors-25.10.0-py3-none-any.whl", hash = "sha256:032c727334856fc0b968f63daa252a1ac93d33db2f5267756623c210e57a4f1d", size = 14905, upload-time = "2025-10-31T07:51:01.778Z" },
+]
+
+[[package]]
+name = "webencodings"
+version = "0.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0b/02/ae6ceac1baeda530866a85075641cec12989bd8d31af6d5ab4a3e8c92f47/webencodings-0.5.1.tar.gz", hash = "sha256:b36a1c245f2d304965eb4e0a82848379241dc04b865afcc4aab16748587e1923", size = 9721, upload-time = "2017-04-05T20:21:34.189Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl", hash = "sha256:a0af1213f3c2226497a97e2b3aa01a7e4bee4f403f95be16fc9acd2947514a78", size = 11774, upload-time = "2017-04-05T20:21:32.581Z" },
+]
+
+[[package]]
+name = "websocket-client"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2c/41/aa4bf9664e4cda14c3b39865b12251e8e7d239f4cd0e3cc1b6c2ccde25c1/websocket_client-1.9.0.tar.gz", hash = "sha256:9e813624b6eb619999a97dc7958469217c3176312b3a16a4bd1bc7e08a46ec98", size = 70576, upload-time = "2025-10-07T21:16:36.495Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/34/db/b10e48aa8fff7407e67470363eac595018441cf32d5e1001567a7aeba5d2/websocket_client-1.9.0-py3-none-any.whl", hash = "sha256:af248a825037ef591efbf6ed20cc5faa03d3b47b9e5a2230a529eeee1c1fc3ef", size = 82616, upload-time = "2025-10-07T21:16:34.951Z" },
+]
+
+[[package]]
+name = "websockets"
+version = "16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/04/24/4b2031d72e840ce4c1ccb255f693b15c334757fc50023e4db9537080b8c4/websockets-16.0.tar.gz", hash = "sha256:5f6261a5e56e8d5c42a4497b364ea24d94d9563e8fbd44e78ac40879c60179b5", size = 179346, upload-time = "2026-01-10T09:23:47.181Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cc/9c/baa8456050d1c1b08dd0ec7346026668cbc6f145ab4e314d707bb845bf0d/websockets-16.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:878b336ac47938b474c8f982ac2f7266a540adc3fa4ad74ae96fea9823a02cc9", size = 177364, upload-time = "2026-01-10T09:22:59.333Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/0c/8811fc53e9bcff68fe7de2bcbe75116a8d959ac699a3200f4847a8925210/websockets-16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:52a0fec0e6c8d9a784c2c78276a48a2bdf099e4ccc2a4cad53b27718dbfd0230", size = 175039, upload-time = "2026-01-10T09:23:01.171Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/82/39a5f910cb99ec0b59e482971238c845af9220d3ab9fa76dd9162cda9d62/websockets-16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e6578ed5b6981005df1860a56e3617f14a6c307e6a71b4fff8c48fdc50f3ed2c", size = 175323, upload-time = "2026-01-10T09:23:02.341Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/28/0a25ee5342eb5d5f297d992a77e56892ecb65e7854c7898fb7d35e9b33bd/websockets-16.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95724e638f0f9c350bb1c2b0a7ad0e83d9cc0c9259f3ea94e40d7b02a2179ae5", size = 184975, upload-time = "2026-01-10T09:23:03.756Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/66/27ea52741752f5107c2e41fda05e8395a682a1e11c4e592a809a90c6a506/websockets-16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0204dc62a89dc9d50d682412c10b3542d748260d743500a85c13cd1ee4bde82", size = 186203, upload-time = "2026-01-10T09:23:05.01Z" },
+ { url = "https://files.pythonhosted.org/packages/37/e5/8e32857371406a757816a2b471939d51c463509be73fa538216ea52b792a/websockets-16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:52ac480f44d32970d66763115edea932f1c5b1312de36df06d6b219f6741eed8", size = 185653, upload-time = "2026-01-10T09:23:06.301Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/67/f926bac29882894669368dc73f4da900fcdf47955d0a0185d60103df5737/websockets-16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6e5a82b677f8f6f59e8dfc34ec06ca6b5b48bc4fcda346acd093694cc2c24d8f", size = 184920, upload-time = "2026-01-10T09:23:07.492Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/a1/3d6ccdcd125b0a42a311bcd15a7f705d688f73b2a22d8cf1c0875d35d34a/websockets-16.0-cp313-cp313-win32.whl", hash = "sha256:abf050a199613f64c886ea10f38b47770a65154dc37181bfaff70c160f45315a", size = 178255, upload-time = "2026-01-10T09:23:09.245Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/ae/90366304d7c2ce80f9b826096a9e9048b4bb760e44d3b873bb272cba696b/websockets-16.0-cp313-cp313-win_amd64.whl", hash = "sha256:3425ac5cf448801335d6fdc7ae1eb22072055417a96cc6b31b3861f455fbc156", size = 178689, upload-time = "2026-01-10T09:23:10.483Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/28/258ebab549c2bf3e64d2b0217b973467394a9cea8c42f70418ca2c5d0d2e/websockets-16.0-py3-none-any.whl", hash = "sha256:1637db62fad1dc833276dded54215f2c7fa46912301a24bd94d45d46a011ceec", size = 171598, upload-time = "2026-01-10T09:23:45.395Z" },
+]
+
+[[package]]
+name = "widgetsnbextension"
+version = "4.0.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/bd/f4/c67440c7fb409a71b7404b7aefcd7569a9c0d6bd071299bf4198ae7a5d95/widgetsnbextension-4.0.15.tar.gz", hash = "sha256:de8610639996f1567952d763a5a41af8af37f2575a41f9852a38f947eb82a3b9", size = 1097402, upload-time = "2025-11-01T21:15:55.178Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/0e/fa3b193432cfc60c93b42f3be03365f5f909d2b3ea410295cf36df739e31/widgetsnbextension-4.0.15-py3-none-any.whl", hash = "sha256:8156704e4346a571d9ce73b84bee86a29906c9abfd7223b7228a28899ccf3366", size = 2196503, upload-time = "2025-11-01T21:15:53.565Z" },
+]
+
+[[package]]
+name = "win32-setctime"
+version = "1.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b3/8f/705086c9d734d3b663af0e9bb3d4de6578d08f46b1b101c2442fd9aecaa2/win32_setctime-1.2.0.tar.gz", hash = "sha256:ae1fdf948f5640aae05c511ade119313fb6a30d7eabe25fef9764dca5873c4c0", size = 4867, upload-time = "2024-12-07T15:28:28.314Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e1/07/c6fe3ad3e685340704d314d765b7912993bcb8dc198f0e7a89382d37974b/win32_setctime-1.2.0-py3-none-any.whl", hash = "sha256:95d644c4e708aba81dc3704a116d8cbc974d70b3bdb8be1d150e36be6e9d1390", size = 4083, upload-time = "2024-12-07T15:28:26.465Z" },
+]
+
+[[package]]
+name = "wrapt"
+version = "1.17.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/95/8f/aeb76c5b46e273670962298c23e7ddde79916cb74db802131d49a85e4b7d/wrapt-1.17.3.tar.gz", hash = "sha256:f66eb08feaa410fe4eebd17f2a2c8e2e46d3476e9f8c783daa8e09e0faa666d0", size = 55547, upload-time = "2025-08-12T05:53:21.714Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fc/f6/759ece88472157acb55fc195e5b116e06730f1b651b5b314c66291729193/wrapt-1.17.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:a47681378a0439215912ef542c45a783484d4dd82bac412b71e59cf9c0e1cea0", size = 54003, upload-time = "2025-08-12T05:51:48.627Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/a9/49940b9dc6d47027dc850c116d79b4155f15c08547d04db0f07121499347/wrapt-1.17.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:54a30837587c6ee3cd1a4d1c2ec5d24e77984d44e2f34547e2323ddb4e22eb77", size = 39025, upload-time = "2025-08-12T05:51:37.156Z" },
+ { url = "https://files.pythonhosted.org/packages/45/35/6a08de0f2c96dcdd7fe464d7420ddb9a7655a6561150e5fc4da9356aeaab/wrapt-1.17.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:16ecf15d6af39246fe33e507105d67e4b81d8f8d2c6598ff7e3ca1b8a37213f7", size = 39108, upload-time = "2025-08-12T05:51:58.425Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/37/6faf15cfa41bf1f3dba80cd3f5ccc6622dfccb660ab26ed79f0178c7497f/wrapt-1.17.3-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6fd1ad24dc235e4ab88cda009e19bf347aabb975e44fd5c2fb22a3f6e4141277", size = 88072, upload-time = "2025-08-12T05:52:37.53Z" },
+ { url = "https://files.pythonhosted.org/packages/78/f2/efe19ada4a38e4e15b6dff39c3e3f3f73f5decf901f66e6f72fe79623a06/wrapt-1.17.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ed61b7c2d49cee3c027372df5809a59d60cf1b6c2f81ee980a091f3afed6a2d", size = 88214, upload-time = "2025-08-12T05:52:15.886Z" },
+ { url = "https://files.pythonhosted.org/packages/40/90/ca86701e9de1622b16e09689fc24b76f69b06bb0150990f6f4e8b0eeb576/wrapt-1.17.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:423ed5420ad5f5529db9ce89eac09c8a2f97da18eb1c870237e84c5a5c2d60aa", size = 87105, upload-time = "2025-08-12T05:52:17.914Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/e0/d10bd257c9a3e15cbf5523025252cc14d77468e8ed644aafb2d6f54cb95d/wrapt-1.17.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e01375f275f010fcbf7f643b4279896d04e571889b8a5b3f848423d91bf07050", size = 87766, upload-time = "2025-08-12T05:52:39.243Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/cf/7d848740203c7b4b27eb55dbfede11aca974a51c3d894f6cc4b865f42f58/wrapt-1.17.3-cp313-cp313-win32.whl", hash = "sha256:53e5e39ff71b3fc484df8a522c933ea2b7cdd0d5d15ae82e5b23fde87d44cbd8", size = 36711, upload-time = "2025-08-12T05:53:10.074Z" },
+ { url = "https://files.pythonhosted.org/packages/57/54/35a84d0a4d23ea675994104e667ceff49227ce473ba6a59ba2c84f250b74/wrapt-1.17.3-cp313-cp313-win_amd64.whl", hash = "sha256:1f0b2f40cf341ee8cc1a97d51ff50dddb9fcc73241b9143ec74b30fc4f44f6cb", size = 38885, upload-time = "2025-08-12T05:53:08.695Z" },
+ { url = "https://files.pythonhosted.org/packages/01/77/66e54407c59d7b02a3c4e0af3783168fff8e5d61def52cda8728439d86bc/wrapt-1.17.3-cp313-cp313-win_arm64.whl", hash = "sha256:7425ac3c54430f5fc5e7b6f41d41e704db073309acfc09305816bc6a0b26bb16", size = 36896, upload-time = "2025-08-12T05:52:55.34Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/f6/a933bd70f98e9cf3e08167fc5cd7aaaca49147e48411c0bd5ae701bb2194/wrapt-1.17.3-py3-none-any.whl", hash = "sha256:7171ae35d2c33d326ac19dd8facb1e82e5fd04ef8c6c0e394d7af55a55051c22", size = 23591, upload-time = "2025-08-12T05:53:20.674Z" },
+]
+
+[[package]]
+name = "xgrammar"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "apache-tvm-ffi" },
+ { name = "numpy" },
+ { name = "pydantic" },
+ { name = "torch" },
+ { name = "transformers" },
+ { name = "triton", version = "3.7.1", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/81/f4/e71693d8cec60b7e36dab660784ecc5a6aa51e478a83b556011645c58c87/xgrammar-0.2.3.tar.gz", hash = "sha256:f76423630ae3ac4e090cb38ce1e30e7bcc69b3dee4d22d94353944386a4c6f18", size = 2447704, upload-time = "2026-06-27T04:45:24.759Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1b/06/a1a3c3a53cebb5f1bdaa1d1452f3327b6ca7413bdeff94ff639c3b9b8378/xgrammar-0.2.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:11108010c54c8f12f0b14c239ef5bca217cf107fa3d0bf0db008bf594ab0a1ff", size = 23240039, upload-time = "2026-06-27T04:44:36.173Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/aa/1c74ff8bad6624c08a33924f7d051a9278a622b48d843e939d574a6b5bf8/xgrammar-0.2.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eeb5e46bd7d3230e5d8e6385793c48ec872a4fb377c19d341dbcaedc41f495e9", size = 44314496, upload-time = "2026-06-27T04:44:38.552Z" },
+ { url = "https://files.pythonhosted.org/packages/28/f8/2407b44049416650c257e22399c32788d3497524a9a899addc74b5d27d1d/xgrammar-0.2.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:29d0fbd4709733b224e6e115d1001fb07124c71fe114a9087a2e3e53ce871517", size = 44855037, upload-time = "2026-06-27T04:44:41.122Z" },
+ { url = "https://files.pythonhosted.org/packages/92/67/271dd31308c5a1d32d9c696044f9d44589f8ac4406faaddc3047765e0c71/xgrammar-0.2.3-cp313-cp313-win_amd64.whl", hash = "sha256:3a718fa2c1bfc06951e1c480a656c28bde6d5f828199308fd577bd264e05b923", size = 15780261, upload-time = "2026-06-27T04:44:43.573Z" },
+]
+
+[[package]]
+name = "xxhash"
+version = "3.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/78/ed/07e560876a4458987511461187b285071f53cde49dd5b25cd8c51091522b/xxhash-3.8.0.tar.gz", hash = "sha256:d72b2204f37840b0f16f34192c09b994b97bd25823d723d47a1eddfacf06eb43", size = 86107, upload-time = "2026-06-27T08:17:28.798Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/1f/96f43c5c7c7c4d44721f8d2e5d74698c667a30283c4b10a7e50a56804ee3/xxhash-3.8.0-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:36434c1d1b0a4729df1fa26ab11bffed1ba52666c0beb605c98a995b470cd143", size = 38508, upload-time = "2026-06-27T08:13:46.152Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/d9/7d5d6af4876c6481f2e0acb2dda64dd5209574bf7ba1ad4f6af7a1f8d473/xxhash-3.8.0-cp313-cp313-android_21_x86_64.whl", hash = "sha256:a5e6497cefcb2d67f1745c66df9718a99112583af6cc2b70da0312a2eb939f1e", size = 36542, upload-time = "2026-06-27T08:13:47.497Z" },
+ { url = "https://files.pythonhosted.org/packages/32/ff/66fed439d78c5a09a1491a85af29bf8923b516530116731a9ac6b14dee2b/xxhash-3.8.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:5b00b82f1be708da9404fefd658cf5cf3be5ee3be2aae4bfe3b874255badd342", size = 31102, upload-time = "2026-06-27T08:13:48.721Z" },
+ { url = "https://files.pythonhosted.org/packages/56/b8/9fae0399281095f8aca1f32b21947b3c3c75ad6021b255c5c6e4b11d3866/xxhash-3.8.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:38b0cb0ab7f283413b7cace2bf710d7cf8f702ea82cbc683908691d52028a89b", size = 32096, upload-time = "2026-06-27T08:13:50.138Z" },
+ { url = "https://files.pythonhosted.org/packages/61/a4/e53d162c74a8a2950dc063969914387b0680da4c7c20ad17744ec03a3b0a/xxhash-3.8.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:084312171a9798dea85e924b2674f5e1a44933050a1ea1cb1c6b1364e004c66c", size = 34585, upload-time = "2026-06-27T08:13:51.572Z" },
+ { url = "https://files.pythonhosted.org/packages/69/f5/e12397e3f2c4917b6572e103a3277cd27cc56330e304bba61d195d7e5224/xxhash-3.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6a1a9e845bd3bbc57d9356819e0d198fe23282e0576b398a6282a0f8fdc75aef", size = 34622, upload-time = "2026-06-27T08:13:52.818Z" },
+ { url = "https://files.pythonhosted.org/packages/70/80/c053dc51af5c942229689a0e9cb66fdc999bbd840f645e761f5ab73cbb17/xxhash-3.8.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9ffbde09743ebaf8957b8426948fbe85eab5e5de0d29eec407fcff5a2812a3cc", size = 32320, upload-time = "2026-06-27T08:13:54.04Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/a3/294171b67dfe770e1293edcf2a3f7e41302cdb8aefb258585312191b3ffe/xxhash-3.8.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:a6dee3952c2b6e82e7f1dbc5dbc6167f9c84126851def7926e32827c2816169c", size = 220532, upload-time = "2026-06-27T08:13:55.448Z" },
+ { url = "https://files.pythonhosted.org/packages/80/c3/d141bfdeca785c8c680abf867d4b52a5e64a55d90df242c3141a3e58c4b2/xxhash-3.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bf8ff8e12416c9fa05b43c7509b9332d6ffc4090413c4e7a1dee8599763b6d59", size = 241215, upload-time = "2026-06-27T08:13:57.047Z" },
+ { url = "https://files.pythonhosted.org/packages/09/5a/aeaf35143a6f3d44db73298e861405bdd9c9dacaedfc369cb43d9fd65282/xxhash-3.8.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:cebbb322df4d97d8ef2704f49ed2f6f21f6702fafa0dc0c2a6ae70e904205689", size = 264615, upload-time = "2026-06-27T08:13:58.912Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/3e/f8ca782bb34f99693faab70a7989bcc84f62ffe93c9a4cca464a33507a4b/xxhash-3.8.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9a8d08707b4100ebce598fc59fadf04b42d79b855818d6994f8f0fffd1df8edb", size = 242682, upload-time = "2026-06-27T08:14:00.483Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/fa/ddbee4ff1542c2e88e72269a5a6bd18c3f26a80c2514e0918f5d1f3e9ec5/xxhash-3.8.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cf5427602dda15d8ce3c6d870d29bf07d43975f59c9d6d3f7f6f93a901b28b12", size = 473551, upload-time = "2026-06-27T08:14:02.17Z" },
+ { url = "https://files.pythonhosted.org/packages/25/f5/a680d48dddab37ab2fd9189ca03f775e29e3627122e30790816d7eb365af/xxhash-3.8.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:97d7bd715ea5050b6c9638b52c62adf3055b648ef6eee6892a4cd9697b530191", size = 220485, upload-time = "2026-06-27T08:14:03.765Z" },
+ { url = "https://files.pythonhosted.org/packages/22/b1/7ac129b74981c07f1ff9c649f204465e86f83f9f29b2ebdc70d91514c365/xxhash-3.8.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0cd25bbbab37d898f6e5a90905ce6ae2c1f8bd6668c07cef406fb3e8c8c570dd", size = 310307, upload-time = "2026-06-27T08:14:05.366Z" },
+ { url = "https://files.pythonhosted.org/packages/67/e6/43e673411249dd63f6cd974523a1b32fad75cf5453e363bc8f44af215fb9/xxhash-3.8.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3e30e5c057f483c3c53a11b53eba091a737cb19dfead36c8b23bf5beb4a169cd", size = 238164, upload-time = "2026-06-27T08:14:07.149Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/95/87f8baf41f63130f3637104b7a610f82b20106332fc6e289c8dbf7955d0e/xxhash-3.8.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:07dd44d992ebd456752bc25b1c42cd172d94bd8cb24049300449ad0716081c3a", size = 269062, upload-time = "2026-06-27T08:14:08.834Z" },
+ { url = "https://files.pythonhosted.org/packages/38/c9/3369b497cd1f926b930c52fd2400606f177790d887b49f9e86bddcc24562/xxhash-3.8.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:3118600a3102d4707dc1c485dbc3acbbbf37819069ad3e7854e77b923745d76b", size = 225007, upload-time = "2026-06-27T08:14:10.689Z" },
+ { url = "https://files.pythonhosted.org/packages/34/c8/03dceb86a8128858ac105bd6e282d62b3db6fd421a79bd8a9f6b8cdc47a7/xxhash-3.8.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:7ed37b0c95d8fb3fbaad5e13cc0a9727eb8739d1d54b2adef28108c250cada3a", size = 240815, upload-time = "2026-06-27T08:14:12.195Z" },
+ { url = "https://files.pythonhosted.org/packages/47/a5/ebd43eeb1af1dd8f0201943688b20958e99d3f6eb36481fb8c37b55ef139/xxhash-3.8.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:bb043da412e478e7b1db3407051124b85b133803794d3809ad6d92870b304fc7", size = 300632, upload-time = "2026-06-27T08:14:13.916Z" },
+ { url = "https://files.pythonhosted.org/packages/df/24/c873e41a3c00dacc385c8ff08c007723f6a528922c1cea7fd9684e86dae7/xxhash-3.8.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:196fc132683d9311a0bdce8388ee52bfa07fdc1987cc428a27956e47ccd7b50d", size = 443293, upload-time = "2026-06-27T08:14:15.446Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/1b/c671272fe28f70574e3c574d58465f26460154bcc68876121872afa1c14d/xxhash-3.8.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dfb5411af3b77c75e99db100aa15c5ba623c85d72c565e4d7a0ed1a986ff766e", size = 217327, upload-time = "2026-06-27T08:14:17.28Z" },
+ { url = "https://files.pythonhosted.org/packages/57/43/b45a52f795812cb769b6ac159e69b605d18b1c067749e63dcac159e90064/xxhash-3.8.0-cp313-cp313-win32.whl", hash = "sha256:6d1d6179e26830c6690fac63f76d372f69714b977e12ca9c42188a60f51c59f5", size = 31898, upload-time = "2026-06-27T08:14:18.952Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/42/2bd70e4eec25dc5990652979d708d4d7c999793d7d5af5d0e48ab4374dc1/xxhash-3.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:7c92427a56a12f4d5c7bb26dbb9e9a4658c313ecb6c2f1dca349902e3822df07", size = 32680, upload-time = "2026-06-27T08:14:20.277Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/c8/2fe61edb6144183cf094035a8c5354c65a073127acf6379655ed1e705b70/xxhash-3.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:9fc8453642c1c6d38b4fbac8901c2452ce1fa88b27f003bfee6703cbfae9bd63", size = 29157, upload-time = "2026-06-27T08:14:21.674Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/b8/81d17a993b9a4750ba426ce966421681bb4b8e82a460cd346756491b8cc2/xxhash-3.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:efcacb644a915f010dc477447b045e5dcde1afaa40d16b2f0f8e7cd99c9e1635", size = 34897, upload-time = "2026-06-27T08:14:23.044Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/3b/f5a368e3273440b3ea58fbd3f0b08c19f552b25ca59f43f5732ca96d2126/xxhash-3.8.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d1e0dbc510cff94c5efbcc2b82c28b41519fad09b5b1f9f3d99c63e3940e49a0", size = 32630, upload-time = "2026-06-27T08:14:24.603Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/ab/f424359c91c55f564fbbe4e454a126eb522471109f67376f20ad19c5e663/xxhash-3.8.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:ff19d016a41c90d1f519005887191896b6da1274e1d5d48b347e17eb798ffc5a", size = 225874, upload-time = "2026-06-27T08:14:25.992Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/c2/434579ef9235123b6c9bfa89c5614e0001e988613b91557b24aa326d9faa/xxhash-3.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aafc3eab99c50508852e34307e9565933bf128cad084cac7d2471b7ab1743de0", size = 249705, upload-time = "2026-06-27T08:14:27.607Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/6c/3c0c917331ca3c71f826cedce2127f230624e2b49b992472dd5e9e72101c/xxhash-3.8.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:5e521368ed79ae6c4d31e1e417726643c49d7d6e286f4fdabf9a8330ed8a8ff7", size = 274716, upload-time = "2026-06-27T08:14:29.495Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/f3/a8bb98d3307c67e88be9642dff52854c3de3f488f95989b60ff69c8dcc42/xxhash-3.8.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6a0127688d116ec0c225e7e1f744e3f206de2b8822ffeb31a9ab5cc6384f92c5", size = 252019, upload-time = "2026-06-27T08:14:31.247Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/73/fab69a2e5b6353dde643209fe9b6adf4fbd64c888e531deffc476bfb2635/xxhash-3.8.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:22c0b17da2f9fea0f8836538512249871b359141616bad44c58d238b5f011f40", size = 482024, upload-time = "2026-06-27T08:14:32.973Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/5b/ba34099b5278097ec9c68c0b740719813553bfd11ca17e7353de6d2a41e3/xxhash-3.8.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3d49465646b1a5e3b1729c5f636e05676a2fb52e203e3b22a5411c416c4c5302", size = 226655, upload-time = "2026-06-27T08:14:34.608Z" },
+ { url = "https://files.pythonhosted.org/packages/76/0c/90aba4708a37fe752b324a7cbf10058eaa33e892cdd62751ff17a5137b93/xxhash-3.8.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c2853dea1e30ed00ca87dd87d76da5da063d302b823b3fb80ccd18421de0f251", size = 319583, upload-time = "2026-06-27T08:14:36.419Z" },
+ { url = "https://files.pythonhosted.org/packages/38/46/42e349e2d3017b2688f4cb301742c37c438e77963e3fef711edce2fc5c65/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:82f0102a2a3760287b7cd7f9e0a30edd4c3b18762ed1a242208d43c8e2bcf30b", size = 246000, upload-time = "2026-06-27T08:14:38.104Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/15/741b947ae3c768e82018c46846f8616f6aa9b5042649f318a1a6897defe3/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:b8414a66a7524596d841cad5dc1adab6ce76848db5ab2b83db911fbdab1417af", size = 275455, upload-time = "2026-06-27T08:14:39.841Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/b4/a9db84c9458fc8f53eaf0051377d1e9eecd9f330fb1225640027417a309d/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:0dbaa73df10414ea1e41b98691a9d8241d4c47ad8d02c726587a3cda05278e53", size = 231209, upload-time = "2026-06-27T08:14:41.543Z" },
+ { url = "https://files.pythonhosted.org/packages/20/92/60a868cd34851746d0b0d95dced0f42867c7c00606f6e5dba85b70b232ce/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:43fc9aaba10ab4267c90793601f60d35c3c9caa1544eceb483618a71ad9ce7da", size = 250416, upload-time = "2026-06-27T08:14:43.193Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/6a/168ca46a4679c32aae9246caa1fddf35981d6304487e45e992b3d4530324/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ec5eb3d28fbb9802c6d2526f772133a06c91d6f03756fcc67c834b642ffdd51d", size = 309764, upload-time = "2026-06-27T08:14:44.79Z" },
+ { url = "https://files.pythonhosted.org/packages/18/0b/13646b348c07679c818791ab2d35415db5cb20f3bc77daaa255909a401b4/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:2b77c301b644cd9b4d0749a3291081ec2048a6bef7fe0487c993bbba3efb9ce0", size = 448650, upload-time = "2026-06-27T08:14:46.562Z" },
+ { url = "https://files.pythonhosted.org/packages/59/9a/3d244b2acf6bbd86a363817ee09084b4684e8e11840663e19869e9e0d952/xxhash-3.8.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:d7ece11a132325353890a144c30119073617a1299c593ca29b96c315b07e1edd", size = 223572, upload-time = "2026-06-27T08:14:48.294Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/c7/143410d026a6e0d86dc69037ec2a3b8db810a54e7f443b340ac17612be2e/xxhash-3.8.0-cp313-cp313t-win32.whl", hash = "sha256:b21db84df7b9d54d9e4195a964243c1b32d745c6fbc0cfcfffee1d4bd297196a", size = 32301, upload-time = "2026-06-27T08:14:49.687Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/db/2240b0638161637b2f310231748a7a6a06c79fb43a3adb34c96f359762bf/xxhash-3.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:0643b7d9f598f6da6f1f6b899f4358250d0fb853242e2d712cbde27bf5a99d29", size = 33221, upload-time = "2026-06-27T08:14:51.404Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/d8/52038e4fa5baf4f00654a225516168d02908edfec7ca104fbefc58af394f/xxhash-3.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:4bbacf2e938526969f8ab3334d4ac3da14ea059e1dfd1339a92f9091467e750f", size = 29294, upload-time = "2026-06-27T08:14:52.778Z" },
+]
+
+[[package]]
+name = "yarl"
+version = "1.24.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "idna" },
+ { name = "multidict" },
+ { name = "propcache" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/79/12/1e8f37460ea0f7eb59c221fdaf0ed75e7ac43e97f8093b9c6f411df50a78/yarl-1.24.2.tar.gz", hash = "sha256:9ac374123c6fd7abf64d1fec93962b0bd4ee2c19751755a762a72dd96c0378f8", size = 210798, upload-time = "2026-05-19T21:31:05.599Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/62/fcf0ce677f17e5c471c06311dd25964be38a4c586993632910d2e75278bc/yarl-1.24.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:491ac9141decf49ee8030199e1ee251cdff0e131f25678817ff6aa5f837a3536", size = 128978, upload-time = "2026-05-19T21:29:23.83Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/58/8e63299bb71ed61a834121d9d3fe6c9fcf2a6a5d09754ff4f20f2d20baf5/yarl-1.24.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e89418f65eda18f99030386305bd44d7d504e328a7945db1ead514fbe03a0607", size = 91733, upload-time = "2026-05-19T21:29:25.375Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/24/16748d5dab6daec8b0ed81ccec639a1cded0f18dcc62a4f696b4fe366c37/yarl-1.24.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cdfcce633b4a4bb8281913c57fcafd4b5933fbc19111a5e3930bbd299d6102f1", size = 91113, upload-time = "2026-05-19T21:29:26.928Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/66/b63fff7b71211e866624b21432d5943cbb633eb0c2872d9ee3070648f22c/yarl-1.24.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:863297ddede92ee49024e9a9b11ecb59f310ca85b60d8537f56bed9bbb5b1986", size = 103899, upload-time = "2026-05-19T21:29:28.842Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/ac/ba1974b8533909636f7733fe86cf677e3619527c3c2fa913e0ea89c48757/yarl-1.24.2-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:374423f70754a2c96942ede36a29d37dc6b0cb8f92f8d009ddf3ed78d3da5488", size = 97862, upload-time = "2026-05-19T21:29:31.086Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/a5/123ac993b5c2ba6f554a140305620cb8f150fa543711bbc49be3ec0a65a4/yarl-1.24.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:33a29b5d00ccbf3219bb3e351d7875739c19481e030779f48cc46a7a71681a9b", size = 111060, upload-time = "2026-05-19T21:29:32.657Z" },
+ { url = "https://files.pythonhosted.org/packages/23/37/c472d3af3509688392134a88a825276770a187f1daa4de3f6dc0a327a751/yarl-1.24.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a9532c57211730c515341af11fef6e9b61d157487272a096d0c04da445642592", size = 110613, upload-time = "2026-05-19T21:29:34.379Z" },
+ { url = "https://files.pythonhosted.org/packages/df/88/09c28dad91e662ccfaa1b78f1c57badde74fc9d0b23e74aef644750ecd73/yarl-1.24.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:91e72cf093fd833483a97ee648e0c053c7c629f51ff4a0e7edd84f806b0c5617", size = 107012, upload-time = "2026-05-19T21:29:36.216Z" },
+ { url = "https://files.pythonhosted.org/packages/07/ab/9d4f69d571a94f4d112fa7e2e007200f5a54d319f58c82ac7b7baa61f5c6/yarl-1.24.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b3177bc0a768ef3bacceb4f272632990b7bea352f1b2f1eee9d6d6ff16516f92", size = 105887, upload-time = "2026-05-19T21:29:38.746Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/9a/000b2b66c0d772a499fc531d21dab92dfeb73b640a12eed6ba89f49bb2d0/yarl-1.24.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e196952aacaf3b232e265ff02980b64d483dc0972bd49bcb061171ff22ac203a", size = 103620, upload-time = "2026-05-19T21:29:40.368Z" },
+ { url = "https://files.pythonhosted.org/packages/41/7c/7c1050f73450fbdaa3f0c72017059f00ce5e13366692f3dba25275a1083d/yarl-1.24.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:204e7a61ce99919c0de1bf904ab5d7aa188a129ea8f690a8f76cfb6e2844dc44", size = 100599, upload-time = "2026-05-19T21:29:42.66Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/b1/29e5756b3926705f5f6089bd5b9f50a56eaac550da6e260bf713ead44d04/yarl-1.24.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4b156914620f0b9d78dc1adb3751141daee561cfec796088abb89ed49d220f1a", size = 110604, upload-time = "2026-05-19T21:29:44.632Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/4b/8415bc96e9b150cde942fbac9a8182985e58f40ce5c54c34ed015407d3ee/yarl-1.24.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:8372a2b976cf70654b2be6619ab6068acabb35f724c0fda7b277fbf53d66a5cf", size = 105161, upload-time = "2026-05-19T21:29:46.755Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/d4/cde059abfa229553b7298a2eadde2752e723d50aeedaef86ce59da2718ee/yarl-1.24.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:f9a1e9b622ca284143aab5d885848686dcd85453bb1ca9abcdb7503e64dc0056", size = 110619, upload-time = "2026-05-19T21:29:48.972Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/2c/d6a6c9a61549f7b6c7e6dc6937d195bcf069582b47b7200dcd0e7b256acf/yarl-1.24.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:810e19b685c8c3c5862f6a38160a1f4e4c0916c9390024ec347b6157a45a0992", size = 107362, upload-time = "2026-05-19T21:29:51Z" },
+ { url = "https://files.pythonhosted.org/packages/92/dd/3ae5fe417e9d1c353a548553326eb9935e76b6b727161563b424cc296df3/yarl-1.24.2-cp313-cp313-win_amd64.whl", hash = "sha256:7d37fb7c38f2b6edab0f845c4f85148d4c44204f52bc127021bd2bc9fdbf1656", size = 92667, upload-time = "2026-05-19T21:29:52.743Z" },
+ { url = "https://files.pythonhosted.org/packages/10/cc/a7beb239f78f27fca1b053c8e8595e4179c02e62249b4687ec218c370c50/yarl-1.24.2-cp313-cp313-win_arm64.whl", hash = "sha256:1e831894be7c2954240e49791fa4b50c05a0dc881de2552cfe3ffd8631c7f461", size = 87069, upload-time = "2026-05-19T21:29:54.442Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/4d/4b880086bd0d3e034d25647be1d830afc3e3f610e98c4ab3490af6b1b6d5/yarl-1.24.2-py3-none-any.whl", hash = "sha256:2783d9226db8797636cd6896e4de81feed252d1db72265686c9558d97a4d94b9", size = 53576, upload-time = "2026-05-19T21:31:03.909Z" },
+]
+
+[[package]]
+name = "z3-solver"
+version = "4.15.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8a/8e/0c8f17309549d2e5cde9a3ccefa6365437f1e7bafe71878eaf9478e47b18/z3_solver-4.15.4.0.tar.gz", hash = "sha256:928c29b58c4eb62106da51c1914f6a4a55d0441f8f48a81b9da07950434a8946", size = 5018600, upload-time = "2025-10-29T18:12:03.062Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/63/33/a3d5d2eaeb0f7b3174d57d405437eabb2075d4d50bd9ea0957696c435c7b/z3_solver-4.15.4.0-py3-none-macosx_13_0_arm64.whl", hash = "sha256:407e825cc9211f95ef46bdc8d151bf630e7ab2d62a21d24cd74c09cc5b73f3aa", size = 37052538, upload-time = "2025-10-29T18:11:46.233Z" },
+ { url = "https://files.pythonhosted.org/packages/47/84/fd7ffac1551cd9f8d44fe41358f738be670fc4c24dfd514fab503f2cf3e7/z3_solver-4.15.4.0-py3-none-macosx_13_0_x86_64.whl", hash = "sha256:00bd10c5a6a5f6112d3a9a810d0799227e52f76caa860dafa5e00966bb47eb13", size = 39807925, upload-time = "2025-10-29T18:11:49.81Z" },
+ { url = "https://files.pythonhosted.org/packages/21/c9/bb51a96af0091324c81b803f16c49f719f9f6ea0b0bb52200f5c97ec4892/z3_solver-4.15.4.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e103a6f203f505b8b8b8e5c931cc407c95b61556512d4921c1ddc0b3f41b08e", size = 29268352, upload-time = "2025-10-29T18:11:53.032Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/2e/0b49f7e4e53817cfb09a0f6585012b782dfe0b666e8abefcb4fac0570606/z3_solver-4.15.4.0-py3-none-manylinux_2_34_aarch64.whl", hash = "sha256:62c7e9cbdd711932301f29919ad9158de9b2f58b4d281dd259bbcd0a2f408ba1", size = 27226534, upload-time = "2025-10-29T18:11:55.59Z" },
+ { url = "https://files.pythonhosted.org/packages/26/91/33de49538444d4aafbe47415c450c2f9abab1733e1226f276b496672f46c/z3_solver-4.15.4.0-py3-none-win32.whl", hash = "sha256:be3bc916545c96ffbf89e00d07104ff14f78336e55db069177a1bfbcc01b269d", size = 13191672, upload-time = "2025-10-29T18:11:58.424Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d6/a0b135e4419df475177ae78fc93c422430b0fd8875649486f9a5989772e6/z3_solver-4.15.4.0-py3-none-win_amd64.whl", hash = "sha256:00e35b02632ed085ea8199fb230f6015e6fc40554a6680c097bd5f060e827431", size = 16259597, upload-time = "2025-10-29T18:12:01.14Z" },
+]
+
+[[package]]
+name = "zipp"
+version = "4.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b9/d8/eab98a517c14134c0b2eb4e2387bc5f457334293ec5d2dd3857ec2966802/zipp-4.1.0.tar.gz", hash = "sha256:4cb57381f544315db7688e976e922a2b18cdb513d21cc194eb42232ba2a3e602", size = 26214, upload-time = "2026-05-18T20:08:57.967Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3a/13/547360d81e6d88d58492968ffda9f9542854f11310ee556fef14260cc886/zipp-4.1.0-py3-none-any.whl", hash = "sha256:25ad4e16390cd314347dd8f1de67a2ac538ae658ed4ab9db16029c07c188e97f", size = 10238, upload-time = "2026-05-18T20:08:57.045Z" },
+]
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json b/aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json
new file mode 100644
index 0000000..4ff61b8
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/audit_summary.json
@@ -0,0 +1,137 @@
+{
+ "run_name": "sk_zero_available_without_nli_lr_0p00010_20261022_185255",
+ "result_file": "results_2026-10-22T21-55-04.268099.json",
+ "task_list_count": 222,
+ "configured_tasks": 222,
+ "scored_task_variants": 222,
+ "result_json_keys": 226,
+ "extra_result_keys_excluded_from_task_summaries": [
+ "all",
+ "mmlu.prompt1_mqa_single_correct_sk:_average|0",
+ "mmlu.prompt2_mqa_single_correct_sk:_average|0",
+ "mmlu.prompt3_mqa_single_correct_sk:_average|0"
+ ],
+ "all_original_task_variants": 412,
+ "omitted_task_variants": 190,
+ "inaccessible_repositories": 12,
+ "inaccessible_task_variants": 187,
+ "additionally_excluded_nli_task_variants": 3,
+ "families": {
+ "arc": 3,
+ "belebele": 3,
+ "financial_exams": 3,
+ "globalmmlu-lite": 3,
+ "mmlu": 171,
+ "movie_review": 3,
+ "multi_wiki_qa": 3,
+ "senti_sk": 6,
+ "skj1": 3,
+ "sklep": 15,
+ "slovakcopa": 3,
+ "toxic_sk": 3,
+ "truthfulqa": 3
+ },
+ "prompt_counts": {
+ "1": 74,
+ "2": 74,
+ "3": 74
+ },
+ "evaluated_instances": 159426,
+ "instances_by_prompt": {
+ "1": 53142,
+ "2": 53142,
+ "3": 53142
+ },
+ "details_files_audited": 222,
+ "truncation_column": "model_response.truncated_tokens_count",
+ "truncation_values_present": 159426,
+ "truncation_null_values": 0,
+ "details_reported_samples_with_truncation": 0,
+ "details_reported_truncated_tokens_total": 0,
+ "details_reported_max_truncated_tokens_per_sample": 0,
+ "context_truncation_observed_in_log": true,
+ "actual_truncated_sample_count": null,
+ "console_context_truncation_warning_count": 1,
+ "console_context_truncation_warnings": [
+ "context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens."
+ ],
+ "truncation_interpretation": "All DETAILS counters are zero, but the console explicitly reports context truncation. Do not infer no truncation or an affected-sample count from these counters.",
+ "model_config": {
+ "model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010",
+ "generation_parameters": {
+ "num_blocks": null,
+ "block_size": null,
+ "early_stopping": null,
+ "repetition_penalty": null,
+ "frequency_penalty": null,
+ "length_penalty": null,
+ "presence_penalty": null,
+ "max_new_tokens": null,
+ "min_new_tokens": null,
+ "seed": null,
+ "stop_tokens": null,
+ "temperature": 0,
+ "top_k": null,
+ "min_p": null,
+ "top_p": null,
+ "truncate_prompt": null,
+ "cache_implementation": null,
+ "response_format": null,
+ "skip_special_tokens": true
+ },
+ "system_prompt": null,
+ "cache_dir": "~/.cache/huggingface/lighteval",
+ "tokenizer": null,
+ "revision": "main",
+ "dtype": "bfloat16",
+ "tensor_parallel_size": 1,
+ "data_parallel_size": 1,
+ "pipeline_parallel_size": 1,
+ "gpu_memory_utilization": 0.85,
+ "enable_prefix_caching": null,
+ "max_model_length": 32768,
+ "quantization": null,
+ "load_format": null,
+ "swap_space": 4,
+ "seed": 1234,
+ "trust_remote_code": false,
+ "add_special_tokens": true,
+ "multichoice_continuations_start_space": true,
+ "pairwise_tokenization": false,
+ "max_num_seqs": 128,
+ "max_num_batched_tokens": 2048,
+ "subfolder": null,
+ "is_async": false,
+ "override_chat_template": null,
+ "enforce_eager": true
+ },
+ "max_samples": null,
+ "generation_size_counts": {
+ "32": 216,
+ "126": 6
+ },
+ "original_summary_general": {
+ "hashes": {
+ "hash_examples": "316f54894dcb6f0d",
+ "hash_full_prompts": "ba7f9b257c923f55",
+ "hash_input_tokens": "8de2f010e9daf7fe",
+ "hash_cont_tokens": "1f06ad3e4b80f7cc"
+ },
+ "truncated": 0,
+ "non_truncated": 0,
+ "padded": 0,
+ "non_padded": 0
+ },
+ "pyarrow_version": "26.0.0",
+ "python_version": "3.12.14 (main, Aug 25 2026, 14:01:42) [MSC v.1944 64 bit (AMD64)]",
+ "aggregation": "Descriptive metric-specific task macro means and sample-weighted means; no official full-SK-bench overall score.",
+ "limitations": [
+ "Partial available zero-shot subset: 222 of 412 task variants.",
+ "Only the validation-selected LR=0.00010 was benchmarked.",
+ "Sample counts are evaluation instances; prompt variants reuse underlying questions.",
+ "One console warning reports context truncation; all DETAILS counters are zero, so the recorded counters do not reliably enumerate affected inputs.",
+ "The audit cannot recover the number of affected tasks/samples or quantify output generation-size termination.",
+ "All-zero summary counters include non_truncated=0 and cannot prove absence of truncation.",
+ "Raw DETAILS and generation-cache parquets are excluded from this curated tree."
+ ]
+}
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv b/aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv
new file mode 100644
index 0000000..ca9bf97
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/family_prompt_summary.csv
@@ -0,0 +1,121 @@
+family,prompt,metric,task_variants,prompt_count,evaluated_instances,task_macro_mean,sample_weighted_mean,min_task_value,max_task_value
+arc,1,em,1,1,1169,0.013686911890504704,0.013686911890504704,0.013686911890504704,0.013686911890504704
+arc,1,em_conservative,1,1,1169,0.1710863986313088,0.1710863986313088,0.1710863986313088,0.1710863986313088
+arc,1,em_relaxed,1,1,1169,0.1710863986313088,0.1710863986313088,0.1710863986313088,0.1710863986313088
+arc,2,em,1,1,1169,0.03507271171941831,0.03507271171941831,0.03507271171941831,0.03507271171941831
+arc,2,em_conservative,1,1,1169,0.223267750213858,0.223267750213858,0.223267750213858,0.223267750213858
+arc,2,em_relaxed,1,1,1169,0.22412318220701455,0.22412318220701455,0.22412318220701455,0.22412318220701455
+arc,3,em,1,1,1169,0.022241231822070145,0.022241231822070145,0.022241231822070145,0.022241231822070145
+arc,3,em_conservative,1,1,1169,0.17621899059024806,0.17621899059024806,0.17621899059024806,0.17621899059024806
+arc,3,em_relaxed,1,1,1169,0.1770744225834046,0.1770744225834046,0.1770744225834046,0.1770744225834046
+belebele,1,em,1,1,900,0.02,0.02,0.02,0.02
+belebele,1,em_conservative,1,1,900,0.21555555555555556,0.21555555555555556,0.21555555555555556,0.21555555555555556
+belebele,1,em_relaxed,1,1,900,0.21555555555555556,0.21555555555555556,0.21555555555555556,0.21555555555555556
+belebele,2,em,1,1,900,0.016666666666666666,0.016666666666666666,0.016666666666666666,0.016666666666666666
+belebele,2,em_conservative,1,1,900,0.2111111111111111,0.2111111111111111,0.2111111111111111,0.2111111111111111
+belebele,2,em_relaxed,1,1,900,0.2111111111111111,0.2111111111111111,0.2111111111111111,0.2111111111111111
+belebele,3,em,1,1,900,0.03,0.03,0.03,0.03
+belebele,3,em_conservative,1,1,900,0.2111111111111111,0.2111111111111111,0.2111111111111111,0.2111111111111111
+belebele,3,em_relaxed,1,1,900,0.21333333333333335,0.21333333333333335,0.21333333333333335,0.21333333333333335
+financial_exams,1,em,1,1,1334,0.009745127436281859,0.009745127436281859,0.009745127436281859,0.009745127436281859
+financial_exams,1,em_conservative,1,1,1334,0.20089955022488756,0.20089955022488756,0.20089955022488756,0.20089955022488756
+financial_exams,1,em_relaxed,1,1,1334,0.20089955022488756,0.20089955022488756,0.20089955022488756,0.20089955022488756
+financial_exams,2,em,1,1,1334,0.07571214392803598,0.07571214392803598,0.07571214392803598,0.07571214392803598
+financial_exams,2,em_conservative,1,1,1334,0.20539730134932535,0.20539730134932535,0.20539730134932535,0.20539730134932535
+financial_exams,2,em_relaxed,1,1,1334,0.2383808095952024,0.2383808095952024,0.2383808095952024,0.2383808095952024
+financial_exams,3,em,1,1,1334,0.0037481259370314842,0.0037481259370314842,0.0037481259370314842,0.0037481259370314842
+financial_exams,3,em_conservative,1,1,1334,0.24887556221889057,0.24887556221889057,0.24887556221889057,0.24887556221889057
+financial_exams,3,em_relaxed,1,1,1334,0.24887556221889057,0.24887556221889057,0.24887556221889057,0.24887556221889057
+globalmmlu-lite,1,em,1,1,400,0.025,0.025,0.025,0.025
+globalmmlu-lite,1,em_conservative,1,1,400,0.1825,0.1825,0.1825,0.1825
+globalmmlu-lite,1,em_relaxed,1,1,400,0.1825,0.1825,0.1825,0.1825
+globalmmlu-lite,2,em,1,1,400,0.035,0.035,0.035,0.035
+globalmmlu-lite,2,em_conservative,1,1,400,0.21,0.21,0.21,0.21
+globalmmlu-lite,2,em_relaxed,1,1,400,0.21,0.21,0.21,0.21
+globalmmlu-lite,3,em,1,1,400,0.025,0.025,0.025,0.025
+globalmmlu-lite,3,em_conservative,1,1,400,0.2025,0.2025,0.2025,0.2025
+globalmmlu-lite,3,em_relaxed,1,1,400,0.2025,0.2025,0.2025,0.2025
+mmlu,1,em,57,1,13062,0.023388116537651125,0.02174245904149441,0.0,0.06818181818181818
+mmlu,1,em_conservative,57,1,13062,0.1403536981271315,0.13910580309294135,0.0053404539385847796,0.3096446700507614
+mmlu,1,em_relaxed,57,1,13062,0.14312818797047927,0.14094319399785638,0.006675567423230975,0.3096446700507614
+mmlu,2,em,57,1,13062,0.04274500178923145,0.040116368090644616,0.0,0.13
+mmlu,2,em_conservative,57,1,13062,0.17602310250998576,0.17432246210381258,0.0026702269692923898,0.3553299492385787
+mmlu,2,em_relaxed,57,1,13062,0.17964687184852454,0.17661920073495635,0.0026702269692923898,0.3553299492385787
+mmlu,3,em,57,1,13062,0.028327141879067468,0.026871841984382177,0.0,0.07575757575757576
+mmlu,3,em_conservative,57,1,13062,0.15358971887332207,0.15051293829428877,0.0053404539385847796,0.3350253807106599
+mmlu,3,em_relaxed,57,1,13062,0.1575073160887174,0.15411116214974735,0.01335113484646195,0.3350253807106599
+movie_review,1,em_conservative_norm,1,1,2008,0.6324701195219123,0.6324701195219123,0.6324701195219123,0.6324701195219123
+movie_review,1,em_norm,1,1,2008,0.38745019920318724,0.38745019920318724,0.38745019920318724,0.38745019920318724
+movie_review,1,em_norm_relaxed,1,1,2008,0.6329681274900398,0.6329681274900398,0.6329681274900398,0.6329681274900398
+movie_review,2,em_conservative_norm,1,1,2008,0.47758964143426297,0.47758964143426297,0.47758964143426297,0.47758964143426297
+movie_review,2,em_norm,1,1,2008,0.01045816733067729,0.01045816733067729,0.01045816733067729,0.01045816733067729
+movie_review,2,em_norm_relaxed,1,1,2008,0.48306772908366535,0.48306772908366535,0.48306772908366535,0.48306772908366535
+movie_review,3,em_conservative_norm,1,1,2008,0.09412350597609562,0.09412350597609562,0.09412350597609562,0.09412350597609562
+movie_review,3,em_norm,1,1,2008,0.04731075697211155,0.04731075697211155,0.04731075697211155,0.04731075697211155
+movie_review,3,em_norm_relaxed,1,1,2008,0.11105577689243028,0.11105577689243028,0.11105577689243028,0.11105577689243028
+multi_wiki_qa,1,exact_match_slk_prefix,1,1,4997,0.36401841104662797,0.36401841104662797,0.36401841104662797,0.36401841104662797
+multi_wiki_qa,1,f1_slk,1,1,4997,0.5156574509584114,0.5156574509584114,0.5156574509584114,0.5156574509584114
+multi_wiki_qa,2,exact_match_slk_prefix,1,1,4997,0.36001600960576347,0.36001600960576347,0.36001600960576347,0.36001600960576347
+multi_wiki_qa,2,f1_slk,1,1,4997,0.5090494926903776,0.5090494926903776,0.5090494926903776,0.5090494926903776
+multi_wiki_qa,3,exact_match_slk_prefix,1,1,4997,0.34000400240144085,0.34000400240144085,0.34000400240144085,0.34000400240144085
+multi_wiki_qa,3,f1_slk,1,1,4997,0.5050071786470199,0.5050071786470199,0.5050071786470199,0.5050071786470199
+senti_sk,1,em_conservative_norm,2,1,8405,0.3151902608050904,0.3156454491374182,0.3111587982832618,0.31922172332691895
+senti_sk,1,em_norm,2,1,8405,0.21165780446913535,0.2133254015466984,0.1968884120171674,0.22642719692110327
+senti_sk,1,em_norm_relaxed,2,1,8405,0.3161543660374346,0.31659726353361095,0.31223175965665234,0.3200769724182168
+senti_sk,2,em_conservative_norm,2,1,8405,0.4350953001676545,0.4285544318857823,0.37716484926234767,0.49302575107296137
+senti_sk,2,em_norm,2,1,8405,0.011392328544121953,0.01225461035098156,0.0037553648068669528,0.019029292281376953
+senti_sk,2,em_norm_relaxed,2,1,8405,0.43993914035536885,0.43355145746579415,0.383365405174257,0.4965128755364807
+senti_sk,3,em_conservative_norm,2,1,8405,0.23176005812390282,0.23545508625817965,0.1990343347639485,0.26448578148385715
+senti_sk,3,em_norm,2,1,8405,0.06853090550874015,0.0726948245092207,0.03165236051502146,0.10540945050245884
+senti_sk,3,em_norm_relaxed,2,1,8405,0.3302092538502268,0.33908387864366446,0.25160944206008584,0.40880906564036773
+skj1,1,em,1,1,100,0.0,0.0,0.0,0.0
+skj1,1,em_conservative,1,1,100,0.02,0.02,0.02,0.02
+skj1,1,em_relaxed,1,1,100,0.02,0.02,0.02,0.02
+skj1,2,em,1,1,100,0.0,0.0,0.0,0.0
+skj1,2,em_conservative,1,1,100,0.05,0.05,0.05,0.05
+skj1,2,em_relaxed,1,1,100,0.06,0.06,0.06,0.06
+skj1,3,em,1,1,100,0.0,0.0,0.0,0.0
+skj1,3,em_conservative,1,1,100,0.03,0.03,0.03,0.03
+skj1,3,em_relaxed,1,1,100,0.05,0.05,0.05,0.05
+sklep,1,em_conservative_norm,4,1,9025,0.4046842034899774,0.2173961218836565,0.016386890487609912,0.8800383877159309
+sklep,1,em_norm,4,1,9025,0.11229466834690793,0.0520775623268698,0.0,0.44817658349328215
+sklep,1,em_norm_relaxed,4,1,9025,0.4050240467784944,0.21772853185595567,0.016786570743405275,0.8809980806142035
+sklep,1,exact_match_slk_prefix,1,1,9583,0.16456224564332672,0.16456224564332672,0.16456224564332672,0.16456224564332672
+sklep,1,f1_slk,1,1,9583,0.3730151221096053,0.3730151221096053,0.3730151221096053,0.3730151221096053
+sklep,2,em_conservative_norm,4,1,9025,0.37570022410310483,0.26204986149584486,0.09952038369304557,0.7247915087187263
+sklep,2,em_norm,4,1,9025,0.0006917304534838009,0.00044321329639889195,0.0,0.0018072289156626507
+sklep,2,em_norm_relaxed,4,1,9025,0.3764699538087837,0.26249307479224376,0.09972022382094324,0.7247915087187263
+sklep,2,exact_match_slk_prefix,1,1,9583,0.16518835437754356,0.16518835437754356,0.16518835437754356,0.16518835437754356
+sklep,2,f1_slk,1,1,9583,0.3751701402109962,0.3751701402109962,0.3751701402109962,0.3751701402109962
+sklep,3,em_conservative_norm,4,1,9025,0.26202719318429357,0.15889196675900277,0.001398880895283773,0.6967399545109931
+sklep,3,em_norm,4,1,9025,0.004139976412367319,0.001994459833795014,0.0,0.015355086372360844
+sklep,3,em_norm_relaxed,4,1,9025,0.2709043524933147,0.16299168975069253,0.001398880895283773,0.6967399545109931
+sklep,3,exact_match_slk_prefix,1,1,9583,0.16143170197224252,0.16143170197224252,0.16143170197224252,0.16143170197224252
+sklep,3,f1_slk,1,1,9583,0.3769392643579436,0.3769392643579436,0.3769392643579436,0.3769392643579436
+slovakcopa,1,em,1,1,500,0.0,0.0,0.0,0.0
+slovakcopa,1,em_conservative,1,1,500,0.508,0.508,0.508,0.508
+slovakcopa,1,em_relaxed,1,1,500,0.618,0.618,0.618,0.618
+slovakcopa,2,em,1,1,500,0.0,0.0,0.0,0.0
+slovakcopa,2,em_conservative,1,1,500,0.326,0.326,0.326,0.326
+slovakcopa,2,em_relaxed,1,1,500,0.51,0.51,0.51,0.51
+slovakcopa,3,em,1,1,500,0.0,0.0,0.0,0.0
+slovakcopa,3,em_conservative,1,1,500,0.084,0.084,0.084,0.084
+slovakcopa,3,em_relaxed,1,1,500,0.086,0.086,0.086,0.086
+toxic_sk,1,em_conservative_norm,1,1,881,0.45629965947786605,0.45629965947786605,0.45629965947786605,0.45629965947786605
+toxic_sk,1,em_norm,1,1,881,0.0,0.0,0.0,0.0
+toxic_sk,1,em_norm_relaxed,1,1,881,0.45629965947786605,0.45629965947786605,0.45629965947786605,0.45629965947786605
+toxic_sk,2,em_conservative_norm,1,1,881,0.47559591373439275,0.47559591373439275,0.47559591373439275,0.47559591373439275
+toxic_sk,2,em_norm,1,1,881,0.0,0.0,0.0,0.0
+toxic_sk,2,em_norm_relaxed,1,1,881,0.47559591373439275,0.47559591373439275,0.47559591373439275,0.47559591373439275
+toxic_sk,3,em_conservative_norm,1,1,881,0.4960272417707151,0.4960272417707151,0.4960272417707151,0.4960272417707151
+toxic_sk,3,em_norm,1,1,881,0.007945516458569807,0.007945516458569807,0.007945516458569807,0.007945516458569807
+toxic_sk,3,em_norm_relaxed,1,1,881,0.4960272417707151,0.4960272417707151,0.4960272417707151,0.4960272417707151
+truthfulqa,1,em,1,1,778,0.0,0.0,0.0,0.0
+truthfulqa,1,em_conservative,1,1,778,0.007712082262210797,0.007712082262210797,0.007712082262210797,0.007712082262210797
+truthfulqa,1,em_relaxed,1,1,778,0.007712082262210797,0.007712082262210797,0.007712082262210797,0.007712082262210797
+truthfulqa,2,em,1,1,778,0.0,0.0,0.0,0.0
+truthfulqa,2,em_conservative,1,1,778,0.016709511568123392,0.016709511568123392,0.016709511568123392,0.016709511568123392
+truthfulqa,2,em_relaxed,1,1,778,0.016709511568123392,0.016709511568123392,0.016709511568123392,0.016709511568123392
+truthfulqa,3,em,1,1,778,0.0,0.0,0.0,0.0
+truthfulqa,3,em_conservative,1,1,778,0.011568123393316195,0.011568123393316195,0.011568123393316195,0.011568123393316195
+truthfulqa,3,em_relaxed,1,1,778,0.011568123393316195,0.011568123393316195,0.011568123393316195,0.011568123393316195
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv b/aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv
new file mode 100644
index 0000000..f7c37fc
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/family_summary.csv
@@ -0,0 +1,41 @@
+family,metric,task_variants,prompt_count,evaluated_instances,task_macro_mean,sample_weighted_mean,min_task_value,max_task_value
+arc,em,3,3,3507,0.023666951810664385,0.023666951810664385,0.013686911890504704,0.03507271171941831
+arc,em_conservative,3,3,3507,0.19019104647847163,0.19019104647847163,0.1710863986313088,0.223267750213858
+arc,em_relaxed,3,3,3507,0.19076133447390933,0.19076133447390933,0.1710863986313088,0.22412318220701455
+belebele,em,3,3,2700,0.022222222222222223,0.022222222222222223,0.016666666666666666,0.03
+belebele,em_conservative,3,3,2700,0.21259259259259258,0.21259259259259258,0.2111111111111111,0.21555555555555556
+belebele,em_relaxed,3,3,2700,0.21333333333333335,0.21333333333333335,0.2111111111111111,0.21555555555555556
+financial_exams,em,3,3,4002,0.029735132433783108,0.029735132433783108,0.0037481259370314842,0.07571214392803598
+financial_exams,em_conservative,3,3,4002,0.21839080459770116,0.21839080459770116,0.20089955022488756,0.24887556221889057
+financial_exams,em_relaxed,3,3,4002,0.22938530734632684,0.22938530734632684,0.20089955022488756,0.24887556221889057
+globalmmlu-lite,em,3,3,1200,0.028333333333333335,0.028333333333333332,0.025,0.035
+globalmmlu-lite,em_conservative,3,3,1200,0.19833333333333333,0.19833333333333333,0.1825,0.21
+globalmmlu-lite,em_relaxed,3,3,1200,0.19833333333333333,0.19833333333333333,0.1825,0.21
+mmlu,em,171,3,39186,0.03148675340198335,0.029576889705507068,0.0,0.13
+mmlu,em_conservative,171,3,39186,0.15665550650347979,0.15464706783034757,0.0026702269692923898,0.3553299492385787
+mmlu,em_relaxed,171,3,39186,0.16009412530257372,0.15722451896085338,0.0026702269692923898,0.3553299492385787
+movie_review,em_conservative_norm,3,3,6024,0.40139442231075695,0.40139442231075695,0.09412350597609562,0.6324701195219123
+movie_review,em_norm,3,3,6024,0.14840637450199204,0.14840637450199204,0.01045816733067729,0.38745019920318724
+movie_review,em_norm_relaxed,3,3,6024,0.40903054448871184,0.40903054448871184,0.11105577689243028,0.6329681274900398
+multi_wiki_qa,exact_match_slk_prefix,3,3,14991,0.35467947435127745,0.35467947435127745,0.34000400240144085,0.36401841104662797
+multi_wiki_qa,f1_slk,3,3,14991,0.5099047074319363,0.5099047074319363,0.5050071786470199,0.5156574509584114
+senti_sk,em_conservative_norm,6,3,25215,0.32734853969888256,0.32655165576046,0.1990343347639485,0.49302575107296137
+senti_sk,em_norm,6,3,25215,0.09719367950733247,0.09942494546896688,0.0037553648068669528,0.22642719692110327
+senti_sk,em_norm_relaxed,6,3,25215,0.3621009200810101,0.36307753321435654,0.25160944206008584,0.4965128755364807
+skj1,em,3,3,300,0.0,0.0,0.0,0.0
+skj1,em_conservative,3,3,300,0.03333333333333333,0.03333333333333333,0.02,0.05
+skj1,em_relaxed,3,3,300,0.043333333333333335,0.043333333333333335,0.02,0.06
+sklep,em_conservative_norm,12,3,27075,0.34747054025912527,0.21277931671283473,0.001398880895283773,0.8800383877159309
+sklep,em_norm,12,3,27075,0.039042125070919684,0.01817174515235457,0.0,0.44817658349328215
+sklep,em_norm_relaxed,12,3,27075,0.3507994510268643,0.21440443213296398,0.001398880895283773,0.8809980806142035
+sklep,exact_match_slk_prefix,3,3,28749,0.16372743399770426,0.16372743399770426,0.16143170197224252,0.16518835437754356
+sklep,f1_slk,3,3,28749,0.3750415088928484,0.3750415088928484,0.3730151221096053,0.3769392643579436
+slovakcopa,em,3,3,1500,0.0,0.0,0.0,0.0
+slovakcopa,em_conservative,3,3,1500,0.306,0.306,0.084,0.508
+slovakcopa,em_relaxed,3,3,1500,0.4046666666666667,0.4046666666666667,0.086,0.618
+toxic_sk,em_conservative_norm,3,3,2643,0.4759742716609913,0.4759742716609913,0.45629965947786605,0.4960272417707151
+toxic_sk,em_norm,3,3,2643,0.0026485054861899358,0.0026485054861899358,0.0,0.007945516458569807
+toxic_sk,em_norm_relaxed,3,3,2643,0.4759742716609913,0.4759742716609913,0.45629965947786605,0.4960272417707151
+truthfulqa,em,3,3,2334,0.0,0.0,0.0,0.0
+truthfulqa,em_conservative,3,3,2334,0.011996572407883462,0.011996572407883462,0.007712082262210797,0.016709511568123392
+truthfulqa,em_relaxed,3,3,2334,0.011996572407883462,0.011996572407883462,0.007712082262210797,0.016709511568123392
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_budget_boundary_rows.csv b/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_budget_boundary_rows.csv
new file mode 100644
index 0000000..8543503
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_budget_boundary_rows.csv
@@ -0,0 +1,70 @@
+task,details_row_index_zero_based,input_token_array_length,generation_size,input_budget,recorded_input_characters
+multi_wiki_qa.prompt1_qa_sk|0,47,32642,126,32642,97590
+multi_wiki_qa.prompt1_qa_sk|0,605,32642,126,32642,71863
+multi_wiki_qa.prompt1_qa_sk|0,724,32642,126,32642,97580
+multi_wiki_qa.prompt1_qa_sk|0,837,32642,126,32642,121781
+multi_wiki_qa.prompt1_qa_sk|0,881,32642,126,32642,71849
+multi_wiki_qa.prompt1_qa_sk|0,1155,32642,126,32642,71849
+multi_wiki_qa.prompt1_qa_sk|0,1479,32642,126,32642,71827
+multi_wiki_qa.prompt1_qa_sk|0,1770,32642,126,32642,121822
+multi_wiki_qa.prompt1_qa_sk|0,2200,32642,126,32642,97575
+multi_wiki_qa.prompt1_qa_sk|0,2352,32642,126,32642,121852
+multi_wiki_qa.prompt1_qa_sk|0,2447,32642,126,32642,71858
+multi_wiki_qa.prompt1_qa_sk|0,2507,32642,126,32642,121802
+multi_wiki_qa.prompt1_qa_sk|0,2890,32642,126,32642,71841
+multi_wiki_qa.prompt1_qa_sk|0,3094,32642,126,32642,121827
+multi_wiki_qa.prompt1_qa_sk|0,3098,32642,126,32642,97594
+multi_wiki_qa.prompt1_qa_sk|0,3354,32642,126,32642,97576
+multi_wiki_qa.prompt1_qa_sk|0,4088,32642,126,32642,121819
+multi_wiki_qa.prompt1_qa_sk|0,4511,32642,126,32642,97623
+multi_wiki_qa.prompt1_qa_sk|0,4584,32642,126,32642,71833
+multi_wiki_qa.prompt1_qa_sk|0,4667,32642,126,32642,71832
+multi_wiki_qa.prompt1_qa_sk|0,4688,32642,126,32642,97578
+multi_wiki_qa.prompt1_qa_sk|0,4752,32642,126,32642,121811
+multi_wiki_qa.prompt1_qa_sk|0,4837,32642,126,32642,71839
+multi_wiki_qa.prompt2_qa_sk|0,47,32642,126,32642,97593
+multi_wiki_qa.prompt2_qa_sk|0,605,32642,126,32642,71866
+multi_wiki_qa.prompt2_qa_sk|0,724,32642,126,32642,97583
+multi_wiki_qa.prompt2_qa_sk|0,837,32642,126,32642,121784
+multi_wiki_qa.prompt2_qa_sk|0,881,32642,126,32642,71852
+multi_wiki_qa.prompt2_qa_sk|0,1155,32642,126,32642,71852
+multi_wiki_qa.prompt2_qa_sk|0,1479,32642,126,32642,71830
+multi_wiki_qa.prompt2_qa_sk|0,1770,32642,126,32642,121825
+multi_wiki_qa.prompt2_qa_sk|0,2200,32642,126,32642,97578
+multi_wiki_qa.prompt2_qa_sk|0,2352,32642,126,32642,121855
+multi_wiki_qa.prompt2_qa_sk|0,2447,32642,126,32642,71861
+multi_wiki_qa.prompt2_qa_sk|0,2507,32642,126,32642,121805
+multi_wiki_qa.prompt2_qa_sk|0,2890,32642,126,32642,71844
+multi_wiki_qa.prompt2_qa_sk|0,3094,32642,126,32642,121830
+multi_wiki_qa.prompt2_qa_sk|0,3098,32642,126,32642,97597
+multi_wiki_qa.prompt2_qa_sk|0,3354,32642,126,32642,97579
+multi_wiki_qa.prompt2_qa_sk|0,4088,32642,126,32642,121822
+multi_wiki_qa.prompt2_qa_sk|0,4511,32642,126,32642,97626
+multi_wiki_qa.prompt2_qa_sk|0,4584,32642,126,32642,71836
+multi_wiki_qa.prompt2_qa_sk|0,4667,32642,126,32642,71835
+multi_wiki_qa.prompt2_qa_sk|0,4688,32642,126,32642,97581
+multi_wiki_qa.prompt2_qa_sk|0,4752,32642,126,32642,121814
+multi_wiki_qa.prompt2_qa_sk|0,4837,32642,126,32642,71842
+multi_wiki_qa.prompt3_qa_sk|0,47,32642,126,32642,97521
+multi_wiki_qa.prompt3_qa_sk|0,605,32642,126,32642,71794
+multi_wiki_qa.prompt3_qa_sk|0,724,32642,126,32642,97511
+multi_wiki_qa.prompt3_qa_sk|0,837,32642,126,32642,121712
+multi_wiki_qa.prompt3_qa_sk|0,881,32642,126,32642,71780
+multi_wiki_qa.prompt3_qa_sk|0,1155,32642,126,32642,71780
+multi_wiki_qa.prompt3_qa_sk|0,1479,32642,126,32642,71758
+multi_wiki_qa.prompt3_qa_sk|0,1770,32642,126,32642,121753
+multi_wiki_qa.prompt3_qa_sk|0,2200,32642,126,32642,97506
+multi_wiki_qa.prompt3_qa_sk|0,2352,32642,126,32642,121783
+multi_wiki_qa.prompt3_qa_sk|0,2447,32642,126,32642,71789
+multi_wiki_qa.prompt3_qa_sk|0,2507,32642,126,32642,121733
+multi_wiki_qa.prompt3_qa_sk|0,2890,32642,126,32642,71772
+multi_wiki_qa.prompt3_qa_sk|0,3094,32642,126,32642,121758
+multi_wiki_qa.prompt3_qa_sk|0,3098,32642,126,32642,97525
+multi_wiki_qa.prompt3_qa_sk|0,3354,32642,126,32642,97507
+multi_wiki_qa.prompt3_qa_sk|0,4088,32642,126,32642,121750
+multi_wiki_qa.prompt3_qa_sk|0,4511,32642,126,32642,97554
+multi_wiki_qa.prompt3_qa_sk|0,4584,32642,126,32642,71764
+multi_wiki_qa.prompt3_qa_sk|0,4667,32642,126,32642,71763
+multi_wiki_qa.prompt3_qa_sk|0,4688,32642,126,32642,97509
+multi_wiki_qa.prompt3_qa_sk|0,4752,32642,126,32642,121742
+multi_wiki_qa.prompt3_qa_sk|0,4837,32642,126,32642,71770
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_per_task.csv b/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_per_task.csv
new file mode 100644
index 0000000..9d78900
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_per_task.csv
@@ -0,0 +1,223 @@
+task,generation_size,context_limit,input_budget,samples,null_token_arrays,null_input_strings,min_recorded_input_tokens,max_recorded_input_tokens,mean_recorded_input_tokens,arrays_at_input_budget,arrays_over_input_budget,max_recorded_input_characters
+arc.prompt1_mqa_single_correct_sk|0,32,32768,32736,1169,0,0,147,539,236.47733105218134,0,0,1275
+arc.prompt2_mqa_single_correct_sk|0,32,32768,32736,1169,0,0,131,523,220.47733105218134,0,0,1233
+arc.prompt3_mqa_single_correct_sk|0,32,32768,32736,1169,0,0,142,534,231.47733105218134,0,0,1261
+belebele.prompt1_mqa_single_correct_sk|0,32,32768,32736,900,0,0,237,905,434.35777777777776,0,0,2118
+belebele.prompt2_mqa_single_correct_sk|0,32,32768,32736,900,0,0,221,889,418.35777777777776,0,0,2076
+belebele.prompt3_mqa_single_correct_sk|0,32,32768,32736,900,0,0,232,900,429.35777777777776,0,0,2104
+financial_exams.prompt1_completion_sk|0,32,32768,32736,1334,0,0,152,682,295.84107946026984,0,0,1642
+financial_exams.prompt2_completion_sk|0,32,32768,32736,1334,0,0,149,679,292.84107946026984,0,0,1633
+financial_exams.prompt3_completion_sk|0,32,32768,32736,1334,0,0,137,667,280.84107946026984,0,0,1610
+globalmmlu-lite.prompt1_mqa_single_correct_sk|0,32,32768,32736,400,0,0,153,1165,301.385,0,0,2646
+globalmmlu-lite.prompt2_mqa_single_correct_sk|0,32,32768,32736,400,0,0,137,1149,285.385,0,0,2604
+globalmmlu-lite.prompt3_mqa_single_correct_sk|0,32,32768,32736,400,0,0,148,1160,296.385,0,0,2632
+mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0,32,32768,32736,100,0,0,154,266,210.72,0,0,626
+mmlu.prompt1_mqa_single_correct_sk:anatomy|0,32,32768,32736,134,0,0,153,384,220.02238805970148,0,0,908
+mmlu.prompt1_mqa_single_correct_sk:astronomy|0,32,32768,32736,152,0,0,163,480,249.75,0,0,1099
+mmlu.prompt1_mqa_single_correct_sk:business_ethics|0,32,32768,32736,100,0,0,160,419,257.8,0,0,1001
+mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0,32,32768,32736,259,0,0,151,377,224.1853281853282,0,0,888
+mmlu.prompt1_mqa_single_correct_sk:college_biology|0,32,32768,32736,142,0,0,171,479,257.1478873239437,0,0,1074
+mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0,32,32768,32736,100,0,0,160,389,234.02,0,0,952
+mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0,32,32768,32736,100,0,0,174,602,307.65,0,0,1433
+mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0,32,32768,32736,99,0,0,163,356,234.63636363636363,0,0,810
+mmlu.prompt1_mqa_single_correct_sk:college_medicine|0,32,32768,32736,172,0,0,166,2217,313.7325581395349,0,0,5040
+mmlu.prompt1_mqa_single_correct_sk:college_physics|0,32,32768,32736,102,0,0,160,364,246.8921568627451,0,0,861
+mmlu.prompt1_mqa_single_correct_sk:computer_security|0,32,32768,32736,100,0,0,161,486,234.18,0,0,1197
+mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0,32,32768,32736,235,0,0,155,280,195.20425531914893,0,0,669
+mmlu.prompt1_mqa_single_correct_sk:econometrics|0,32,32768,32736,112,0,0,164,484,289.25892857142856,0,0,1078
+mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0,32,32768,32736,145,0,0,148,270,195.91034482758621,0,0,647
+mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0,32,32768,32736,373,0,0,152,393,212.76139410187668,0,0,927
+mmlu.prompt1_mqa_single_correct_sk:formal_logic|0,32,32768,32736,115,0,0,168,572,295.5304347826087,0,0,1336
+mmlu.prompt1_mqa_single_correct_sk:global_facts|0,32,32768,32736,100,0,0,166,376,207.62,0,0,860
+mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0,32,32768,32736,282,0,0,165,469,264.54609929078015,0,0,1078
+mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0,32,32768,32736,198,0,0,153,484,247.45454545454547,0,0,1110
+mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0,32,32768,32736,99,0,0,157,760,311.77777777777777,0,0,1787
+mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0,32,32768,32736,135,0,0,355,1239,745.1703703703704,0,0,2795
+mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0,32,32768,32736,198,0,0,158,358,207.76262626262627,0,0,817
+mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0,32,32768,32736,193,0,0,188,470,259.4974093264249,0,0,1096
+mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0,32,32768,32736,387,0,0,169,441,248.3798449612403,0,0,984
+mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0,32,32768,32736,266,0,0,156,430,219.24436090225564,0,0,948
+mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0,32,32768,32736,217,0,0,171,542,253.58986175115209,0,0,1203
+mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0,32,32768,32736,150,0,0,179,575,284.43333333333334,0,0,1342
+mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0,32,32768,32736,536,0,0,159,670,238.3320895522388,0,0,1588
+mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0,32,32768,32736,214,0,0,187,778,343.33644859813086,0,0,1769
+mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0,32,32768,32736,57,0,0,370,1075,585.6842105263158,0,0,2520
+mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0,32,32768,32736,180,0,0,261,1734,729.6333333333333,0,0,4002
+mmlu.prompt1_mqa_single_correct_sk:human_aging|0,32,32768,32736,219,0,0,160,341,206.5068493150685,0,0,825
+mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0,32,32768,32736,109,0,0,159,334,211.24770642201835,0,0,855
+mmlu.prompt1_mqa_single_correct_sk:international_law|0,32,32768,32736,118,0,0,171,454,302.9491525423729,0,0,1019
+mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0,32,32768,32736,107,0,0,158,481,244.4018691588785,0,0,1116
+mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0,32,32768,32736,158,0,0,163,412,247.30379746835442,0,0,979
+mmlu.prompt1_mqa_single_correct_sk:machine_learning|0,32,32768,32736,112,0,0,170,435,255.64285714285714,0,0,909
+mmlu.prompt1_mqa_single_correct_sk:management|0,32,32768,32736,100,0,0,154,285,197.01,0,0,728
+mmlu.prompt1_mqa_single_correct_sk:marketing|0,32,32768,32736,234,0,0,170,361,230.77777777777777,0,0,853
+mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0,32,32768,32736,100,0,0,161,304,207.78,0,0,728
+mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0,32,32768,32736,779,0,0,147,565,196.65340179717586,0,0,1348
+mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0,32,32768,32736,326,0,0,164,411,249.94171779141104,0,0,992
+mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0,32,32768,32736,749,0,0,261,382,310.8558077436582,0,0,890
+mmlu.prompt1_mqa_single_correct_sk:nutrition|0,32,32768,32736,303,0,0,151,476,250.0990099009901,0,0,1123
+mmlu.prompt1_mqa_single_correct_sk:philosophy|0,32,32768,32736,308,0,0,161,548,226.2922077922078,0,0,1306
+mmlu.prompt1_mqa_single_correct_sk:prehistory|0,32,32768,32736,319,0,0,150,495,236.16614420062695,0,0,1118
+mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0,32,32768,32736,276,0,0,184,582,315.0217391304348,0,0,1193
+mmlu.prompt1_mqa_single_correct_sk:professional_law|0,32,32768,32736,1171,0,0,168,1660,593.5713065755764,0,0,3903
+mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0,32,32768,32736,264,0,0,261,855,456.2537878787879,0,0,1946
+mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0,32,32768,32736,596,0,0,161,595,267.18624161073825,0,0,1449
+mmlu.prompt1_mqa_single_correct_sk:public_relations|0,32,32768,32736,109,0,0,152,513,224.1559633027523,0,0,1207
+mmlu.prompt1_mqa_single_correct_sk:security_studies|0,32,32768,32736,222,0,0,164,1068,439.8063063063063,0,0,2495
+mmlu.prompt1_mqa_single_correct_sk:sociology|0,32,32768,32736,197,0,0,170,440,254.248730964467,0,0,1038
+mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0,32,32768,32736,99,0,0,163,389,233.3939393939394,0,0,900
+mmlu.prompt1_mqa_single_correct_sk:virology|0,32,32768,32736,164,0,0,162,673,219.10365853658536,0,0,1384
+mmlu.prompt1_mqa_single_correct_sk:world_religions|0,32,32768,32736,171,0,0,155,255,178.2046783625731,0,0,611
+mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0,32,32768,32736,100,0,0,138,250,194.72,0,0,584
+mmlu.prompt2_mqa_single_correct_sk:anatomy|0,32,32768,32736,134,0,0,137,368,204.02238805970148,0,0,866
+mmlu.prompt2_mqa_single_correct_sk:astronomy|0,32,32768,32736,152,0,0,147,464,233.75,0,0,1057
+mmlu.prompt2_mqa_single_correct_sk:business_ethics|0,32,32768,32736,100,0,0,144,403,241.8,0,0,959
+mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0,32,32768,32736,259,0,0,135,361,208.1853281853282,0,0,846
+mmlu.prompt2_mqa_single_correct_sk:college_biology|0,32,32768,32736,142,0,0,155,463,241.14788732394365,0,0,1032
+mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0,32,32768,32736,100,0,0,144,373,218.02,0,0,910
+mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0,32,32768,32736,100,0,0,158,586,291.65,0,0,1391
+mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0,32,32768,32736,99,0,0,147,340,218.63636363636363,0,0,768
+mmlu.prompt2_mqa_single_correct_sk:college_medicine|0,32,32768,32736,172,0,0,150,2201,297.7325581395349,0,0,4998
+mmlu.prompt2_mqa_single_correct_sk:college_physics|0,32,32768,32736,102,0,0,144,348,230.8921568627451,0,0,819
+mmlu.prompt2_mqa_single_correct_sk:computer_security|0,32,32768,32736,100,0,0,145,470,218.18,0,0,1155
+mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0,32,32768,32736,235,0,0,139,264,179.20425531914893,0,0,627
+mmlu.prompt2_mqa_single_correct_sk:econometrics|0,32,32768,32736,112,0,0,148,468,273.25892857142856,0,0,1036
+mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0,32,32768,32736,145,0,0,132,254,179.91034482758621,0,0,605
+mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0,32,32768,32736,373,0,0,136,377,196.76139410187668,0,0,885
+mmlu.prompt2_mqa_single_correct_sk:formal_logic|0,32,32768,32736,115,0,0,152,556,279.5304347826087,0,0,1294
+mmlu.prompt2_mqa_single_correct_sk:global_facts|0,32,32768,32736,100,0,0,150,360,191.62,0,0,818
+mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0,32,32768,32736,282,0,0,149,453,248.54609929078015,0,0,1036
+mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0,32,32768,32736,198,0,0,137,468,231.45454545454547,0,0,1068
+mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0,32,32768,32736,99,0,0,141,744,295.77777777777777,0,0,1745
+mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0,32,32768,32736,135,0,0,339,1223,729.1703703703704,0,0,2753
+mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0,32,32768,32736,198,0,0,142,342,191.76262626262627,0,0,775
+mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0,32,32768,32736,193,0,0,172,454,243.49740932642487,0,0,1054
+mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0,32,32768,32736,387,0,0,153,425,232.3798449612403,0,0,942
+mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0,32,32768,32736,266,0,0,140,414,203.24436090225564,0,0,906
+mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0,32,32768,32736,217,0,0,155,526,237.58986175115209,0,0,1161
+mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0,32,32768,32736,150,0,0,163,559,268.43333333333334,0,0,1300
+mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0,32,32768,32736,536,0,0,143,654,222.3320895522388,0,0,1546
+mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0,32,32768,32736,214,0,0,171,762,327.33644859813086,0,0,1727
+mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0,32,32768,32736,57,0,0,354,1059,569.6842105263158,0,0,2478
+mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0,32,32768,32736,180,0,0,245,1718,713.6333333333333,0,0,3960
+mmlu.prompt2_mqa_single_correct_sk:human_aging|0,32,32768,32736,219,0,0,144,325,190.5068493150685,0,0,783
+mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0,32,32768,32736,109,0,0,143,318,195.24770642201835,0,0,813
+mmlu.prompt2_mqa_single_correct_sk:international_law|0,32,32768,32736,118,0,0,155,438,286.9491525423729,0,0,977
+mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0,32,32768,32736,107,0,0,142,465,228.4018691588785,0,0,1074
+mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0,32,32768,32736,158,0,0,147,396,231.30379746835442,0,0,937
+mmlu.prompt2_mqa_single_correct_sk:machine_learning|0,32,32768,32736,112,0,0,154,419,239.64285714285714,0,0,867
+mmlu.prompt2_mqa_single_correct_sk:management|0,32,32768,32736,100,0,0,138,269,181.01,0,0,686
+mmlu.prompt2_mqa_single_correct_sk:marketing|0,32,32768,32736,234,0,0,154,345,214.77777777777777,0,0,811
+mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0,32,32768,32736,100,0,0,145,288,191.78,0,0,686
+mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0,32,32768,32736,779,0,0,131,549,180.65340179717586,0,0,1306
+mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0,32,32768,32736,326,0,0,148,395,233.94171779141104,0,0,950
+mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0,32,32768,32736,749,0,0,245,366,294.8558077436582,0,0,848
+mmlu.prompt2_mqa_single_correct_sk:nutrition|0,32,32768,32736,303,0,0,135,460,234.0990099009901,0,0,1081
+mmlu.prompt2_mqa_single_correct_sk:philosophy|0,32,32768,32736,308,0,0,145,532,210.2922077922078,0,0,1264
+mmlu.prompt2_mqa_single_correct_sk:prehistory|0,32,32768,32736,319,0,0,134,479,220.16614420062695,0,0,1076
+mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0,32,32768,32736,276,0,0,168,566,299.0217391304348,0,0,1151
+mmlu.prompt2_mqa_single_correct_sk:professional_law|0,32,32768,32736,1171,0,0,152,1644,577.5713065755764,0,0,3861
+mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0,32,32768,32736,264,0,0,245,839,440.2537878787879,0,0,1904
+mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0,32,32768,32736,596,0,0,145,579,251.18624161073825,0,0,1407
+mmlu.prompt2_mqa_single_correct_sk:public_relations|0,32,32768,32736,109,0,0,136,497,208.1559633027523,0,0,1165
+mmlu.prompt2_mqa_single_correct_sk:security_studies|0,32,32768,32736,222,0,0,148,1052,423.8063063063063,0,0,2453
+mmlu.prompt2_mqa_single_correct_sk:sociology|0,32,32768,32736,197,0,0,154,424,238.248730964467,0,0,996
+mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0,32,32768,32736,99,0,0,147,373,217.3939393939394,0,0,858
+mmlu.prompt2_mqa_single_correct_sk:virology|0,32,32768,32736,164,0,0,146,657,203.10365853658536,0,0,1342
+mmlu.prompt2_mqa_single_correct_sk:world_religions|0,32,32768,32736,171,0,0,139,239,162.2046783625731,0,0,569
+mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0,32,32768,32736,100,0,0,149,261,205.72,0,0,612
+mmlu.prompt3_mqa_single_correct_sk:anatomy|0,32,32768,32736,134,0,0,148,379,215.02238805970148,0,0,894
+mmlu.prompt3_mqa_single_correct_sk:astronomy|0,32,32768,32736,152,0,0,158,475,244.75,0,0,1085
+mmlu.prompt3_mqa_single_correct_sk:business_ethics|0,32,32768,32736,100,0,0,155,414,252.8,0,0,987
+mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0,32,32768,32736,259,0,0,146,372,219.1853281853282,0,0,874
+mmlu.prompt3_mqa_single_correct_sk:college_biology|0,32,32768,32736,142,0,0,166,474,252.14788732394365,0,0,1060
+mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0,32,32768,32736,100,0,0,155,384,229.02,0,0,938
+mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0,32,32768,32736,100,0,0,169,597,302.65,0,0,1419
+mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0,32,32768,32736,99,0,0,158,351,229.63636363636363,0,0,796
+mmlu.prompt3_mqa_single_correct_sk:college_medicine|0,32,32768,32736,172,0,0,161,2212,308.7325581395349,0,0,5026
+mmlu.prompt3_mqa_single_correct_sk:college_physics|0,32,32768,32736,102,0,0,155,359,241.8921568627451,0,0,847
+mmlu.prompt3_mqa_single_correct_sk:computer_security|0,32,32768,32736,100,0,0,156,481,229.18,0,0,1183
+mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0,32,32768,32736,235,0,0,150,275,190.20425531914893,0,0,655
+mmlu.prompt3_mqa_single_correct_sk:econometrics|0,32,32768,32736,112,0,0,159,479,284.25892857142856,0,0,1064
+mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0,32,32768,32736,145,0,0,143,265,190.91034482758621,0,0,633
+mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0,32,32768,32736,373,0,0,147,388,207.76139410187668,0,0,913
+mmlu.prompt3_mqa_single_correct_sk:formal_logic|0,32,32768,32736,115,0,0,163,567,290.5304347826087,0,0,1322
+mmlu.prompt3_mqa_single_correct_sk:global_facts|0,32,32768,32736,100,0,0,161,371,202.62,0,0,846
+mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0,32,32768,32736,282,0,0,160,464,259.54609929078015,0,0,1064
+mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0,32,32768,32736,198,0,0,148,479,242.45454545454547,0,0,1096
+mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0,32,32768,32736,99,0,0,152,755,306.77777777777777,0,0,1773
+mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0,32,32768,32736,135,0,0,350,1234,740.1703703703704,0,0,2781
+mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0,32,32768,32736,198,0,0,153,353,202.76262626262627,0,0,803
+mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0,32,32768,32736,193,0,0,183,465,254.49740932642487,0,0,1082
+mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0,32,32768,32736,387,0,0,164,436,243.3798449612403,0,0,970
+mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0,32,32768,32736,266,0,0,151,425,214.24436090225564,0,0,934
+mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0,32,32768,32736,217,0,0,166,537,248.58986175115209,0,0,1189
+mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0,32,32768,32736,150,0,0,174,570,279.43333333333334,0,0,1328
+mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0,32,32768,32736,536,0,0,154,665,233.3320895522388,0,0,1574
+mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0,32,32768,32736,214,0,0,182,773,338.33644859813086,0,0,1755
+mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0,32,32768,32736,57,0,0,365,1070,580.6842105263158,0,0,2506
+mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0,32,32768,32736,180,0,0,256,1729,724.6333333333333,0,0,3988
+mmlu.prompt3_mqa_single_correct_sk:human_aging|0,32,32768,32736,219,0,0,155,336,201.5068493150685,0,0,811
+mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0,32,32768,32736,109,0,0,154,329,206.24770642201835,0,0,841
+mmlu.prompt3_mqa_single_correct_sk:international_law|0,32,32768,32736,118,0,0,166,449,297.9491525423729,0,0,1005
+mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0,32,32768,32736,107,0,0,153,476,239.4018691588785,0,0,1102
+mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0,32,32768,32736,158,0,0,158,407,242.30379746835442,0,0,965
+mmlu.prompt3_mqa_single_correct_sk:machine_learning|0,32,32768,32736,112,0,0,165,430,250.64285714285714,0,0,895
+mmlu.prompt3_mqa_single_correct_sk:management|0,32,32768,32736,100,0,0,149,280,192.01,0,0,714
+mmlu.prompt3_mqa_single_correct_sk:marketing|0,32,32768,32736,234,0,0,165,356,225.77777777777777,0,0,839
+mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0,32,32768,32736,100,0,0,156,299,202.78,0,0,714
+mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0,32,32768,32736,779,0,0,142,560,191.65340179717586,0,0,1334
+mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0,32,32768,32736,326,0,0,159,406,244.94171779141104,0,0,978
+mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0,32,32768,32736,749,0,0,256,377,305.8558077436582,0,0,876
+mmlu.prompt3_mqa_single_correct_sk:nutrition|0,32,32768,32736,303,0,0,146,471,245.0990099009901,0,0,1109
+mmlu.prompt3_mqa_single_correct_sk:philosophy|0,32,32768,32736,308,0,0,156,543,221.2922077922078,0,0,1292
+mmlu.prompt3_mqa_single_correct_sk:prehistory|0,32,32768,32736,319,0,0,145,490,231.16614420062695,0,0,1104
+mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0,32,32768,32736,276,0,0,179,577,310.0217391304348,0,0,1179
+mmlu.prompt3_mqa_single_correct_sk:professional_law|0,32,32768,32736,1171,0,0,163,1655,588.5713065755764,0,0,3889
+mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0,32,32768,32736,264,0,0,256,850,451.2537878787879,0,0,1932
+mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0,32,32768,32736,596,0,0,156,590,262.18624161073825,0,0,1435
+mmlu.prompt3_mqa_single_correct_sk:public_relations|0,32,32768,32736,109,0,0,147,508,219.1559633027523,0,0,1193
+mmlu.prompt3_mqa_single_correct_sk:security_studies|0,32,32768,32736,222,0,0,159,1063,434.8063063063063,0,0,2481
+mmlu.prompt3_mqa_single_correct_sk:sociology|0,32,32768,32736,197,0,0,165,435,249.248730964467,0,0,1024
+mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0,32,32768,32736,99,0,0,158,384,228.3939393939394,0,0,886
+mmlu.prompt3_mqa_single_correct_sk:virology|0,32,32768,32736,164,0,0,157,668,214.10365853658536,0,0,1370
+mmlu.prompt3_mqa_single_correct_sk:world_religions|0,32,32768,32736,171,0,0,150,250,173.2046783625731,0,0,597
+movie_review.prompt1_sentiment_binary_sk|0,32,32768,32736,2008,0,0,91,1799,249.22211155378486,0,0,4162
+movie_review.prompt2_sentiment_binary_sk|0,32,32768,32736,2008,0,0,102,1810,260.2221115537849,0,0,4184
+movie_review.prompt3_sentiment_binary_sk|0,32,32768,32736,2008,0,0,70,1778,228.22211155378486,0,0,4118
+multi_wiki_qa.prompt1_qa_sk|0,126,32768,32642,4997,0,0,512,32642,1964.61997198319,23,0,121852
+multi_wiki_qa.prompt2_qa_sk|0,126,32768,32642,4997,0,0,517,32642,1969.596958174905,23,0,121855
+multi_wiki_qa.prompt3_qa_sk|0,126,32768,32642,4997,0,0,483,32642,1935.7534520712427,23,0,121783
+senti_sk.prompt1_sentiment_3_class_sk|0,32,32768,32736,4677,0,0,94,551,123.0096215522771,0,0,1271
+senti_sk.prompt1_sentiment_binary_sk|0,32,32768,32736,3728,0,0,88,545,117.15665236051503,0,0,1258
+senti_sk.prompt2_sentiment_3_class_sk|0,32,32768,32736,4677,0,0,109,566,138.0096215522771,0,0,1304
+senti_sk.prompt2_sentiment_binary_sk|0,32,32768,32736,3728,0,0,99,556,128.156652360515,0,0,1280
+senti_sk.prompt3_sentiment_3_class_sk|0,32,32768,32736,4677,0,0,72,529,101.0096215522771,0,0,1224
+senti_sk.prompt3_sentiment_binary_sk|0,32,32768,32736,3728,0,0,67,524,96.15665236051503,0,0,1214
+skj1.prompt1_skj1_sk|0,32,32768,32736,100,0,0,175,317,215.89,0,0,721
+skj1.prompt2_skj1_sk|0,32,32768,32736,100,0,0,162,304,202.89,0,0,691
+skj1.prompt3_skj1_sk|0,32,32768,32736,100,0,0,163,305,203.89,0,0,696
+sklep.prompt1_hate_speech_sk:hate-speech|0,32,32768,32736,1319,0,0,94,728,129.9871114480667,0,0,1831
+sklep.prompt1_nli_3_classes_sk:nli|0,32,32768,32736,5004,0,0,136,292,192.7312150279776,0,0,698
+sklep.prompt1_nli_binary_sk:rte|0,32,32768,32736,1660,0,0,143,622,270.4018072289157,0,0,1500
+sklep.prompt1_qa_sk:question-answering|0,126,32768,32642,9583,0,0,330,1724,505.43765000521756,0,0,3895
+sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0,32,32768,32736,1042,0,0,88,557,126.23800383877159,0,0,1280
+sklep.prompt2_hate_speech_sk:hate-speech|0,32,32768,32736,1319,0,0,78,712,113.98711144806671,0,0,1804
+sklep.prompt2_nli_3_classes_sk:nli|0,32,32768,32736,5004,0,0,186,342,242.7312150279776,0,0,802
+sklep.prompt2_nli_binary_sk:rte|0,32,32768,32736,1660,0,0,117,596,244.40180722891566,0,0,1438
+sklep.prompt2_qa_sk:question-answering|0,126,32768,32642,9583,0,0,335,1729,510.43765000521756,0,0,3898
+sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0,32,32768,32736,1042,0,0,99,568,137.2380038387716,0,0,1302
+sklep.prompt3_hate_speech_sk:hate-speech|0,32,32768,32736,1319,0,0,102,736,137.9871114480667,0,0,1861
+sklep.prompt3_nli_3_classes_sk:nli|0,32,32768,32736,5004,0,0,130,286,186.7312150279776,0,0,673
+sklep.prompt3_nli_binary_sk:rte|0,32,32768,32736,1660,0,0,168,647,295.4018072289157,0,0,1546
+sklep.prompt3_qa_sk:question-answering|0,126,32768,32642,9583,0,0,302,1696,477.43765000521756,0,0,3826
+sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0,32,32768,32736,1042,0,0,67,536,105.23800383877159,0,0,1236
+slovakcopa.prompt1_copa_sk|0,32,32768,32736,500,0,0,152,207,172.342,0,0,479
+slovakcopa.prompt2_copa_sk|0,32,32768,32736,500,0,0,157,212,177.344,0,0,494
+slovakcopa.prompt3_copa_sk|0,32,32768,32736,500,0,0,152,208,172.65,0,0,503
+toxic_sk.prompt1_toxicity_sk|0,32,32768,32736,881,0,0,80,360,102.18501702610669,0,0,981
+toxic_sk.prompt2_toxicity_sk|0,32,32768,32736,881,0,0,72,352,94.18501702610669,0,0,965
+toxic_sk.prompt3_toxicity_sk|0,32,32768,32736,881,0,0,105,385,127.18501702610669,0,0,1034
+truthfulqa.prompt1_mqa_single_correct_sk:mc1|0,32,32768,32736,778,0,0,145,544,264.5141388174807,0,0,1275
+truthfulqa.prompt2_mqa_single_correct_sk:mc1|0,32,32768,32736,778,0,0,129,528,248.51413881748073,0,0,1233
+truthfulqa.prompt3_mqa_single_correct_sk:mc1|0,32,32768,32736,778,0,0,140,539,259.5141388174807,0,0,1261
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_summary.json b/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_summary.json
new file mode 100644
index 0000000..abb8934
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/input_length_diagnostics_summary.json
@@ -0,0 +1,14 @@
+{
+ "task_count": 222,
+ "evaluated_instances": 159426,
+ "max_recorded_input_tokens": 32642,
+ "arrays_at_input_budget": 69,
+ "arrays_over_input_budget": 0,
+ "log_messages_matching_truncat_case_insensitive": [
+ {
+ "log_lf_line_number": 16033,
+ "message": "Splits: 50%|█████ | 1/2 [1:14:51<1:14:51, 4491.94s/it]context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens."
+ }
+ ],
+ "limitation": "Input strings and token arrays are recorded artifacts, not a separately tokenized original source prompt. Their lengths cannot recover how much text was discarded or establish the number of truly truncated inputs."
+}
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/log_truncation_evidence.json b/aya_300k_lr_experiments/benchmarks/skbench/results/log_truncation_evidence.json
new file mode 100644
index 0000000..ed8e228
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/log_truncation_evidence.json
@@ -0,0 +1,7 @@
+[
+ {
+ "line_number": 16033,
+ "message": "context_size + max_new_tokens=53860 which is greater than self.max_length=32768. Truncating context to 32642 tokens.",
+ "source": "logs/sk_zero_available_without_nli_lr_0p00010_20261022_185255.log"
+ }
+]
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv b/aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv
new file mode 100644
index 0000000..b9fa624
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/per_task_metrics.csv
@@ -0,0 +1,223 @@
+task,family,task_group_without_prompt,prompt,num_fewshots,sample_count,primary_metric,primary_value,generation_size,evaluation_splits,hf_repo,hf_subset,hf_revision,em,em_conservative,em_conservative_norm,em_conservative_norm_stderr,em_conservative_stderr,em_norm,em_norm_relaxed,em_norm_relaxed_stderr,em_norm_stderr,em_relaxed,em_relaxed_stderr,em_stderr,exact_match_slk_prefix,exact_match_slk_prefix_stderr,f1_slk,f1_slk_stderr
+arc.prompt1_mqa_single_correct_sk|0,arc,arc.mqa_single_correct_sk,1,0,1169,em_relaxed,0.1710863986313088,32,test,jon-tow/okapi_arc_challenge,sk,56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e,0.013686911890504704,0.1710863986313088,,,0.011018968811092407,,,,,0.1710863986313088,0.011018968811092407,0.0033996852610928244,,,,
+arc.prompt2_mqa_single_correct_sk|0,arc,arc.mqa_single_correct_sk,2,0,1169,em_relaxed,0.22412318220701455,32,test,jon-tow/okapi_arc_challenge,sk,56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e,0.03507271171941831,0.223267750213858,,,0.012185048029719049,,,,,0.22412318220701455,0.012201644195165715,0.005382828530594407,,,,
+arc.prompt3_mqa_single_correct_sk|0,arc,arc.mqa_single_correct_sk,3,0,1169,em_relaxed,0.1770744225834046,32,test,jon-tow/okapi_arc_challenge,sk,56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e,0.022241231822070145,0.17621899059024806,,,0.011148355810710043,,,,,0.1770744225834046,0.011169578281264629,0.004314931013145922,,,,
+belebele.prompt1_mqa_single_correct_sk|0,belebele,belebele.mqa_single_correct_sk,1,0,900,em_relaxed,0.21555555555555556,32,test,facebook/belebele,slk_Latn,7899cdfa4e1e0d733fd77c848e2c273cb1d32be2,0.02,0.21555555555555556,,,0.01371452783236968,,,,,0.21555555555555556,0.01371452783236968,0.004669261421755523,,,,
+belebele.prompt2_mqa_single_correct_sk|0,belebele,belebele.mqa_single_correct_sk,2,0,900,em_relaxed,0.2111111111111111,32,test,facebook/belebele,slk_Latn,7899cdfa4e1e0d733fd77c848e2c273cb1d32be2,0.016666666666666666,0.2111111111111111,,,0.013610798969328482,,,,,0.2111111111111111,0.013610798969328482,0.004269675894690937,,,,
+belebele.prompt3_mqa_single_correct_sk|0,belebele,belebele.mqa_single_correct_sk,3,0,900,em_relaxed,0.21333333333333335,32,test,facebook/belebele,slk_Latn,7899cdfa4e1e0d733fd77c848e2c273cb1d32be2,0.03,0.2111111111111111,,,0.013610798969328486,,,,,0.21333333333333335,0.013662962863011163,0.005689402360648914,,,,
+financial_exams.prompt1_completion_sk|0,financial_exams,financial_exams.completion_sk,1,0,1334,em_relaxed,0.20089955022488756,32,test,TUKE-KEMT/slovak-financial-exam,default,7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c,0.009745127436281859,0.20089955022488756,,,0.010974256206370726,,,,,0.20089955022488756,0.010974256206370726,0.002690618851324012,,,,
+financial_exams.prompt2_completion_sk|0,financial_exams,financial_exams.completion_sk,2,0,1334,em_relaxed,0.2383808095952024,32,test,TUKE-KEMT/slovak-financial-exam,default,7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c,0.07571214392803598,0.20539730134932535,,,0.011065150105060274,,,,,0.2383808095952024,0.011670501255680532,0.007245549384888845,,,,
+financial_exams.prompt3_completion_sk|0,financial_exams,financial_exams.completion_sk,3,0,1334,em_relaxed,0.24887556221889057,32,test,TUKE-KEMT/slovak-financial-exam,default,7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c,0.0037481259370314842,0.24887556221889057,,,0.01184218927916494,,,,,0.24887556221889057,0.01184218927916494,0.0016736960391157028,,,,
+globalmmlu-lite.prompt1_mqa_single_correct_sk|0,globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,1,0,400,em_relaxed,0.1825,32,test,CohereLabs/Global-MMLU-Lite,sk,600af1bff2d447c7fc3aee22f11f607560d5fcb7,0.025,0.1825,,,0.01933698975881026,,,,,0.1825,0.01933698975881026,0.007816023641463065,,,,
+globalmmlu-lite.prompt2_mqa_single_correct_sk|0,globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,2,0,400,em_relaxed,0.21,32,test,CohereLabs/Global-MMLU-Lite,sk,600af1bff2d447c7fc3aee22f11f607560d5fcb7,0.035,0.21,,,0.020390916450326858,,,,,0.21,0.020390916450326858,0.009200495791370004,,,,
+globalmmlu-lite.prompt3_mqa_single_correct_sk|0,globalmmlu-lite,globalmmlu-lite.mqa_single_correct_sk,3,0,400,em_relaxed,0.2025,32,test,CohereLabs/Global-MMLU-Lite,sk,600af1bff2d447c7fc3aee22f11f607560d5fcb7,0.025,0.2025,,,0.02011830610666578,,,,,0.2025,0.02011830610666578,0.007816023641463063,,,,
+mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0,mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,1,0,100,em_relaxed,0.01,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.01,,,0.009999999999999997,,,,,0.01,0.009999999999999997,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:anatomy|0,mmlu,mmlu.mqa_single_correct_sk:anatomy,1,0,134,em_relaxed,0.12686567164179105,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.022388059701492536,0.12686567164179105,,,0.028859365115889928,,,,,0.12686567164179105,0.028859365115889928,0.012828198053329663,,,,
+mmlu.prompt1_mqa_single_correct_sk:astronomy|0,mmlu,mmlu.mqa_single_correct_sk:astronomy,1,0,152,em_relaxed,0.14473684210526316,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03289473684210526,0.13815789473684212,,,0.028081042939576552,,,,,0.14473684210526316,0.028631951845930412,0.014514818445457586,,,,
+mmlu.prompt1_mqa_single_correct_sk:business_ethics|0,mmlu,mmlu.mqa_single_correct_sk:business_ethics,1,0,100,em_relaxed,0.14,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.14,,,0.03487350880197772,,,,,0.14,0.03487350880197772,0.014070529413628957,,,,
+mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0,mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,1,0,259,em_relaxed,0.15444015444015444,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.011583011583011582,0.15444015444015444,,,0.022497923516297567,,,,,0.15444015444015444,0.022497923516297567,0.006661484058899665,,,,
+mmlu.prompt1_mqa_single_correct_sk:college_biology|0,mmlu,mmlu.mqa_single_correct_sk:college_biology,1,0,142,em_relaxed,0.15492957746478872,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.028169014084507043,0.15492957746478872,,,0.030472217725828407,,,,,0.15492957746478872,0.030472217725828407,0.013933866272175799,,,,
+mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:college_chemistry,1,0,100,em_relaxed,0.08,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.08,,,0.027265992434429093,,,,,0.08,0.027265992434429093,0.019694638556693227,,,,
+mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:college_computer_science,1,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.11,,,0.031446603773522035,,,,,0.15,0.035887028128263714,0.009999999999999998,,,,
+mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:college_mathematics,1,0,99,em_relaxed,0.06060606060606061,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.050505050505050504,,,0.02212079324207198,,,,,0.06060606060606061,0.024102876029567216,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:college_medicine|0,mmlu,mmlu.mqa_single_correct_sk:college_medicine,1,0,172,em_relaxed,0.1511627906976744,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.011627906976744186,0.1511627906976744,,,0.027392810919920886,,,,,0.1511627906976744,0.027392810919920886,0.008198095184581854,,,,
+mmlu.prompt1_mqa_single_correct_sk:college_physics|0,mmlu,mmlu.mqa_single_correct_sk:college_physics,1,0,102,em_relaxed,0.0784313725490196,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.00980392156862745,0.0784313725490196,,,0.026751495257385273,,,,,0.0784313725490196,0.026751495257385273,0.009803921568627454,,,,
+mmlu.prompt1_mqa_single_correct_sk:computer_security|0,mmlu,mmlu.mqa_single_correct_sk:computer_security,1,0,100,em_relaxed,0.08,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.08,,,0.027265992434429082,,,,,0.08,0.027265992434429082,0.014070529413628966,,,,
+mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0,mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,1,0,235,em_relaxed,0.09361702127659574,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03829787234042553,0.09361702127659574,,,0.01904256081095342,,,,,0.09361702127659574,0.01904256081095342,0.012545838230269625,,,,
+mmlu.prompt1_mqa_single_correct_sk:econometrics|0,mmlu,mmlu.mqa_single_correct_sk:econometrics,1,0,112,em_relaxed,0.125,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.026785714285714284,0.125,,,0.03139045014587016,,,,,0.125,0.03139045014587016,0.015324784082343043,,,,
+mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0,mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,1,0,145,em_relaxed,0.0896551724137931,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.041379310344827586,0.0896551724137931,,,0.023807258824376967,,,,,0.0896551724137931,0.023807258824376967,0.016597159859992713,,,,
+mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,1,0,373,em_relaxed,0.06166219839142091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005361930294906166,0.058981233243967826,,,0.012214750400069162,,,,,0.06166219839142091,0.012471469772330128,0.003786357797685454,,,,
+mmlu.prompt1_mqa_single_correct_sk:formal_logic|0,mmlu,mmlu.mqa_single_correct_sk:formal_logic,1,0,115,em_relaxed,0.12173913043478261,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0782608695652174,,,0.025154976812194616,,,,,0.12173913043478261,0.030624883036764184,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:global_facts|0,mmlu,mmlu.mqa_single_correct_sk:global_facts,1,0,100,em_relaxed,0.11,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06,0.11,,,0.031446603773522035,,,,,0.11,0.031446603773522035,0.02386832565759416,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_biology,1,0,282,em_relaxed,0.18439716312056736,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02127659574468085,0.1773049645390071,,,0.022783829728850895,,,,,0.18439716312056736,0.023134670670309505,0.00860850820920177,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,1,0,198,em_relaxed,0.08585858585858586,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.010101010101010102,0.08080808080808081,,,0.019417681889724533,,,,,0.08585858585858586,0.019960225563172885,0.007124341521250813,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,1,0,99,em_relaxed,0.10101010101010101,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.10101010101010101,,,0.030440148787487714,,,,,0.10101010101010101,0.030440148787487714,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,1,0,135,em_relaxed,0.24444444444444444,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.014814814814814815,0.24444444444444444,,,0.03712537833614866,,,,,0.24444444444444444,0.03712537833614866,0.010436494549594362,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0,mmlu,mmlu.mqa_single_correct_sk:high_school_geography,1,0,198,em_relaxed,0.15656565656565657,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.15656565656565657,,,0.025890520358141454,,,,,0.15656565656565657,0.025890520358141454,0.012213156893572809,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,1,0,193,em_relaxed,0.30569948186528495,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.025906735751295335,0.30569948186528495,,,0.03324837939758159,,,,,0.30569948186528495,0.03324837939758159,0.011464523356953171,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,1,0,387,em_relaxed,0.12919896640826872,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.012919896640826873,0.12661498708010335,,,0.016925900082204414,,,,,0.12919896640826872,0.017072429785340966,0.005747937884708704,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,1,0,266,em_relaxed,0.03759398496240601,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.03759398496240601,,,0.011684641953170118,,,,,0.03759398496240601,0.011684641953170118,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,1,0,217,em_relaxed,0.14285714285714285,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.004608294930875576,0.14285714285714285,,,0.0238095238095238,,,,,0.14285714285714285,0.0238095238095238,0.004608294930875571,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_physics,1,0,150,em_relaxed,0.13333333333333333,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02666666666666667,0.13333333333333333,,,0.027848530325897826,,,,,0.13333333333333333,0.027848530325897826,0.013198422611557032,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,1,0,536,em_relaxed,0.16791044776119404,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.016791044776119403,0.16791044776119404,,,0.016160210122502162,,,,,0.16791044776119404,0.016160210122502162,0.005555010467354857,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,1,0,214,em_relaxed,0.1822429906542056,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.17289719626168223,,,0.025910984835548666,,,,,0.1822429906542056,0.026451344562470138,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,1,0,57,em_relaxed,0.19298245614035087,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.19298245614035087,,,0.05273590328947574,,,,,0.19298245614035087,0.05273590328947574,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,1,0,180,em_relaxed,0.2777777777777778,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005555555555555556,0.2777777777777778,,,0.033477857599386325,,,,,0.2777777777777778,0.033477857599386325,0.005555555555555546,,,,
+mmlu.prompt1_mqa_single_correct_sk:human_aging|0,mmlu,mmlu.mqa_single_correct_sk:human_aging,1,0,219,em_relaxed,0.1187214611872146,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0365296803652968,0.1187214611872146,,,0.021907507389103187,,,,,0.1187214611872146,0.021907507389103187,0.012706146532849703,,,,
+mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0,mmlu,mmlu.mqa_single_correct_sk:human_sexuality,1,0,109,em_relaxed,0.13761467889908258,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.045871559633027525,0.13761467889908258,,,0.03314905716923366,,,,,0.13761467889908258,0.03314905716923366,0.020130904850286974,,,,
+mmlu.prompt1_mqa_single_correct_sk:international_law|0,mmlu,mmlu.mqa_single_correct_sk:international_law,1,0,118,em_relaxed,0.13559322033898305,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03389830508474576,0.13559322033898305,,,0.03165083260501198,,,,,0.13559322033898305,0.03165083260501198,0.016730444637044904,,,,
+mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0,mmlu,mmlu.mqa_single_correct_sk:jurisprudence,1,0,107,em_relaxed,0.14018691588785046,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.056074766355140186,0.14018691588785046,,,0.03372118539369118,,,,,0.14018691588785046,0.03372118539369118,0.02234599008672909,,,,
+mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0,mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,1,0,158,em_relaxed,0.13924050632911392,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02531645569620253,0.13924050632911392,,,0.027629554439252756,,,,,0.13924050632911392,0.027629554439252756,0.012536706047327604,,,,
+mmlu.prompt1_mqa_single_correct_sk:machine_learning|0,mmlu,mmlu.mqa_single_correct_sk:machine_learning,1,0,112,em_relaxed,0.11607142857142858,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.008928571428571428,0.10714285714285714,,,0.02935696201881377,,,,,0.11607142857142858,0.03040253249894973,0.008928571428571423,,,,
+mmlu.prompt1_mqa_single_correct_sk:management|0,mmlu,mmlu.mqa_single_correct_sk:management,1,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.15,,,0.0358870281282637,,,,,0.15,0.0358870281282637,0.01407052941362896,,,,
+mmlu.prompt1_mqa_single_correct_sk:marketing|0,mmlu,mmlu.mqa_single_correct_sk:marketing,1,0,234,em_relaxed,0.15384615384615385,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03418803418803419,0.15384615384615385,,,0.023636873317489277,,,,,0.15384615384615385,0.023636873317489277,0.011904341997629792,,,,
+mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0,mmlu,mmlu.mqa_single_correct_sk:medical_genetics,1,0,100,em_relaxed,0.18,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.18,,,0.038612291966536955,,,,,0.18,0.038612291966536955,0.019694638556693223,,,,
+mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0,mmlu,mmlu.mqa_single_correct_sk:miscellaneous,1,0,779,em_relaxed,0.14505776636713735,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03209242618741977,0.14505776636713735,,,0.01262551435912769,,,,,0.14505776636713735,0.01262551435912769,0.006318709962464618,,,,
+mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0,mmlu,mmlu.mqa_single_correct_sk:moral_disputes,1,0,326,em_relaxed,0.20245398773006135,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06134969325153374,0.20245398773006135,,,0.022289450459613447,,,,,0.20245398773006135,0.022289450459613447,0.01331117614747688,,,,
+mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0,mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,1,0,749,em_relaxed,0.006675567423230975,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0053404539385847796,,,0.0026648668562206292,,,,,0.006675567423230975,0.0029774114441068625,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:nutrition|0,mmlu,mmlu.mqa_single_correct_sk:nutrition,1,0,303,em_relaxed,0.1848184818481848,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.036303630363036306,0.1848184818481848,,,0.022335536574954167,,,,,0.1848184818481848,0.022335536574954167,0.010763206393659719,,,,
+mmlu.prompt1_mqa_single_correct_sk:philosophy|0,mmlu,mmlu.mqa_single_correct_sk:philosophy,1,0,308,em_relaxed,0.21103896103896103,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06818181818181818,0.21103896103896103,,,0.023288413887090093,,,,,0.21103896103896103,0.023288413887090093,0.014385689805382164,,,,
+mmlu.prompt1_mqa_single_correct_sk:prehistory|0,mmlu,mmlu.mqa_single_correct_sk:prehistory,1,0,319,em_relaxed,0.15987460815047022,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04075235109717868,0.15987460815047022,,,0.02055172319966096,,,,,0.15987460815047022,0.02055172319966096,0.011087359931149773,,,,
+mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0,mmlu,mmlu.mqa_single_correct_sk:professional_accounting,1,0,276,em_relaxed,0.15217391304347827,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0036231884057971015,0.14130434782608695,,,0.02100540442245809,,,,,0.15217391304347827,0.02165993479655374,0.0036231884057970937,,,,
+mmlu.prompt1_mqa_single_correct_sk:professional_law|0,mmlu,mmlu.mqa_single_correct_sk:professional_law,1,0,1171,em_relaxed,0.12724167378309137,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0017079419299743809,0.12724167378309137,,,0.009742462799429212,,,,,0.12724167378309137,0.009742462799429212,0.0012071811002646103,,,,
+mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0,mmlu,mmlu.mqa_single_correct_sk:professional_medicine,1,0,264,em_relaxed,0.13257575757575757,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.13257575757575757,,,0.020910766968363862,,,,,0.13257575757575757,0.020910766968363862,0.0,,,,
+mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0,mmlu,mmlu.mqa_single_correct_sk:professional_psychology,1,0,596,em_relaxed,0.20134228187919462,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03355704697986577,0.20134228187919462,,,0.016439528475054862,,,,,0.20134228187919462,0.016439528475054862,0.0073828066874755535,,,,
+mmlu.prompt1_mqa_single_correct_sk:public_relations|0,mmlu,mmlu.mqa_single_correct_sk:public_relations,1,0,109,em_relaxed,0.1743119266055046,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.027522935779816515,0.1743119266055046,,,0.03650561586028542,,,,,0.1743119266055046,0.03650561586028542,0.015742553732396008,,,,
+mmlu.prompt1_mqa_single_correct_sk:security_studies|0,mmlu,mmlu.mqa_single_correct_sk:security_studies,1,0,222,em_relaxed,0.1981981981981982,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04504504504504504,0.1981981981981982,,,0.026815582053277207,,,,,0.1981981981981982,0.026815582053277207,0.01395143304433832,,,,
+mmlu.prompt1_mqa_single_correct_sk:sociology|0,mmlu,mmlu.mqa_single_correct_sk:sociology,1,0,197,em_relaxed,0.3096446700507614,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.050761421319796954,0.3096446700507614,,,0.03302479343505637,,,,,0.3096446700507614,0.03302479343505637,0.015679295366526198,,,,
+mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0,mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,1,0,99,em_relaxed,0.23232323232323232,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.2222222222222222,,,0.041996052556580815,,,,,0.23232323232323232,0.04266016017054687,0.01731601731601731,,,,
+mmlu.prompt1_mqa_single_correct_sk:virology|0,mmlu,mmlu.mqa_single_correct_sk:virology,1,0,164,em_relaxed,0.12195121951219512,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03048780487804878,0.12195121951219512,,,0.025630589418441155,,,,,0.12195121951219512,0.025630589418441155,0.013466226472535378,,,,
+mmlu.prompt1_mqa_single_correct_sk:world_religions|0,mmlu,mmlu.mqa_single_correct_sk:world_religions,1,0,171,em_relaxed,0.05263157894736842,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.023391812865497075,0.05263157894736842,,,0.017126088775157082,,,,,0.05263157894736842,0.017126088775157082,0.01159224790573493,,,,
+mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0,mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,2,0,100,em_relaxed,0.05,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.04,,,0.019694638556693237,,,,,0.05,0.021904291355759026,0.009999999999999997,,,,
+mmlu.prompt2_mqa_single_correct_sk:anatomy|0,mmlu,mmlu.mqa_single_correct_sk:anatomy,2,0,134,em_relaxed,0.1791044776119403,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05970149253731343,0.1791044776119403,,,0.03324844546213894,,,,,0.1791044776119403,0.03324844546213894,0.020544692585423446,,,,
+mmlu.prompt2_mqa_single_correct_sk:astronomy|0,mmlu,mmlu.mqa_single_correct_sk:astronomy,2,0,152,em_relaxed,0.17763157894736842,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05263157894736842,0.17105263157894737,,,0.03064360707167709,,,,,0.17763157894736842,0.031103182383123398,0.01817164224152954,,,,
+mmlu.prompt2_mqa_single_correct_sk:business_ethics|0,mmlu,mmlu.mqa_single_correct_sk:business_ethics,2,0,100,em_relaxed,0.22,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03,0.21,,,0.04093601807403326,,,,,0.22,0.0416333199893227,0.017144660799776522,,,,
+mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0,mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,2,0,259,em_relaxed,0.24324324324324326,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04633204633204633,0.23938223938223938,,,0.026565582359277095,,,,,0.24324324324324326,0.026710910573068247,0.013086679986931563,,,,
+mmlu.prompt2_mqa_single_correct_sk:college_biology|0,mmlu,mmlu.mqa_single_correct_sk:college_biology,2,0,142,em_relaxed,0.19014084507042253,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04225352112676056,0.19014084507042253,,,0.03304706552420638,,,,,0.19014084507042253,0.03304706552420638,0.01694131738100396,,,,
+mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:college_chemistry,2,0,100,em_relaxed,0.14,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.09,0.14,,,0.03487350880197772,,,,,0.14,0.03487350880197772,0.028762349126466143,,,,
+mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:college_computer_science,2,0,100,em_relaxed,0.17,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.12,,,0.032659863237109066,,,,,0.17,0.03775251680686371,0.009999999999999995,,,,
+mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:college_mathematics,2,0,99,em_relaxed,0.04040404040404041,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.030303030303030304,,,0.017316017316017295,,,,,0.04040404040404041,0.019890402239668436,0.0,,,,
+mmlu.prompt2_mqa_single_correct_sk:college_medicine|0,mmlu,mmlu.mqa_single_correct_sk:college_medicine,2,0,172,em_relaxed,0.20930232558139536,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06395348837209303,0.20930232558139536,,,0.031109583909764628,,,,,0.20930232558139536,0.031109583909764628,0.018710387348210598,,,,
+mmlu.prompt2_mqa_single_correct_sk:college_physics|0,mmlu,mmlu.mqa_single_correct_sk:college_physics,2,0,102,em_relaxed,0.12745098039215685,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0392156862745098,0.12745098039215685,,,0.03318224921942078,,,,,0.12745098039215685,0.03318224921942078,0.01931444240436096,,,,
+mmlu.prompt2_mqa_single_correct_sk:computer_security|0,mmlu,mmlu.mqa_single_correct_sk:computer_security,2,0,100,em_relaxed,0.11,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.11,,,0.03144660377352203,,,,,0.11,0.03144660377352203,0.014070529413628963,,,,
+mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0,mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,2,0,235,em_relaxed,0.14042553191489363,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0851063829787234,0.14042553191489363,,,0.022712077616627854,,,,,0.14042553191489363,0.022712077616627854,0.018241411344572286,,,,
+mmlu.prompt2_mqa_single_correct_sk:econometrics|0,mmlu,mmlu.mqa_single_correct_sk:econometrics,2,0,112,em_relaxed,0.11607142857142858,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.017857142857142856,0.11607142857142858,,,0.03040253249894973,,,,,0.11607142857142858,0.03040253249894973,0.012569900164490047,,,,
+mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0,mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,2,0,145,em_relaxed,0.1310344827586207,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06896551724137931,0.1310344827586207,,,0.028119838672931184,,,,,0.1310344827586207,0.028119838672931184,0.021116290886061897,,,,
+mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,2,0,373,em_relaxed,0.10187667560321716,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0160857908847185,0.09651474530831099,,,0.015310377502350168,,,,,0.10187667560321716,0.015683171805464465,0.006522714286001283,,,,
+mmlu.prompt2_mqa_single_correct_sk:formal_logic|0,mmlu,mmlu.mqa_single_correct_sk:formal_logic,2,0,115,em_relaxed,0.13043478260869565,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.008695652173913044,0.08695652173913043,,,0.026390303420299282,,,,,0.13043478260869565,0.03154244565695702,0.008695652173913031,,,,
+mmlu.prompt2_mqa_single_correct_sk:global_facts|0,mmlu,mmlu.mqa_single_correct_sk:global_facts,2,0,100,em_relaxed,0.18,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.13,0.18,,,0.03861229196653694,,,,,0.18,0.03861229196653694,0.03379976689896309,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_biology,2,0,282,em_relaxed,0.24822695035460993,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04964539007092199,0.24113475177304963,,,0.02551873104953778,,,,,0.24822695035460993,0.025770015644290406,0.012957736571882147,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,2,0,198,em_relaxed,0.11616161616161616,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.020202020202020204,0.1111111111111111,,,0.022390787638216763,,,,,0.11616161616161616,0.022828881775249377,0.01002380379631872,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,2,0,99,em_relaxed,0.12121212121212122,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.12121212121212122,,,0.03296871474399962,,,,,0.12121212121212122,0.03296871474399962,0.01731601731601732,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,2,0,135,em_relaxed,0.23703703703703705,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.022222222222222223,0.23703703703703705,,,0.03673731683969506,,,,,0.23703703703703705,0.03673731683969506,0.012733899715059683,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0,mmlu,mmlu.mqa_single_correct_sk:high_school_geography,2,0,198,em_relaxed,0.22727272727272727,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0707070707070707,0.22727272727272727,,,0.02985751567338641,,,,,0.22727272727272727,0.02985751567338641,0.01826310542019951,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,2,0,193,em_relaxed,0.32124352331606215,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.046632124352331605,0.32124352331606215,,,0.033699508685490674,,,,,0.32124352331606215,0.033699508685490674,0.015216761819262577,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,2,0,387,em_relaxed,0.1731266149870801,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.031007751937984496,0.17054263565891473,,,0.019143434938568855,,,,,0.1731266149870801,0.01925784882518206,0.00882270258926331,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,2,0,266,em_relaxed,0.06766917293233082,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.011278195488721804,0.06766917293233082,,,0.015429701322264242,,,,,0.06766917293233082,0.015429701322264242,0.0064868510822409155,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,2,0,217,em_relaxed,0.17511520737327188,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02304147465437788,0.17511520737327188,,,0.025860189581375218,,,,,0.17511520737327188,0.025860189581375218,0.010208603193741356,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_physics,2,0,150,em_relaxed,0.14666666666666667,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.14666666666666667,,,0.02898223889753507,,,,,0.14666666666666667,0.02898223889753507,0.016053601490175616,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,2,0,536,em_relaxed,0.22388059701492538,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04477611940298507,0.22388059701492538,,,0.01802169384796668,,,,,0.22388059701492538,0.01802169384796668,0.008941264053122127,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,2,0,214,em_relaxed,0.18691588785046728,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.004672897196261682,0.17757009345794392,,,0.02618451638732211,,,,,0.18691588785046728,0.026711670245826564,0.004672897196261678,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,2,0,57,em_relaxed,0.21052631578947367,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.21052631578947367,,,0.05447885994808057,,,,,0.21052631578947367,0.05447885994808057,0.0,,,,
+mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,2,0,180,em_relaxed,0.2777777777777778,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005555555555555556,0.2777777777777778,,,0.03347785759938633,,,,,0.2777777777777778,0.03347785759938633,0.005555555555555554,,,,
+mmlu.prompt2_mqa_single_correct_sk:human_aging|0,mmlu,mmlu.mqa_single_correct_sk:human_aging,2,0,219,em_relaxed,0.1735159817351598,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0867579908675799,0.1735159817351598,,,0.02564830822412722,,,,,0.1735159817351598,0.02564830822412722,0.01906423736413682,,,,
+mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0,mmlu,mmlu.mqa_single_correct_sk:human_sexuality,2,0,109,em_relaxed,0.21100917431192662,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05504587155963303,0.21100917431192662,,,0.039262212380368526,,,,,0.21100917431192662,0.039262212380368526,0.0219460246091855,,,,
+mmlu.prompt2_mqa_single_correct_sk:international_law|0,mmlu,mmlu.mqa_single_correct_sk:international_law,2,0,118,em_relaxed,0.16101694915254236,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05084745762711865,0.16101694915254236,,,0.03397969368949715,,,,,0.16101694915254236,0.03397969368949715,0.020309989475094187,,,,
+mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0,mmlu,mmlu.mqa_single_correct_sk:jurisprudence,2,0,107,em_relaxed,0.19626168224299065,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.056074766355140186,0.19626168224299065,,,0.038576441428227824,,,,,0.19626168224299065,0.038576441428227824,0.02234599008672909,,,,
+mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0,mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,2,0,158,em_relaxed,0.25316455696202533,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05063291139240506,0.25316455696202533,,,0.0347027655481443,,,,,0.25316455696202533,0.0347027655481443,0.017497810541168043,,,,
+mmlu.prompt2_mqa_single_correct_sk:machine_learning|0,mmlu,mmlu.mqa_single_correct_sk:machine_learning,2,0,112,em_relaxed,0.14285714285714285,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.026785714285714284,0.125,,,0.03139045014587016,,,,,0.14285714285714285,0.033213611069662675,0.015324784082343059,,,,
+mmlu.prompt2_mqa_single_correct_sk:management|0,mmlu,mmlu.mqa_single_correct_sk:management,2,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.15,,,0.035887028128263686,,,,,0.15,0.035887028128263686,0.019694638556693237,,,,
+mmlu.prompt2_mqa_single_correct_sk:marketing|0,mmlu,mmlu.mqa_single_correct_sk:marketing,2,0,234,em_relaxed,0.2094017094017094,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0811965811965812,0.20512820512820512,,,0.026453508054040356,,,,,0.2094017094017094,0.02665569965392276,0.017893784904018422,,,,
+mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0,mmlu,mmlu.mqa_single_correct_sk:medical_genetics,2,0,100,em_relaxed,0.18,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07,0.18,,,0.03861229196653694,,,,,0.18,0.03861229196653694,0.02564323999762429,,,,
+mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0,mmlu,mmlu.mqa_single_correct_sk:miscellaneous,2,0,779,em_relaxed,0.2092426187419769,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0693196405648267,0.2092426187419769,,,0.014583336673745648,,,,,0.2092426187419769,0.014583336673745648,0.009106230246473392,,,,
+mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0,mmlu,mmlu.mqa_single_correct_sk:moral_disputes,2,0,326,em_relaxed,0.25766871165644173,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.08895705521472393,0.25766871165644173,,,0.0242598463239315,,,,,0.25766871165644173,0.0242598463239315,0.015791302137788298,,,,
+mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0,mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,2,0,749,em_relaxed,0.0026702269692923898,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0026702269692923898,,,0.001886873052453708,,,,,0.0026702269692923898,0.001886873052453708,0.0,,,,
+mmlu.prompt2_mqa_single_correct_sk:nutrition|0,mmlu,mmlu.mqa_single_correct_sk:nutrition,2,0,303,em_relaxed,0.19801980198019803,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0297029702970297,0.19801980198019803,,,0.02293151046482573,,,,,0.19801980198019803,0.02293151046482573,0.00976897099052845,,,,
+mmlu.prompt2_mqa_single_correct_sk:philosophy|0,mmlu,mmlu.mqa_single_correct_sk:philosophy,2,0,308,em_relaxed,0.2922077922077922,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.10064935064935066,0.2922077922077922,,,0.02595551182455085,,,,,0.2922077922077922,0.02595551182455085,0.017171207138792156,,,,
+mmlu.prompt2_mqa_single_correct_sk:prehistory|0,mmlu,mmlu.mqa_single_correct_sk:prehistory,2,0,319,em_relaxed,0.18808777429467086,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06583072100313479,0.18808777429467086,,,0.021913980163123535,,,,,0.18808777429467086,0.021913980163123535,0.013906362189890747,,,,
+mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0,mmlu,mmlu.mqa_single_correct_sk:professional_accounting,2,0,276,em_relaxed,0.15217391304347827,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.14130434782608695,,,0.02100540442245809,,,,,0.15217391304347827,0.02165993479655374,0.0,,,,
+mmlu.prompt2_mqa_single_correct_sk:professional_law|0,mmlu,mmlu.mqa_single_correct_sk:professional_law,2,0,1171,em_relaxed,0.14175918018787362,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.004269854824935952,0.14175918018787362,,,0.01019734819812428,,,,,0.14175918018787362,0.01019734819812428,0.001906270167826036,,,,
+mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0,mmlu,mmlu.mqa_single_correct_sk:professional_medicine,2,0,264,em_relaxed,0.19696969696969696,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.003787878787878788,0.19696969696969696,,,0.024523802005628457,,,,,0.19696969696969696,0.024523802005628457,0.003787878787878786,,,,
+mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0,mmlu,mmlu.mqa_single_correct_sk:professional_psychology,2,0,596,em_relaxed,0.21308724832214765,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.050335570469798654,0.21308724832214765,,,0.01678740388759121,,,,,0.21308724832214765,0.01678740388759121,0.008963220911107225,,,,
+mmlu.prompt2_mqa_single_correct_sk:public_relations|0,mmlu,mmlu.mqa_single_correct_sk:public_relations,2,0,109,em_relaxed,0.23853211009174313,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.09174311926605505,0.23853211009174313,,,0.041009771029340336,,,,,0.23853211009174313,0.041009771029340336,0.02777660875318803,,,,
+mmlu.prompt2_mqa_single_correct_sk:security_studies|0,mmlu,mmlu.mqa_single_correct_sk:security_studies,2,0,222,em_relaxed,0.21621621621621623,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04504504504504504,0.21621621621621623,,,0.02769147769493752,,,,,0.21621621621621623,0.02769147769493752,0.013951433044338322,,,,
+mmlu.prompt2_mqa_single_correct_sk:sociology|0,mmlu,mmlu.mqa_single_correct_sk:sociology,2,0,197,em_relaxed,0.3553299492385787,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07106598984771574,0.3553299492385787,,,0.034186655739878676,,,,,0.3553299492385787,0.034186655739878676,0.018352502878671184,,,,
+mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0,mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,2,0,99,em_relaxed,0.25252525252525254,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.030303030303030304,0.24242424242424243,,,0.04329004329004326,,,,,0.25252525252525254,0.043887175543277195,0.01731601731601731,,,,
+mmlu.prompt2_mqa_single_correct_sk:virology|0,mmlu,mmlu.mqa_single_correct_sk:virology,2,0,164,em_relaxed,0.16463414634146342,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03048780487804878,0.16463414634146342,,,0.029047216079384597,,,,,0.16463414634146342,0.029047216079384597,0.013466226472535378,,,,
+mmlu.prompt2_mqa_single_correct_sk:world_religions|0,mmlu,mmlu.mqa_single_correct_sk:world_religions,2,0,171,em_relaxed,0.0935672514619883,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04678362573099415,0.0935672514619883,,,0.022335993231163274,,,,,0.0935672514619883,0.022335993231163274,0.016196389956235627,,,,
+mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0,mmlu,mmlu.mqa_single_correct_sk:abstract_algebra,3,0,100,em_relaxed,0.06,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.06,,,0.023868325657594173,,,,,0.06,0.023868325657594173,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:anatomy|0,mmlu,mmlu.mqa_single_correct_sk:anatomy,3,0,134,em_relaxed,0.13432835820895522,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.029850746268656716,0.13432835820895522,,,0.029568863219678807,,,,,0.13432835820895522,0.029568863219678807,0.01475608176283378,,,,
+mmlu.prompt3_mqa_single_correct_sk:astronomy|0,mmlu,mmlu.mqa_single_correct_sk:astronomy,3,0,152,em_relaxed,0.1513157894736842,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.019736842105263157,0.14473684210526316,,,0.028631951845930412,,,,,0.1513157894736842,0.029162631596843985,0.011319355505772411,,,,
+mmlu.prompt3_mqa_single_correct_sk:business_ethics|0,mmlu,mmlu.mqa_single_correct_sk:business_ethics,3,0,100,em_relaxed,0.13,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.13,,,0.03379976689896308,,,,,0.13,0.03379976689896308,0.009999999999999998,,,,
+mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0,mmlu,mmlu.mqa_single_correct_sk:clinical_knowledge,3,0,259,em_relaxed,0.1969111969111969,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03088803088803089,0.1969111969111969,,,0.024757519994692216,,,,,0.1969111969111969,0.024757519994692216,0.010771402065873048,,,,
+mmlu.prompt3_mqa_single_correct_sk:college_biology|0,mmlu,mmlu.mqa_single_correct_sk:college_biology,3,0,142,em_relaxed,0.176056338028169,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.035211267605633804,0.176056338028169,,,0.03207487957537854,,,,,0.176056338028169,0.03207487957537854,0.015521989489917001,,,,
+mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:college_chemistry,3,0,100,em_relaxed,0.16,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04,0.16,,,0.03684529491774709,,,,,0.16,0.03684529491774709,0.01969463855669323,,,,
+mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:college_computer_science,3,0,100,em_relaxed,0.15,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03,0.11,,,0.031446603773522035,,,,,0.15,0.035887028128263714,0.01714466079977653,,,,
+mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:college_mathematics,3,0,99,em_relaxed,0.09090909090909091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.08080808080808081,,,0.027530712883605948,,,,,0.09090909090909091,0.02903984386363362,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:college_medicine|0,mmlu,mmlu.mqa_single_correct_sk:college_medicine,3,0,172,em_relaxed,0.1569767441860465,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.040697674418604654,0.1569767441860465,,,0.027818863343493188,,,,,0.1569767441860465,0.027818863343493188,0.01511000089316197,,,,
+mmlu.prompt3_mqa_single_correct_sk:college_physics|0,mmlu,mmlu.mqa_single_correct_sk:college_physics,3,0,102,em_relaxed,0.12745098039215685,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.12745098039215685,,,0.03318224921942076,,,,,0.12745098039215685,0.03318224921942076,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:computer_security|0,mmlu,mmlu.mqa_single_correct_sk:computer_security,3,0,100,em_relaxed,0.13,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02,0.13,,,0.03379976689896309,,,,,0.13,0.03379976689896309,0.014070529413628963,,,,
+mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0,mmlu,mmlu.mqa_single_correct_sk:conceptual_physics,3,0,235,em_relaxed,0.1276595744680851,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05106382978723404,0.1276595744680851,,,0.021815328994143183,,,,,0.1276595744680851,0.021815328994143183,0.014390214367756187,,,,
+mmlu.prompt3_mqa_single_correct_sk:econometrics|0,mmlu,mmlu.mqa_single_correct_sk:econometrics,3,0,112,em_relaxed,0.10714285714285714,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.017857142857142856,0.09821428571428571,,,0.02824733740077989,,,,,0.10714285714285714,0.02935696201881377,0.012569900164490043,,,,
+mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0,mmlu,mmlu.mqa_single_correct_sk:electrical_engineering,3,0,145,em_relaxed,0.12413793103448276,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.041379310344827586,0.12413793103448276,,,0.0274782369836366,,,,,0.12413793103448276,0.0274782369836366,0.016597159859992713,,,,
+mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:elementary_mathematics,3,0,373,em_relaxed,0.08847184986595175,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.08579088471849866,,,0.01452017587959949,,,,,0.08847184986595175,0.014723671750091158,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:formal_logic|0,mmlu,mmlu.mqa_single_correct_sk:formal_logic,3,0,115,em_relaxed,0.11304347826086956,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.06956521739130435,,,0.02382794649967283,,,,,0.11304347826086956,0.029656609316837397,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:global_facts|0,mmlu,mmlu.mqa_single_correct_sk:global_facts,3,0,100,em_relaxed,0.12,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.12,,,0.03265986323710906,,,,,0.12,0.03265986323710906,0.009999999999999997,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_biology,3,0,282,em_relaxed,0.1950354609929078,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02127659574468085,0.1879432624113475,,,0.023305230769714243,,,,,0.1950354609929078,0.02363698094391562,0.00860850820920177,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0,mmlu,mmlu.mqa_single_correct_sk:high_school_chemistry,3,0,198,em_relaxed,0.09090909090909091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005050505050505051,0.08585858585858586,,,0.01996022556317289,,,,,0.09090909090909091,0.020482086775424218,0.00505050505050505,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0,mmlu,mmlu.mqa_single_correct_sk:high_school_computer_science,3,0,99,em_relaxed,0.13131313131313133,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.010101010101010102,0.12121212121212122,,,0.03296871474399963,,,,,0.13131313131313133,0.034117144062155554,0.010101010101010109,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_european_history,3,0,135,em_relaxed,0.21481481481481482,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02962962962962963,0.2074074074074074,,,0.03502553170678318,,,,,0.21481481481481482,0.03547854198560824,0.014648038602753806,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0,mmlu,mmlu.mqa_single_correct_sk:high_school_geography,3,0,198,em_relaxed,0.20202020202020202,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07575757575757576,0.20202020202020202,,,0.028606204289229872,,,,,0.20202020202020202,0.028606204289229872,0.018852670234993107,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_government_and_politics,3,0,193,em_relaxed,0.30569948186528495,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.046632124352331605,0.3005181347150259,,,0.0330881859441575,,,,,0.30569948186528495,0.033248379397581594,0.015216761819262604,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_macroeconomics,3,0,387,em_relaxed,0.15762273901808785,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01808785529715762,0.14987080103359174,,,0.018168008667456638,,,,,0.15762273901808785,0.018546804758294254,0.006783224657656192,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_mathematics,3,0,266,em_relaxed,0.05639097744360902,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.05639097744360902,,,0.014170263190664145,,,,,0.05639097744360902,0.014170263190664145,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_microeconomics,3,0,217,em_relaxed,0.17050691244239632,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.013824884792626729,0.16589861751152074,,,0.025310686717794278,,,,,0.17050691244239632,0.025588834483805927,0.007944762237164815,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_physics,3,0,150,em_relaxed,0.17333333333333334,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.02666666666666667,0.17333333333333334,,,0.031010810941145068,,,,,0.17333333333333334,0.031010810941145068,0.013198422611557013,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0,mmlu,mmlu.mqa_single_correct_sk:high_school_psychology,3,0,536,em_relaxed,0.18470149253731344,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.03731343283582089,0.18470149253731344,,,0.016777090030491517,,,,,0.18470149253731344,0.016777090030491517,0.008194041665419557,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0,mmlu,mmlu.mqa_single_correct_sk:high_school_statistics,3,0,214,em_relaxed,0.16355140186915887,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.1542056074766355,,,0.024745297661074854,,,,,0.16355140186915887,0.025342938086817392,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_us_history,3,0,57,em_relaxed,0.12280701754385964,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.12280701754385964,,,0.04385964912280699,,,,,0.12280701754385964,0.04385964912280699,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0,mmlu,mmlu.mqa_single_correct_sk:high_school_world_history,3,0,180,em_relaxed,0.26666666666666666,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.005555555555555556,0.26666666666666666,,,0.03305282343736876,,,,,0.26666666666666666,0.03305282343736876,0.005555555555555554,,,,
+mmlu.prompt3_mqa_single_correct_sk:human_aging|0,mmlu,mmlu.mqa_single_correct_sk:human_aging,3,0,219,em_relaxed,0.1278538812785388,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0684931506849315,0.1278538812785388,,,0.02261639276396452,,,,,0.1278538812785388,0.02261639276396452,0.017107571020219335,,,,
+mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0,mmlu,mmlu.mqa_single_correct_sk:human_sexuality,3,0,109,em_relaxed,0.1559633027522936,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05504587155963303,0.1559633027522936,,,0.03491242362801975,,,,,0.1559633027522936,0.03491242362801975,0.021946024609185506,,,,
+mmlu.prompt3_mqa_single_correct_sk:international_law|0,mmlu,mmlu.mqa_single_correct_sk:international_law,3,0,118,em_relaxed,0.15254237288135594,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0423728813559322,0.15254237288135594,,,0.03324002290477697,,,,,0.15254237288135594,0.03324002290477697,0.018622984668462274,,,,
+mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0,mmlu,mmlu.mqa_single_correct_sk:jurisprudence,3,0,107,em_relaxed,0.18691588785046728,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.037383177570093455,0.18691588785046728,,,0.03786499587281739,,,,,0.18691588785046728,0.03786499587281739,0.01842518672106723,,,,
+mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0,mmlu,mmlu.mqa_single_correct_sk:logical_fallacies,3,0,158,em_relaxed,0.17721518987341772,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0379746835443038,0.17721518987341772,,,0.030475004517281346,,,,,0.17721518987341772,0.030475004517281346,0.015254237576434418,,,,
+mmlu.prompt3_mqa_single_correct_sk:machine_learning|0,mmlu,mmlu.mqa_single_correct_sk:machine_learning,3,0,112,em_relaxed,0.125,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.008928571428571428,0.11607142857142858,,,0.03040253249894973,,,,,0.125,0.03139045014587016,0.008928571428571423,,,,
+mmlu.prompt3_mqa_single_correct_sk:management|0,mmlu,mmlu.mqa_single_correct_sk:management,3,0,100,em_relaxed,0.2,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06,0.2,,,0.040201512610368445,,,,,0.2,0.040201512610368445,0.023868325657594162,,,,
+mmlu.prompt3_mqa_single_correct_sk:marketing|0,mmlu,mmlu.mqa_single_correct_sk:marketing,3,0,234,em_relaxed,0.1623931623931624,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05555555555555555,0.1623931623931624,,,0.02416161812798774,,,,,0.1623931623931624,0.02416161812798774,0.01500631280644691,,,,
+mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0,mmlu,mmlu.mqa_single_correct_sk:medical_genetics,3,0,100,em_relaxed,0.14,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.01,0.14,,,0.034873508801977704,,,,,0.14,0.034873508801977704,0.009999999999999997,,,,
+mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0,mmlu,mmlu.mqa_single_correct_sk:miscellaneous,3,0,779,em_relaxed,0.17843388960205392,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.055198973042362,0.17715019255455713,,,0.013688041138952063,,,,,0.17843388960205392,0.013726826073077934,0.00818739755268085,,,,
+mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0,mmlu,mmlu.mqa_single_correct_sk:moral_disputes,3,0,326,em_relaxed,0.2147239263803681,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.06748466257668712,0.2116564417177914,,,0.02265853497858724,,,,,0.2147239263803681,0.022777692313730433,0.013915180741090785,,,,
+mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0,mmlu,mmlu.mqa_single_correct_sk:moral_scenarios,3,0,749,em_relaxed,0.01335113484646195,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.0053404539385847796,,,0.0026648668562206297,,,,,0.01335113484646195,0.004196522950335422,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:nutrition|0,mmlu,mmlu.mqa_single_correct_sk:nutrition,3,0,303,em_relaxed,0.18811881188118812,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.036303630363036306,0.18811881188118812,,,0.022488416775417838,,,,,0.18811881188118812,0.022488416775417838,0.010763206393659715,,,,
+mmlu.prompt3_mqa_single_correct_sk:philosophy|0,mmlu,mmlu.mqa_single_correct_sk:philosophy,3,0,308,em_relaxed,0.237012987012987,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05194805194805195,0.237012987012987,,,0.024270317456654856,,,,,0.237012987012987,0.024270317456654856,0.01266576753511598,,,,
+mmlu.prompt3_mqa_single_correct_sk:prehistory|0,mmlu,mmlu.mqa_single_correct_sk:prehistory,3,0,319,em_relaxed,0.15047021943573669,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04075235109717868,0.14733542319749215,,,0.019876006858615635,,,,,0.15047021943573669,0.020049383410796876,0.011087359931149772,,,,
+mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0,mmlu,mmlu.mqa_single_correct_sk:professional_accounting,3,0,276,em_relaxed,0.14492753623188406,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.13043478260869565,,,0.020308668108561446,,,,,0.14492753623188406,0.02122807279169736,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:professional_law|0,mmlu,mmlu.mqa_single_correct_sk:professional_law,3,0,1171,em_relaxed,0.1340734415029889,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0017079419299743809,0.12809564474807855,,,0.009770317447740377,,,,,0.1340734415029889,0.009961367847349073,0.0012071811002646103,,,,
+mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0,mmlu,mmlu.mqa_single_correct_sk:professional_medicine,3,0,264,em_relaxed,0.09090909090909091,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.0,0.09090909090909091,,,0.01772676309015986,,,,,0.09090909090909091,0.01772676309015986,0.0,,,,
+mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0,mmlu,mmlu.mqa_single_correct_sk:professional_psychology,3,0,596,em_relaxed,0.19630872483221476,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.018456375838926176,0.19630872483221476,,,0.016283806947934177,,,,,0.19630872483221476,0.016283806947934177,0.005517845461546564,,,,
+mmlu.prompt3_mqa_single_correct_sk:public_relations|0,mmlu,mmlu.mqa_single_correct_sk:public_relations,3,0,109,em_relaxed,0.21100917431192662,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.07339449541284404,0.21100917431192662,,,0.03926221238036853,,,,,0.21100917431192662,0.03926221238036853,0.025093850407933094,,,,
+mmlu.prompt3_mqa_single_correct_sk:security_studies|0,mmlu,mmlu.mqa_single_correct_sk:security_studies,3,0,222,em_relaxed,0.15765765765765766,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04954954954954955,0.15765765765765766,,,0.024513522045494097,,,,,0.15765765765765766,0.024513522045494097,0.014597835283794243,,,,
+mmlu.prompt3_mqa_single_correct_sk:sociology|0,mmlu,mmlu.mqa_single_correct_sk:sociology,3,0,197,em_relaxed,0.3350253807106599,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.05583756345177665,0.3350253807106599,,,0.03371423842235921,,,,,0.3350253807106599,0.03371423842235921,0.016400555297895067,,,,
+mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0,mmlu,mmlu.mqa_single_correct_sk:us_foreign_policy,3,0,99,em_relaxed,0.2727272727272727,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.050505050505050504,0.26262626262626265,,,0.044452876769839444,,,,,0.2727272727272727,0.044988332664126655,0.022120793242071984,,,,
+mmlu.prompt3_mqa_single_correct_sk:virology|0,mmlu,mmlu.mqa_single_correct_sk:virology,3,0,164,em_relaxed,0.1402439024390244,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.024390243902439025,0.1402439024390244,,,0.027197928271392544,,,,,0.1402439024390244,0.027197928271392544,0.012082375722391943,,,,
+mmlu.prompt3_mqa_single_correct_sk:world_religions|0,mmlu,mmlu.mqa_single_correct_sk:world_religions,3,0,171,em_relaxed,0.10526315789473684,32,test,lighteval/okapi_mmlu,sk,b3e738ae24d5d96c4b425d4893f377451196cd2e,0.04678362573099415,0.10526315789473684,,,0.02353755765789256,,,,,0.10526315789473684,0.02353755765789256,0.01619638995623562,,,,
+movie_review.prompt1_sentiment_binary_sk|0,movie_review,movie_review.sentiment_binary_sk,1,0,2008,em_norm_relaxed,0.6329681274900398,32,test,mteb/slovak_movie_review_sentiment,default,29a7405aabcfd4860a51ae6f65a5650d63108f26,,,0.6324701195219123,0.010761991548945318,,0.38745019920318724,0.6329681274900398,0.010758931054934645,0.010874391376667836,,,,,,,
+movie_review.prompt2_sentiment_binary_sk|0,movie_review,movie_review.sentiment_binary_sk,2,0,2008,em_norm_relaxed,0.48306772908366535,32,test,mteb/slovak_movie_review_sentiment,default,29a7405aabcfd4860a51ae6f65a5650d63108f26,,,0.47758964143426297,0.011149609394535145,,0.01045816733067729,0.48306772908366535,0.011154424009707972,0.002270759742696006,,,,,,,
+movie_review.prompt3_sentiment_binary_sk|0,movie_review,movie_review.sentiment_binary_sk,3,0,2008,em_norm_relaxed,0.11105577689243028,32,test,mteb/slovak_movie_review_sentiment,default,29a7405aabcfd4860a51ae6f65a5650d63108f26,,,0.09412350597609562,0.006517932474820501,,0.04731075697211155,0.11105577689243028,0.007013489433691037,0.004738947450976647,,,,,,,
+multi_wiki_qa.prompt1_qa_sk|0,multi_wiki_qa,multi_wiki_qa.qa_sk,1,0,4997,exact_match_slk_prefix,0.36401841104662797,126,train,alexandrainst/multi-wiki-qa,sk,9bdb3350487e399bd1cd873e1395290d636db5ce,,,,,,,,,,,,,0.36401841104662797,0.00680726616133214,0.5156574509584114,0.0058319674487835125
+multi_wiki_qa.prompt2_qa_sk|0,multi_wiki_qa,multi_wiki_qa.qa_sk,2,0,4997,exact_match_slk_prefix,0.36001600960576347,126,train,alexandrainst/multi-wiki-qa,sk,9bdb3350487e399bd1cd873e1395290d636db5ce,,,,,,,,,,,,,0.36001600960576347,0.006791008078277957,0.5090494926903776,0.005852156127470673
+multi_wiki_qa.prompt3_qa_sk|0,multi_wiki_qa,multi_wiki_qa.qa_sk,3,0,4997,exact_match_slk_prefix,0.34000400240144085,126,train,alexandrainst/multi-wiki-qa,sk,9bdb3350487e399bd1cd873e1395290d636db5ce,,,,,,,,,,,,,0.34000400240144085,0.006701954125609285,0.5050071786470199,0.005797838571943781
+senti_sk.prompt1_sentiment_3_class_sk|0,senti_sk,senti_sk.sentiment_3_class_sk,1,0,4677,em_norm_relaxed,0.3200769724182168,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.31922172332691895,0.006817290681927466,,0.22642719692110327,0.3200769724182168,0.0068221276139399335,0.006120369470744044,,,,,,,
+senti_sk.prompt1_sentiment_binary_sk|0,senti_sk,senti_sk.sentiment_binary_sk,1,0,3728,em_norm_relaxed,0.31223175965665234,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.3111587982832618,0.007583521183648285,,0.1968884120171674,0.31223175965665234,0.0075906663188793026,0.006513558648330005,,,,,,,
+senti_sk.prompt2_sentiment_3_class_sk|0,senti_sk,senti_sk.sentiment_3_class_sk,2,0,4677,em_norm_relaxed,0.383365405174257,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.37716484926234767,0.007087856422819165,,0.019029292281376953,0.383365405174257,0.007110221850748201,0.001998030377272694,,,,,,,
+senti_sk.prompt2_sentiment_binary_sk|0,senti_sk,senti_sk.sentiment_binary_sk,2,0,3728,em_norm_relaxed,0.4965128755364807,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.49302575107296137,0.008189324033951761,,0.0037553648068669528,0.4965128755364807,0.008189921623320459,0.001001911513818721,,,,,,,
+senti_sk.prompt3_sentiment_3_class_sk|0,senti_sk,senti_sk.sentiment_3_class_sk,3,0,4677,em_norm_relaxed,0.40880906564036773,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.26448578148385715,0.006449996068936491,,0.10540945050245884,0.40880906564036773,0.0071893047031815266,0.004490706548553942,,,,,,,
+senti_sk.prompt3_sentiment_binary_sk|0,senti_sk,senti_sk.sentiment_binary_sk,3,0,3728,em_norm_relaxed,0.25160944206008584,32,test,TUKE-KEMT/senti-sk,default,3c8a92168ff2b692c653de65c97bda573127d15d,,,0.1990343347639485,0.0065402033880141534,,0.03165236051502146,0.25160944206008584,0.0071080082101389235,0.0028677349189843607,,,,,,,
+skj1.prompt1_skj1_sk|0,skj1,skj1.skj1_sk,1,0,100,em_relaxed,0.02,32,train,slovak-nlp/SKJ1,default,e615c60b415389b629703f82d3a79810dbde6734,0.0,0.02,,,0.014070529413628957,,,,,0.02,0.014070529413628957,0.0,,,,
+skj1.prompt2_skj1_sk|0,skj1,skj1.skj1_sk,2,0,100,em_relaxed,0.06,32,train,slovak-nlp/SKJ1,default,e615c60b415389b629703f82d3a79810dbde6734,0.0,0.05,,,0.021904291355759067,,,,,0.06,0.023868325657594166,0.0,,,,
+skj1.prompt3_skj1_sk|0,skj1,skj1.skj1_sk,3,0,100,em_relaxed,0.05,32,train,slovak-nlp/SKJ1,default,e615c60b415389b629703f82d3a79810dbde6734,0.0,0.03,,,0.017144660799776543,,,,,0.05,0.021904291355759067,0.0,,,,
+sklep.prompt1_hate_speech_sk:hate-speech|0,sklep,sklep.hate_speech_sk:hate-speech,1,0,1319,em_norm_relaxed,0.6921910538286581,32,test,slovak-nlp/sklep,hate-speech,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.6921910538286581,0.012714401009923647,,0.0,0.6921910538286581,0.012714401009923647,0.0,,,,,,,
+sklep.prompt1_nli_3_classes_sk:nli|0,sklep,sklep.nli_3_classes_sk:nli,1,0,5004,em_norm_relaxed,0.016786570743405275,32,test,slovak-nlp/sklep,nli,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.016386890487609912,0.0017949203375471519,,0.0003996802557953637,0.016786570743405275,0.0018163086111764766,0.00028258837305266234,,,,,,,
+sklep.prompt1_nli_binary_sk:rte|0,sklep,sklep.nli_binary_sk:rte,1,0,1660,em_norm_relaxed,0.030120481927710843,32,test,slovak-nlp/sklep,rte,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.030120481927710843,0.004196301257202765,,0.0006024096385542169,0.030120481927710843,0.004196301257202765,0.0006024096385542043,,,,,,,
+sklep.prompt1_qa_sk:question-answering|0,sklep,sklep.qa_sk:question-answering,1,0,9583,exact_match_slk_prefix,0.16456224564332672,126,test,slovak-nlp/sklep,question-answering,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,,,,,,,,,,,0.16456224564332672,0.003787861300566259,0.3730151221096053,0.003855179221962441
+sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0,sklep,sklep.sentiment_binary_sk:sentiment-analysis,1,0,1042,em_norm_relaxed,0.8809980806142035,32,test,slovak-nlp/sklep,sentiment-analysis,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.8800383877159309,0.010070396971355563,,0.44817658349328215,0.8809980806142035,0.010035501945449067,0.015413429449614385,,,,,,,
+sklep.prompt2_hate_speech_sk:hate-speech|0,sklep,sklep.hate_speech_sk:hate-speech,2,0,1319,em_norm_relaxed,0.7247915087187263,32,test,slovak-nlp/sklep,hate-speech,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.7247915087187263,0.012302114305862654,,0.0,0.7247915087187263,0.012302114305862654,0.0,,,,,,,
+sklep.prompt2_nli_3_classes_sk:nli|0,sklep,sklep.nli_3_classes_sk:nli,2,0,5004,em_norm_relaxed,0.09972022382094324,32,test,slovak-nlp/sklep,nli,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.09952038369304557,0.004232312358957502,,0.0,0.09972022382094324,0.004236089409386789,0.0,,,,,,,
+sklep.prompt2_nli_binary_sk:rte|0,sklep,sklep.nli_binary_sk:rte,2,0,1660,em_norm_relaxed,0.3301204819277108,32,test,slovak-nlp/sklep,rte,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.3301204819277108,0.011545464009906971,,0.0018072289156626507,0.3301204819277108,0.011545464009906971,0.001042774975713993,,,,,,,
+sklep.prompt2_qa_sk:question-answering|0,sklep,sklep.qa_sk:question-answering,2,0,9583,exact_match_slk_prefix,0.16518835437754356,126,test,slovak-nlp/sklep,question-answering,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,,,,,,,,,,,0.16518835437754356,0.003793637935655721,0.3751701402109962,0.0038611217329234445
+sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0,sklep,sklep.sentiment_binary_sk:sentiment-analysis,2,0,1042,em_norm_relaxed,0.3512476007677543,32,test,slovak-nlp/sklep,sentiment-analysis,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.34836852207293667,0.014767096602802959,,0.0009596928982725527,0.3512476007677543,0.014795198864354053,0.0009596928982725573,,,,,,,
+sklep.prompt3_hate_speech_sk:hate-speech|0,sklep,sklep.hate_speech_sk:hate-speech,3,0,1319,em_norm_relaxed,0.6967399545109931,32,test,slovak-nlp/sklep,hate-speech,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.6967399545109931,0.012661502663418697,,0.0,0.6967399545109931,0.012661502663418697,0.0,,,,,,,
+sklep.prompt3_nli_3_classes_sk:nli|0,sklep,sklep.nli_3_classes_sk:nli,3,0,5004,em_norm_relaxed,0.001398880895283773,32,test,slovak-nlp/sklep,nli,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.001398880895283773,0.0005284101391342323,,0.0,0.001398880895283773,0.0005284101391342323,0.0,,,,,,,
+sklep.prompt3_nli_binary_sk:rte|0,sklep,sklep.nli_binary_sk:rte,3,0,1660,em_norm_relaxed,0.2319277108433735,32,test,slovak-nlp/sklep,rte,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.2319277108433735,0.010362252629508847,,0.0012048192771084338,0.2319277108433735,0.010362252629508847,0.0008516790803748009,,,,,,,
+sklep.prompt3_qa_sk:question-answering|0,sklep,sklep.qa_sk:question-answering,3,0,9583,exact_match_slk_prefix,0.16143170197224252,126,test,slovak-nlp/sklep,question-answering,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,,,,,,,,,,,0.16143170197224252,0.0037586816998114083,0.3769392643579436,0.003773157275405368
+sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0,sklep,sklep.sentiment_binary_sk:sentiment-analysis,3,0,1042,em_norm_relaxed,0.15355086372360843,32,test,slovak-nlp/sklep,sentiment-analysis,10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9,,,0.118042226487524,0.010000396690181516,,0.015355086372360844,0.15355086372360843,0.011173807886451477,0.0038110143844746576,,,,,,,
+slovakcopa.prompt1_copa_sk|0,slovakcopa,slovakcopa.copa_sk,1,0,500,em_relaxed,0.618,32,test,NaiveNeuron/SlovakCOPA,default,5c61e4cd9343f9b4fab623bdc50bb895cc170771,0.0,0.508,,,0.022380208834928028,,,,,0.618,0.021750820591250834,0.0,,,,
+slovakcopa.prompt2_copa_sk|0,slovakcopa,slovakcopa.copa_sk,2,0,500,em_relaxed,0.51,32,test,NaiveNeuron/SlovakCOPA,default,5c61e4cd9343f9b4fab623bdc50bb895cc170771,0.0,0.326,,,0.020984009562393567,,,,,0.51,0.02237859698923078,0.0,,,,
+slovakcopa.prompt3_copa_sk|0,slovakcopa,slovakcopa.copa_sk,3,0,500,em_relaxed,0.086,32,test,NaiveNeuron/SlovakCOPA,default,5c61e4cd9343f9b4fab623bdc50bb895cc170771,0.0,0.084,,,0.01241758401564368,,,,,0.086,0.01255081854271602,0.0,,,,
+toxic_sk.prompt1_toxicity_sk|0,toxic_sk,toxic_sk.toxicity_sk,1,0,881,em_norm_relaxed,0.45629965947786605,32,test,TUKE-KEMT/toxic-sk,default,2fb416e18785c128e7ec934c07a14c803a88cdb3,,,0.45629965947786605,0.016790496508382532,,0.0,0.45629965947786605,0.016790496508382532,0.0,,,,,,,
+toxic_sk.prompt2_toxicity_sk|0,toxic_sk,toxic_sk.toxicity_sk,2,0,881,em_norm_relaxed,0.47559591373439275,32,test,TUKE-KEMT/toxic-sk,default,2fb416e18785c128e7ec934c07a14c803a88cdb3,,,0.47559591373439275,0.016834908286515442,,0.0,0.47559591373439275,0.016834908286515442,0.0,,,,,,,
+toxic_sk.prompt3_toxicity_sk|0,toxic_sk,toxic_sk.toxicity_sk,3,0,881,em_norm_relaxed,0.4960272417707151,32,test,TUKE-KEMT/toxic-sk,default,2fb416e18785c128e7ec934c07a14c803a88cdb3,,,0.4960272417707151,0.016854464514836336,,0.007945516458569807,0.4960272417707151,0.016854464514836336,0.00299286751118699,,,,,,,
+truthfulqa.prompt1_mqa_single_correct_sk:mc1|0,truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,1,0,778,em_relaxed,0.007712082262210797,32,val,alexandrainst/m_truthfulqa,sk,f0445d470f1925882b990f5f247fdcf288972f60,0.0,0.007712082262210797,,,0.003138297919968842,,,,,0.007712082262210797,0.003138297919968842,0.0,,,,
+truthfulqa.prompt2_mqa_single_correct_sk:mc1|0,truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,2,0,778,em_relaxed,0.016709511568123392,32,val,alexandrainst/m_truthfulqa,sk,f0445d470f1925882b990f5f247fdcf288972f60,0.0,0.016709511568123392,,,0.004598458668423289,,,,,0.016709511568123392,0.004598458668423289,0.0,,,,
+truthfulqa.prompt3_mqa_single_correct_sk:mc1|0,truthfulqa,truthfulqa.mqa_single_correct_sk:mc1,3,0,778,em_relaxed,0.011568123393316195,32,val,alexandrainst/m_truthfulqa,sk,f0445d470f1925882b990f5f247fdcf288972f60,0.0,0.011568123393316195,,,0.00383613885044431,,,,,0.011568123393316195,0.00383613885044431,0.0,,,,
diff --git a/aya_300k_lr_experiments/benchmarks/skbench/results/results_2026-10-22T21-55-04.268099.json b/aya_300k_lr_experiments/benchmarks/skbench/results/results_2026-10-22T21-55-04.268099.json
new file mode 100644
index 0000000..a56512f
--- /dev/null
+++ b/aya_300k_lr_experiments/benchmarks/skbench/results/results_2026-10-22T21-55-04.268099.json
@@ -0,0 +1,18023 @@
+{
+ "config_general": {
+ "lighteval_sha": "?",
+ "num_fewshot_seeds": 1,
+ "max_samples": null,
+ "job_id": "0",
+ "start_time": 796668.90423246,
+ "end_time": 807588.555648873,
+ "total_evaluation_time_secondes": "10919.65141641302",
+ "model_config": {
+ "model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010",
+ "generation_parameters": {
+ "num_blocks": null,
+ "block_size": null,
+ "early_stopping": null,
+ "repetition_penalty": null,
+ "frequency_penalty": null,
+ "length_penalty": null,
+ "presence_penalty": null,
+ "max_new_tokens": null,
+ "min_new_tokens": null,
+ "seed": null,
+ "stop_tokens": null,
+ "temperature": 0,
+ "top_k": null,
+ "min_p": null,
+ "top_p": null,
+ "truncate_prompt": null,
+ "cache_implementation": null,
+ "response_format": null,
+ "skip_special_tokens": true
+ },
+ "system_prompt": null,
+ "cache_dir": "~/.cache/huggingface/lighteval",
+ "tokenizer": null,
+ "revision": "main",
+ "dtype": "bfloat16",
+ "tensor_parallel_size": 1,
+ "data_parallel_size": 1,
+ "pipeline_parallel_size": 1,
+ "gpu_memory_utilization": 0.85,
+ "enable_prefix_caching": null,
+ "max_model_length": 32768,
+ "quantization": null,
+ "load_format": null,
+ "swap_space": 4,
+ "seed": 1234,
+ "trust_remote_code": false,
+ "add_special_tokens": true,
+ "multichoice_continuations_start_space": true,
+ "pairwise_tokenization": false,
+ "max_num_seqs": 128,
+ "max_num_batched_tokens": 2048,
+ "subfolder": null,
+ "is_async": false,
+ "override_chat_template": null,
+ "enforce_eager": true
+ },
+ "model_name": "/home/schwarc/diplomovka/skbench_evaluation/models/aya_300k_lr_0p00010"
+ },
+ "results": {
+ "mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0": {
+ "em_relaxed": 0.17843388960205392,
+ "em_relaxed_stderr": 0.013726826073077934,
+ "em": 0.055198973042362,
+ "em_stderr": 0.00818739755268085,
+ "em_conservative": 0.17715019255455713,
+ "em_conservative_stderr": 0.013688041138952063
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0": {
+ "em_relaxed": 0.06166219839142091,
+ "em_relaxed_stderr": 0.012471469772330128,
+ "em": 0.005361930294906166,
+ "em_stderr": 0.003786357797685454,
+ "em_conservative": 0.058981233243967826,
+ "em_conservative_stderr": 0.012214750400069162
+ },
+ "financial_exams.prompt2_completion_sk|0": {
+ "em_relaxed": 0.2383808095952024,
+ "em_relaxed_stderr": 0.011670501255680532,
+ "em": 0.07571214392803598,
+ "em_stderr": 0.007245549384888845,
+ "em_conservative": 0.20539730134932535,
+ "em_conservative_stderr": 0.011065150105060274
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:machine_learning|0": {
+ "em_relaxed": 0.11607142857142858,
+ "em_relaxed_stderr": 0.03040253249894973,
+ "em": 0.008928571428571428,
+ "em_stderr": 0.008928571428571423,
+ "em_conservative": 0.10714285714285714,
+ "em_conservative_stderr": 0.02935696201881377
+ },
+ "belebele.prompt3_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.21333333333333335,
+ "em_relaxed_stderr": 0.013662962863011163,
+ "em": 0.03,
+ "em_stderr": 0.005689402360648914,
+ "em_conservative": 0.2111111111111111,
+ "em_conservative_stderr": 0.013610798969328486
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0": {
+ "em_relaxed": 0.15444015444015444,
+ "em_relaxed_stderr": 0.022497923516297567,
+ "em": 0.011583011583011582,
+ "em_stderr": 0.006661484058899665,
+ "em_conservative": 0.15444015444015444,
+ "em_conservative_stderr": 0.022497923516297567
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_law|0": {
+ "em_relaxed": 0.1340734415029889,
+ "em_relaxed_stderr": 0.009961367847349073,
+ "em": 0.0017079419299743809,
+ "em_stderr": 0.0012071811002646103,
+ "em_conservative": 0.12809564474807855,
+ "em_conservative_stderr": 0.009770317447740377
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0": {
+ "em_relaxed": 0.15,
+ "em_relaxed_stderr": 0.035887028128263714,
+ "em": 0.01,
+ "em_stderr": 0.009999999999999998,
+ "em_conservative": 0.11,
+ "em_conservative_stderr": 0.031446603773522035
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0": {
+ "em_relaxed": 0.15217391304347827,
+ "em_relaxed_stderr": 0.02165993479655374,
+ "em": 0.0036231884057971015,
+ "em_stderr": 0.0036231884057970937,
+ "em_conservative": 0.14130434782608695,
+ "em_conservative_stderr": 0.02100540442245809
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0": {
+ "em_relaxed": 0.12280701754385964,
+ "em_relaxed_stderr": 0.04385964912280699,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.12280701754385964,
+ "em_conservative_stderr": 0.04385964912280699
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0": {
+ "em_relaxed": 0.08585858585858586,
+ "em_relaxed_stderr": 0.019960225563172885,
+ "em": 0.010101010101010102,
+ "em_stderr": 0.007124341521250813,
+ "em_conservative": 0.08080808080808081,
+ "em_conservative_stderr": 0.019417681889724533
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0": {
+ "em_relaxed": 0.21052631578947367,
+ "em_relaxed_stderr": 0.05447885994808057,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.21052631578947367,
+ "em_conservative_stderr": 0.05447885994808057
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_law|0": {
+ "em_relaxed": 0.14175918018787362,
+ "em_relaxed_stderr": 0.01019734819812428,
+ "em": 0.004269854824935952,
+ "em_stderr": 0.001906270167826036,
+ "em_conservative": 0.14175918018787362,
+ "em_conservative_stderr": 0.01019734819812428
+ },
+ "sklep.prompt1_qa_sk:question-answering|0": {
+ "exact_match_slk_prefix": 0.16456224564332672,
+ "exact_match_slk_prefix_stderr": 0.003787861300566259,
+ "f1_slk": 0.3730151221096053,
+ "f1_slk_stderr": 0.003855179221962441
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0": {
+ "em_relaxed": 0.14,
+ "em_relaxed_stderr": 0.03487350880197772,
+ "em": 0.09,
+ "em_stderr": 0.028762349126466143,
+ "em_conservative": 0.14,
+ "em_conservative_stderr": 0.03487350880197772
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "em_relaxed": 0.17511520737327188,
+ "em_relaxed_stderr": 0.025860189581375218,
+ "em": 0.02304147465437788,
+ "em_stderr": 0.010208603193741356,
+ "em_conservative": 0.17511520737327188,
+ "em_conservative_stderr": 0.025860189581375218
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0": {
+ "em_relaxed": 0.04040404040404041,
+ "em_relaxed_stderr": 0.019890402239668436,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.030303030303030304,
+ "em_conservative_stderr": 0.017316017316017295
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0": {
+ "em_relaxed": 0.14666666666666667,
+ "em_relaxed_stderr": 0.02898223889753507,
+ "em": 0.04,
+ "em_stderr": 0.016053601490175616,
+ "em_conservative": 0.14666666666666667,
+ "em_conservative_stderr": 0.02898223889753507
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0": {
+ "em_relaxed": 0.1822429906542056,
+ "em_relaxed_stderr": 0.026451344562470138,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.17289719626168223,
+ "em_conservative_stderr": 0.025910984835548666
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0": {
+ "em_relaxed": 0.19696969696969696,
+ "em_relaxed_stderr": 0.024523802005628457,
+ "em": 0.003787878787878788,
+ "em_stderr": 0.003787878787878786,
+ "em_conservative": 0.19696969696969696,
+ "em_conservative_stderr": 0.024523802005628457
+ },
+ "slovakcopa.prompt3_copa_sk|0": {
+ "em_relaxed": 0.086,
+ "em_relaxed_stderr": 0.01255081854271602,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.084,
+ "em_conservative_stderr": 0.01241758401564368
+ },
+ "movie_review.prompt2_sentiment_binary_sk|0": {
+ "em_norm_relaxed": 0.48306772908366535,
+ "em_norm_relaxed_stderr": 0.011154424009707972,
+ "em_norm": 0.01045816733067729,
+ "em_norm_stderr": 0.002270759742696006,
+ "em_conservative_norm": 0.47758964143426297,
+ "em_conservative_norm_stderr": 0.011149609394535145
+ },
+ "sklep.prompt2_qa_sk:question-answering|0": {
+ "exact_match_slk_prefix": 0.16518835437754356,
+ "exact_match_slk_prefix_stderr": 0.003793637935655721,
+ "f1_slk": 0.3751701402109962,
+ "f1_slk_stderr": 0.0038611217329234445
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_biology|0": {
+ "em_relaxed": 0.15492957746478872,
+ "em_relaxed_stderr": 0.030472217725828407,
+ "em": 0.028169014084507043,
+ "em_stderr": 0.013933866272175799,
+ "em_conservative": 0.15492957746478872,
+ "em_conservative_stderr": 0.030472217725828407
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:nutrition|0": {
+ "em_relaxed": 0.18811881188118812,
+ "em_relaxed_stderr": 0.022488416775417838,
+ "em": 0.036303630363036306,
+ "em_stderr": 0.010763206393659715,
+ "em_conservative": 0.18811881188118812,
+ "em_conservative_stderr": 0.022488416775417838
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0": {
+ "em_relaxed": 0.24822695035460993,
+ "em_relaxed_stderr": 0.025770015644290406,
+ "em": 0.04964539007092199,
+ "em_stderr": 0.012957736571882147,
+ "em_conservative": 0.24113475177304963,
+ "em_conservative_stderr": 0.02551873104953778
+ },
+ "arc.prompt3_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.1770744225834046,
+ "em_relaxed_stderr": 0.011169578281264629,
+ "em": 0.022241231822070145,
+ "em_stderr": 0.004314931013145922,
+ "em_conservative": 0.17621899059024806,
+ "em_conservative_stderr": 0.011148355810710043
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:econometrics|0": {
+ "em_relaxed": 0.10714285714285714,
+ "em_relaxed_stderr": 0.02935696201881377,
+ "em": 0.017857142857142856,
+ "em_stderr": 0.012569900164490043,
+ "em_conservative": 0.09821428571428571,
+ "em_conservative_stderr": 0.02824733740077989
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:prehistory|0": {
+ "em_relaxed": 0.18808777429467086,
+ "em_relaxed_stderr": 0.021913980163123535,
+ "em": 0.06583072100313479,
+ "em_stderr": 0.013906362189890747,
+ "em_conservative": 0.18808777429467086,
+ "em_conservative_stderr": 0.021913980163123535
+ },
+ "toxic_sk.prompt3_toxicity_sk|0": {
+ "em_norm_relaxed": 0.4960272417707151,
+ "em_norm_relaxed_stderr": 0.016854464514836336,
+ "em_norm": 0.007945516458569807,
+ "em_norm_stderr": 0.00299286751118699,
+ "em_conservative_norm": 0.4960272417707151,
+ "em_conservative_norm_stderr": 0.016854464514836336
+ },
+ "belebele.prompt1_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.21555555555555556,
+ "em_relaxed_stderr": 0.01371452783236968,
+ "em": 0.02,
+ "em_stderr": 0.004669261421755523,
+ "em_conservative": 0.21555555555555556,
+ "em_conservative_stderr": 0.01371452783236968
+ },
+ "arc.prompt2_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.22412318220701455,
+ "em_relaxed_stderr": 0.012201644195165715,
+ "em": 0.03507271171941831,
+ "em_stderr": 0.005382828530594407,
+ "em_conservative": 0.223267750213858,
+ "em_conservative_stderr": 0.012185048029719049
+ },
+ "multi_wiki_qa.prompt2_qa_sk|0": {
+ "exact_match_slk_prefix": 0.36001600960576347,
+ "exact_match_slk_prefix_stderr": 0.006791008078277957,
+ "f1_slk": 0.5090494926903776,
+ "f1_slk_stderr": 0.005852156127470673
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:management|0": {
+ "em_relaxed": 0.15,
+ "em_relaxed_stderr": 0.035887028128263686,
+ "em": 0.04,
+ "em_stderr": 0.019694638556693237,
+ "em_conservative": 0.15,
+ "em_conservative_stderr": 0.035887028128263686
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0": {
+ "em_relaxed": 0.06,
+ "em_relaxed_stderr": 0.023868325657594173,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.06,
+ "em_conservative_stderr": 0.023868325657594173
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0": {
+ "em_relaxed": 0.08,
+ "em_relaxed_stderr": 0.027265992434429093,
+ "em": 0.04,
+ "em_stderr": 0.019694638556693227,
+ "em_conservative": 0.08,
+ "em_conservative_stderr": 0.027265992434429093
+ },
+ "sklep.prompt3_nli_3_classes_sk:nli|0": {
+ "em_norm_relaxed": 0.001398880895283773,
+ "em_norm_relaxed_stderr": 0.0005284101391342323,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.001398880895283773,
+ "em_conservative_norm_stderr": 0.0005284101391342323
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_medicine|0": {
+ "em_relaxed": 0.1511627906976744,
+ "em_relaxed_stderr": 0.027392810919920886,
+ "em": 0.011627906976744186,
+ "em_stderr": 0.008198095184581854,
+ "em_conservative": 0.1511627906976744,
+ "em_conservative_stderr": 0.027392810919920886
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:business_ethics|0": {
+ "em_relaxed": 0.22,
+ "em_relaxed_stderr": 0.0416333199893227,
+ "em": 0.03,
+ "em_stderr": 0.017144660799776522,
+ "em_conservative": 0.21,
+ "em_conservative_stderr": 0.04093601807403326
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0": {
+ "em_relaxed": 0.15,
+ "em_relaxed_stderr": 0.035887028128263714,
+ "em": 0.03,
+ "em_stderr": 0.01714466079977653,
+ "em_conservative": 0.11,
+ "em_conservative_stderr": 0.031446603773522035
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:international_law|0": {
+ "em_relaxed": 0.15254237288135594,
+ "em_relaxed_stderr": 0.03324002290477697,
+ "em": 0.0423728813559322,
+ "em_stderr": 0.018622984668462274,
+ "em_conservative": 0.15254237288135594,
+ "em_conservative_stderr": 0.03324002290477697
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:international_law|0": {
+ "em_relaxed": 0.13559322033898305,
+ "em_relaxed_stderr": 0.03165083260501198,
+ "em": 0.03389830508474576,
+ "em_stderr": 0.016730444637044904,
+ "em_conservative": 0.13559322033898305,
+ "em_conservative_stderr": 0.03165083260501198
+ },
+ "truthfulqa.prompt3_mqa_single_correct_sk:mc1|0": {
+ "em_relaxed": 0.011568123393316195,
+ "em_relaxed_stderr": 0.00383613885044431,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.011568123393316195,
+ "em_conservative_stderr": 0.00383613885044431
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:prehistory|0": {
+ "em_relaxed": 0.15047021943573669,
+ "em_relaxed_stderr": 0.020049383410796876,
+ "em": 0.04075235109717868,
+ "em_stderr": 0.011087359931149772,
+ "em_conservative": 0.14733542319749215,
+ "em_conservative_stderr": 0.019876006858615635
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:nutrition|0": {
+ "em_relaxed": 0.1848184818481848,
+ "em_relaxed_stderr": 0.022335536574954167,
+ "em": 0.036303630363036306,
+ "em_stderr": 0.010763206393659719,
+ "em_conservative": 0.1848184818481848,
+ "em_conservative_stderr": 0.022335536574954167
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0": {
+ "em_relaxed": 0.09090909090909091,
+ "em_relaxed_stderr": 0.020482086775424218,
+ "em": 0.005050505050505051,
+ "em_stderr": 0.00505050505050505,
+ "em_conservative": 0.08585858585858586,
+ "em_conservative_stderr": 0.01996022556317289
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0": {
+ "em_relaxed": 0.09361702127659574,
+ "em_relaxed_stderr": 0.01904256081095342,
+ "em": 0.03829787234042553,
+ "em_stderr": 0.012545838230269625,
+ "em_conservative": 0.09361702127659574,
+ "em_conservative_stderr": 0.01904256081095342
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0": {
+ "em_relaxed": 0.17,
+ "em_relaxed_stderr": 0.03775251680686371,
+ "em": 0.01,
+ "em_stderr": 0.009999999999999995,
+ "em_conservative": 0.12,
+ "em_conservative_stderr": 0.032659863237109066
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_medicine|0": {
+ "em_relaxed": 0.1569767441860465,
+ "em_relaxed_stderr": 0.027818863343493188,
+ "em": 0.040697674418604654,
+ "em_stderr": 0.01511000089316197,
+ "em_conservative": 0.1569767441860465,
+ "em_conservative_stderr": 0.027818863343493188
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_medicine|0": {
+ "em_relaxed": 0.20930232558139536,
+ "em_relaxed_stderr": 0.031109583909764628,
+ "em": 0.06395348837209303,
+ "em_stderr": 0.018710387348210598,
+ "em_conservative": 0.20930232558139536,
+ "em_conservative_stderr": 0.031109583909764628
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0": {
+ "em_relaxed": 0.18470149253731344,
+ "em_relaxed_stderr": 0.016777090030491517,
+ "em": 0.03731343283582089,
+ "em_stderr": 0.008194041665419557,
+ "em_conservative": 0.18470149253731344,
+ "em_conservative_stderr": 0.016777090030491517
+ },
+ "sklep.prompt2_nli_binary_sk:rte|0": {
+ "em_norm_relaxed": 0.3301204819277108,
+ "em_norm_relaxed_stderr": 0.011545464009906971,
+ "em_norm": 0.0018072289156626507,
+ "em_norm_stderr": 0.001042774975713993,
+ "em_conservative_norm": 0.3301204819277108,
+ "em_conservative_norm_stderr": 0.011545464009906971
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "em_relaxed": 0.14285714285714285,
+ "em_relaxed_stderr": 0.0238095238095238,
+ "em": 0.004608294930875576,
+ "em_stderr": 0.004608294930875571,
+ "em_conservative": 0.14285714285714285,
+ "em_conservative_stderr": 0.0238095238095238
+ },
+ "sklep.prompt2_hate_speech_sk:hate-speech|0": {
+ "em_norm_relaxed": 0.7247915087187263,
+ "em_norm_relaxed_stderr": 0.012302114305862654,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.7247915087187263,
+ "em_conservative_norm_stderr": 0.012302114305862654
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0": {
+ "em_relaxed": 0.13761467889908258,
+ "em_relaxed_stderr": 0.03314905716923366,
+ "em": 0.045871559633027525,
+ "em_stderr": 0.020130904850286974,
+ "em_conservative": 0.13761467889908258,
+ "em_conservative_stderr": 0.03314905716923366
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0": {
+ "em_relaxed": 0.22388059701492538,
+ "em_relaxed_stderr": 0.01802169384796668,
+ "em": 0.04477611940298507,
+ "em_stderr": 0.008941264053122127,
+ "em_conservative": 0.22388059701492538,
+ "em_conservative_stderr": 0.01802169384796668
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0": {
+ "em_relaxed": 0.19298245614035087,
+ "em_relaxed_stderr": 0.05273590328947574,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.19298245614035087,
+ "em_conservative_stderr": 0.05273590328947574
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:marketing|0": {
+ "em_relaxed": 0.1623931623931624,
+ "em_relaxed_stderr": 0.02416161812798774,
+ "em": 0.05555555555555555,
+ "em_stderr": 0.01500631280644691,
+ "em_conservative": 0.1623931623931624,
+ "em_conservative_stderr": 0.02416161812798774
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0": {
+ "em_relaxed": 0.17721518987341772,
+ "em_relaxed_stderr": 0.030475004517281346,
+ "em": 0.0379746835443038,
+ "em_stderr": 0.015254237576434418,
+ "em_conservative": 0.17721518987341772,
+ "em_conservative_stderr": 0.030475004517281346
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0": {
+ "em_relaxed": 0.14042553191489363,
+ "em_relaxed_stderr": 0.022712077616627854,
+ "em": 0.0851063829787234,
+ "em_stderr": 0.018241411344572286,
+ "em_conservative": 0.14042553191489363,
+ "em_conservative_stderr": 0.022712077616627854
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:virology|0": {
+ "em_relaxed": 0.16463414634146342,
+ "em_relaxed_stderr": 0.029047216079384597,
+ "em": 0.03048780487804878,
+ "em_stderr": 0.013466226472535378,
+ "em_conservative": 0.16463414634146342,
+ "em_conservative_stderr": 0.029047216079384597
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0": {
+ "em_relaxed": 0.14,
+ "em_relaxed_stderr": 0.034873508801977704,
+ "em": 0.01,
+ "em_stderr": 0.009999999999999997,
+ "em_conservative": 0.14,
+ "em_conservative_stderr": 0.034873508801977704
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0": {
+ "em_relaxed": 0.006675567423230975,
+ "em_relaxed_stderr": 0.0029774114441068625,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.0053404539385847796,
+ "em_conservative_stderr": 0.0026648668562206292
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0": {
+ "em_relaxed": 0.25252525252525254,
+ "em_relaxed_stderr": 0.043887175543277195,
+ "em": 0.030303030303030304,
+ "em_stderr": 0.01731601731601731,
+ "em_conservative": 0.24242424242424243,
+ "em_conservative_stderr": 0.04329004329004326
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:world_religions|0": {
+ "em_relaxed": 0.05263157894736842,
+ "em_relaxed_stderr": 0.017126088775157082,
+ "em": 0.023391812865497075,
+ "em_stderr": 0.01159224790573493,
+ "em_conservative": 0.05263157894736842,
+ "em_conservative_stderr": 0.017126088775157082
+ },
+ "senti_sk.prompt3_sentiment_binary_sk|0": {
+ "em_norm_relaxed": 0.25160944206008584,
+ "em_norm_relaxed_stderr": 0.0071080082101389235,
+ "em_norm": 0.03165236051502146,
+ "em_norm_stderr": 0.0028677349189843607,
+ "em_conservative_norm": 0.1990343347639485,
+ "em_conservative_norm_stderr": 0.0065402033880141534
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0": {
+ "em_relaxed": 0.13333333333333333,
+ "em_relaxed_stderr": 0.027848530325897826,
+ "em": 0.02666666666666667,
+ "em_stderr": 0.013198422611557032,
+ "em_conservative": 0.13333333333333333,
+ "em_conservative_stderr": 0.027848530325897826
+ },
+ "sklep.prompt1_nli_3_classes_sk:nli|0": {
+ "em_norm_relaxed": 0.016786570743405275,
+ "em_norm_relaxed_stderr": 0.0018163086111764766,
+ "em_norm": 0.0003996802557953637,
+ "em_norm_stderr": 0.00028258837305266234,
+ "em_conservative_norm": 0.016386890487609912,
+ "em_conservative_norm_stderr": 0.0017949203375471519
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "em_relaxed": 0.12919896640826872,
+ "em_relaxed_stderr": 0.017072429785340966,
+ "em": 0.012919896640826873,
+ "em_stderr": 0.005747937884708704,
+ "em_conservative": 0.12661498708010335,
+ "em_conservative_stderr": 0.016925900082204414
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0": {
+ "em_relaxed": 0.01335113484646195,
+ "em_relaxed_stderr": 0.004196522950335422,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.0053404539385847796,
+ "em_conservative_stderr": 0.0026648668562206297
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0": {
+ "em_relaxed": 0.11616161616161616,
+ "em_relaxed_stderr": 0.022828881775249377,
+ "em": 0.020202020202020204,
+ "em_stderr": 0.01002380379631872,
+ "em_conservative": 0.1111111111111111,
+ "em_conservative_stderr": 0.022390787638216763
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "em_relaxed": 0.32124352331606215,
+ "em_relaxed_stderr": 0.033699508685490674,
+ "em": 0.046632124352331605,
+ "em_stderr": 0.015216761819262577,
+ "em_conservative": 0.32124352331606215,
+ "em_conservative_stderr": 0.033699508685490674
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:business_ethics|0": {
+ "em_relaxed": 0.13,
+ "em_relaxed_stderr": 0.03379976689896308,
+ "em": 0.01,
+ "em_stderr": 0.009999999999999998,
+ "em_conservative": 0.13,
+ "em_conservative_stderr": 0.03379976689896308
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:public_relations|0": {
+ "em_relaxed": 0.23853211009174313,
+ "em_relaxed_stderr": 0.041009771029340336,
+ "em": 0.09174311926605505,
+ "em_stderr": 0.02777660875318803,
+ "em_conservative": 0.23853211009174313,
+ "em_conservative_stderr": 0.041009771029340336
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0": {
+ "em_relaxed": 0.16355140186915887,
+ "em_relaxed_stderr": 0.025342938086817392,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.1542056074766355,
+ "em_conservative_stderr": 0.024745297661074854
+ },
+ "senti_sk.prompt2_sentiment_binary_sk|0": {
+ "em_norm_relaxed": 0.4965128755364807,
+ "em_norm_relaxed_stderr": 0.008189921623320459,
+ "em_norm": 0.0037553648068669528,
+ "em_norm_stderr": 0.001001911513818721,
+ "em_conservative_norm": 0.49302575107296137,
+ "em_conservative_norm_stderr": 0.008189324033951761
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0": {
+ "em_relaxed": 0.18,
+ "em_relaxed_stderr": 0.03861229196653694,
+ "em": 0.07,
+ "em_stderr": 0.02564323999762429,
+ "em_conservative": 0.18,
+ "em_conservative_stderr": 0.03861229196653694
+ },
+ "belebele.prompt2_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.2111111111111111,
+ "em_relaxed_stderr": 0.013610798969328482,
+ "em": 0.016666666666666666,
+ "em_stderr": 0.004269675894690937,
+ "em_conservative": 0.2111111111111111,
+ "em_conservative_stderr": 0.013610798969328482
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:virology|0": {
+ "em_relaxed": 0.12195121951219512,
+ "em_relaxed_stderr": 0.025630589418441155,
+ "em": 0.03048780487804878,
+ "em_stderr": 0.013466226472535378,
+ "em_conservative": 0.12195121951219512,
+ "em_conservative_stderr": 0.025630589418441155
+ },
+ "sklep.prompt2_nli_3_classes_sk:nli|0": {
+ "em_norm_relaxed": 0.09972022382094324,
+ "em_norm_relaxed_stderr": 0.004236089409386789,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.09952038369304557,
+ "em_conservative_norm_stderr": 0.004232312358957502
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0": {
+ "em_relaxed": 0.24444444444444444,
+ "em_relaxed_stderr": 0.03712537833614866,
+ "em": 0.014814814814814815,
+ "em_stderr": 0.010436494549594362,
+ "em_conservative": 0.24444444444444444,
+ "em_conservative_stderr": 0.03712537833614866
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "em_relaxed": 0.1731266149870801,
+ "em_relaxed_stderr": 0.01925784882518206,
+ "em": 0.031007751937984496,
+ "em_stderr": 0.00882270258926331,
+ "em_conservative": 0.17054263565891473,
+ "em_conservative_stderr": 0.019143434938568855
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0": {
+ "em_relaxed": 0.06766917293233082,
+ "em_relaxed_stderr": 0.015429701322264242,
+ "em": 0.011278195488721804,
+ "em_stderr": 0.0064868510822409155,
+ "em_conservative": 0.06766917293233082,
+ "em_conservative_stderr": 0.015429701322264242
+ },
+ "sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0": {
+ "em_norm_relaxed": 0.15355086372360843,
+ "em_norm_relaxed_stderr": 0.011173807886451477,
+ "em_norm": 0.015355086372360844,
+ "em_norm_stderr": 0.0038110143844746576,
+ "em_conservative_norm": 0.118042226487524,
+ "em_conservative_norm_stderr": 0.010000396690181516
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0": {
+ "em_relaxed": 0.1276595744680851,
+ "em_relaxed_stderr": 0.021815328994143183,
+ "em": 0.05106382978723404,
+ "em_stderr": 0.014390214367756187,
+ "em_conservative": 0.1276595744680851,
+ "em_conservative_stderr": 0.021815328994143183
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0": {
+ "em_relaxed": 0.12121212121212122,
+ "em_relaxed_stderr": 0.03296871474399962,
+ "em": 0.030303030303030304,
+ "em_stderr": 0.01731601731601732,
+ "em_conservative": 0.12121212121212122,
+ "em_conservative_stderr": 0.03296871474399962
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0": {
+ "em_relaxed": 0.20245398773006135,
+ "em_relaxed_stderr": 0.022289450459613447,
+ "em": 0.06134969325153374,
+ "em_stderr": 0.01331117614747688,
+ "em_conservative": 0.20245398773006135,
+ "em_conservative_stderr": 0.022289450459613447
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:security_studies|0": {
+ "em_relaxed": 0.1981981981981982,
+ "em_relaxed_stderr": 0.026815582053277207,
+ "em": 0.04504504504504504,
+ "em_stderr": 0.01395143304433832,
+ "em_conservative": 0.1981981981981982,
+ "em_conservative_stderr": 0.026815582053277207
+ },
+ "skj1.prompt1_skj1_sk|0": {
+ "em_relaxed": 0.02,
+ "em_relaxed_stderr": 0.014070529413628957,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.02,
+ "em_conservative_stderr": 0.014070529413628957
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_physics|0": {
+ "em_relaxed": 0.0784313725490196,
+ "em_relaxed_stderr": 0.026751495257385273,
+ "em": 0.00980392156862745,
+ "em_stderr": 0.009803921568627454,
+ "em_conservative": 0.0784313725490196,
+ "em_conservative_stderr": 0.026751495257385273
+ },
+ "sklep.prompt3_hate_speech_sk:hate-speech|0": {
+ "em_norm_relaxed": 0.6967399545109931,
+ "em_norm_relaxed_stderr": 0.012661502663418697,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.6967399545109931,
+ "em_conservative_norm_stderr": 0.012661502663418697
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0": {
+ "em_relaxed": 0.03759398496240601,
+ "em_relaxed_stderr": 0.011684641953170118,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.03759398496240601,
+ "em_conservative_stderr": 0.011684641953170118
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:virology|0": {
+ "em_relaxed": 0.1402439024390244,
+ "em_relaxed_stderr": 0.027197928271392544,
+ "em": 0.024390243902439025,
+ "em_stderr": 0.012082375722391943,
+ "em_conservative": 0.1402439024390244,
+ "em_conservative_stderr": 0.027197928271392544
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0": {
+ "em_relaxed": 0.05639097744360902,
+ "em_relaxed_stderr": 0.014170263190664145,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.05639097744360902,
+ "em_conservative_stderr": 0.014170263190664145
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:security_studies|0": {
+ "em_relaxed": 0.21621621621621623,
+ "em_relaxed_stderr": 0.02769147769493752,
+ "em": 0.04504504504504504,
+ "em_stderr": 0.013951433044338322,
+ "em_conservative": 0.21621621621621623,
+ "em_conservative_stderr": 0.02769147769493752
+ },
+ "financial_exams.prompt3_completion_sk|0": {
+ "em_relaxed": 0.24887556221889057,
+ "em_relaxed_stderr": 0.01184218927916494,
+ "em": 0.0037481259370314842,
+ "em_stderr": 0.0016736960391157028,
+ "em_conservative": 0.24887556221889057,
+ "em_conservative_stderr": 0.01184218927916494
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0": {
+ "em_relaxed": 0.18691588785046728,
+ "em_relaxed_stderr": 0.03786499587281739,
+ "em": 0.037383177570093455,
+ "em_stderr": 0.01842518672106723,
+ "em_conservative": 0.18691588785046728,
+ "em_conservative_stderr": 0.03786499587281739
+ },
+ "senti_sk.prompt3_sentiment_3_class_sk|0": {
+ "em_norm_relaxed": 0.40880906564036773,
+ "em_norm_relaxed_stderr": 0.0071893047031815266,
+ "em_norm": 0.10540945050245884,
+ "em_norm_stderr": 0.004490706548553942,
+ "em_conservative_norm": 0.26448578148385715,
+ "em_conservative_norm_stderr": 0.006449996068936491
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0": {
+ "em_relaxed": 0.1310344827586207,
+ "em_relaxed_stderr": 0.028119838672931184,
+ "em": 0.06896551724137931,
+ "em_stderr": 0.021116290886061897,
+ "em_conservative": 0.1310344827586207,
+ "em_conservative_stderr": 0.028119838672931184
+ },
+ "sklep.prompt3_qa_sk:question-answering|0": {
+ "exact_match_slk_prefix": 0.16143170197224252,
+ "exact_match_slk_prefix_stderr": 0.0037586816998114083,
+ "f1_slk": 0.3769392643579436,
+ "f1_slk_stderr": 0.003773157275405368
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0": {
+ "em_relaxed": 0.19630872483221476,
+ "em_relaxed_stderr": 0.016283806947934177,
+ "em": 0.018456375838926176,
+ "em_stderr": 0.005517845461546564,
+ "em_conservative": 0.19630872483221476,
+ "em_conservative_stderr": 0.016283806947934177
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:global_facts|0": {
+ "em_relaxed": 0.12,
+ "em_relaxed_stderr": 0.03265986323710906,
+ "em": 0.01,
+ "em_stderr": 0.009999999999999997,
+ "em_conservative": 0.12,
+ "em_conservative_stderr": 0.03265986323710906
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:computer_security|0": {
+ "em_relaxed": 0.11,
+ "em_relaxed_stderr": 0.03144660377352203,
+ "em": 0.02,
+ "em_stderr": 0.014070529413628963,
+ "em_conservative": 0.11,
+ "em_conservative_stderr": 0.03144660377352203
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:computer_security|0": {
+ "em_relaxed": 0.08,
+ "em_relaxed_stderr": 0.027265992434429082,
+ "em": 0.02,
+ "em_stderr": 0.014070529413628966,
+ "em_conservative": 0.08,
+ "em_conservative_stderr": 0.027265992434429082
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_physics|0": {
+ "em_relaxed": 0.12745098039215685,
+ "em_relaxed_stderr": 0.03318224921942078,
+ "em": 0.0392156862745098,
+ "em_stderr": 0.01931444240436096,
+ "em_conservative": 0.12745098039215685,
+ "em_conservative_stderr": 0.03318224921942078
+ },
+ "globalmmlu-lite.prompt2_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.21,
+ "em_relaxed_stderr": 0.020390916450326858,
+ "em": 0.035,
+ "em_stderr": 0.009200495791370004,
+ "em_conservative": 0.21,
+ "em_conservative_stderr": 0.020390916450326858
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0": {
+ "em_relaxed": 0.26666666666666666,
+ "em_relaxed_stderr": 0.03305282343736876,
+ "em": 0.005555555555555556,
+ "em_stderr": 0.005555555555555554,
+ "em_conservative": 0.26666666666666666,
+ "em_conservative_stderr": 0.03305282343736876
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:anatomy|0": {
+ "em_relaxed": 0.13432835820895522,
+ "em_relaxed_stderr": 0.029568863219678807,
+ "em": 0.029850746268656716,
+ "em_stderr": 0.01475608176283378,
+ "em_conservative": 0.13432835820895522,
+ "em_conservative_stderr": 0.029568863219678807
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:security_studies|0": {
+ "em_relaxed": 0.15765765765765766,
+ "em_relaxed_stderr": 0.024513522045494097,
+ "em": 0.04954954954954955,
+ "em_stderr": 0.014597835283794243,
+ "em_conservative": 0.15765765765765766,
+ "em_conservative_stderr": 0.024513522045494097
+ },
+ "movie_review.prompt3_sentiment_binary_sk|0": {
+ "em_norm_relaxed": 0.11105577689243028,
+ "em_norm_relaxed_stderr": 0.007013489433691037,
+ "em_norm": 0.04731075697211155,
+ "em_norm_stderr": 0.004738947450976647,
+ "em_conservative_norm": 0.09412350597609562,
+ "em_conservative_norm_stderr": 0.006517932474820501
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0": {
+ "em_relaxed": 0.16,
+ "em_relaxed_stderr": 0.03684529491774709,
+ "em": 0.04,
+ "em_stderr": 0.01969463855669323,
+ "em_conservative": 0.16,
+ "em_conservative_stderr": 0.03684529491774709
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:astronomy|0": {
+ "em_relaxed": 0.17763157894736842,
+ "em_relaxed_stderr": 0.031103182383123398,
+ "em": 0.05263157894736842,
+ "em_stderr": 0.01817164224152954,
+ "em_conservative": 0.17105263157894737,
+ "em_conservative_stderr": 0.03064360707167709
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0": {
+ "em_relaxed": 0.18,
+ "em_relaxed_stderr": 0.038612291966536955,
+ "em": 0.04,
+ "em_stderr": 0.019694638556693223,
+ "em_conservative": 0.18,
+ "em_conservative_stderr": 0.038612291966536955
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0": {
+ "em_relaxed": 0.05,
+ "em_relaxed_stderr": 0.021904291355759026,
+ "em": 0.01,
+ "em_stderr": 0.009999999999999997,
+ "em_conservative": 0.04,
+ "em_conservative_stderr": 0.019694638556693237
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0": {
+ "em_relaxed": 0.0896551724137931,
+ "em_relaxed_stderr": 0.023807258824376967,
+ "em": 0.041379310344827586,
+ "em_stderr": 0.016597159859992713,
+ "em_conservative": 0.0896551724137931,
+ "em_conservative_stderr": 0.023807258824376967
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0": {
+ "em_relaxed": 0.2777777777777778,
+ "em_relaxed_stderr": 0.033477857599386325,
+ "em": 0.005555555555555556,
+ "em_stderr": 0.005555555555555546,
+ "em_conservative": 0.2777777777777778,
+ "em_conservative_stderr": 0.033477857599386325
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:philosophy|0": {
+ "em_relaxed": 0.237012987012987,
+ "em_relaxed_stderr": 0.024270317456654856,
+ "em": 0.05194805194805195,
+ "em_stderr": 0.01266576753511598,
+ "em_conservative": 0.237012987012987,
+ "em_conservative_stderr": 0.024270317456654856
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0": {
+ "em_relaxed": 0.21100917431192662,
+ "em_relaxed_stderr": 0.039262212380368526,
+ "em": 0.05504587155963303,
+ "em_stderr": 0.0219460246091855,
+ "em_conservative": 0.21100917431192662,
+ "em_conservative_stderr": 0.039262212380368526
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0": {
+ "em_relaxed": 0.01,
+ "em_relaxed_stderr": 0.009999999999999997,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.01,
+ "em_conservative_stderr": 0.009999999999999997
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:world_religions|0": {
+ "em_relaxed": 0.10526315789473684,
+ "em_relaxed_stderr": 0.02353755765789256,
+ "em": 0.04678362573099415,
+ "em_stderr": 0.01619638995623562,
+ "em_conservative": 0.10526315789473684,
+ "em_conservative_stderr": 0.02353755765789256
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0": {
+ "em_relaxed": 0.24324324324324326,
+ "em_relaxed_stderr": 0.026710910573068247,
+ "em": 0.04633204633204633,
+ "em_stderr": 0.013086679986931563,
+ "em_conservative": 0.23938223938223938,
+ "em_conservative_stderr": 0.026565582359277095
+ },
+ "toxic_sk.prompt1_toxicity_sk|0": {
+ "em_norm_relaxed": 0.45629965947786605,
+ "em_norm_relaxed_stderr": 0.016790496508382532,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.45629965947786605,
+ "em_conservative_norm_stderr": 0.016790496508382532
+ },
+ "senti_sk.prompt1_sentiment_3_class_sk|0": {
+ "em_norm_relaxed": 0.3200769724182168,
+ "em_norm_relaxed_stderr": 0.0068221276139399335,
+ "em_norm": 0.22642719692110327,
+ "em_norm_stderr": 0.006120369470744044,
+ "em_conservative_norm": 0.31922172332691895,
+ "em_conservative_norm_stderr": 0.006817290681927466
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_biology|0": {
+ "em_relaxed": 0.19014084507042253,
+ "em_relaxed_stderr": 0.03304706552420638,
+ "em": 0.04225352112676056,
+ "em_stderr": 0.01694131738100396,
+ "em_conservative": 0.19014084507042253,
+ "em_conservative_stderr": 0.03304706552420638
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:public_relations|0": {
+ "em_relaxed": 0.21100917431192662,
+ "em_relaxed_stderr": 0.03926221238036853,
+ "em": 0.07339449541284404,
+ "em_stderr": 0.025093850407933094,
+ "em_conservative": 0.21100917431192662,
+ "em_conservative_stderr": 0.03926221238036853
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0": {
+ "em_relaxed": 0.1969111969111969,
+ "em_relaxed_stderr": 0.024757519994692216,
+ "em": 0.03088803088803089,
+ "em_stderr": 0.010771402065873048,
+ "em_conservative": 0.1969111969111969,
+ "em_conservative_stderr": 0.024757519994692216
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:formal_logic|0": {
+ "em_relaxed": 0.13043478260869565,
+ "em_relaxed_stderr": 0.03154244565695702,
+ "em": 0.008695652173913044,
+ "em_stderr": 0.008695652173913031,
+ "em_conservative": 0.08695652173913043,
+ "em_conservative_stderr": 0.026390303420299282
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0": {
+ "em_relaxed": 0.20202020202020202,
+ "em_relaxed_stderr": 0.028606204289229872,
+ "em": 0.07575757575757576,
+ "em_stderr": 0.018852670234993107,
+ "em_conservative": 0.20202020202020202,
+ "em_conservative_stderr": 0.028606204289229872
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:prehistory|0": {
+ "em_relaxed": 0.15987460815047022,
+ "em_relaxed_stderr": 0.02055172319966096,
+ "em": 0.04075235109717868,
+ "em_stderr": 0.011087359931149773,
+ "em_conservative": 0.15987460815047022,
+ "em_conservative_stderr": 0.02055172319966096
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:econometrics|0": {
+ "em_relaxed": 0.125,
+ "em_relaxed_stderr": 0.03139045014587016,
+ "em": 0.026785714285714284,
+ "em_stderr": 0.015324784082343043,
+ "em_conservative": 0.125,
+ "em_conservative_stderr": 0.03139045014587016
+ },
+ "arc.prompt1_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.1710863986313088,
+ "em_relaxed_stderr": 0.011018968811092407,
+ "em": 0.013686911890504704,
+ "em_stderr": 0.0033996852610928244,
+ "em_conservative": 0.1710863986313088,
+ "em_conservative_stderr": 0.011018968811092407
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:business_ethics|0": {
+ "em_relaxed": 0.14,
+ "em_relaxed_stderr": 0.03487350880197772,
+ "em": 0.02,
+ "em_stderr": 0.014070529413628957,
+ "em_conservative": 0.14,
+ "em_conservative_stderr": 0.03487350880197772
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0": {
+ "em_relaxed": 0.14018691588785046,
+ "em_relaxed_stderr": 0.03372118539369118,
+ "em": 0.056074766355140186,
+ "em_stderr": 0.02234599008672909,
+ "em_conservative": 0.14018691588785046,
+ "em_conservative_stderr": 0.03372118539369118
+ },
+ "truthfulqa.prompt2_mqa_single_correct_sk:mc1|0": {
+ "em_relaxed": 0.016709511568123392,
+ "em_relaxed_stderr": 0.004598458668423289,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.016709511568123392,
+ "em_conservative_stderr": 0.004598458668423289
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0": {
+ "em_relaxed": 0.14505776636713735,
+ "em_relaxed_stderr": 0.01262551435912769,
+ "em": 0.03209242618741977,
+ "em_stderr": 0.006318709962464618,
+ "em_conservative": 0.14505776636713735,
+ "em_conservative_stderr": 0.01262551435912769
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:astronomy|0": {
+ "em_relaxed": 0.14473684210526316,
+ "em_relaxed_stderr": 0.028631951845930412,
+ "em": 0.03289473684210526,
+ "em_stderr": 0.014514818445457586,
+ "em_conservative": 0.13815789473684212,
+ "em_conservative_stderr": 0.028081042939576552
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0": {
+ "em_relaxed": 0.21481481481481482,
+ "em_relaxed_stderr": 0.03547854198560824,
+ "em": 0.02962962962962963,
+ "em_stderr": 0.014648038602753806,
+ "em_conservative": 0.2074074074074074,
+ "em_conservative_stderr": 0.03502553170678318
+ },
+ "senti_sk.prompt2_sentiment_3_class_sk|0": {
+ "em_norm_relaxed": 0.383365405174257,
+ "em_norm_relaxed_stderr": 0.007110221850748201,
+ "em_norm": 0.019029292281376953,
+ "em_norm_stderr": 0.001998030377272694,
+ "em_conservative_norm": 0.37716484926234767,
+ "em_conservative_norm_stderr": 0.007087856422819165
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0": {
+ "em_relaxed": 0.23703703703703705,
+ "em_relaxed_stderr": 0.03673731683969506,
+ "em": 0.022222222222222223,
+ "em_stderr": 0.012733899715059683,
+ "em_conservative": 0.23703703703703705,
+ "em_conservative_stderr": 0.03673731683969506
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:formal_logic|0": {
+ "em_relaxed": 0.11304347826086956,
+ "em_relaxed_stderr": 0.029656609316837397,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.06956521739130435,
+ "em_conservative_stderr": 0.02382794649967283
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "em_relaxed": 0.30569948186528495,
+ "em_relaxed_stderr": 0.033248379397581594,
+ "em": 0.046632124352331605,
+ "em_stderr": 0.015216761819262604,
+ "em_conservative": 0.3005181347150259,
+ "em_conservative_stderr": 0.0330881859441575
+ },
+ "sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0": {
+ "em_norm_relaxed": 0.3512476007677543,
+ "em_norm_relaxed_stderr": 0.014795198864354053,
+ "em_norm": 0.0009596928982725527,
+ "em_norm_stderr": 0.0009596928982725573,
+ "em_conservative_norm": 0.34836852207293667,
+ "em_conservative_norm_stderr": 0.014767096602802959
+ },
+ "sklep.prompt1_sentiment_binary_sk:sentiment-analysis|0": {
+ "em_norm_relaxed": 0.8809980806142035,
+ "em_norm_relaxed_stderr": 0.010035501945449067,
+ "em_norm": 0.44817658349328215,
+ "em_norm_stderr": 0.015413429449614385,
+ "em_conservative_norm": 0.8800383877159309,
+ "em_conservative_norm_stderr": 0.010070396971355563
+ },
+ "skj1.prompt2_skj1_sk|0": {
+ "em_relaxed": 0.06,
+ "em_relaxed_stderr": 0.023868325657594166,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.05,
+ "em_conservative_stderr": 0.021904291355759067
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:international_law|0": {
+ "em_relaxed": 0.16101694915254236,
+ "em_relaxed_stderr": 0.03397969368949715,
+ "em": 0.05084745762711865,
+ "em_stderr": 0.020309989475094187,
+ "em_conservative": 0.16101694915254236,
+ "em_conservative_stderr": 0.03397969368949715
+ },
+ "sklep.prompt1_nli_binary_sk:rte|0": {
+ "em_norm_relaxed": 0.030120481927710843,
+ "em_norm_relaxed_stderr": 0.004196301257202765,
+ "em_norm": 0.0006024096385542169,
+ "em_norm_stderr": 0.0006024096385542043,
+ "em_conservative_norm": 0.030120481927710843,
+ "em_conservative_norm_stderr": 0.004196301257202765
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:human_aging|0": {
+ "em_relaxed": 0.1278538812785388,
+ "em_relaxed_stderr": 0.02261639276396452,
+ "em": 0.0684931506849315,
+ "em_stderr": 0.017107571020219335,
+ "em_conservative": 0.1278538812785388,
+ "em_conservative_stderr": 0.02261639276396452
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:management|0": {
+ "em_relaxed": 0.2,
+ "em_relaxed_stderr": 0.040201512610368445,
+ "em": 0.06,
+ "em_stderr": 0.023868325657594162,
+ "em_conservative": 0.2,
+ "em_conservative_stderr": 0.040201512610368445
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_accounting|0": {
+ "em_relaxed": 0.15217391304347827,
+ "em_relaxed_stderr": 0.02165993479655374,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.14130434782608695,
+ "em_conservative_stderr": 0.02100540442245809
+ },
+ "truthfulqa.prompt1_mqa_single_correct_sk:mc1|0": {
+ "em_relaxed": 0.007712082262210797,
+ "em_relaxed_stderr": 0.003138297919968842,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.007712082262210797,
+ "em_conservative_stderr": 0.003138297919968842
+ },
+ "senti_sk.prompt1_sentiment_binary_sk|0": {
+ "em_norm_relaxed": 0.31223175965665234,
+ "em_norm_relaxed_stderr": 0.0075906663188793026,
+ "em_norm": 0.1968884120171674,
+ "em_norm_stderr": 0.006513558648330005,
+ "em_conservative_norm": 0.3111587982832618,
+ "em_conservative_norm_stderr": 0.007583521183648285
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:astronomy|0": {
+ "em_relaxed": 0.1513157894736842,
+ "em_relaxed_stderr": 0.029162631596843985,
+ "em": 0.019736842105263157,
+ "em_stderr": 0.011319355505772411,
+ "em_conservative": 0.14473684210526316,
+ "em_conservative_stderr": 0.028631951845930412
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:human_aging|0": {
+ "em_relaxed": 0.1735159817351598,
+ "em_relaxed_stderr": 0.02564830822412722,
+ "em": 0.0867579908675799,
+ "em_stderr": 0.01906423736413682,
+ "em_conservative": 0.1735159817351598,
+ "em_conservative_stderr": 0.02564830822412722
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:formal_logic|0": {
+ "em_relaxed": 0.12173913043478261,
+ "em_relaxed_stderr": 0.030624883036764184,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.0782608695652174,
+ "em_conservative_stderr": 0.025154976812194616
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:sociology|0": {
+ "em_relaxed": 0.3350253807106599,
+ "em_relaxed_stderr": 0.03371423842235921,
+ "em": 0.05583756345177665,
+ "em_stderr": 0.016400555297895067,
+ "em_conservative": 0.3350253807106599,
+ "em_conservative_stderr": 0.03371423842235921
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:econometrics|0": {
+ "em_relaxed": 0.11607142857142858,
+ "em_relaxed_stderr": 0.03040253249894973,
+ "em": 0.017857142857142856,
+ "em_stderr": 0.012569900164490047,
+ "em_conservative": 0.11607142857142858,
+ "em_conservative_stderr": 0.03040253249894973
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:elementary_mathematics|0": {
+ "em_relaxed": 0.08847184986595175,
+ "em_relaxed_stderr": 0.014723671750091158,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.08579088471849866,
+ "em_conservative_stderr": 0.01452017587959949
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_mathematics|0": {
+ "em_relaxed": 0.06060606060606061,
+ "em_relaxed_stderr": 0.024102876029567216,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.050505050505050504,
+ "em_conservative_stderr": 0.02212079324207198
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_psychology|0": {
+ "em_relaxed": 0.20134228187919462,
+ "em_relaxed_stderr": 0.016439528475054862,
+ "em": 0.03355704697986577,
+ "em_stderr": 0.0073828066874755535,
+ "em_conservative": 0.20134228187919462,
+ "em_conservative_stderr": 0.016439528475054862
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_mathematics|0": {
+ "em_relaxed": 0.09090909090909091,
+ "em_relaxed_stderr": 0.02903984386363362,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.08080808080808081,
+ "em_conservative_stderr": 0.027530712883605948
+ },
+ "globalmmlu-lite.prompt3_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.2025,
+ "em_relaxed_stderr": 0.02011830610666578,
+ "em": 0.025,
+ "em_stderr": 0.007816023641463063,
+ "em_conservative": 0.2025,
+ "em_conservative_stderr": 0.02011830610666578
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "em_relaxed": 0.15762273901808785,
+ "em_relaxed_stderr": 0.018546804758294254,
+ "em": 0.01808785529715762,
+ "em_stderr": 0.006783224657656192,
+ "em_conservative": 0.14987080103359174,
+ "em_conservative_stderr": 0.018168008667456638
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "em_relaxed": 0.17050691244239632,
+ "em_relaxed_stderr": 0.025588834483805927,
+ "em": 0.013824884792626729,
+ "em_stderr": 0.007944762237164815,
+ "em_conservative": 0.16589861751152074,
+ "em_conservative_stderr": 0.025310686717794278
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:human_sexuality|0": {
+ "em_relaxed": 0.1559633027522936,
+ "em_relaxed_stderr": 0.03491242362801975,
+ "em": 0.05504587155963303,
+ "em_stderr": 0.021946024609185506,
+ "em_conservative": 0.1559633027522936,
+ "em_conservative_stderr": 0.03491242362801975
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:computer_security|0": {
+ "em_relaxed": 0.13,
+ "em_relaxed_stderr": 0.03379976689896309,
+ "em": 0.02,
+ "em_stderr": 0.014070529413628963,
+ "em_conservative": 0.13,
+ "em_conservative_stderr": 0.03379976689896309
+ },
+ "toxic_sk.prompt2_toxicity_sk|0": {
+ "em_norm_relaxed": 0.47559591373439275,
+ "em_norm_relaxed_stderr": 0.016834908286515442,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.47559591373439275,
+ "em_conservative_norm_stderr": 0.016834908286515442
+ },
+ "multi_wiki_qa.prompt3_qa_sk|0": {
+ "exact_match_slk_prefix": 0.34000400240144085,
+ "exact_match_slk_prefix_stderr": 0.006701954125609285,
+ "f1_slk": 0.5050071786470199,
+ "f1_slk_stderr": 0.005797838571943781
+ },
+ "financial_exams.prompt1_completion_sk|0": {
+ "em_relaxed": 0.20089955022488756,
+ "em_relaxed_stderr": 0.010974256206370726,
+ "em": 0.009745127436281859,
+ "em_stderr": 0.002690618851324012,
+ "em_conservative": 0.20089955022488756,
+ "em_conservative_stderr": 0.010974256206370726
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:philosophy|0": {
+ "em_relaxed": 0.21103896103896103,
+ "em_relaxed_stderr": 0.023288413887090093,
+ "em": 0.06818181818181818,
+ "em_stderr": 0.014385689805382164,
+ "em_conservative": 0.21103896103896103,
+ "em_conservative_stderr": 0.023288413887090093
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:moral_disputes|0": {
+ "em_relaxed": 0.2147239263803681,
+ "em_relaxed_stderr": 0.022777692313730433,
+ "em": 0.06748466257668712,
+ "em_stderr": 0.013915180741090785,
+ "em_conservative": 0.2116564417177914,
+ "em_conservative_stderr": 0.02265853497858724
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:marketing|0": {
+ "em_relaxed": 0.15384615384615385,
+ "em_relaxed_stderr": 0.023636873317489277,
+ "em": 0.03418803418803419,
+ "em_stderr": 0.011904341997629792,
+ "em_conservative": 0.15384615384615385,
+ "em_conservative_stderr": 0.023636873317489277
+ },
+ "globalmmlu-lite.prompt1_mqa_single_correct_sk|0": {
+ "em_relaxed": 0.1825,
+ "em_relaxed_stderr": 0.01933698975881026,
+ "em": 0.025,
+ "em_stderr": 0.007816023641463065,
+ "em_conservative": 0.1825,
+ "em_conservative_stderr": 0.01933698975881026
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_law|0": {
+ "em_relaxed": 0.12724167378309137,
+ "em_relaxed_stderr": 0.009742462799429212,
+ "em": 0.0017079419299743809,
+ "em_stderr": 0.0012071811002646103,
+ "em_conservative": 0.12724167378309137,
+ "em_conservative_stderr": 0.009742462799429212
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:machine_learning|0": {
+ "em_relaxed": 0.14285714285714285,
+ "em_relaxed_stderr": 0.033213611069662675,
+ "em": 0.026785714285714284,
+ "em_stderr": 0.015324784082343059,
+ "em_conservative": 0.125,
+ "em_conservative_stderr": 0.03139045014587016
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:us_foreign_policy|0": {
+ "em_relaxed": 0.2727272727272727,
+ "em_relaxed_stderr": 0.044988332664126655,
+ "em": 0.050505050505050504,
+ "em_stderr": 0.022120793242071984,
+ "em_conservative": 0.26262626262626265,
+ "em_conservative_stderr": 0.044452876769839444
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "em_relaxed": 0.30569948186528495,
+ "em_relaxed_stderr": 0.03324837939758159,
+ "em": 0.025906735751295335,
+ "em_stderr": 0.011464523356953171,
+ "em_conservative": 0.30569948186528495,
+ "em_conservative_stderr": 0.03324837939758159
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_physics|0": {
+ "em_relaxed": 0.17333333333333334,
+ "em_relaxed_stderr": 0.031010810941145068,
+ "em": 0.02666666666666667,
+ "em_stderr": 0.013198422611557013,
+ "em_conservative": 0.17333333333333334,
+ "em_conservative_stderr": 0.031010810941145068
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:management|0": {
+ "em_relaxed": 0.15,
+ "em_relaxed_stderr": 0.0358870281282637,
+ "em": 0.02,
+ "em_stderr": 0.01407052941362896,
+ "em_conservative": 0.15,
+ "em_conservative_stderr": 0.0358870281282637
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:sociology|0": {
+ "em_relaxed": 0.3096446700507614,
+ "em_relaxed_stderr": 0.03302479343505637,
+ "em": 0.050761421319796954,
+ "em_stderr": 0.015679295366526198,
+ "em_conservative": 0.3096446700507614,
+ "em_conservative_stderr": 0.03302479343505637
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:us_foreign_policy|0": {
+ "em_relaxed": 0.23232323232323232,
+ "em_relaxed_stderr": 0.04266016017054687,
+ "em": 0.030303030303030304,
+ "em_stderr": 0.01731601731601731,
+ "em_conservative": 0.2222222222222222,
+ "em_conservative_stderr": 0.041996052556580815
+ },
+ "skj1.prompt3_skj1_sk|0": {
+ "em_relaxed": 0.05,
+ "em_relaxed_stderr": 0.021904291355759067,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.03,
+ "em_conservative_stderr": 0.017144660799776543
+ },
+ "movie_review.prompt1_sentiment_binary_sk|0": {
+ "em_norm_relaxed": 0.6329681274900398,
+ "em_norm_relaxed_stderr": 0.010758931054934645,
+ "em_norm": 0.38745019920318724,
+ "em_norm_stderr": 0.010874391376667836,
+ "em_conservative_norm": 0.6324701195219123,
+ "em_conservative_norm_stderr": 0.010761991548945318
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:anatomy|0": {
+ "em_relaxed": 0.12686567164179105,
+ "em_relaxed_stderr": 0.028859365115889928,
+ "em": 0.022388059701492536,
+ "em_stderr": 0.012828198053329663,
+ "em_conservative": 0.12686567164179105,
+ "em_conservative_stderr": 0.028859365115889928
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_computer_science|0": {
+ "em_relaxed": 0.10101010101010101,
+ "em_relaxed_stderr": 0.030440148787487714,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.10101010101010101,
+ "em_conservative_stderr": 0.030440148787487714
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:philosophy|0": {
+ "em_relaxed": 0.2922077922077922,
+ "em_relaxed_stderr": 0.02595551182455085,
+ "em": 0.10064935064935066,
+ "em_stderr": 0.017171207138792156,
+ "em_conservative": 0.2922077922077922,
+ "em_conservative_stderr": 0.02595551182455085
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:human_aging|0": {
+ "em_relaxed": 0.1187214611872146,
+ "em_relaxed_stderr": 0.021907507389103187,
+ "em": 0.0365296803652968,
+ "em_stderr": 0.012706146532849703,
+ "em_conservative": 0.1187214611872146,
+ "em_conservative_stderr": 0.021907507389103187
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:elementary_mathematics|0": {
+ "em_relaxed": 0.10187667560321716,
+ "em_relaxed_stderr": 0.015683171805464465,
+ "em": 0.0160857908847185,
+ "em_stderr": 0.006522714286001283,
+ "em_conservative": 0.09651474530831099,
+ "em_conservative_stderr": 0.015310377502350168
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_computer_science|0": {
+ "em_relaxed": 0.13131313131313133,
+ "em_relaxed_stderr": 0.034117144062155554,
+ "em": 0.010101010101010102,
+ "em_stderr": 0.010101010101010109,
+ "em_conservative": 0.12121212121212122,
+ "em_conservative_stderr": 0.03296871474399963
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_medicine|0": {
+ "em_relaxed": 0.13257575757575757,
+ "em_relaxed_stderr": 0.020910766968363862,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.13257575757575757,
+ "em_conservative_stderr": 0.020910766968363862
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_psychology|0": {
+ "em_relaxed": 0.21308724832214765,
+ "em_relaxed_stderr": 0.01678740388759121,
+ "em": 0.050335570469798654,
+ "em_stderr": 0.008963220911107225,
+ "em_conservative": 0.21308724832214765,
+ "em_conservative_stderr": 0.01678740388759121
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:electrical_engineering|0": {
+ "em_relaxed": 0.12413793103448276,
+ "em_relaxed_stderr": 0.0274782369836366,
+ "em": 0.041379310344827586,
+ "em_stderr": 0.016597159859992713,
+ "em_conservative": 0.12413793103448276,
+ "em_conservative_stderr": 0.0274782369836366
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:logical_fallacies|0": {
+ "em_relaxed": 0.25316455696202533,
+ "em_relaxed_stderr": 0.0347027655481443,
+ "em": 0.05063291139240506,
+ "em_stderr": 0.017497810541168043,
+ "em_conservative": 0.25316455696202533,
+ "em_conservative_stderr": 0.0347027655481443
+ },
+ "sklep.prompt1_hate_speech_sk:hate-speech|0": {
+ "em_norm_relaxed": 0.6921910538286581,
+ "em_norm_relaxed_stderr": 0.012714401009923647,
+ "em_norm": 0.0,
+ "em_norm_stderr": 0.0,
+ "em_conservative_norm": 0.6921910538286581,
+ "em_conservative_norm_stderr": 0.012714401009923647
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:miscellaneous|0": {
+ "em_relaxed": 0.2092426187419769,
+ "em_relaxed_stderr": 0.014583336673745648,
+ "em": 0.0693196405648267,
+ "em_stderr": 0.009106230246473392,
+ "em_conservative": 0.2092426187419769,
+ "em_conservative_stderr": 0.014583336673745648
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:world_religions|0": {
+ "em_relaxed": 0.0935672514619883,
+ "em_relaxed_stderr": 0.022335993231163274,
+ "em": 0.04678362573099415,
+ "em_stderr": 0.016196389956235627,
+ "em_conservative": 0.0935672514619883,
+ "em_conservative_stderr": 0.022335993231163274
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_biology|0": {
+ "em_relaxed": 0.18439716312056736,
+ "em_relaxed_stderr": 0.023134670670309505,
+ "em": 0.02127659574468085,
+ "em_stderr": 0.00860850820920177,
+ "em_conservative": 0.1773049645390071,
+ "em_conservative_stderr": 0.022783829728850895
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_world_history|0": {
+ "em_relaxed": 0.2777777777777778,
+ "em_relaxed_stderr": 0.03347785759938633,
+ "em": 0.005555555555555556,
+ "em_stderr": 0.005555555555555554,
+ "em_conservative": 0.2777777777777778,
+ "em_conservative_stderr": 0.03347785759938633
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_biology|0": {
+ "em_relaxed": 0.176056338028169,
+ "em_relaxed_stderr": 0.03207487957537854,
+ "em": 0.035211267605633804,
+ "em_stderr": 0.015521989489917001,
+ "em_conservative": 0.176056338028169,
+ "em_conservative_stderr": 0.03207487957537854
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:global_facts|0": {
+ "em_relaxed": 0.18,
+ "em_relaxed_stderr": 0.03861229196653694,
+ "em": 0.13,
+ "em_stderr": 0.03379976689896309,
+ "em_conservative": 0.18,
+ "em_conservative_stderr": 0.03861229196653694
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:moral_disputes|0": {
+ "em_relaxed": 0.25766871165644173,
+ "em_relaxed_stderr": 0.0242598463239315,
+ "em": 0.08895705521472393,
+ "em_stderr": 0.015791302137788298,
+ "em_conservative": 0.25766871165644173,
+ "em_conservative_stderr": 0.0242598463239315
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_geography|0": {
+ "em_relaxed": 0.15656565656565657,
+ "em_relaxed_stderr": 0.025890520358141454,
+ "em": 0.030303030303030304,
+ "em_stderr": 0.012213156893572809,
+ "em_conservative": 0.15656565656565657,
+ "em_conservative_stderr": 0.025890520358141454
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:moral_scenarios|0": {
+ "em_relaxed": 0.0026702269692923898,
+ "em_relaxed_stderr": 0.001886873052453708,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.0026702269692923898,
+ "em_conservative_stderr": 0.001886873052453708
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_physics|0": {
+ "em_relaxed": 0.12745098039215685,
+ "em_relaxed_stderr": 0.03318224921942076,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.12745098039215685,
+ "em_conservative_stderr": 0.03318224921942076
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_medicine|0": {
+ "em_relaxed": 0.09090909090909091,
+ "em_relaxed_stderr": 0.01772676309015986,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.09090909090909091,
+ "em_conservative_stderr": 0.01772676309015986
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:logical_fallacies|0": {
+ "em_relaxed": 0.13924050632911392,
+ "em_relaxed_stderr": 0.027629554439252756,
+ "em": 0.02531645569620253,
+ "em_stderr": 0.012536706047327604,
+ "em_conservative": 0.13924050632911392,
+ "em_conservative_stderr": 0.027629554439252756
+ },
+ "sklep.prompt3_nli_binary_sk:rte|0": {
+ "em_norm_relaxed": 0.2319277108433735,
+ "em_norm_relaxed_stderr": 0.010362252629508847,
+ "em_norm": 0.0012048192771084338,
+ "em_norm_stderr": 0.0008516790803748009,
+ "em_conservative_norm": 0.2319277108433735,
+ "em_conservative_norm_stderr": 0.010362252629508847
+ },
+ "slovakcopa.prompt2_copa_sk|0": {
+ "em_relaxed": 0.51,
+ "em_relaxed_stderr": 0.02237859698923078,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.326,
+ "em_conservative_stderr": 0.020984009562393567
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:jurisprudence|0": {
+ "em_relaxed": 0.19626168224299065,
+ "em_relaxed_stderr": 0.038576441428227824,
+ "em": 0.056074766355140186,
+ "em_stderr": 0.02234599008672909,
+ "em_conservative": 0.19626168224299065,
+ "em_conservative_stderr": 0.038576441428227824
+ },
+ "multi_wiki_qa.prompt1_qa_sk|0": {
+ "exact_match_slk_prefix": 0.36401841104662797,
+ "exact_match_slk_prefix_stderr": 0.00680726616133214,
+ "f1_slk": 0.5156574509584114,
+ "f1_slk_stderr": 0.0058319674487835125
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_psychology|0": {
+ "em_relaxed": 0.16791044776119404,
+ "em_relaxed_stderr": 0.016160210122502162,
+ "em": 0.016791044776119403,
+ "em_stderr": 0.005555010467354857,
+ "em_conservative": 0.16791044776119404,
+ "em_conservative_stderr": 0.016160210122502162
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:global_facts|0": {
+ "em_relaxed": 0.11,
+ "em_relaxed_stderr": 0.031446603773522035,
+ "em": 0.06,
+ "em_stderr": 0.02386832565759416,
+ "em_conservative": 0.11,
+ "em_conservative_stderr": 0.031446603773522035
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:sociology|0": {
+ "em_relaxed": 0.3553299492385787,
+ "em_relaxed_stderr": 0.034186655739878676,
+ "em": 0.07106598984771574,
+ "em_stderr": 0.018352502878671184,
+ "em_conservative": 0.3553299492385787,
+ "em_conservative_stderr": 0.034186655739878676
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_biology|0": {
+ "em_relaxed": 0.1950354609929078,
+ "em_relaxed_stderr": 0.02363698094391562,
+ "em": 0.02127659574468085,
+ "em_stderr": 0.00860850820920177,
+ "em_conservative": 0.1879432624113475,
+ "em_conservative_stderr": 0.023305230769714243
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:public_relations|0": {
+ "em_relaxed": 0.1743119266055046,
+ "em_relaxed_stderr": 0.03650561586028542,
+ "em": 0.027522935779816515,
+ "em_stderr": 0.015742553732396008,
+ "em_conservative": 0.1743119266055046,
+ "em_conservative_stderr": 0.03650561586028542
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_statistics|0": {
+ "em_relaxed": 0.18691588785046728,
+ "em_relaxed_stderr": 0.026711670245826564,
+ "em": 0.004672897196261682,
+ "em_stderr": 0.004672897196261678,
+ "em_conservative": 0.17757009345794392,
+ "em_conservative_stderr": 0.02618451638732211
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:anatomy|0": {
+ "em_relaxed": 0.1791044776119403,
+ "em_relaxed_stderr": 0.03324844546213894,
+ "em": 0.05970149253731343,
+ "em_stderr": 0.020544692585423446,
+ "em_conservative": 0.1791044776119403,
+ "em_conservative_stderr": 0.03324844546213894
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:marketing|0": {
+ "em_relaxed": 0.2094017094017094,
+ "em_relaxed_stderr": 0.02665569965392276,
+ "em": 0.0811965811965812,
+ "em_stderr": 0.017893784904018422,
+ "em_conservative": 0.20512820512820512,
+ "em_conservative_stderr": 0.026453508054040356
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_accounting|0": {
+ "em_relaxed": 0.14492753623188406,
+ "em_relaxed_stderr": 0.02122807279169736,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.13043478260869565,
+ "em_conservative_stderr": 0.020308668108561446
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_geography|0": {
+ "em_relaxed": 0.22727272727272727,
+ "em_relaxed_stderr": 0.02985751567338641,
+ "em": 0.0707070707070707,
+ "em_stderr": 0.01826310542019951,
+ "em_conservative": 0.22727272727272727,
+ "em_conservative_stderr": 0.02985751567338641
+ },
+ "slovakcopa.prompt1_copa_sk|0": {
+ "em_relaxed": 0.618,
+ "em_relaxed_stderr": 0.021750820591250834,
+ "em": 0.0,
+ "em_stderr": 0.0,
+ "em_conservative": 0.508,
+ "em_conservative_stderr": 0.022380208834928028
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:nutrition|0": {
+ "em_relaxed": 0.19801980198019803,
+ "em_relaxed_stderr": 0.02293151046482573,
+ "em": 0.0297029702970297,
+ "em_stderr": 0.00976897099052845,
+ "em_conservative": 0.19801980198019803,
+ "em_conservative_stderr": 0.02293151046482573
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:machine_learning|0": {
+ "em_relaxed": 0.125,
+ "em_relaxed_stderr": 0.03139045014587016,
+ "em": 0.008928571428571428,
+ "em_stderr": 0.008928571428571423,
+ "em_conservative": 0.11607142857142858,
+ "em_conservative_stderr": 0.03040253249894973
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:_average|0": {
+ "em_relaxed": 0.15750731608871743,
+ "em_relaxed_stderr": 0.027457423641973053,
+ "em": 0.028327141879067475,
+ "em_stderr": 0.010791129591908543,
+ "em_conservative": 0.15358971887332207,
+ "em_conservative_stderr": 0.027071657841089895
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:_average|0": {
+ "em_relaxed": 0.1431281879704793,
+ "em_relaxed_stderr": 0.02601762436348415,
+ "em": 0.023388116537651122,
+ "em_stderr": 0.010092739621009001,
+ "em_conservative": 0.14035369812713153,
+ "em_conservative_stderr": 0.02572013192933555
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:_average|0": {
+ "em_relaxed": 0.1796468718485245,
+ "em_relaxed_stderr": 0.02870969018742619,
+ "em": 0.04274500178923145,
+ "em_stderr": 0.01407239223594206,
+ "em_conservative": 0.17602310250998576,
+ "em_conservative_stderr": 0.028335799048817806
+ },
+ "all": {
+ "em_relaxed": 0.16276835973658582,
+ "em_relaxed_stderr": 0.02594967705725322,
+ "em": 0.029667227870516465,
+ "em_stderr": 0.010711849450086212,
+ "em_conservative": 0.15781564927252603,
+ "em_conservative_stderr": 0.025592515877326957,
+ "exact_match_slk_prefix": 0.25920345417449087,
+ "exact_match_slk_prefix_stderr": 0.005273401550208795,
+ "f1_slk": 0.4424731081623923,
+ "f1_slk_stderr": 0.004828570063081536,
+ "em_norm_relaxed": 0.37655055755239747,
+ "em_norm_relaxed_stderr": 0.009574346535835498,
+ "em_norm": 0.06270134241081571,
+ "em_norm_stderr": 0.0027847027649703544,
+ "em_conservative_norm": 0.3652434918007519,
+ "em_conservative_norm_stderr": 0.009448048478463963
+ }
+ },
+ "versions": {},
+ "config_tasks": {
+ "mmlu.prompt3_mqa_single_correct_sk:miscellaneous|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:miscellaneous",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:elementary_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "financial_exams.prompt2_completion_sk|0": {
+ "name": "financial_exams.prompt2_completion_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/slovak-financial-exam",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:machine_learning|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:machine_learning",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "belebele.prompt3_mqa_single_correct_sk|0": {
+ "name": "belebele.prompt3_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "facebook/belebele",
+ "hf_subset": "slk_Latn",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:clinical_knowledge",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_law|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:professional_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_computer_science|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:professional_accounting|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:professional_accounting",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_us_history|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_us_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_us_history|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_us_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_law|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:professional_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_qa_sk:question-answering|0": {
+ "name": "sklep.prompt1_qa_sk:question-answering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "question-answering",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_chemistry|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_microeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_mathematics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_physics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_statistics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_statistics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:professional_medicine|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:professional_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "slovakcopa.prompt3_copa_sk|0": {
+ "name": "slovakcopa.prompt3_copa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "NaiveNeuron/SlovakCOPA",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "5c61e4cd9343f9b4fab623bdc50bb895cc170771",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "validation",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "movie_review.prompt2_sentiment_binary_sk|0": {
+ "name": "movie_review.prompt2_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "mteb/slovak_movie_review_sentiment",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_qa_sk:question-answering|0": {
+ "name": "sklep.prompt2_qa_sk:question-answering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "question-answering",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_biology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:nutrition|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:nutrition",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_biology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "arc.prompt3_mqa_single_correct_sk|0": {
+ "name": "arc.prompt3_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "jon-tow/okapi_arc_challenge",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": [
+ "solve"
+ ],
+ "scorer": "score",
+ "sample_fields": "record_to_sample",
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:econometrics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:econometrics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:prehistory|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:prehistory",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "toxic_sk.prompt3_toxicity_sk|0": {
+ "name": "toxic_sk.prompt3_toxicity_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/toxic-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "belebele.prompt1_mqa_single_correct_sk|0": {
+ "name": "belebele.prompt1_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "facebook/belebele",
+ "hf_subset": "slk_Latn",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "arc.prompt2_mqa_single_correct_sk|0": {
+ "name": "arc.prompt2_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "jon-tow/okapi_arc_challenge",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": [
+ "solve"
+ ],
+ "scorer": "score",
+ "sample_fields": "record_to_sample",
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "multi_wiki_qa.prompt2_qa_sk|0": {
+ "name": "multi_wiki_qa.prompt2_qa_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/multi-wiki-qa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "9bdb3350487e399bd1cd873e1395290d636db5ce",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:management|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:management",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:abstract_algebra|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:abstract_algebra",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_chemistry|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_nli_3_classes_sk:nli|0": {
+ "name": "sklep.prompt3_nli_3_classes_sk:nli",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "nli",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_medicine|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:business_ethics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:business_ethics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_computer_science|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:international_law|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:international_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:international_law|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:international_law",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "truthfulqa.prompt3_mqa_single_correct_sk:mc1|0": {
+ "name": "truthfulqa.prompt3_mqa_single_correct_sk:mc1",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/m_truthfulqa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "f0445d470f1925882b990f5f247fdcf288972f60",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "val"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:prehistory|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:prehistory",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:nutrition|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:nutrition",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:conceptual_physics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:conceptual_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_computer_science|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_medicine|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_medicine|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_medicine",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_psychology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_nli_binary_sk:rte|0": {
+ "name": "sklep.prompt2_nli_binary_sk:rte",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "rte",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_microeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_hate_speech_sk:hate-speech|0": {
+ "name": "sklep.prompt2_hate_speech_sk:hate-speech",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "hate-speech",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:human_sexuality|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:human_sexuality",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_psychology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_us_history|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_us_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:marketing|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:marketing",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:logical_fallacies|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:logical_fallacies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:conceptual_physics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:conceptual_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:virology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:virology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:medical_genetics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:medical_genetics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:moral_scenarios|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:moral_scenarios",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:us_foreign_policy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:world_religions|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:world_religions",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt3_sentiment_binary_sk|0": {
+ "name": "senti_sk.prompt3_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_physics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt1_nli_3_classes_sk:nli|0": {
+ "name": "sklep.prompt1_nli_3_classes_sk:nli",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "nli",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_macroeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:moral_scenarios|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:moral_scenarios",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_government_and_politics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:business_ethics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:business_ethics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:public_relations|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:public_relations",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_statistics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_statistics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt2_sentiment_binary_sk|0": {
+ "name": "senti_sk.prompt2_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": "",
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:medical_genetics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:medical_genetics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "belebele.prompt2_mqa_single_correct_sk|0": {
+ "name": "belebele.prompt2_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "facebook/belebele",
+ "hf_subset": "slk_Latn",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7899cdfa4e1e0d733fd77c848e2c273cb1d32be2",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:virology|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:virology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_nli_3_classes_sk:nli|0": {
+ "name": "sklep.prompt2_nli_3_classes_sk:nli",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "nli",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_european_history|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_european_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_macroeconomics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_sentiment_binary_sk:sentiment-analysis|0": {
+ "name": "sklep.prompt3_sentiment_binary_sk:sentiment-analysis",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "sentiment-analysis",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:conceptual_physics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:conceptual_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_computer_science",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:moral_disputes|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:moral_disputes",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:security_studies|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:security_studies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "skj1.prompt1_skj1_sk|0": {
+ "name": "skj1.prompt1_skj1_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/SKJ1",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "e615c60b415389b629703f82d3a79810dbde6734",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train"
+ ],
+ "evaluation_splits": [
+ "train"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:college_physics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:college_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_hate_speech_sk:hate-speech|0": {
+ "name": "sklep.prompt3_hate_speech_sk:hate-speech",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "hate-speech",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:virology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:virology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_mathematics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:security_studies|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:security_studies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "financial_exams.prompt3_completion_sk|0": {
+ "name": "financial_exams.prompt3_completion_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/slovak-financial-exam",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "7cdbfe635c3ed0d8b4df2ee04429d8afa651b33c",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:jurisprudence|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:jurisprudence",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt3_sentiment_3_class_sk|0": {
+ "name": "senti_sk.prompt3_sentiment_3_class_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:electrical_engineering|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:electrical_engineering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt3_qa_sk:question-answering|0": {
+ "name": "sklep.prompt3_qa_sk:question-answering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "question-answering",
+ "metrics": [
+ {
+ "metric_name": "exact_match_slk_prefix",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=_inner_normalizer, normalize_pred=_inner_normalize_answer, strip_strings=False, type_exact_match=prefix)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "f1_slk",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "F1_score(aggregation_function=max, normalize_gold=_inner_normalize_answer, normalize_pred=_inner_normalize_answer, strip_strings=False)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "10549a8c63542e6a0db4fcf5fcdc29b3e1b8c4e9",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": "random_sampling_from_train",
+ "generation_size": 126,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:professional_psychology|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:professional_psychology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:global_facts|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:global_facts",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:computer_security|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:computer_security",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:computer_security|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:computer_security",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_physics|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_physics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "globalmmlu-lite.prompt2_mqa_single_correct_sk|0": {
+ "name": "globalmmlu-lite.prompt2_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "CohereLabs/Global-MMLU-Lite",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "600af1bff2d447c7fc3aee22f11f607560d5fcb7",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": "1"
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_world_history|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_world_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:anatomy|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:anatomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:security_studies|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:security_studies",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "movie_review.prompt3_sentiment_binary_sk|0": {
+ "name": "movie_review.prompt3_sentiment_binary_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "mteb/slovak_movie_review_sentiment",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "29a7405aabcfd4860a51ae6f65a5650d63108f26",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:college_chemistry|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:college_chemistry",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:astronomy|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:astronomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:medical_genetics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:medical_genetics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:abstract_algebra|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:abstract_algebra",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:electrical_engineering|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:electrical_engineering",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:high_school_world_history|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:high_school_world_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:philosophy|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:philosophy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:human_sexuality|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:human_sexuality",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:abstract_algebra|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:abstract_algebra",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:world_religions|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:world_religions",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:clinical_knowledge",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "toxic_sk.prompt1_toxicity_sk|0": {
+ "name": "toxic_sk.prompt1_toxicity_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/toxic-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "2fb416e18785c128e7ec934c07a14c803a88cdb3",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt1_sentiment_3_class_sk|0": {
+ "name": "senti_sk.prompt1_sentiment_3_class_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:college_biology|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:college_biology",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:public_relations|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:public_relations",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:clinical_knowledge",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:formal_logic|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:formal_logic",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_geography|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_geography",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:prehistory|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:prehistory",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:econometrics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:econometrics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "arc.prompt1_mqa_single_correct_sk|0": {
+ "name": "arc.prompt1_mqa_single_correct_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "jon-tow/okapi_arc_challenge",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": [
+ "solve"
+ ],
+ "scorer": "score",
+ "sample_fields": "record_to_sample",
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "56aa31aedeb55f0635cc7c3c13f7f86b276f9a4e",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "train",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:business_ethics|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:business_ethics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:jurisprudence|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:jurisprudence",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "truthfulqa.prompt2_mqa_single_correct_sk:mc1|0": {
+ "name": "truthfulqa.prompt2_mqa_single_correct_sk:mc1",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "alexandrainst/m_truthfulqa",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "f0445d470f1925882b990f5f247fdcf288972f60",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "val"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:miscellaneous|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:miscellaneous",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt1_mqa_single_correct_sk:astronomy|0": {
+ "name": "mmlu.prompt1_mqa_single_correct_sk:astronomy",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_european_history|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_european_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "senti_sk.prompt2_sentiment_3_class_sk|0": {
+ "name": "senti_sk.prompt2_sentiment_3_class_sk",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "TUKE-KEMT/senti-sk",
+ "hf_subset": "default",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "3c8a92168ff2b692c653de65c97bda573127d15d",
+ "hf_filter": null,
+ "hf_avail_splits": [
+ "train",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": null,
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt2_mqa_single_correct_sk:high_school_european_history|0": {
+ "name": "mmlu.prompt2_mqa_single_correct_sk:high_school_european_history",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:formal_logic|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:formal_logic",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics|0": {
+ "name": "mmlu.prompt3_mqa_single_correct_sk:high_school_government_and_politics",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "lighteval/okapi_mmlu",
+ "hf_subset": "sk",
+ "metrics": [
+ {
+ "metric_name": "em_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ExactMatches(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=True, type_exact_match=full)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_conservative",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "ConservativeExactMatch(aggregation_function=max, normalize_gold=None, normalize_pred=None, strip_strings=False, type_exact_match=full, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ }
+ ],
+ "hf_data_files": null,
+ "solver": null,
+ "scorer": null,
+ "sample_fields": null,
+ "sample_to_fewshot": null,
+ "filter": null,
+ "hf_revision": "b3e738ae24d5d96c4b425d4893f377451196cd2e",
+ "hf_filter": "mmlu_filter",
+ "hf_avail_splits": [
+ "train",
+ "validation",
+ "test"
+ ],
+ "evaluation_splits": [
+ "test"
+ ],
+ "few_shots_split": "dev",
+ "few_shots_select": null,
+ "generation_size": 32,
+ "generation_grammar": null,
+ "stop_sequence": [
+ "\n"
+ ],
+ "num_samples": null,
+ "original_num_docs": -1,
+ "effective_num_docs": -1,
+ "must_remove_duplicate_docs": false,
+ "num_fewshots": 0,
+ "version": 0
+ },
+ "sklep.prompt2_sentiment_binary_sk:sentiment-analysis|0": {
+ "name": "sklep.prompt2_sentiment_binary_sk:sentiment-analysis",
+ "prompt_function": "prompt_fn",
+ "hf_repo": "slovak-nlp/sklep",
+ "hf_subset": "sentiment-analysis",
+ "metrics": [
+ {
+ "metric_name": "em_norm_relaxed",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=, normalize_pred=, strip_strings=True, type_exact_match=relaxed, fallback_mode=first_match)",
+ "corpus_level_fn": "mean",
+ "batched_compute": false
+ },
+ {
+ "metric_name": "em_norm",
+ "higher_is_better": true,
+ "category": "GENERATIVE",
+ "sample_level_fn": "RelaxedExtractMatch(aggregation_function=max, normalize_gold=