30k_experiment_hruska

README.md

@@ -799,3 +799,69 @@ Pri interpretácii výsledkov treba zohľadniť:
 - nebol zatiaľ zrobený benchmark základného modelu `slovak-nlp/mistral-sk-7b` bez adaptéru.
 
 
+
+
+---
+
+# Optuna – výber learning rate na 30k Aya príkladoch
+
+Po sérii 100k porovnávacích experimentov som na Hruške vyskúšal menší skúšobný experiment s automatickým výberom learning rate pomocou framewrku Optuna. Cieľom bolo vybrať sľubnú oblasť LR pre následné overenie na väčšej dátovej množine. Táto časť sumarizuje **osem úspešne dokončených trialov**.
+
+Konfigurácie, súhrnné výsledky, grafy a podrobné záznamy úspešných trialov sú v priečinku [optuna_30k_experiments/](optuna_30k_experiments/).
+
+Tréningové logy všetkých ôsmich dokončených trialov sú v priečinku [logs/](optuna_30k_experiments/logs/).
+
+## Metodika a spoločné nastavenia
+
+Optimalizovaný bol iba parameter `learning_rate` v rozsahu **2e-5 až 2e-4**. Prvé štyri referenčné hodnoty boli vložené ručne: `2e-5`, `5e-5`, `1e-4` a `2e-4`. Ďalšie hodnoty navrhovala Optuna.
+
+
+| Parameter | Hodnota |
+|---|---|
+| Základný model | `slovak-nlp/mistral-sk-7b` |
+| Dataset | fixný `aya_slovak_hpo_30k`, bez streamingu |
+| Rozdelenie dát | 29 000 tréningových + 1 000 validačných príkladov |
+| Hardvér | 1× NVIDIA RTX A4500 na Hruške, GPU 0 |
+| Compute type | `bf16` |
+| Framework | LLaMA-Factory + Unsloth |
+| Fine-tuning | LoRA, bez nastavenej 4-bit kvantizácie |
+| Template / max. dĺžka | `alpaca` / 1024 tokenov |
+| Packing / train on prompt | `false` / `false` |
+| LoRA rank / alpha / dropout | 16 / 32 / 0.0 |
+| LoRA target moduly | `q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj` |
+| Batch size / gradient accumulation | 1 / 16, effective batch 16 |
+| Rozsah tréningu | 1 epocha, 1 813 optimizer krokov |
+| Optimizer / scheduler | AdamW (`adamw_torch`) / cosine |
+| Warmup | 150 krokov |
+| Max grad norm | 0.5 |
+| Eval interval / eval batch size | každých 300 krokov + finálna evaluácia / 1 |
+| Seed | 42 |
+
+
+## Výsledky ôsmich dokončených trialov
+
+| Poradie | Trial | Learning rate | Finálny eval loss | Maximum zaznamenaného grad_norm |
+|---:|---:|---:|---:|---:|
+| 1 | 3 | 0.0002000000 | 0.740409 | 14.7127 |
+| 2 | 8 | 0.0001864520 | 0.740567 | 3.4996 |
+| 3 | 4 | 0.0001869045 | 0.741250 | 29.0258 |
+| 4 | 2 | 0.0001000000 | 0.752890 | 7.8035 |
+| 5 | 6 | 0.0000625750 | 0.766744 | 8.2603 |
+| 6 | 1 | 0.0000500000 | 0.774108 | 9.5442 |
+| 7 | 7 | 0.0000202781 | 0.800064 | 16.1318 |
+| 8 | 0 | 0.0000200000 | 0.800524 | 17.8756 |
+
+Presné hodnoty LR, finálne aj minimálne eval loss, kroky, training loss a runtime sú v [CSV súhrne](optuna_30k_experiments/results/optuna_lr_results.csv).
+
+![Finálny eval loss v závislosti od learning rate](optuna_30k_experiments/results/lr_vs_eval_loss.png)
+
+![Maximum zaznamenaného grad_norm v závislosti od learning rate](optuna_30k_experiments/results/lr_vs_max_grad_norm.png)
+
+
+## Súhrn
+
+- Najnižší finálny eval loss dosiahol **trial 3 s LR 0.0002: 0.740409**.
+- Trial 8 s LR približne **0.00018645** dosiahol veľmi blízky výsledok **0.740567**; rozdiel je približne 0.000157.
+- Na tomto 30k tréningu mali nižšie skúšané LR vyšší finálny eval loss. 
+- Najlepší skúšaný LR leží na hornej hranici rozsahu.
+

optuna_30k_experiments/configs/base_hpo_config.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 0.0001
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/template
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_000_lr_2.000e-05.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 2.0e-05
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_000_lr_2.000e-05
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_001_lr_5.000e-05.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 5.0e-05
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_001_lr_5.000e-05
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_002_lr_1.000e-04.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 0.0001
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_002_lr_1.000e-04
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_003_lr_2.000e-04.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 0.0002
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_003_lr_2.000e-04
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_004_lr_1.869e-04.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 0.00018690446893209942
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_004_lr_1.869e-04
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_006_lr_6.257e-05.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 6.257495330311199e-05
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_006_lr_6.257e-05
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_007_lr_2.028e-05.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 2.0278096180761755e-05
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_007_lr_2.028e-05
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/configs/trial_008_lr_1.865e-04.yaml

@@ -0,0 +1,50 @@
+stage: sft
+do_train: true
+do_eval: true
+model_name_or_path: slovak-nlp/mistral-sk-7b
+trust_remote_code: true
+dataset_dir: data
+dataset: aya_slovak_hpo_30k
+template: alpaca
+cutoff_len: 1024
+packing: false
+train_on_prompt: false
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.0
+lora_target: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+gradient_accumulation_steps: 16
+learning_rate: 0.00018645202585249604
+lr_scheduler_type: cosine
+warmup_steps: 150
+max_grad_norm: 0.5
+bf16: true
+fp16: false
+optim: adamw_torch
+flash_attn: auto
+plot_loss: true
+ddp_timeout: 180000000
+ddp_find_unused_parameters: false
+include_num_input_tokens_seen: true
+enable_thinking: false
+use_unsloth: true
+use_unsloth_gc: true
+streaming: false
+eval_strategy: steps
+val_size: 1000
+eval_steps: 300
+overwrite_cache: false
+overwrite_output_dir: true
+output_dir: /home/schwarc/diplomovka/optuna_lr_hpo_1gpu/trials/trial_008_lr_1.865e-04
+logging_steps: 10
+save_steps: 1800
+save_total_limit: 1
+report_to: none
+seed: 42
+accelerator_config:
+  dispatch_batches: false
+  split_batches: false
+num_train_epochs: 1.0

optuna_30k_experiments/results/optuna_lr_results.csv

@@ -0,0 +1,9 @@
+rank,trial,state,learning_rate,final_eval_loss,final_eval_step,min_eval_loss,min_eval_step,max_grad_norm,grad_status,train_loss,train_runtime_seconds,global_step,config,results_dir
+1,3,COMPLETE,0.0002,0.7404094338417053,1813,0.7404094338417053,1813,14.712687492370605,OK,0.9520373807319014,12634.3569,1813,configs/trial_003_lr_2.000e-04.yaml,trials/trial_003_lr_2.000e-04
+2,8,COMPLETE,0.00018645202585249604,0.7405665516853333,1813,0.7405665516853333,1813,3.499601125717163,OK,0.9515243444532172,12255.1774,1813,configs/trial_008_lr_1.865e-04.yaml,trials/trial_008_lr_1.865e-04
+3,4,COMPLETE,0.00018690446893209942,0.741250216960907,1813,0.741250216960907,1813,29.025802612304688,OK,0.9521475063130399,12306.5607,1813,configs/trial_004_lr_1.869e-04.yaml,trials/trial_004_lr_1.869e-04
+4,2,COMPLETE,0.0001,0.7528900504112244,1813,0.7528843879699707,1800,7.803459644317627,OK,0.9646503870091383,12301.2835,1813,configs/trial_002_lr_1.000e-04.yaml,trials/trial_002_lr_1.000e-04
+5,6,COMPLETE,6.257495330311199e-05,0.7667444944381714,1813,0.7667444944381714,1813,8.260318756103516,OK,0.9794918145517979,12296.8874,1813,configs/trial_006_lr_6.257e-05.yaml,trials/trial_006_lr_6.257e-05
+6,1,COMPLETE,5e-05,0.7741077542304993,1813,0.7741077542304993,1813,9.544248580932617,OK,0.9867739501212489,12328.0877,1813,configs/trial_001_lr_5.000e-05.yaml,trials/trial_001_lr_5.000e-05
+7,7,COMPLETE,2.0278096180761755e-05,0.8000637888908386,1813,0.8000125288963318,1800,16.131792068481445,OK,1.0186243988306432,12284.0985,1813,configs/trial_007_lr_2.028e-05.yaml,trials/trial_007_lr_2.028e-05
+8,0,COMPLETE,2e-05,0.8005238175392151,1813,0.8004471659660339,1800,17.875633239746094,OK,1.0190646677338084,12292.1318,1813,configs/trial_000_lr_2.000e-05.yaml,trials/trial_000_lr_2.000e-05

optuna_30k_experiments/results/optuna_lr_results.json

@@ -0,0 +1,130 @@
+[
+  {
+    "trial": 3,
+    "state": "COMPLETE",
+    "learning_rate": 0.0002,
+    "final_eval_loss": 0.7404094338417053,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.7404094338417053,
+    "min_eval_step": 1813,
+    "max_grad_norm": 14.712687492370605,
+    "grad_status": "OK",
+    "train_loss": 0.9520373807319014,
+    "train_runtime_seconds": 12634.3569,
+    "global_step": 1813,
+    "config": "configs/trial_003_lr_2.000e-04.yaml",
+    "results_dir": "trials/trial_003_lr_2.000e-04"
+  },
+  {
+    "trial": 8,
+    "state": "COMPLETE",
+    "learning_rate": 0.00018645202585249604,
+    "final_eval_loss": 0.7405665516853333,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.7405665516853333,
+    "min_eval_step": 1813,
+    "max_grad_norm": 3.499601125717163,
+    "grad_status": "OK",
+    "train_loss": 0.9515243444532172,
+    "train_runtime_seconds": 12255.1774,
+    "global_step": 1813,
+    "config": "configs/trial_008_lr_1.865e-04.yaml",
+    "results_dir": "trials/trial_008_lr_1.865e-04"
+  },
+  {
+    "trial": 4,
+    "state": "COMPLETE",
+    "learning_rate": 0.00018690446893209942,
+    "final_eval_loss": 0.741250216960907,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.741250216960907,
+    "min_eval_step": 1813,
+    "max_grad_norm": 29.025802612304688,
+    "grad_status": "OK",
+    "train_loss": 0.9521475063130399,
+    "train_runtime_seconds": 12306.5607,
+    "global_step": 1813,
+    "config": "configs/trial_004_lr_1.869e-04.yaml",
+    "results_dir": "trials/trial_004_lr_1.869e-04"
+  },
+  {
+    "trial": 2,
+    "state": "COMPLETE",
+    "learning_rate": 0.0001,
+    "final_eval_loss": 0.7528900504112244,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.7528843879699707,
+    "min_eval_step": 1800,
+    "max_grad_norm": 7.803459644317627,
+    "grad_status": "OK",
+    "train_loss": 0.9646503870091383,
+    "train_runtime_seconds": 12301.2835,
+    "global_step": 1813,
+    "config": "configs/trial_002_lr_1.000e-04.yaml",
+    "results_dir": "trials/trial_002_lr_1.000e-04"
+  },
+  {
+    "trial": 6,
+    "state": "COMPLETE",
+    "learning_rate": 6.257495330311199e-05,
+    "final_eval_loss": 0.7667444944381714,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.7667444944381714,
+    "min_eval_step": 1813,
+    "max_grad_norm": 8.260318756103516,
+    "grad_status": "OK",
+    "train_loss": 0.9794918145517979,
+    "train_runtime_seconds": 12296.8874,
+    "global_step": 1813,
+    "config": "configs/trial_006_lr_6.257e-05.yaml",
+    "results_dir": "trials/trial_006_lr_6.257e-05"
+  },
+  {
+    "trial": 1,
+    "state": "COMPLETE",
+    "learning_rate": 5e-05,
+    "final_eval_loss": 0.7741077542304993,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.7741077542304993,
+    "min_eval_step": 1813,
+    "max_grad_norm": 9.544248580932617,
+    "grad_status": "OK",
+    "train_loss": 0.9867739501212489,
+    "train_runtime_seconds": 12328.0877,
+    "global_step": 1813,
+    "config": "configs/trial_001_lr_5.000e-05.yaml",
+    "results_dir": "trials/trial_001_lr_5.000e-05"
+  },
+  {
+    "trial": 7,
+    "state": "COMPLETE",
+    "learning_rate": 2.0278096180761755e-05,
+    "final_eval_loss": 0.8000637888908386,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.8000125288963318,
+    "min_eval_step": 1800,
+    "max_grad_norm": 16.131792068481445,
+    "grad_status": "OK",
+    "train_loss": 1.0186243988306432,
+    "train_runtime_seconds": 12284.0985,
+    "global_step": 1813,
+    "config": "configs/trial_007_lr_2.028e-05.yaml",
+    "results_dir": "trials/trial_007_lr_2.028e-05"
+  },
+  {
+    "trial": 0,
+    "state": "COMPLETE",
+    "learning_rate": 2e-05,
+    "final_eval_loss": 0.8005238175392151,
+    "final_eval_step": 1813,
+    "min_eval_loss": 0.8004471659660339,
+    "min_eval_step": 1800,
+    "max_grad_norm": 17.875633239746094,
+    "grad_status": "OK",
+    "train_loss": 1.0190646677338084,
+    "train_runtime_seconds": 12292.1318,
+    "global_step": 1813,
+    "config": "configs/trial_000_lr_2.000e-05.yaml",
+    "results_dir": "trials/trial_000_lr_2.000e-05"
+  }
+]

optuna_30k_experiments/trials/trial_000_lr_2.000e-05/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.8005238175392151,
+    "eval_runtime": 135.4368,
+    "eval_samples_per_second": 7.384,
+    "eval_steps_per_second": 7.384,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_000_lr_2.000e-05/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 0,
+  "learning_rate": 2e-05,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.8005238175392151,
+  "min_eval_step": 1800,
+  "min_eval_loss": 0.8004471659660339,
+  "max_grad_norm": 17.875633239746094,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_000_lr_2.000e-05/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 1.0190646677338084,
+    "train_runtime": 12292.1318,
+    "train_samples_per_second": 2.359,
+    "train_steps_per_second": 0.147
+}

optuna_30k_experiments/trials/trial_001_lr_5.000e-05/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.7741077542304993,
+    "eval_runtime": 135.3685,
+    "eval_samples_per_second": 7.387,
+    "eval_steps_per_second": 7.387,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_001_lr_5.000e-05/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 1,
+  "learning_rate": 5e-05,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.7741077542304993,
+  "min_eval_step": 1813,
+  "min_eval_loss": 0.7741077542304993,
+  "max_grad_norm": 9.544248580932617,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_001_lr_5.000e-05/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 0.9867739501212489,
+    "train_runtime": 12328.0877,
+    "train_samples_per_second": 2.352,
+    "train_steps_per_second": 0.147
+}

optuna_30k_experiments/trials/trial_002_lr_1.000e-04/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.7528900504112244,
+    "eval_runtime": 135.5345,
+    "eval_samples_per_second": 7.378,
+    "eval_steps_per_second": 7.378,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_002_lr_1.000e-04/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 2,
+  "learning_rate": 0.0001,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.7528900504112244,
+  "min_eval_step": 1800,
+  "min_eval_loss": 0.7528843879699707,
+  "max_grad_norm": 7.803459644317627,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_002_lr_1.000e-04/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 0.9646503870091383,
+    "train_runtime": 12301.2835,
+    "train_samples_per_second": 2.357,
+    "train_steps_per_second": 0.147
+}

optuna_30k_experiments/trials/trial_003_lr_2.000e-04/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.7404094338417053,
+    "eval_runtime": 158.3367,
+    "eval_samples_per_second": 6.316,
+    "eval_steps_per_second": 6.316,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_003_lr_2.000e-04/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 3,
+  "learning_rate": 0.0002,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.7404094338417053,
+  "min_eval_step": 1813,
+  "min_eval_loss": 0.7404094338417053,
+  "max_grad_norm": 14.712687492370605,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_003_lr_2.000e-04/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 0.9520373807319014,
+    "train_runtime": 12634.3569,
+    "train_samples_per_second": 2.295,
+    "train_steps_per_second": 0.143
+}

optuna_30k_experiments/trials/trial_004_lr_1.869e-04/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.741250216960907,
+    "eval_runtime": 135.4566,
+    "eval_samples_per_second": 7.382,
+    "eval_steps_per_second": 7.382,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_004_lr_1.869e-04/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 4,
+  "learning_rate": 0.00018690446893209942,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.741250216960907,
+  "min_eval_step": 1813,
+  "min_eval_loss": 0.741250216960907,
+  "max_grad_norm": 29.025802612304688,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_004_lr_1.869e-04/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 0.9521475063130399,
+    "train_runtime": 12306.5607,
+    "train_samples_per_second": 2.356,
+    "train_steps_per_second": 0.147
+}

optuna_30k_experiments/trials/trial_006_lr_6.257e-05/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.7667444944381714,
+    "eval_runtime": 135.2654,
+    "eval_samples_per_second": 7.393,
+    "eval_steps_per_second": 7.393,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_006_lr_6.257e-05/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 6,
+  "learning_rate": 6.257495330311199e-05,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.7667444944381714,
+  "min_eval_step": 1813,
+  "min_eval_loss": 0.7667444944381714,
+  "max_grad_norm": 8.260318756103516,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_006_lr_6.257e-05/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 0.9794918145517979,
+    "train_runtime": 12296.8874,
+    "train_samples_per_second": 2.358,
+    "train_steps_per_second": 0.147
+}

optuna_30k_experiments/trials/trial_007_lr_2.028e-05/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.8000637888908386,
+    "eval_runtime": 135.1761,
+    "eval_samples_per_second": 7.398,
+    "eval_steps_per_second": 7.398,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_007_lr_2.028e-05/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 7,
+  "learning_rate": 2.0278096180761755e-05,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.8000637888908386,
+  "min_eval_step": 1800,
+  "min_eval_loss": 0.8000125288963318,
+  "max_grad_norm": 16.131792068481445,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_007_lr_2.028e-05/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 1.0186243988306432,
+    "train_runtime": 12284.0985,
+    "train_samples_per_second": 2.361,
+    "train_steps_per_second": 0.148
+}

optuna_30k_experiments/trials/trial_008_lr_1.865e-04/eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.7405665516853333,
+    "eval_runtime": 135.1617,
+    "eval_samples_per_second": 7.399,
+    "eval_steps_per_second": 7.399,
+    "num_input_tokens_seen": 12201816
+}

optuna_30k_experiments/trials/trial_008_lr_1.865e-04/optuna_summary.json

@@ -0,0 +1,11 @@
+{
+  "trial": 8,
+  "learning_rate": 0.00018645202585249604,
+  "final_eval_step": 1813,
+  "final_eval_loss": 0.7405665516853333,
+  "min_eval_step": 1813,
+  "min_eval_loss": 0.7405665516853333,
+  "max_grad_norm": 3.499601125717163,
+  "grad_nonfinite": false,
+  "grad_status": "OK"
+}

optuna_30k_experiments/trials/trial_008_lr_1.865e-04/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 1.0,
+    "num_input_tokens_seen": 12201816,
+    "total_flos": 5.2364848431896986e+17,
+    "train_loss": 0.9515243444532172,
+    "train_runtime": 12255.1774,
+    "train_samples_per_second": 2.366,
+    "train_steps_per_second": 0.148
+}