lp832ut/zpwiki
1
0
Fork 0
forked from KEMT/zpwiki
Code Issues Pull Requests Releases Wiki Activity

Update 'pages/topics/bert/README.md'

Browse Source
This commit is contained in:
dano 2021-10-20 06:45:31 +00:00
parent 87828f160b
commit a2f8b2b110
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
pages/topics/bert/README.md
View File

@ -22,9 +22,13 @@ author: Daniel Hládek
[https://medium.com/nvidia-ai/how-to-scale-the-bert-training-with-nvidia-gpus-c1575e8eaf71](zz): [https://medium.com/nvidia-ai/how-to-scale-the-bert-training-with-nvidia-gpus-c1575e8eaf71](zz):
When the mini-batch size n is multiplied by k, we should multiply the starting learning rate η by the square root of k as some theories may suggest. However, with experiments from multiple researchers, linear scaling shows better results, i.e. multiply the starting learning rate by k instead.
| BERT Large | 330M | | BERT Large | 330M |
| BERT Base | 110M | | BERT Base | 110M |
Väčšia veľkosť vstupného vektora => menšia veľkosť dávky => menší parameter učenia => pomalšie učenie
## Hotové úlohy ## Hotové úlohy