metadata
language: nl
widget:
- text: 'Een zalig kerstfeest en '
- text: >-
Na een lange reeks vertragingen zal eind volgende week de James Webb Space
Telescope (JWST) de aarde verlaten. Met een vergulde spiegel van
tags:
- gpt2-medium
- gpt2
pipeline_tag: text-generation
datasets:
- yhavinga/mc4_nl_cleaned
GPT2-Medium pre-trained on cleaned Dutch mC4 🇳🇱
Training details:
- trained for 240k steps (29 dec 2021)
- block size: 512
- optimizer: adam, lr 8e-4, beta1 0.9, beta2 0.98
- warmup 5000 steps
- weight decay 0.01
Work in progress. Dec 2021.
- Many thanks to the Google TPU Research Cloud for providing access to a TPU cluster!
- Thanks to @gsarti for creating the t5-flax-gcp repository.
- Also thanks to the creators of gpt2-medium-persian and gpt2-medium-indonesian for sharing their training scripts!