2.9 kB
| --- | |
| language: da | |
| widget: | |
| - text: "Jeg elsker livet" | |
| --- | |
| # GPT2-svenska-wikipedia | |
| A Danish GPT2 style model trained using Flax CLM pipeline on the Danish | |
| part of the wiki40b dataset. | |
| https://huggingface.co/datasets/wiki40b | |
| ## Model series | |
| This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge. | |
| ## Gpt models | |
| ## Swedish Gpt | |
| https://huggingface.co/birgermoell/swedish-gpt/ | |
| ## Swedish gpt wiki | |
| https://huggingface.co/flax-community/swe-gpt-wiki | |
| # Nordic gpt wiki | |
| https://huggingface.co/flax-community/nordic-gpt-wiki | |
| ## Dansk gpt wiki | |
| https://huggingface.co/flax-community/dansk-gpt-wiki | |
| ## Norsk gpt wiki | |
| https://huggingface.co/flax-community/norsk-gpt-wiki | |
| ## Roberta models | |
| ## Nordic Roberta Wiki | |
| https://huggingface.co/flax-community/nordic-roberta-wiki | |
| ## Swe Roberta Wiki Oscar | |
| https://huggingface.co/flax-community/swe-roberta-wiki-oscar | |
| ## Roberta Swedish Scandi | |
| https://huggingface.co/birgermoell/roberta-swedish-scandi | |
| ## Roberta Swedish | |
| https://huggingface.co/birgermoell/roberta-swedish | |
| ## Swedish T5 model | |
| https://huggingface.co/birgermoell/t5-base-swedish | |
| ## Data cleaning and preprocessing | |
| The data was cleaned and preprocessed using the following script. Make sure to install depencies for beam_runner to make the dataset work. | |
| ```python | |
| from datasets import load_dataset | |
| def load_and_clean_wiki(): | |
| dataset = load_dataset('wiki40b', 'da', beam_runner='DirectRunner', split="train") | |
| #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner') | |
| dataset = dataset.remove_columns(['wikidata_id', 'version_id']) | |
| filtered_dataset = dataset.map(filter_wikipedia) | |
| # filtered_dataset[:3] | |
| # print(filtered_dataset[:3]) | |
| return filtered_dataset | |
| def filter_wikipedia(batch): | |
| batch["text"] = " ".join(batch["text"].split("\ | |
| _START_SECTION_\ | |
| ")) | |
| batch["text"] = " ".join(batch["text"].split("\ | |
| _START_ARTICLE_\ | |
| ")) | |
| batch["text"] = " ".join(batch["text"].split("\ | |
| _START_ARTICLE_\ | |
| ")) | |
| batch["text"] = " ".join(batch["text"].split("\ | |
| _START_PARAGRAPH_\ | |
| ")) | |
| batch["text"] = " ".join(batch["text"].split("_NEWLINE_")) | |
| batch["text"] = " ".join(batch["text"].split("\xa0")) | |
| return batch | |
| ``` | |
| ## Training script | |
| The following training script was used to train the model. | |
| ```bash | |
| ./run_clm_flax.py --output_dir="${MODEL_DIR}" --model_type="gpt2" --config_name="${MODEL_DIR}" --tokenizer_name="${MODEL_DIR}" --dataset_name="wiki40b" --dataset_config_name="da" --do_train --do_eval --block_size="512" --per_device_train_batch_size="64" --per_device_eval_batch_size="64" --learning_rate="5e-3" --warmup_steps="1000" --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01" --overwrite_output_dir --num_train_epochs="20" --logging_steps="500" --save_steps="1000" --eval_steps="2500" --push_to_hub | |
| ``` | |