- learning_rate: 7.5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 10.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 3.0304        | 4.81  | 500   | 1.5676          | 1.0554 |
| 1.5263        | 9.61  | 1000  | 0.4693          | 0.8023 |
| 1.5299        | 14.42 | 1500  | 0.4368          | 0.7311 |
| 1.5063        | 19.23 | 2000  | 0.4360          | 0.7302 |
| 1.455         | 24.04 | 2500  | 0.4213          | 0.6692 |
| 1.4755        | 28.84 | 3000  | 0.4329          | 0.5943 |
| 1.352         | 33.65 | 3500  | 0.4074          | 0.5765 |
| 1.3122        | 38.46 | 4000  | 0.3866          | 0.5630 |
| 1.2799        | 43.27 | 4500  | 0.3860          | 0.5480 |
| 1.212         | 48.08 | 5000  | 0.3590          | 0.5317 |
| 1.1645        | 52.88 | 5500  | 0.3283          | 0.4757 |
| 1.0854        | 57.69 | 6000  | 0.3162          | 0.4687 |
| 1.0292        | 62.5  | 6500  | 0.3126          | 0.4416 |
| 0.9607        | 67.31 | 7000  | 0.2990          | 0.4066 |
| 0.9156        | 72.12 | 7500  | 0.2870          | 0.4009 |
| 0.8329        | 76.92 | 8000  | 0.2791          | 0.3909 |
| 0.7979        | 81.73 | 8500  | 0.2770          | 0.3670 |
| 0.7144        | 86.54 | 9000  | 0.2841          | 0.3661 |
| 0.6997        | 91.35 | 9500  | 0.2721          | 0.3485 |
| 0.6568        | 96.15 | 10000 | 0.2681          | 0.3437 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


model name: xls-r-300m-te


model author: chmanoj


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, te, dataset:openslr, dataset:SLR66, transformers, license:apache-2.0, openslr_SLR66, generated_from_trainer, robust-speech-event, model-index


tags:


- flair
- token-classification
- sequence-tagger-model
language: de
widget:
- text: "Herr Oberst Brunner ist nämlich Hauptagent für den Kanton Zürich."
license: mit


# Triple E - Effective Ensembling of Embeddings and Language Models for NER of Historical German


Based on [our paper](http://ceur-ws.org/Vol-2696/paper_173.pdf) we release a new baseline model for the German
[CLEF-HIPE shared task](https://impresso.github.io/CLEF-HIPE-2020/).


In contrast to the models used in the paper, we manually sentence-segmented and normalize hyphenations and
trained a NER model using the German Europeana BERT model.


Additionally, we perform experiments with different context sizes. This approach is described in
more detail in [this paper](https://arxiv.org/abs/2011.06993).


# Results


The results with different context sizes can be seen in the following table:


| Model                      | Run 1           | Run 2           | Run 3           | Run 4               | Run 5           | Avg.
| -------------------------- | --------------- | --------------- | --------------- | ------------------- | --------------- | ---------------
| German Europeana BERT      | (81.45) / 76.92 | (**81.53**) / 77.03 | (80.49) / 77.83 | (80.88) / 77.19 | (81.39) / 77.00 | (81.15 ± 0.45) / 77.19 ± 0.34
| German Europeana BERT (16) | (**82.56**) / 77.38 | (81.19) / 77.76 | (80.99) / 76.34 | (81.27) / 77.70 | (81.28) / 77.22 | (81.46 ± 0.63) / 77.28 ± 0.57
| German Europeana BERT (32) | (**82.04**) / 78.50 | (81.14) / 76.56 | (81.81) / 78.28 | (81.50) / 76.90 | (81.64) / 77.94 | (81.63 ± 0.34) / 77.64 ± 0.86
| German Europeana BERT (64) | (81.21) / 78.39 | (81.27) / 75.98 | (**81.88**) / 78.40 | (81.66) / 77.35 | (81.29) / 76.70 | (81.46 ± 0.29) / 77.36 ± 1.06
| German Europeana BERT (80) | (82.13) / 77.77 | (81.31) / 76.81 | (82.09) / 78.69 | (**82.30**) / 76.79 | (80.65) / 77.10 | (81.70 ± 0.70) / 77.43 ± 0.81


For model upload, we choose the best model on development score: 82.56 with a context length of 16.


## Comparisons


The following figure shows the results with different context sized (on development dataset):


![German CLEF-HIPE Development Results](figures/clef_hipe_f1_score_development.png)


We perform "Almost Stochastic Order" tests as proposed in the
["Deep Dominance - How to Properly Compare Deep Neural Models"](https://www.aclweb.org/anthology/P19-1266/) paper.
The heatmap figure is heavily inspired by the ["CharacterBERT"](https://arxiv.org/abs/2010.10392) paper.


![Almost Stochastic Order Tests on Development set](figures/clef_hipe_asd_development.png)


model name: flair-clef-hipe-german-base


model author: dbmdz


task: token-classification


tags: pytorch, de, arxiv:2011.06993, arxiv:2010.10392, flair, token-classification, sequence-tagger-model, license:mit


## language: tr
license: mit


# 🤗 + 📚 dbmdz Turkish BERT model


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources a cased model for Turkish 🎉


# 🇹🇷 BERTurk


BERTurk is a community-driven cased BERT model for Turkish.


Some datasets used for pretraining and evaluation are contributed from the
awesome Turkish NLP community, as well as the decision for the model name: BERTurk.


## Stats


The current version of the model is trained on a filtered and sentence
segmented version of the Turkish [OSCAR corpus](https://traces1.inria.fr/oscar/),
a recent Wikipedia dump, various [OPUS corpora](http://opus.nlpl.eu/) and a
special corpus provided by [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/).


The final training corpus has a size of 35GB and 44,04,976,662 tokens.


Thanks to Google's TensorFlow Research Cloud (TFRC) we could train a cased model
on a TPU v3-8 for 2M steps.


For this model we use a vocab size of 128k.


## Model weights


Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
compatible weights are available. If you need access to TensorFlow checkpoints,
please raise an issue!


| Model                                | Downloads
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------
| `dbmdz/bert-base-turkish-128k-cased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-turkish-128k-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-turkish-128k-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-turkish-128k-cased/vocab.txt)


## Usage


With Transformers >= 2.3 our BERTurk cased model can be loaded like:


```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-turkish-128k-cased")
model = AutoModel.from_pretrained("dbmdz/bert-base-turkish-128k-cased")
```


## Results


For results on PoS tagging or NER tasks, please refer to
[this repository](https://github.com/stefan-it/turkish-bert).


# Huggingface model hub


All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).


# Contact (Bugs, Feedback, Contribution and more)


For questions about our BERT models just open an issue
[here](https://github.com/dbmdz/berts/issues/new) 🤗


# Acknowledgments


Thanks to [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/) for providing us
additional large corpora for Turkish. Many thanks to Reyyan Yeniterzi for providing
us the Turkish NER dataset for evaluation.


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-turkish-128k-cased


model author: dbmdz


tags: pytorch, tf, jax, bert, tr, transformers, license:mit, infinity_compatible


language: tr
license: mit
datasets:


- allenai/c4


# 🇹🇷 Turkish ELECTRA model


We present community-driven BERT, DistilBERT, ELECTRA and ConvBERT models for Turkish 🎉


Some datasets used for pretraining and evaluation are contributed from the
awesome Turkish NLP community, as well as the decision for the BERT model name: BERTurk.


Logo is provided by [Merve Noyan](https://twitter.com/mervenoyann).


# Stats


We've also trained an ELECTRA (uncased) model on the recently released Turkish part of the
[multiligual C4 (mC4) corpus](https://github.com/allenai/allennlp/discussions/5265) from the AI2 team.


After filtering documents with a broken encoding, the training corpus has a size of 242GB resulting
in 31,240,963,926 tokens.


We used the original 32k vocab (instead of creating a new one).


# mC4 ELECTRA


In addition to the ELEC**TR**A base cased model, we also trained an ELECTRA uncased model on the Turkish part of the mC4 corpus. We use a
sequence length of 512 over the full training time and train the model for 1M steps on a v3-32 TPU.


# Model usage


All trained models can be used from the [DBMDZ](https://github.com/dbmdz) Hugging Face [model hub page](https://huggingface.co/dbmdz)
using their model name.


Example usage with 🤗/Transformers:


```python
tokenizer = AutoTokenizer.from_pretrained("electra-base-turkish-mc4-uncased-discriminator")

model = AutoModel.from_pretrained("electra-base-turkish-mc4-uncased-discriminator")
```


# Citation


You can use the following BibTeX entry for citation:


```bibtex
@software{stefan_schweter_2020_3770924,
  author       = {Stefan Schweter},
  title        = {BERTurk - BERT models for Turkish},
  month        = apr,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.3770924},
  url          = {https://doi.org/10.5281/zenodo.3770924}
}
```


# Acknowledgments


We would like to thank [Merve Noyan](https://twitter.com/mervenoyann) for the
awesome logo!


model name: electra-base-turkish-mc4-uncased-discriminator


model author: dbmdz


tags: pytorch, tf, electra, pretraining, tr, dataset:allenai/c4, transformers, license:mit, infinity_compatible


language: swedish
license: mit
widget:


- text: "Det vore [MASK] häller nödvändigt att be"


# Historic Language Models (HLMs)


## Languages


Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:


| Language | Training data | Size
| -------- | ------------- | ----
| German   | [Europeana](http://www.europeana-newspapers.eu/)       | 13-28GB (filtered)
| French   | [Europeana](http://www.europeana-newspapers.eu/)       | 11-31GB (filtered)
| English  | [British Library](https://data.bl.uk/digbks/db14.html) | 24GB (year filtered)
| Finnish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.2GB
| Swedish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.1GB


## Models


At the moment, the following models are available on the model hub:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
| `dbmdz/bert-base-historic-english-cased`      | [here](https://huggingface.co/dbmdz/bert-base-historic-english-cased)
| `dbmdz/bert-base-finnish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-finnish-europeana-cased)
| `dbmdz/bert-base-swedish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-swedish-europeana-cased)


# Corpora Stats


## German Europeana Corpus


We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size
and use less-noisier data:


| OCR confidence | Size
| -------------- | ----
| **0.60**       | 28GB
| 0.65           | 18GB
| 0.70           | 13GB


For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:


![German Europeana Corpus Stats](stats/figures/german_europeana_corpus_stats.png)


## French Europeana Corpus


Like German, we use different ocr confidence thresholds:


| OCR confidence | Size
| -------------- | ----
| 0.60           | 31GB
| 0.65           | 27GB
| **0.70**       | 27GB
| 0.75           | 23GB
| 0.80           | 11GB


For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:


![French Europeana Corpus Stats](stats/figures/french_europeana_corpus_stats.png)


## British Library Corpus


Metadata is taken from [here](https://data.bl.uk/digbks/DB21.html). Stats incl. year filtering:


| Years             | Size
| ----------------- | ----
| ALL               | 24GB
| >= 1800 && < 1900 | 24GB


We use the year filtered variant. The following plot shows a tokens per year distribution:


![British Library Corpus Stats](stats/figures/bl_corpus_stats.png)


## Finnish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.2GB


The following plot shows a tokens per year distribution:


![Finnish Europeana Corpus Stats](stats/figures/finnish_europeana_corpus_stats.png)


## Swedish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.1GB


![Swedish Europeana Corpus Stats](stats/figures/swedish_europeana_corpus_stats.png)


## All Corpora


The following plot shows a tokens per year distribution of the complete training corpus:


![All Corpora Stats](stats/figures/all_corpus_stats.png)


# Multilingual Vocab generation


For the first attempt, we use the first 10GB of each pretraining corpus. We upsample both Finnish and Swedish to ~10GB.
The following tables shows the exact size that is used for generating a 32k and 64k subword vocabs:


| Language | Size
| -------- | ----
| German   | 10GB
| French   | 10GB
| English  | 10GB
| Finnish  | 9.5GB
| Swedish  | 9.7GB


We then calculate the subword fertility rate and portion of `[UNK]`s over the following NER corpora:


| Language | NER corpora
| -------- | ------------------
| German   | CLEF-HIPE, NewsEye
| French   | CLEF-HIPE, NewsEye
| English  | CLEF-HIPE
| Finnish  | NewsEye
| Swedish  | NewsEye


Breakdown of subword fertility rate and unknown portion per language for the 32k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.43               | 0.0004
| French   | 1.25               | 0.0001
| English  | 1.25               | 0.0
| Finnish  | 1.69               | 0.0007
| Swedish  | 1.43               | 0.0


Breakdown of subword fertility rate and unknown portion per language for the 64k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.31               | 0.0004
| French   | 1.16               | 0.0001
| English  | 1.17               | 0.0
| Finnish  | 1.54               | 0.0007
| Swedish  | 1.32               | 0.0


# Final pretraining corpora


We upsample Swedish and Finnish to ~27GB. The final stats for all pretraining corpora can be seen here:


| Language | Size
| -------- | ----
| German   | 28GB
| French   | 27GB
| English  | 24GB
| Finnish  | 27GB
| Swedish  | 27GB


Total size is 130GB.


# Pretraining


## Multilingual model


We train a multilingual BERT model using the 32k vocab with the official BERT implementation
on a v3-32 TPU using the following parameters:


```bash
python3 run_pretraining.py --input_file gs://histolectra/historic-multilingual-tfrecords/*.tfrecord \
--output_dir gs://histolectra/bert-base-historic-multilingual-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_historic-multilingual.png)


## English model


The English BERT model - with texts from British Library corpus - was trained with the Hugging Face
JAX/FLAX implementation for 10 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-historic-english-cased/ \
--tokenizer_name /mnt/datasets/bert-base-historic-english-cased/ \
--train_file /mnt/datasets/bl-corpus/bl_1800-1900_extracted.txt \
--validation_file /mnt/datasets/bl-corpus/english_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 10 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-historic-english-cased-512-noadafactor-10e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_historic_english.png)


## Finnish model


The BERT model - with texts from Finnish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Finnish_0.6.txt \
--validation_file /mnt/datasets/hlms/finnish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-finnish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_finnish_europeana.png)


## Swedish model


The BERT model - with texts from Swedish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 660K steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Swedish_0.6.txt \
--validation_file /mnt/datasets/hlms/swedish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-swedish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_swedish_europeana.png)


# Acknowledgments


Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as
TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-swedish-europeana-cased


model author: dbmdz


task: fill-mask


tags: pytorch, jax, tensorboard, bert, fill-mask, swedish, transformers, license:mit, autonlp_compatible, infinity_compatible


language: en
license: mit
datasets:


- conll2003
widget:
- text: My name is Clara Clever and I live in Berkeley , California .


# T5 Base Model for Named Entity Recognition (NER, CoNLL-2003)


In this repository, we open source a T5 Base model, that was fine-tuned on the official CoNLL-2003 NER dataset.


We use the great [TANL library](https://github.com/amazon-research/tanl) from Amazon for fine-tuning the model.


The exact approach of fine-tuning is presented in the "TANL: Structured Prediction as Translation between Augmented Natural Languages"
paper from Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang and Stefano Soatto.


# Fine-Tuning


We use the same hyper-parameter settings as used in the official implementation with one minor change. Instead of using 8 V100 GPUs, we train the model
on one V100 GPU and used gradient accumulation. The slighly modified configuration file (`config.ini`) then looks like:


```ini
[conll03]
datasets = conll03
model_name_or_path = t5-base
num_train_epochs = 10
max_seq_length = 256
max_seq_length_eval = 512
per_device_train_batch_size = 4
per_device_eval_batch_size = 4
do_train = True
do_eval = True
do_predict = True
gradient_accumulation_steps = 8
```


It took around 2 hours to fine-tune that model on the 14,041 training sentences of CoNLL-2003 dataset.


# Evaluation


On the development set, the following evaluation results could be achieved:


```json
{
"entity_precision": 0.9536446086664427,
"entity_recall": 0.9555705149781218,
"entity_f1": 0.9546065904505716,
"entity_precision_no_type": 0.9773261672824992,
"entity_recall_no_type": 0.9792998990238977,
"entity_f1_no_type": 0.9783120376597176
}
```


The evaluation results on the test set looks like:


```json
{
"entity_precision": 0.912182296231376,
"entity_recall": 0.9213881019830028,
"entity_f1": 0.9167620893155995,
"entity_precision_no_type": 0.953900087642419,
"entity_recall_no_type": 0.9635269121813032,
"entity_f1_no_type": 0.9586893332158901
}
```


To summarize: On the development set, 95.46% F1-Score and 91.68% on test set were achieved with this model. The paper reported a F1-Score of 91.7%.


# License


The models is licensed under [MIT](https://choosealicense.com/licenses/mit/).


# Acknowledgments


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: t5-base-conll03-english


model author: dbmdz


task: text2text-generation


tags: pytorch, t5, text2text-generation, en, dataset:conll2003, transformers, license:mit, autonlp_compatible


language: finnish
license: mit
widget:


- text: "Täkäläinen sanomalehdistö [MASK] erit - täin"


# Historic Language Models (HLMs)


## Languages


Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:


| Language | Training data | Size
| -------- | ------------- | ----
| German   | [Europeana](http://www.europeana-newspapers.eu/)       | 13-28GB (filtered)
| French   | [Europeana](http://www.europeana-newspapers.eu/)       | 11-31GB (filtered)
| English  | [British Library](https://data.bl.uk/digbks/db14.html) | 24GB (year filtered)
| Finnish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.2GB
| Swedish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.1GB


## Models


At the moment, the following models are available on the model hub:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
| `dbmdz/bert-base-historic-english-cased`      | [here](https://huggingface.co/dbmdz/bert-base-historic-english-cased)
| `dbmdz/bert-base-finnish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-finnish-europeana-cased)
| `dbmdz/bert-base-swedish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-swedish-europeana-cased)


# Corpora Stats


## German Europeana Corpus


We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size
and use less-noisier data:


| OCR confidence | Size
| -------------- | ----
| **0.60**       | 28GB
| 0.65           | 18GB
| 0.70           | 13GB


For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:


![German Europeana Corpus Stats](stats/figures/german_europeana_corpus_stats.png)


## French Europeana Corpus


Like German, we use different ocr confidence thresholds:


| OCR confidence | Size
| -------------- | ----
| 0.60           | 31GB
| 0.65           | 27GB
| **0.70**       | 27GB
| 0.75           | 23GB
| 0.80           | 11GB


For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:


![French Europeana Corpus Stats](stats/figures/french_europeana_corpus_stats.png)


## British Library Corpus


Metadata is taken from [here](https://data.bl.uk/digbks/DB21.html). Stats incl. year filtering:


| Years             | Size
| ----------------- | ----
| ALL               | 24GB
| >= 1800 && < 1900 | 24GB


We use the year filtered variant. The following plot shows a tokens per year distribution:


![British Library Corpus Stats](stats/figures/bl_corpus_stats.png)


## Finnish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.2GB


The following plot shows a tokens per year distribution:


![Finnish Europeana Corpus Stats](stats/figures/finnish_europeana_corpus_stats.png)


## Swedish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.1GB


![Swedish Europeana Corpus Stats](stats/figures/swedish_europeana_corpus_stats.png)


## All Corpora


The following plot shows a tokens per year distribution of the complete training corpus:


![All Corpora Stats](stats/figures/all_corpus_stats.png)


# Multilingual Vocab generation


For the first attempt, we use the first 10GB of each pretraining corpus. We upsample both Finnish and Swedish to ~10GB.
The following tables shows the exact size that is used for generating a 32k and 64k subword vocabs:


| Language | Size
| -------- | ----
| German   | 10GB
| French   | 10GB
| English  | 10GB
| Finnish  | 9.5GB
| Swedish  | 9.7GB


We then calculate the subword fertility rate and portion of `[UNK]`s over the following NER corpora:


| Language | NER corpora
| -------- | ------------------
| German   | CLEF-HIPE, NewsEye
| French   | CLEF-HIPE, NewsEye
| English  | CLEF-HIPE
| Finnish  | NewsEye
| Swedish  | NewsEye


Breakdown of subword fertility rate and unknown portion per language for the 32k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.43               | 0.0004
| French   | 1.25               | 0.0001
| English  | 1.25               | 0.0
| Finnish  | 1.69               | 0.0007
| Swedish  | 1.43               | 0.0


Breakdown of subword fertility rate and unknown portion per language for the 64k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.31               | 0.0004
| French   | 1.16               | 0.0001
| English  | 1.17               | 0.0
| Finnish  | 1.54               | 0.0007
| Swedish  | 1.32               | 0.0


# Final pretraining corpora


We upsample Swedish and Finnish to ~27GB. The final stats for all pretraining corpora can be seen here:


| Language | Size
| -------- | ----
| German   | 28GB
| French   | 27GB
| English  | 24GB
| Finnish  | 27GB
| Swedish  | 27GB


Total size is 130GB.


# Pretraining


## Multilingual model


We train a multilingual BERT model using the 32k vocab with the official BERT implementation
on a v3-32 TPU using the following parameters:


```bash
python3 run_pretraining.py --input_file gs://histolectra/historic-multilingual-tfrecords/*.tfrecord \
--output_dir gs://histolectra/bert-base-historic-multilingual-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_historic-multilingual.png)


## English model


The English BERT model - with texts from British Library corpus - was trained with the Hugging Face
JAX/FLAX implementation for 10 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-historic-english-cased/ \
--tokenizer_name /mnt/datasets/bert-base-historic-english-cased/ \
--train_file /mnt/datasets/bl-corpus/bl_1800-1900_extracted.txt \
--validation_file /mnt/datasets/bl-corpus/english_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 10 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-historic-english-cased-512-noadafactor-10e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_historic_english.png)


## Finnish model


The BERT model - with texts from Finnish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Finnish_0.6.txt \
--validation_file /mnt/datasets/hlms/finnish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-finnish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_finnish_europeana.png)


## Swedish model


The BERT model - with texts from Swedish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 660K steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Swedish_0.6.txt \
--validation_file /mnt/datasets/hlms/swedish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-swedish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_swedish_europeana.png)


# Acknowledgments


Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as
TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-finnish-europeana-cased


model author: dbmdz


task: fill-mask


tags: pytorch, jax, tensorboard, bert, fill-mask, finnish, transformers, license:mit, autonlp_compatible, infinity_compatible


language: de
license: mit
tags:


- "historic german"


# 🤗 + 📚 dbmdz ConvBERT model


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources a German Europeana ConvBERT model 🎉


# German Europeana ConvBERT


We use the open source [Europeana newspapers](http://www.europeana-newspapers.eu/)
that were provided by *The European Library*. The final
training corpus has a size of 51GB and consists of 8,035,986,369 tokens.


Detailed information about the data and pretraining steps can be found in
[this repository](https://github.com/stefan-it/europeana-bert).


## Results


For results on Historic NER, please refer to [this repository](https://github.com/stefan-it/europeana-bert).


## Usage


With Transformers >= 4.3 our German Europeana ConvBERT model can be loaded like:


```python
from transformers import AutoModel, AutoTokenizer

model_name = "dbmdz/convbert-base-german-europeana-cased"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
```


# Huggingface model hub


All other German Europeana models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).


# Contact (Bugs, Feedback, Contribution and more)


For questions about our Europeana BERT, ELECTRA and ConvBERT models just open a new discussion
[here](https://github.com/stefan-it/europeana-bert/discussions) 🤗


# Acknowledgments


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


model name: convbert-base-german-europeana-cased


model author: dbmdz


task: feature-extraction


tags: pytorch, tf, convbert, feature-extraction, de, transformers, license:mit, historic german


language: it
license: mit
datasets:


- wikipedia


# 🤗 + 📚 dbmdz BERT and ELECTRA models


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources Italian BERT and ELECTRA models 🎉


# Italian BERT


The source data for the Italian BERT model consists of a recent Wikipedia dump and
various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
training corpus has a size of 13GB and 2,050,057,573 tokens.


For sentence splitting, we use NLTK (faster compared to spacy).
Our cased and uncased models are training with an initial sequence length of 512
subwords for ~2-3M steps.


For the XXL Italian models, we use the same training data from OPUS and extend
it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.


Note: Unfortunately, a wrong vocab size was used when training the XXL models.
This explains the mismatch of the "real" vocab size of 31102, compared to the
vocab size specified in `config.json`. However, the model is working and all
evaluations were done under those circumstances.
See [this issue](https://github.com/dbmdz/berts/issues/7) for more information.


The Italian ELECTRA model was trained on the "XXL" corpus for 1M steps in total using a batch
size of 128. We pretty much following the ELECTRA training procedure as used for
[BERTurk](https://github.com/stefan-it/turkish-bert/tree/master/electra).


## Model weights


Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
compatible weights are available. If you need access to TensorFlow checkpoints,
please raise an issue!


| Model                                                | Downloads
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------
| `dbmdz/bert-base-italian-cased`                      | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/config.json)                                               • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/pytorch_model.bin)                      • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-cased/vocab.txt)
| `dbmdz/bert-base-italian-uncased`                    | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/config.json)                                             • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/pytorch_model.bin)                    • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-uncased/vocab.txt)
| `dbmdz/bert-base-italian-xxl-cased`                  | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/config.json)                                           • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/pytorch_model.bin)                  • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-cased/vocab.txt)
| `dbmdz/bert-base-italian-xxl-uncased`                | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/config.json)                                         • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/pytorch_model.bin)                • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-italian-xxl-uncased/vocab.txt)
| `dbmdz/electra-base-italian-xxl-cased-discriminator` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/dbmdz/electra-base-italian-xxl-cased-discriminator/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/electra-base-italian-xxl-cased-discriminator/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/electra-base-italian-xxl-cased-discriminator/vocab.txt)
| `dbmdz/electra-base-italian-xxl-cased-generator`     | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/dbmdz/electra-base-italian-xxl-cased-generator/config.json)     • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/electra-base-italian-xxl-cased-generator/pytorch_model.bin)     • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/electra-base-italian-xxl-cased-generator/vocab.txt)


## Results


For results on downstream tasks like NER or PoS tagging, please refer to
[this repository](https://github.com/stefan-it/italian-bertelectra).


## Usage


With Transformers >= 2.3 our Italian BERT models can be loaded like:


```python
from transformers import AutoModel, AutoTokenizer

model_name = "dbmdz/bert-base-italian-cased"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModel.from_pretrained(model_name)
```


To load the (recommended) Italian XXL BERT models, just use:


```python
from transformers import AutoModel, AutoTokenizer

model_name = "dbmdz/bert-base-italian-xxl-cased"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModel.from_pretrained(model_name)
```


To load the Italian XXL ELECTRA model (discriminator), just use:


```python
from transformers import AutoModel, AutoTokenizer

model_name = "dbmdz/electra-base-italian-xxl-cased-discriminator"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelWithLMHead.from_pretrained(model_name)
```


# Huggingface model hub


All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).


# Contact (Bugs, Feedback, Contribution and more)


For questions about our BERT/ELECTRA models just open an issue
[here](https://github.com/dbmdz/berts/issues/new) 🤗


# Acknowledgments


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-italian-xxl-uncased


model author: dbmdz


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, it, dataset:wikipedia, transformers, license:mit, autonlp_compatible, infinity_compatible


language: tr
license: mit
datasets:


- allenai/c4


# 🇹🇷 Turkish ELECTRA model


We present community-driven BERT, DistilBERT, ELECTRA and ConvBERT models for Turkish 🎉


Some datasets used for pretraining and evaluation are contributed from the
awesome Turkish NLP community, as well as the decision for the BERT model name: BERTurk.


Logo is provided by [Merve Noyan](https://twitter.com/mervenoyann).


# Stats


We've also trained an ELECTRA (cased) model on the recently released Turkish part of the
[multiligual C4 (mC4) corpus](https://github.com/allenai/allennlp/discussions/5265) from the AI2 team.


After filtering documents with a broken encoding, the training corpus has a size of 242GB resulting
in 31,240,963,926 tokens.


We used the original 32k vocab (instead of creating a new one).


# mC4 ELECTRA


In addition to the ELEC**TR**A base model, we also trained an ELECTRA model on the Turkish part of the mC4 corpus. We use a
sequence length of 512 over the full training time and train the model for 1M steps on a v3-32 TPU.


# Model usage


All trained models can be used from the [DBMDZ](https://github.com/dbmdz) Hugging Face [model hub page](https://huggingface.co/dbmdz)
using their model name.


Example usage with 🤗/Transformers:


```python
tokenizer = AutoTokenizer.from_pretrained("dbmdz/electra-base-turkish-mc4-cased-discriminator")

model = AutoModel.from_pretrained("dbmdz/electra-base-turkish-mc4-cased-discriminator")
```


# Citation


You can use the following BibTeX entry for citation:


```bibtex
@software{stefan_schweter_2020_3770924,
  author       = {Stefan Schweter},
  title        = {BERTurk - BERT models for Turkish},
  month        = apr,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.3770924},
  url          = {https://doi.org/10.5281/zenodo.3770924}
}
```


# Acknowledgments


Thanks to [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/) for providing us
additional large corpora for Turkish. Many thanks to Reyyan Yeniterzi for providing
us the Turkish NER dataset for evaluation.


We would like to thank [Merve Noyan](https://twitter.com/mervenoyann) for the
awesome logo!


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


model name: electra-base-turkish-mc4-cased-discriminator


model author: dbmdz


tags: pytorch, tf, tensorboard, electra, pretraining, tr, dataset:allenai/c4, transformers, license:mit, infinity_compatible


language: multilingual
license: mit
widget:


- text: "and I cannot conceive the reafon why [MASK] hath"
- text: "Täkäläinen sanomalehdistö [MASK] erit - täin"
- text: "Det vore [MASK] häller nödvändigt att be"
- text: "Comme, à cette époque [MASK] était celle de la"
- text: "In [MASK] an atmosphärischen Nahrungsmitteln"


# Historic Language Models (HLMs)


## Languages


Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:


| Language | Training data | Size
| -------- | ------------- | ----
| German   | [Europeana](http://www.europeana-newspapers.eu/)       | 13-28GB (filtered)
| French   | [Europeana](http://www.europeana-newspapers.eu/)       | 11-31GB (filtered)
| English  | [British Library](https://data.bl.uk/digbks/db14.html) | 24GB (year filtered)
| Finnish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.2GB
| Swedish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.1GB


## Models


At the moment, the following models are available on the model hub:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
| `dbmdz/bert-base-historic-english-cased`      | [here](https://huggingface.co/dbmdz/bert-base-historic-english-cased)
| `dbmdz/bert-base-finnish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-finnish-europeana-cased)
| `dbmdz/bert-base-swedish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-swedish-europeana-cased)


We also released smaller models for the multilingual model:


| Model identifier | Model Hub link
| ----------------------------------------------- | ---------------------------------------------------------------------------
| `dbmdz/bert-tiny-historic-multilingual-cased`   | [here](https://huggingface.co/dbmdz/bert-tiny-historic-multilingual-cased)
| `dbmdz/bert-mini-historic-multilingual-cased`   | [here](https://huggingface.co/dbmdz/bert-mini-historic-multilingual-cased)
| `dbmdz/bert-small-historic-multilingual-cased`  | [here](https://huggingface.co/dbmdz/bert-small-historic-multilingual-cased)
| `dbmdz/bert-medium-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)


**Notice**: We have released language models for Historic German and French trained on more noisier data earlier - see
[this repo](https://github.com/stefan-it/europeana-bert) for more information:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-german-europeana-cased`      | [here](https://huggingface.co/dbmdz/bert-base-german-europeana-cased)
| `dbmdz/bert-base-french-europeana-cased`      | [here](https://huggingface.co/dbmdz/bert-base-french-europeana-cased)


# Corpora Stats


## German Europeana Corpus


We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size
and use less-noisier data:


| OCR confidence | Size
| -------------- | ----
| **0.60**       | 28GB
| 0.65           | 18GB
| 0.70           | 13GB


For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:


![German Europeana Corpus Stats](stats/figures/german_europeana_corpus_stats.png)


## French Europeana Corpus


Like German, we use different ocr confidence thresholds:


| OCR confidence | Size
| -------------- | ----
| 0.60           | 31GB
| 0.65           | 27GB
| **0.70**       | 27GB
| 0.75           | 23GB
| 0.80           | 11GB


For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:


![French Europeana Corpus Stats](stats/figures/french_europeana_corpus_stats.png)


## British Library Corpus


Metadata is taken from [here](https://data.bl.uk/digbks/DB21.html). Stats incl. year filtering:


| Years             | Size
| ----------------- | ----
| ALL               | 24GB
| >= 1800 && < 1900 | 24GB


We use the year filtered variant. The following plot shows a tokens per year distribution:


![British Library Corpus Stats](stats/figures/bl_corpus_stats.png)


## Finnish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.2GB


The following plot shows a tokens per year distribution:


![Finnish Europeana Corpus Stats](stats/figures/finnish_europeana_corpus_stats.png)


## Swedish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.1GB


![Swedish Europeana Corpus Stats](stats/figures/swedish_europeana_corpus_stats.png)


## All Corpora


The following plot shows a tokens per year distribution of the complete training corpus:


![All Corpora Stats](stats/figures/all_corpus_stats.png)


# Multilingual Vocab generation


For the first attempt, we use the first 10GB of each pretraining corpus. We upsample both Finnish and Swedish to ~10GB.
The following tables shows the exact size that is used for generating a 32k and 64k subword vocabs:


| Language | Size
| -------- | ----
| German   | 10GB
| French   | 10GB
| English  | 10GB
| Finnish  | 9.5GB
| Swedish  | 9.7GB


We then calculate the subword fertility rate and portion of `[UNK]`s over the following NER corpora:


| Language | NER corpora
| -------- | ------------------
| German   | CLEF-HIPE, NewsEye
| French   | CLEF-HIPE, NewsEye
| English  | CLEF-HIPE
| Finnish  | NewsEye
| Swedish  | NewsEye


Breakdown of subword fertility rate and unknown portion per language for the 32k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.43               | 0.0004
| French   | 1.25               | 0.0001
| English  | 1.25               | 0.0
| Finnish  | 1.69               | 0.0007
| Swedish  | 1.43               | 0.0


Breakdown of subword fertility rate and unknown portion per language for the 64k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.31               | 0.0004
| French   | 1.16               | 0.0001
| English  | 1.17               | 0.0
| Finnish  | 1.54               | 0.0007
| Swedish  | 1.32               | 0.0


# Final pretraining corpora


We upsample Swedish and Finnish to ~27GB. The final stats for all pretraining corpora can be seen here:


| Language | Size
| -------- | ----
| German   | 28GB
| French   | 27GB
| English  | 24GB
| Finnish  | 27GB
| Swedish  | 27GB


Total size is 130GB.


# Smaller multilingual models


Inspired by the ["Well-Read Students Learn Better: On the Importance of Pre-training Compact Models"](https://arxiv.org/abs/1908.08962)
paper, we train smaller models (different layers and hidden sizes), and report number of parameters and pre-training costs:


| Model (Layer / Hidden size) | Parameters  | Pre-Training time
| --------------------------- | ----------: | ----------------------:
| hmBERT Tiny   ( 2/128)      |   4.58M     |  4.3 sec / 1,000 steps
| hmBERT Mini   ( 4/256)      |  11.55M     | 10.5 sec / 1,000 steps
| hmBERT Small  ( 4/512)      |  29.52M     | 20.7 sec / 1,000 steps
| hmBERT Medium ( 8/512)      |  42.13M     | 35.0 sec / 1,000 steps
| hmBERT Base   (12/768)      | 110.62M     | 80.0 sec / 1,000 steps


We then perform downstream evaluations on the multilingual [NewsEye](https://zenodo.org/record/4573313#.Ya3oVr-ZNzU) dataset:


![NewsEye hmBERT Evaluation](stats/figures/newseye-hmbert-evaluation.png)


# Pretraining


## Multilingual model - hmBERT Base


We train a multilingual BERT model using the 32k vocab with the official BERT implementation
on a v3-32 TPU using the following parameters:


```bash
python3 run_pretraining.py --input_file gs://histolectra/historic-multilingual-tfrecords/*.tfrecord \
--output_dir gs://histolectra/bert-base-historic-multilingual-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_historic-multilingual.png)


## Smaller multilingual models


We use the same parameters as used for training the base model.


### hmBERT Tiny


The following plot shows the pretraining loss curve for the tiny model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-tiny.png)


### hmBERT Mini


The following plot shows the pretraining loss curve for the mini model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-mini.png)


### hmBERT Small


The following plot shows the pretraining loss curve for the small model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-small.png)


### hmBERT Medium


The following plot shows the pretraining loss curve for the medium model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-medium.png)


## English model


The English BERT model - with texts from British Library corpus - was trained with the Hugging Face
JAX/FLAX implementation for 10 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-historic-english-cased/ \
--tokenizer_name /mnt/datasets/bert-base-historic-english-cased/ \
--train_file /mnt/datasets/bl-corpus/bl_1800-1900_extracted.txt \
--validation_file /mnt/datasets/bl-corpus/english_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 10 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-historic-english-cased-512-noadafactor-10e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_historic_english.png)


## Finnish model


The BERT model - with texts from Finnish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Finnish_0.6.txt \
--validation_file /mnt/datasets/hlms/finnish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-finnish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_finnish_europeana.png)


## Swedish model


The BERT model - with texts from Swedish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 660K steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Swedish_0.6.txt \
--validation_file /mnt/datasets/hlms/swedish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-swedish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_swedish_europeana.png)


# Acknowledgments


Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as
TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-small-historic-multilingual-cased


model author: dbmdz


task: fill-mask


tags: pytorch, tf, tensorboard, bert, fill-mask, multilingual, arxiv:1908.08962, transformers, license:mit, autonlp_compatible, infinity_compatible


language: multilingual
license: mit
widget:


- text: "and I cannot conceive the reafon why [MASK] hath"
- text: "Täkäläinen sanomalehdistö [MASK] erit - täin"
- text: "Det vore [MASK] häller nödvändigt att be"
- text: "Comme, à cette époque [MASK] était celle de la"
- text: "In [MASK] an atmosphärischen Nahrungsmitteln"


# Historic Language Models (HLMs)


## Languages


Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:


| Language | Training data | Size
| -------- | ------------- | ----
| German   | [Europeana](http://www.europeana-newspapers.eu/)       | 13-28GB (filtered)
| French   | [Europeana](http://www.europeana-newspapers.eu/)       | 11-31GB (filtered)
| English  | [British Library](https://data.bl.uk/digbks/db14.html) | 24GB (year filtered)
| Finnish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.2GB
| Swedish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.1GB


## Models


At the moment, the following models are available on the model hub:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
| `dbmdz/bert-base-historic-english-cased`      | [here](https://huggingface.co/dbmdz/bert-base-historic-english-cased)
| `dbmdz/bert-base-finnish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-finnish-europeana-cased)
| `dbmdz/bert-base-swedish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-swedish-europeana-cased)


# Corpora Stats


## German Europeana Corpus


We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size
and use less-noisier data:


| OCR confidence | Size
| -------------- | ----
| **0.60**       | 28GB
| 0.65           | 18GB
| 0.70           | 13GB


For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:


![German Europeana Corpus Stats](stats/figures/german_europeana_corpus_stats.png)


## French Europeana Corpus


Like German, we use different ocr confidence thresholds:


| OCR confidence | Size
| -------------- | ----
| 0.60           | 31GB
| 0.65           | 27GB
| **0.70**       | 27GB
| 0.75           | 23GB
| 0.80           | 11GB


For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:


![French Europeana Corpus Stats](stats/figures/french_europeana_corpus_stats.png)


## British Library Corpus


Metadata is taken from [here](https://data.bl.uk/digbks/DB21.html). Stats incl. year filtering:


| Years             | Size
| ----------------- | ----
| ALL               | 24GB
| >= 1800 && < 1900 | 24GB


We use the year filtered variant. The following plot shows a tokens per year distribution:


![British Library Corpus Stats](stats/figures/bl_corpus_stats.png)


## Finnish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.2GB


The following plot shows a tokens per year distribution:


![Finnish Europeana Corpus Stats](stats/figures/finnish_europeana_corpus_stats.png)


## Swedish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.1GB


![Swedish Europeana Corpus Stats](stats/figures/swedish_europeana_corpus_stats.png)


## All Corpora


The following plot shows a tokens per year distribution of the complete training corpus:


![All Corpora Stats](stats/figures/all_corpus_stats.png)


# Multilingual Vocab generation


For the first attempt, we use the first 10GB of each pretraining corpus. We upsample both Finnish and Swedish to ~10GB.
The following tables shows the exact size that is used for generating a 32k and 64k subword vocabs:


| Language | Size
| -------- | ----
| German   | 10GB
| French   | 10GB
| English  | 10GB
| Finnish  | 9.5GB
| Swedish  | 9.7GB


We then calculate the subword fertility rate and portion of `[UNK]`s over the following NER corpora:


| Language | NER corpora
| -------- | ------------------
| German   | CLEF-HIPE, NewsEye
| French   | CLEF-HIPE, NewsEye
| English  | CLEF-HIPE
| Finnish  | NewsEye
| Swedish  | NewsEye


Breakdown of subword fertility rate and unknown portion per language for the 32k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.43               | 0.0004
| French   | 1.25               | 0.0001
| English  | 1.25               | 0.0
| Finnish  | 1.69               | 0.0007
| Swedish  | 1.43               | 0.0


Breakdown of subword fertility rate and unknown portion per language for the 64k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.31               | 0.0004
| French   | 1.16               | 0.0001
| English  | 1.17               | 0.0
| Finnish  | 1.54               | 0.0007
| Swedish  | 1.32               | 0.0


# Final pretraining corpora


We upsample Swedish and Finnish to ~27GB. The final stats for all pretraining corpora can be seen here:


| Language | Size
| -------- | ----
| German   | 28GB
| French   | 27GB
| English  | 24GB
| Finnish  | 27GB
| Swedish  | 27GB


Total size is 130GB.


# Pretraining


## Multilingual model


We train a multilingual BERT model using the 32k vocab with the official BERT implementation
on a v3-32 TPU using the following parameters:


```bash
python3 run_pretraining.py --input_file gs://histolectra/historic-multilingual-tfrecords/*.tfrecord \
--output_dir gs://histolectra/bert-base-historic-multilingual-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_historic-multilingual.png)


## English model


The English BERT model - with texts from British Library corpus - was trained with the Hugging Face
JAX/FLAX implementation for 10 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-historic-english-cased/ \
--tokenizer_name /mnt/datasets/bert-base-historic-english-cased/ \
--train_file /mnt/datasets/bl-corpus/bl_1800-1900_extracted.txt \
--validation_file /mnt/datasets/bl-corpus/english_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 10 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-historic-english-cased-512-noadafactor-10e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_historic_english.png)


## Finnish model


The BERT model - with texts from Finnish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Finnish_0.6.txt \
--validation_file /mnt/datasets/hlms/finnish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-finnish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_finnish_europeana.png)


## Swedish model


The BERT model - with texts from Swedish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 660K steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Swedish_0.6.txt \
--validation_file /mnt/datasets/hlms/swedish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-swedish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_swedish_europeana.png)


# Acknowledgments


Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as
TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-historic-multilingual-cased


model author: dbmdz


task: fill-mask


tags: pytorch, jax, tensorboard, bert, fill-mask, multilingual, transformers, license:mit, autonlp_compatible, infinity_compatible


datasets:


- germeval_14
tags:
- flair
- token-classification
- sequence-tagger-model
language: de
widget:
- text: "Hugging Face ist eine französische Firma mit Sitz in New York."
license: mit


# Flair NER model trained on GermEval14 dataset


This model was trained on the official [GermEval14](https://sites.google.com/site/germeval2014ner/data)
dataset using the [Flair](https://github.com/flairNLP/flair) framework.


It uses a fine-tuned German DistilBERT model from [here](https://huggingface.co/distilbert-base-german-cased).


# Results


| Dataset \ Run | Run 1 | Run 2 | Run 3†    | Run 4 | Run 5 | Avg.
| ------------- | ----- | ----- | --------- | ----- | ----- | ----
| Development   | 87.05 | 86.52 | **87.34** | 86.85 | 86.46 | 86.84
| Test          | 85.43 | 85.88 | 85.72     | 85.47 | 85.62 | 85.62


† denotes that this model is selected for upload.


# Flair Fine-Tuning


We used the following script to fine-tune the model on the GermEval14 dataset:


```python
from argparse import ArgumentParser
import torch, flair

# dataset, model and embedding imports
from flair.datasets import GERMEVAL_14
from flair.embeddings import TransformerWordEmbeddings
from flair.models import SequenceTagger
from flair.trainers import ModelTrainer

if __name__ == "__main__":

    # All arguments that can be passed
    parser = ArgumentParser()
    parser.add_argument("-s", "--seeds", nargs='+', type=int, default='42')  # pass list of seeds for experiments
    parser.add_argument("-c", "--cuda", type=int, default=0, help="CUDA device")  # which cuda device to use
    parser.add_argument("-m", "--model", type=str, help="Model name (such as Hugging Face model hub name")

    # Parse experimental arguments
    args = parser.parse_args()

    # use cuda device as passed
    flair.device = f'cuda:{str(args.cuda)}'

    # for each passed seed, do one experimental run
    for seed in args.seeds:
        flair.set_seed(seed)

        # model
        hf_model = args.model

        # initialize embeddings
        embeddings = TransformerWordEmbeddings(
            model=hf_model,
            layers="-1",
            subtoken_pooling="first",
            fine_tune=True,
            use_context=False,
            respect_document_boundaries=False,
        )

        # select dataset depending on which language variable is passed
        corpus = GERMEVAL_14()

        # make the dictionary of tags to predict
        tag_dictionary = corpus.make_tag_dictionary('ner')

        # init bare-bones sequence tagger (no reprojection, LSTM or CRF)
        tagger: SequenceTagger = SequenceTagger(
            hidden_size=256,
            embeddings=embeddings,
            tag_dictionary=tag_dictionary,
            tag_type='ner',
            use_crf=False,
            use_rnn=False,
            reproject_embeddings=False,
        )

        # init the model trainer
        trainer = ModelTrainer(tagger, corpus, optimizer=torch.optim.AdamW)

        # make string for output folder
        output_folder = f"flert-ner-{hf_model}-{seed}"

        # train with XLM parameters (AdamW, 20 epochs, small LR)
        from torch.optim.lr_scheduler import OneCycleLR

        trainer.train(
            output_folder,
            learning_rate=5.0e-5,
            mini_batch_size=16,
            mini_batch_chunk_size=1,
            max_epochs=10,
            scheduler=OneCycleLR,
            embeddings_storage_mode='none',
            weight_decay=0.,
            train_with_dev=False,
        )
```


model name: flair-distilbert-ner-germeval14


task: token-classification


tags: pytorch, de, dataset:germeval_14, flair, token-classification, sequence-tagger-model, license:mit


language: tr
license: mit
datasets:


- allenai/c4
widget:
- text: "[MASK] sözcüğü Türkçe kökenlidir"


# 🇹🇷 Turkish ELECTRA model


We present community-driven BERT, DistilBERT, ELECTRA and ConvBERT models for Turkish 🎉


Some datasets used for pretraining and evaluation are contributed from the
awesome Turkish NLP community, as well as the decision for the BERT model name: BERTurk.


Logo is provided by [Merve Noyan](https://twitter.com/mervenoyann).


# Stats


We've also trained an ELECTRA (cased) model on the recently released Turkish part of the
[multiligual C4 (mC4) corpus](https://github.com/allenai/allennlp/discussions/5265) from the AI2 team.


After filtering documents with a broken encoding, the training corpus has a size of 242GB resulting
in 31,240,963,926 tokens.


We used the original 32k vocab (instead of creating a new one).


# mC4 ELECTRA


In addition to the ELEC**TR**A base model, we also trained an ELECTRA model on the Turkish part of the mC4 corpus. We use a
sequence length of 512 over the full training time and train the model for 1M steps on a v3-32 TPU.


# Model usage


All trained models can be used from the [DBMDZ](https://github.com/dbmdz) Hugging Face [model hub page](https://huggingface.co/dbmdz)
using their model name.


Example usage with 🤗/Transformers:


```python
tokenizer = AutoTokenizer.from_pretrained("dbmdz/electra-base-turkish-mc4-cased-generator")

model = AutoModel.from_pretrained("dbmdz/electra-base-turkish-mc4-cased-generator")
```


# Citation


You can use the following BibTeX entry for citation:


```bibtex
@software{stefan_schweter_2020_3770924,
  author       = {Stefan Schweter},
  title        = {BERTurk - BERT models for Turkish},
  month        = apr,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.3770924},
  url          = {https://doi.org/10.5281/zenodo.3770924}
}
```


# Acknowledgments


Thanks to [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/) for providing us
additional large corpora for Turkish. Many thanks to Reyyan Yeniterzi for providing
us the Turkish NER dataset for evaluation.


We would like to thank [Merve Noyan](https://twitter.com/mervenoyann) for the
awesome logo!


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


model name: electra-base-turkish-mc4-cased-generator


model author: dbmdz


task: fill-mask


tags: pytorch, tf, electra, fill-mask, tr, dataset:allenai/c4, transformers, license:mit, autonlp_compatible, infinity_compatible


language: multilingual
license: mit
widget:


- text: "and I cannot conceive the reafon why [MASK] hath"
- text: "Täkäläinen sanomalehdistö [MASK] erit - täin"
- text: "Det vore [MASK] häller nödvändigt att be"
- text: "Comme, à cette époque [MASK] était celle de la"
- text: "In [MASK] an atmosphärischen Nahrungsmitteln"


# Historic Language Models (HLMs)


## Languages


Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:


| Language | Training data | Size
| -------- | ------------- | ----
| German   | [Europeana](http://www.europeana-newspapers.eu/)       | 13-28GB (filtered)
| French   | [Europeana](http://www.europeana-newspapers.eu/)       | 11-31GB (filtered)
| English  | [British Library](https://data.bl.uk/digbks/db14.html) | 24GB (year filtered)
| Finnish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.2GB
| Swedish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.1GB


## Models


At the moment, the following models are available on the model hub:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
| `dbmdz/bert-base-historic-english-cased`      | [here](https://huggingface.co/dbmdz/bert-base-historic-english-cased)
| `dbmdz/bert-base-finnish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-finnish-europeana-cased)
| `dbmdz/bert-base-swedish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-swedish-europeana-cased)


We also released smaller models for the multilingual model:


| Model identifier | Model Hub link
| ----------------------------------------------- | ---------------------------------------------------------------------------
| `dbmdz/bert-tiny-historic-multilingual-cased`   | [here](https://huggingface.co/dbmdz/bert-tiny-historic-multilingual-cased)
| `dbmdz/bert-mini-historic-multilingual-cased`   | [here](https://huggingface.co/dbmdz/bert-mini-historic-multilingual-cased)
| `dbmdz/bert-small-historic-multilingual-cased`  | [here](https://huggingface.co/dbmdz/bert-small-historic-multilingual-cased)
| `dbmdz/bert-medium-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)


**Notice**: We have released language models for Historic German and French trained on more noisier data earlier - see
[this repo](https://github.com/stefan-it/europeana-bert) for more information:


| Model identifier                              | Model Hub link
| --------------------------------------------- | --------------------------------------------------------------------------
| `dbmdz/bert-base-german-europeana-cased`      | [here](https://huggingface.co/dbmdz/bert-base-german-europeana-cased)
| `dbmdz/bert-base-french-europeana-cased`      | [here](https://huggingface.co/dbmdz/bert-base-french-europeana-cased)


# Corpora Stats


## German Europeana Corpus


We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size
and use less-noisier data:


| OCR confidence | Size
| -------------- | ----
| **0.60**       | 28GB
| 0.65           | 18GB
| 0.70           | 13GB


For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:


![German Europeana Corpus Stats](stats/figures/german_europeana_corpus_stats.png)


## French Europeana Corpus


Like German, we use different ocr confidence thresholds:


| OCR confidence | Size
| -------------- | ----
| 0.60           | 31GB
| 0.65           | 27GB
| **0.70**       | 27GB
| 0.75           | 23GB
| 0.80           | 11GB


For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:


![French Europeana Corpus Stats](stats/figures/french_europeana_corpus_stats.png)


## British Library Corpus


Metadata is taken from [here](https://data.bl.uk/digbks/DB21.html). Stats incl. year filtering:


| Years             | Size
| ----------------- | ----
| ALL               | 24GB
| >= 1800 && < 1900 | 24GB


We use the year filtered variant. The following plot shows a tokens per year distribution:


![British Library Corpus Stats](stats/figures/bl_corpus_stats.png)


## Finnish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.2GB


The following plot shows a tokens per year distribution:


![Finnish Europeana Corpus Stats](stats/figures/finnish_europeana_corpus_stats.png)


## Swedish Europeana Corpus


| OCR confidence | Size
| -------------- | ----
| 0.60           | 1.1GB


![Swedish Europeana Corpus Stats](stats/figures/swedish_europeana_corpus_stats.png)


## All Corpora


The following plot shows a tokens per year distribution of the complete training corpus:


![All Corpora Stats](stats/figures/all_corpus_stats.png)


# Multilingual Vocab generation


For the first attempt, we use the first 10GB of each pretraining corpus. We upsample both Finnish and Swedish to ~10GB.
The following tables shows the exact size that is used for generating a 32k and 64k subword vocabs:


| Language | Size
| -------- | ----
| German   | 10GB
| French   | 10GB
| English  | 10GB
| Finnish  | 9.5GB
| Swedish  | 9.7GB


We then calculate the subword fertility rate and portion of `[UNK]`s over the following NER corpora:


| Language | NER corpora
| -------- | ------------------
| German   | CLEF-HIPE, NewsEye
| French   | CLEF-HIPE, NewsEye
| English  | CLEF-HIPE
| Finnish  | NewsEye
| Swedish  | NewsEye


Breakdown of subword fertility rate and unknown portion per language for the 32k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.43               | 0.0004
| French   | 1.25               | 0.0001
| English  | 1.25               | 0.0
| Finnish  | 1.69               | 0.0007
| Swedish  | 1.43               | 0.0


Breakdown of subword fertility rate and unknown portion per language for the 64k vocab:


| Language | Subword fertility  | Unknown portion
| -------- | ------------------ | ---------------
| German   | 1.31               | 0.0004
| French   | 1.16               | 0.0001
| English  | 1.17               | 0.0
| Finnish  | 1.54               | 0.0007
| Swedish  | 1.32               | 0.0


# Final pretraining corpora


We upsample Swedish and Finnish to ~27GB. The final stats for all pretraining corpora can be seen here:


| Language | Size
| -------- | ----
| German   | 28GB
| French   | 27GB
| English  | 24GB
| Finnish  | 27GB
| Swedish  | 27GB


Total size is 130GB.


# Smaller multilingual models


Inspired by the ["Well-Read Students Learn Better: On the Importance of Pre-training Compact Models"](https://arxiv.org/abs/1908.08962)
paper, we train smaller models (different layers and hidden sizes), and report number of parameters and pre-training costs:


| Model (Layer / Hidden size) | Parameters  | Pre-Training time
| --------------------------- | ----------: | ----------------------:
| hmBERT Tiny   ( 2/128)      |   4.58M     |  4.3 sec / 1,000 steps
| hmBERT Mini   ( 4/256)      |  11.55M     | 10.5 sec / 1,000 steps
| hmBERT Small  ( 4/512)      |  29.52M     | 20.7 sec / 1,000 steps
| hmBERT Medium ( 8/512)      |  42.13M     | 35.0 sec / 1,000 steps
| hmBERT Base   (12/768)      | 110.62M     | 80.0 sec / 1,000 steps


We then perform downstream evaluations on the multilingual [NewsEye](https://zenodo.org/record/4573313#.Ya3oVr-ZNzU) dataset:


![NewsEye hmBERT Evaluation](stats/figures/newseye-hmbert-evaluation.png)


# Pretraining


## Multilingual model - hmBERT Base


We train a multilingual BERT model using the 32k vocab with the official BERT implementation
on a v3-32 TPU using the following parameters:


```bash
python3 run_pretraining.py --input_file gs://histolectra/historic-multilingual-tfrecords/*.tfrecord \
--output_dir gs://histolectra/bert-base-historic-multilingual-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_historic-multilingual.png)


## Smaller multilingual models


We use the same parameters as used for training the base model.


### hmBERT Tiny


The following plot shows the pretraining loss curve for the tiny model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-tiny.png)


### hmBERT Mini


The following plot shows the pretraining loss curve for the mini model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-mini.png)


### hmBERT Small


The following plot shows the pretraining loss curve for the small model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-small.png)


### hmBERT Medium


The following plot shows the pretraining loss curve for the medium model:


![Training loss curve](stats/figures/pretraining_loss_hmbert-medium.png)


## English model


The English BERT model - with texts from British Library corpus - was trained with the Hugging Face
JAX/FLAX implementation for 10 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-historic-english-cased/ \
--tokenizer_name /mnt/datasets/bert-base-historic-english-cased/ \
--train_file /mnt/datasets/bl-corpus/bl_1800-1900_extracted.txt \
--validation_file /mnt/datasets/bl-corpus/english_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 10 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-historic-english-cased-512-noadafactor-10e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_historic_english.png)


## Finnish model


The BERT model - with texts from Finnish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Finnish_0.6.txt \
--validation_file /mnt/datasets/hlms/finnish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-finnish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


![Training loss curve](stats/figures/pretraining_loss_finnish_europeana.png)


## Swedish model


The BERT model - with texts from Swedish part of Europeana - was trained with the Hugging Face
JAX/FLAX implementation for 40 epochs (approx. 660K steps) on a v3-8 TPU, using the following command:


```bash
python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Swedish_0.6.txt \
--validation_file /mnt/datasets/hlms/swedish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-swedish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length
```


The following plot shows the pretraining loss curve:


![Training loss curve](stats/figures/pretraining_loss_swedish_europeana.png)


# Acknowledgments


Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as
TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-tiny-historic-multilingual-cased


model author: dbmdz


task: fill-mask


tags: pytorch, tf, tensorboard, bert, fill-mask, multilingual, arxiv:1908.08962, transformers, license:mit, autonlp_compatible, infinity_compatible


tags:


- flair
- token-classification
- sequence-tagger-model
language: de
inference: false
license: mit


# Towards Robust Named Entity Recognition for Historic German


Based on [our paper](https://www.aclweb.org/anthology/W19-4312/)
we release a new model trained on the LFT dataset.


**Note:** We use BPEmbeddings instead of the combination of
Wikipedia, Common Crawl and character embeddings (as used in the paper),
so save space and training/inferencing time.


# Results


| Dataset \ Run | Run 1 | Run 2 | Run 3†    | Avg.
| ------------- | ----- | ----- | --------- | ------------
| Development   | 76.32 | 76.13 | **76.36** | 76.27
| Test          | 77.07 | 77.35 | 77.20     | 77.21


Paper reported an averaged F1-score of 77.51.


† denotes that this model is selected for upload.


model name: flair-historic-ner-lft


task: token-classification


tags: pytorch, de, flair, token-classification, sequence-tagger-model, license:mit


language: fr
license: mit
tags:


- "historic french"


# 🤗 + 📚 dbmdz ELECTRA models


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources French Europeana ELECTRA models 🎉


# French Europeana ELECTRA


We extracted all French texts using the `language` metadata attribute from the Europeana corpus.


The resulting corpus has a size of 63GB and consists of 11,052,528,456 tokens.


Based on the metadata information, texts from the 18th - 20th century are mainly included in the
training corpus.


Detailed information about the data and pretraining steps can be found in
[this repository](https://github.com/stefan-it/europeana-bert).


## Model weights


ELECTRA model weights for PyTorch and TensorFlow are available.


* French Europeana ELECTRA (discriminator): `dbmdz/electra-base-french-europeana-cased-discriminator` - [model hub page](https://huggingface.co/dbmdz/electra-base-french-europeana-cased-discriminator/tree/main)
* French Europeana ELECTRA (generator): `dbmdz/electra-base-french-europeana-cased-generator` - [model hub page](https://huggingface.co/dbmdz/electra-base-french-europeana-cased-generator/tree/main)


## Results


For results on Historic NER, please refer to [this repository](https://github.com/stefan-it/europeana-bert).


## Usage


With Transformers >= 2.3 our French Europeana ELECTRA model can be loaded like:


```python
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("dbmdz/electra-base-french-europeana-cased-discriminator")
model = AutoModel.from_pretrained("dbmdz/electra-base-french-europeana-cased-discriminator")
```


# Huggingface model hub


All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).


# Contact (Bugs, Feedback, Contribution and more)


For questions about our ELECTRA models just open an issue
[here](https://github.com/dbmdz/berts/issues/new) 🤗


# Acknowledgments


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download our models from their S3 storage 🤗


model name: electra-base-french-europeana-cased-discriminator


model author: dbmdz


tags: pytorch, tf, electra, pretraining, fr, transformers, license:mit, historic french, infinity_compatible


language: tr
license: mit
datasets:


- allenai/c4


# 🇹🇷 Turkish ELECTRA model


We present community-driven BERT, DistilBERT, ELECTRA and ConvBERT models for Turkish 🎉


Some datasets used for pretraining and evaluation are contributed from the
awesome Turkish NLP community, as well as the decision for the BERT model name: BERTurk.


Logo is provided by [Merve Noyan](https://twitter.com/mervenoyann).


# Stats


We've also trained an ELECTRA (uncased) model on the recently released Turkish part of the
[multiligual C4 (mC4) corpus](https://github.com/allenai/allennlp/discussions/5265) from the AI2 team.


After filtering documents with a broken encoding, the training corpus has a size of 242GB resulting
in 31,240,963,926 tokens.


We used the original 32k vocab (instead of creating a new one).


# mC4 ELECTRA


In addition to the ELEC**TR**A base cased model, we also trained an ELECTRA uncased model on the Turkish part of the mC4 corpus. We use a
sequence length of 512 over the full training time and train the model for 1M steps on a v3-32 TPU.


# Model usage


All trained models can be used from the [DBMDZ](https://github.com/dbmdz) Hugging Face [model hub page](https://huggingface.co/dbmdz)
using their model name.


Example usage with 🤗/Transformers:


```python
tokenizer = AutoTokenizer.from_pretrained("electra-base-turkish-mc4-uncased-generator")

model = AutoModel.from_pretrained("electra-base-turkish-mc4-uncased-generator")
```


# Citation


You can use the following BibTeX entry for citation:


```bibtex
@software{stefan_schweter_2020_3770924,
  author       = {Stefan Schweter},
  title        = {BERTurk - BERT models for Turkish},
  month        = apr,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.3770924},
  url          = {https://doi.org/10.5281/zenodo.3770924}
}
```


# Acknowledgments


Thanks to [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/) for providing us
additional large corpora for Turkish. Many thanks to Reyyan Yeniterzi for providing
us the Turkish NER dataset for evaluation.


We would like to thank [Merve Noyan](https://twitter.com/mervenoyann) for the
awesome logo!


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


model name: electra-base-turkish-mc4-uncased-generator


model author: dbmdz


task: fill-mask


tags: pytorch, tf, electra, fill-mask, tr, dataset:allenai/c4, transformers, license:mit, autonlp_compatible, infinity_compatible


language: tr
license: mit
datasets:


- allenai/c4


# 🇹🇷 Turkish ConvBERT model


# Stats


We've trained an (uncased) ConvBERT model on the recently released Turkish part of the
[multiligual C4 (mC4) corpus](https://github.com/allenai/allennlp/discussions/5265) from the AI2 team.


# mC4 ConvBERT


In addition to the ELEC**TR**A base model, we also trained an ConvBERT model on the Turkish part of the mC4 corpus. We use a
sequence length of 512 over the full training time and train the model for 1M steps on a v3-32 TPU.


# Model usage


Example usage with 🤗/Transformers:


```python
tokenizer = AutoTokenizer.from_pretrained("dbmdz/convbert-base-turkish-mc4-uncased")

model = AutoModel.from_pretrained("dbmdz/convbert-base-turkish-mc4-uncased")
```


# Citation


# Acknowledgments


model name: convbert-base-turkish-mc4-uncased


tags: pytorch, tf, convbert, fill-mask, tr, dataset:allenai/c4, transformers, license:mit, autonlp_compatible


## language: de
license: mit


# 🤗 + 📚 dbmdz German BERT models


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources another German BERT models 🎉


# German BERT


## Stats


In addition to the recently released [German BERT](https://deepset.ai/german-bert)
model by [deepset](https://deepset.ai/) we provide another German-language model.


The source data for the model consists of a recent Wikipedia dump, EU Bookshop corpus,
Open Subtitles, CommonCrawl, ParaCrawl and News Crawl. This results in a dataset with
a size of 16GB and 2,350,234,427 tokens.


For sentence splitting, we use [spacy](https://spacy.io/). Our preprocessing steps
(sentence piece model for vocab generation) follow those used for training
[SciBERT](https://github.com/allenai/scibert). The model is trained with an initial
sequence length of 512 subwords and was performed for 1.5M steps.


This release includes both cased and uncased models.


## Model weights


Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
compatible weights are available. If you need access to TensorFlow checkpoints,
please raise an issue!


| Model                            | Downloads
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------
| `bert-base-german-dbmdz-cased`   | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-vocab.txt)
| `bert-base-german-dbmdz-uncased` | [`config.json`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json) • [`pytorch_model.bin`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin) • [`vocab.txt`](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt)


## Usage


With Transformers >= 2.3 our German BERT models can be loaded like:


```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")
model = AutoModel.from_pretrained("dbmdz/bert-base-german-cased")
```


## Results


For results on downstream tasks like NER or PoS tagging, please refer to
[this repository](https://github.com/stefan-it/fine-tuned-berts-seq).


# Huggingface model hub


All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).


# Contact (Bugs, Feedback, Contribution and more)


For questions about our BERT models just open an issue
[here](https://github.com/dbmdz/berts/issues/new) 🤗


# Acknowledgments


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-german-uncased


model author: dbmdz


task: fill-mask


tags: pytorch, tf, jax, bert, fill-mask, de, transformers, license:mit, autonlp_compatible, infinity_compatible


## language: tr
license: mit


# 🤗 + 📚 dbmdz Turkish BERT model


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources a cased model for Turkish 🎉


# 🇹🇷 BERTurk


BERTurk is a community-driven cased BERT model for Turkish.


Some datasets used for pretraining and evaluation are contributed from the
awesome Turkish NLP community, as well as the decision for the model name: BERTurk.


## Stats


The current version of the model is trained on a filtered and sentence
segmented version of the Turkish [OSCAR corpus](https://traces1.inria.fr/oscar/),
a recent Wikipedia dump, various [OPUS corpora](http://opus.nlpl.eu/) and a
special corpus provided by [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/).


The final training corpus has a size of 35GB and 44,04,976,662 tokens.


Thanks to Google's TensorFlow Research Cloud (TFRC) we could train a cased model
on a TPU v3-8 for 2M steps.


## Model weights


Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
compatible weights are available. If you need access to TensorFlow checkpoints,
please raise an issue!


| Model                             | Downloads
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------
| `dbmdz/bert-base-turkish-cased`   | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-turkish-cased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-turkish-cased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-turkish-cased/vocab.txt)


## Usage


With Transformers >= 2.3 our BERTurk cased model can be loaded like:


```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-turkish-cased")
model = AutoModel.from_pretrained("dbmdz/bert-base-turkish-cased")
```


## Results


For results on PoS tagging or NER tasks, please refer to
[this repository](https://github.com/stefan-it/turkish-bert).


# Huggingface model hub


All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).


# Contact (Bugs, Feedback, Contribution and more)


For questions about our BERT models just open an issue
[here](https://github.com/dbmdz/berts/issues/new) 🤗


# Acknowledgments


Thanks to [Kemal Oflazer](http://www.andrew.cmu.edu/user/ko/) for providing us
additional large corpora for Turkish. Many thanks to Reyyan Yeniterzi for providing
us the Turkish NER dataset for evaluation.


Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
Thanks for providing access to the TFRC ❤


Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
it is possible to download both cased and uncased models from their S3 storage 🤗


model name: bert-base-turkish-cased


model author: dbmdz


tags: pytorch, tf, jax, bert, tr, transformers, license:mit, infinity_compatible


language: it
license: mit
datasets:


- wikipedia


# 🤗 + 📚 dbmdz BERT and ELECTRA models


In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
Library open sources Italian BERT and ELECTRA models 🎉


# Italian BERT


The source data for the Italian BERT model consists of a recent Wikipedia dump and
various texts from the [OPUS corpora](http://opus.nlpl.eu/) collection. The final
training corpus has a size of 13GB and 2,050,057,573 tokens.


For sentence splitting, we use NLTK (faster compared to spacy).
Our cased and uncased models are training with an initial sequence length of 512
subwords for ~2-3M steps.


For the XXL Italian models, we use the same training data from OPUS and extend
it with data from the Italian part of the [OSCAR corpus](https://traces1.inria.fr/oscar/).
Thus, the final training corpus has a size of 81GB and 13,138,379,147 tokens.


Note: Unfortunately, a wrong vocab size was used when training the XXL models.
This explains the mismatch of the "real" vocab size of 31102, compared to the
vocab size specified in `config.json`. However, the model is working and all
evaluations were done under those circumstances.
See [this issue](https://github.com/dbmdz/berts/issues/7) for more information.


The Italian ELECTRA model was trained on the "XXL" corpus for 1M steps in total using a batch
size of 128. We pretty much following the ELECTRA training procedure as used for
[BERTurk](https://github.com/stefan-it/turkish-bert/tree/master/electra).


## Model weights


Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
compatible weights are available. If you need access to TensorFlow checkpoints,
please raise an issue!