Edit model card

NOTE: This repository is now superseded by https://huggingface.co/bertin-project/bertin-roberta-base-spanish. This model corresponds to the beta version of the model using stepwise over sampling trained for 200k steps with 128 sequence lengths. Version 1 is now available and should be used instead.


BERTIN is a series of BERT-based models for Spanish. This one is a RoBERTa-large model trained from scratch on the Spanish portion of mC4 using Flax, including training scripts.

This is part of the Flax/Jax Community Week, organised by HuggingFace and TPU usage sponsored by Google.

Spanish mC4

The Spanish portion of mC4 containes about 416 million records and 235 billion words.

$ zcat c4/multilingual/c4-es*.tfrecord*.json.gz | wc -l
$ zcat c4/multilingual/c4-es*.tfrecord-*.json.gz | jq -r '.text | split(" ") | length' | paste -s -d+ - | bc

Team members

Useful links

Downloads last month
Hosted inference API
Mask token: <mask>
This model can be loaded on the Inference API on-demand.