README.md

---
language: en
license: apache-2.0
datasets:
- bookcorpus
- wikipedia
- cc_news
---

# BigBird base model

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this [paper](https://arxiv.org/abs/2007.14062) and first released in this [repository](https://github.com/google-research/bigbird).

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.

## Model description

BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.

## How to use `TODO: Update`

Here is how to use this model to get the features of a given text in Flax:

```python
from transformers import BigBirdTokenizer, FlaxBigBirdModel

model_id = "flax-community/bigband"

# by default its in `block_sparse` mode with num_random_blocks=3, block_size=64
model = FlaxBigBirdModel.from_pretrained(model_id)

# you can change `attention_type` to full attention like this:
model = FlaxBigBirdModel.from_pretrained(model_id, attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = FlaxBigBirdModel.from_pretrained(model_id, block_size=16, num_random_blocks=2)

tokenizer = BigBirdTokenizer.from_pretrained(model_id)

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors="jax")
output = model(**inputs)
```

## Training Data `TODO: Update`

This model is pre-trained on four publicly available datasets: **Books**, **CC-News**, **Stories** and **Wikipedia**. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2).

## Training Procedure `TODO: Update`

Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.

Model is warm started from RoBERTa’s checkpoint.