Longformer¶

DISCLAIMER: This model is still a work in progress, if you see something strange, file a Github Issue

Overview¶

The Longformer model was presented in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan. Here the abstract:

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

The Authors’ code can be found here .

Longformer Self Attention¶

Longformer self attention employs self attention on both a “local” context and a “global” context. Most tokens only attend “locally” to each other meaning that each token attends to its \(\frac{1}{2} w\) previous tokens and \(\frac{1}{2} w\) succeding tokens with \(w\) being the window length as defined in config.attention_window. Note that config.attention_window can be of type list to define a different \(w\) for each layer. A selecetd few tokens attend “globally” to all other tokens, as it is conventionally done for all tokens in e.g. BertSelfAttention.

Note that “locally” and “globally” attending tokens are projected by different query, key and value matrices. Also note that every “locally” attending token not only attends to tokens within its window \(w\), but also to all “globally” attending tokens so that global attention is symmetric.

The user can define which tokens are masked, which tokens attend “locally” and which tokens attend “globally” by setting the config.attention_mask torch.Tensor appropriately. In contrast to other models Longformer accepts the following values in config.attention_mask: 0 - the token is masked and not attended at all (as is done in other models), 1 - the token attends “locally”, 2 - token attends “globally”. For more information please also refer to forward() method.

Using Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually represents the memory and time bottleneck, can be reduced from \(\mathcal{O}(n_s \times n_s)\) to \(\mathcal{O}(n_s \times w)\), with \(n_s\) being the sequence length and \(w\) being the average window size. It is assumed that the number of “globally” attending tokens is insignificant as compared to the number of “locally” attending tokens.

For more information, please refer to the official paper .

Training¶

LongformerForMaskedLM is trained the exact same way, RobertaForMaskedLM is trained and should be used as follows:

input_ids = tokenizer.encode('This is a sentence from [MASK] training data', return_tensors='pt')
mlm_labels = tokenizer.encode('This is a sentence from the training data', return_tensors='pt')

loss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]

LongformerConfig¶

class transformers.LongformerConfig(attention_window: Union[List[int], int] = 512, **kwargs)[source]¶

This is the configuration class to store the configuration of an LongformerModel. It is used to instantiate an Longformer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the RoBERTa roberta-base architecture with a sequence length 4,096.

The LongformerConfig class directly inherits RobertaConfig. It reuses the same defaults. Please check the parent class for more information.

Parameters: attention_window (int or List[int], optional, defaults to 512) – Size of an attention window around each token. If int, use the same size for all layers. To specify a different window size for each layer, use a List[int] where len(attention_window) == num_hidden_layers.

Example:

from transformers import LongformerConfig, LongformerModel

# Initializing a Longformer configuration
configuration = LongformerConfig()

# Initializing a model from the configuration
model = LongformerModel(configuration)

# Accessing the model configuration
configuration = model.config

pretrained_config_archive_map¶

A dictionary containing all the available pre-trained checkpoints.

Type: Dict[str, str]

LongformerTokenizer¶

class transformers.LongformerTokenizer(vocab_file, merges_file, errors='replace', bos_token='<s>', eos_token='</s>', sep_token='</s>', cls_token='<s>', unk_token='<unk>', pad_token='<pad>', mask_token='<mask>', **kwargs)[source]¶

LongformerModel¶

class transformers.LongformerModel(config)[source]¶

The bare Longformer Model outputting raw hidden-states without any specific head on top.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters: config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

This class overrides RobertaModel to provide the ability to process long sequences following the selfattention approach described in `Longformer: the Long-Document Transformer`_by Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer selfattention combines a local (sliding window) and global attention to extend to long documents without the O(n^2) increase in memory and compute.

The selfattention module LongformerSelfAttention implemented here supports the combination of local and global attention but it lacks support for autoregressive attention and dilated attention. Autoregressive and dilated attention are more relevant for autoregressive language modeling than finetuning on downstream tasks. Future release will add support for autoregressive attention, but the support for dilated attention requires a custom CUDA kernel to be memory and compute efficient.

config_class¶: alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, masked_lm_labels=None)[source]¶

The LongformerModel forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –
Indices of input sequence tokens in the vocabulary.

Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

What are input IDs?
attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) –
Mask to decide the attention given on each token, local attention, global attenion, or no attention (for padding tokens). Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1, 2]: 0 for no attention (padding tokens), 1 for local attention (a sliding window attention), 2 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

What are attention masks?
token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

What are token type IDs?
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

What are position IDs?
inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

Returns

masked_lm_loss (optional, returned when masked_lm_labels is provided) torch.FloatTensor of shape (1,):

Masked language modeling loss.

prediction_scores (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size))

Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):

Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

attentions (tuple(torch.FloatTensor), optional, returned when config.output_attentions=True):

Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Return type

tuple(torch.FloatTensor) comprising various elements depending on the configuration (RobertaConfig) and inputs

Examples:

import torch
from transformers import LongformerModel, LongformerTokenizer

model = LongformerModel.from_pretrained('longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('longformer-base-4096')

SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000)  # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1

# Attention mask values -- 0: no attention, 1: local attention, 2: global attention
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
                                    # classification: the <s> token
                                    # QA: question tokens
                                    # LM: potentially on the beginning of sentences and paragraphs
sequence_output, pooled_output = model(input_ids, attention_mask=attention_mask)

LongformerForMaskedLM¶

class transformers.LongformerForMaskedLM(config)[source]¶

Longformer Model with a language modeling head on top.

This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters: config (LongformerConfig) – Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

config_class¶: alias of transformers.configuration_longformer.LongformerConfig

forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, masked_lm_labels=None)[source]¶

The LongformerForMaskedLM forward method, overrides the __call__() special method.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Parameters

input_ids (torch.LongTensor of shape (batch_size, sequence_length)) –
Indices of input sequence tokens in the vocabulary.

Indices can be obtained using transformers.LonmgformerTokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.encode_plus() for details.

What are input IDs?
attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional, defaults to None) –
Mask to decide the attention given on each token, local attention, global attenion, or no attention (for padding tokens). Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the <s> token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper https://arxiv.org/abs/2004.05150 for more details. Mask values selected in [0, 1, 2]: 0 for no attention (padding tokens), 1 for local attention (a sliding window attention), 2 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).

What are attention masks?
token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

What are token type IDs?
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) –
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

What are position IDs?
inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) – Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
masked_lm_labels (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) – Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]