# Attention Rollout -- RoBERTa

In this demo, we use the RoBERTa language model (optimized for masked language modelling and finetuned
for sentiment analysis). The model predicts for a given sentences whether it expresses a positive,
negative or neutral sentiment. But how does it arrive at its classification?  This is, surprisingly
perhaps, very difficult to determine.

Abnar & Zuidema (2020) proposed a method for Transformers called **Attention Rollout**, which was further
refined by Chefer et al. (2021) into **Gradient-weighted Attention Rollout**. Here we compare them to
another popular method called **Integrated Gradients**.