R ead T wice: Reading Very Large Documents with Memories (2021)

Abstract: Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose ReadTwice, a simple and effective technique that combines several strengths of prior approaches to model long-range dependencies with Transformers. The main idea is to read text in small segments, in parallel, summarizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of comparable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books.

## introduction transformer-based models such as bert are very effective in capturing long-range dependencies in text passages through the attention mechanism @xcite . however, the amount of compute in attention depends quadratically on the number of tokens in an input text passage. as such, the standard bert implementation limits input size to a fixed number (often 512) of tokens. in reality, dependencies over significantly longer ranges are common and modeling them is crucial. for instance, in a sentence like inside the sammath naur, the ring-bearer struggled to throw the ring into the volcano, the narrative interweaves several prior storylines from a book. comprehending this sentence therefore requires looking up previous mentions of ring-bearer and sammath naur, located many tokens away. several methods have been proposed to address this challenge; see @xcite for a survey and §3 for a detailed discussion. one popular strategy is to reduce the number of tokens attended to. longer inputs can in fact be processed in this way -but only up to a limit of around 5,000 tokens, as used in @xcite @xcite ) -far below the context sizes required to model long documents such as books. another strategy such as hibert @xcite splits inputs into smaller segments which are processed individually, then assembled into a hierarchical representation. as a downside, intersegment context is unavailable during encoding. we propose readtwice, a simple approach that combines the strengths of both strategies. as its name suggests, the main idea is to process the input twice: a long text input (such as a document, or even a book) is treated as a collection of shorter text segments which are read independently and in parallel. then, the encoder reads each segement again, now augmented with compressed information from other segments. the crucial component in readtwice, as illustrated in figure 1 , is a memory module that holds compressed information from all segments. that compressed information is used only once: in the second pass. thus, readtwice is much more computationally efficient than models like etc that rely on memory for all segments, in every layer. while readtwice requires two passes, it differs from hierarchical models such as hibert that do not condition segment encoding on other segments. ## method we first describe the readtwice model, followed by its pre-training procedure. ## conclusion & future work readtwice performs well on several qa tasks, particularly narrativeqa where long-range dependencies among entities appear to be very important. the proposed method is conceptually simple, easy to implement and is capable of reading entire books. for future work, we plan to explore new memory types, hierarchies and aggregation functions. we also aim to apply the model to other tasks, particularly long text summarization, likely to benefit from a memory-forming mechanism.