R ead O nce Transformers: Reusable Representations of Text for Transformers (2021)

Abstract: We present ReadOnce Transformers, an approach to convert a transformer-based model into one that can build an information-capturing, task-independent, and compressed representation of text. The resulting representation is reusable across different examples and tasks, thereby requiring a document shared across many examples or tasks to only be read once. This leads to faster training and evaluation of models. Additionally, we extend standard text-to-text transformer models to Representation+Text-to-text models, and evaluate on multiple downstream tasks: multi-hop QA, abstractive QA, and long-document summarization. Our one-time computed representation results in a 2x-5x speedup compared to standard text-to-text models, while the compression also allows existing language models to handle longer documents without the need for designing new pre-trained models.

## introduction transformer-based large scale language models (lms) @xcite are task-independent models that are surprisingly effective when directly fine-tuned on many different end-tasks @xcite . however, this approach relies heavily on using end-task supervision to learn to solve two sub-problems simultaneously: extract information 1 from an input document d and solve the end-task (e.g., answer a question about d). this incentivizes lm-based models to learn to extract only taskspecific-and even example-specific-information when fine-tuned on the end-task. for example, a question answering (qa) model may learn to only extract the answer from d given the input question. text-to-text model text-to-text model text-to-text model d 1 d 1 d 1 document encoder d 1 ... task-independent representations task-specific models readonce transformers reading comprehension summarization abstractive qa repr. + text model repr. + text model this strategy, while effective on many datasets, is also inefficient. first, it requires model's pretrained weights to be fine-tuned separately for each end-task, even though the sub-problem of gathering the information content of the input document d is shared across tasks. second, each d must be re-read from scratch in the context of each example (e.g., once for each question) even when many examples share d. not only is this computational redundancy undesirable, slow inference can quickly become a bottleneck in deployed, real-time systems if models with billions of parameters must re-read d for every input query. inspired by humans' ability to read a document and extract key information from it without having to know the use case in advance, we ask the following question: can we use transformer-based lms to build compressed representations of text that are example-and task-independent, and hence reusable? further, can we extend text-to-text transformer architectures to consume such representa-tions in conjunction with text? prior representation learning approaches attempt to capture the meaning of sentences into a continuous vector @xcite @xcite . while they have been effective on downstream classification tasks, it is unclear whether they can capture the information content of entire paragraphs. moreover, these approaches focus on building fixed-length representations that are used as the input features for task-specific classifiers. in contrast, our goal is to (a) use transformer-based lms to build compressed representations that scale with the document size, and (b) combine them with example-specific text inputs to produce the more general text output. to this end, we propose an approach to convert any encoder-decoder based transformer lm (such as bart @xcite ) into a new architecture termed readonce transformer, with two key parts: (1) a document encoder that reads documents only once to create compressed, informationcapturing, reusable representations that we refer to as readonce representations (2) a representa-tion+text model that consumes these document representations together with task-and examplespecific plain text (e.g., a question) to produce text output (e.g. an answer). to ensure that our compressed representations capture the key facts, we use supervision from two factoid qa datasets, squad @xcite and unsuper-visedqa @xcite to train readonce transformers. to solve an end-task, we only need to compute the readonce representations of the documents once and only train the representa-tion+text model to perform the end-task. our experiments demonstrate that these representations are more effective at capturing information compared to baseline approaches. our representations also generalize to other tasks such as multihop qa @xcite , abstractive qa @xcite , and summarization @xcite . since readonce representations are computed only once, we can train and infer with models 2x-5x faster than standard approaches, with only a marginal drop in accuracy (about 3 f1 points on qa and 4 rouge-l points on summarization for a 2x speedup). moreover, the compression ratio parameter k of our representations provides an easy way to trade off computation time with accuracy. specifically, our analysis suggests that the resulting model has a computation cost of roughly 1/2r + 3/4k 2 of the base lm, where r is the frequency of document reuse. additionally, our compressed representation enables us to efficiently combine information from long (or multiple) documents enabling more accurate long-document summarization @xcite without needing costly pre-training of new lms @xcite . ## related work representation learning approaches are commonly used to extract fixed-length sentence embeddings @xcite @xcite ) from variable-length text inputs. such fixed length representations have enabled the development of simpler downstream models that do not have to deal with the variable-lengths of textual inputs. however, these representations have mainly been used for simple classification tasks on short input texts @xcite . the word-level representations from rnns or transformers are also variable-length, but uncompressed. while such representations have been re-used with rnns @xcite and are easy to combine with text input, it is not immediately clear how to combine representations from transformers with text, which is what we propose. recent work @xcite @xcite has tried building document-embedding using large-scale language models as well. however these fixed-length representations have mostly been built to identify similar documents @xcite and are not used directly for qa. quase @xcite , also used questionanswering supervision for transfer learning but do not produce re-usable representations. @xcite learned multi-lingual sentence embeddings that may be able to capture the knowledge present in a sentence but they were designed for bilstms. some large-scale lms have been especially designed to handle long documents @xcite @xcite too but need to be pre-trained on large corpora, whereas we can use any pre-trained lm. aspects of our work also bears resemblance to domain adaptation @xcite , transfer learning (pan and yang, 2010) and multi-task learning @xcite but focuses on learning information-capturing represen-tations from transformer-based models that has not been explored by prior work. while model distillation @xcite can also result in speedups, these techniques are orthogonal and can be easily incorporated in our framework (as we show in our experiments). ## readonce transformers our goal in this work is to identify the optimal architecture to extract information-capturing reusable representations. at the same time, we also need to find the optimal architecture to use such representation in conjunction with text inputs. so at a high level (as shown in fig. 1 ), we need to develop two systems: (1) a model to compute the representation, document encoder and (2) a general model for tasks that can consume vector representations and text, representation+text model. given the recent success and generality of encoder-decoder models @xcite @xcite , we focus on developing models for such an architecture. we present the potential choices for each model, with the final model used in our system indicated by a *. ## representation learning experiments we first evaluate the different potential architectural choices for extracting and using document representations discussed in §3.1 and §3.2, respectively. while our main interest is in learning effective representations, we also need to find the optimal representation+text model architecture that can consume the representation. ## conclusion this work introduced readonce transformers, a novel approach for using large scale transformerbased language models to both build and consume reusable document representations. akin to humans' ability to read a document and extract useful information without knowing the enduse, readonce representations are compact, information-capturing document representations that can be pre-computed once, in a task-and example-independent fashion. our results on extractive qa, summarization, and abstractive qa tasks demonstrate that using readonce representations, in lieu of re-reading document text in the context of every example, results in substantially faster training and inference, at a modest cost in accuracy. the readonce framework also offers an easy way to control the trade off between speed and accuracy (via the compression ratio parameter), and enables the use of standard transformer architectures on long documents beyond the model's token limit. identifying the ideal compact document representations in our controlled setting opens up the possibility of efficient open-domain qa, where models retrieve and reason directly over these representations. we leave an exploration of the training of the retrieval function, often with only answer supervision and ideally in an end-to-end setting, to future work.