Beyond Paragraphs: NLP for Long Sequences (2021)

Abstract: In this tutorial, we aim at bringing interested NLP researchers up to speed about the recent and ongoing techniques for document-level representation learning. Additionally, our goal is to reveal new research opportunities to the audience, which will hopefully bring us closer to address existing challenges in this domain.

## introduction a significant subset of natural language data includes documents that span thousands of tokens. the ability to process such long sequences is critical for many nlp tasks including document classification, summarization, multi-hop, and opendomain question answering, and document-level or multi-document relationship extraction and coreference resolution. these tasks have important practical applications in domains such as scientific document understanding and the digital humanities @xcite @xcite @xcite . yet, scaling state-of-the-art models to long sequences is challenging as many models are designed and tested for shorter sequences. one notable example is transformer models @xcite that have o(n 2 ) computational cost in the sequence length n , making them prohibitively expensive to run for many long sequence tasks. this is reflected in many widely-used models such as roberta and bert where the sequence length is limited to only 512 tokens. in this tutorial, we aim at bringing interested nlp researchers up to speed about the recent and ongoing techniques for document-level representation learning. additionally, our goal is to reveal new research opportunities to the audience, which will hopefully bring us closer to address existing challenges in this domain. we will first provide an overview of established long sequence nlp techniques, including hierarchical, graph-based, and retrieval-based methods. we will then focus on the recent long-sequence transformer methods, how they compare to each other, and how they can be applied to nlp tasks (see @xcite for a recent survey). we will also discuss various memory-saving methods that are key to processing long sequences. throughout the tutorial, we will use classification, question answering, and information extraction as motivating tasks. in the end, we will have a hands-on coding exercise focused on summarization. foot_0 2 description tutorial content this tutorial covers methods for long-sequence processing and their application to nlp tasks. we will start by explaining why processing long sequences is difficult. many popular models scale poorly with the sequence length, either in computational or memory requirements, making them too expensive or impossible to run on current hardware. another reason is that we want models that can capture long-distance information while ignoring large amounts of irrelevant text. the introduction also covers the tasks that we will use throughout the tutorial, namely information extraction (relation extraction @xcite and coreference resolution @xcite ), question answering (especially the multi-hop setting as in hotpotqa @xcite and wikihop @xcite ), and document classification, and summarization. the next section will review well-established methods for dealing with long sequences, namely chunking and graph based methods. chunking refers to splitting the sequence into smaller chunks, processing each one independently, then aggregating them in a task-specific way @xcite . hierarchical models are a special case of chunking where the chunks are linguistic constructs (usually sentences) that are aggregated following the document hierarchy @xcite . finally, retrieval-based methods use a recall-optimized simple model to retrieve short text snippets relevant for the task, then follow up with a stronger, more expensive model. retrieval methods have been discussed in detail in the open domain qa tutorial (chen and yih, 2020) so we will cover it here very briefly. graph-based methods will also be discussed, with a focus on question answering. these methods usually use local context to identify potentially relevant information across the document, heuristically connect the identified information in a graph, then apply a graph neural network (kipf and welling, 2017) to propagate information across the document between the snippets. this is particularly effective for the multi-hop reasoning setting @xcite . next, we will focus on the recent transformerbased methods for efficient processing of long sequences. the key question these models are addressing is how to perform the expensive o(n 2 ) self-attention computation efficiently. all models make this computation faster by approximating the full self-attention leading to different models with different behaviors and applications. we will survey a few of the key papers summarized in @xcite . in particular, we will talk about transformer-xl @xcite , longformer @xcite , reformer @xcite and linformer @xcite ). we will also discuss how they apply to nlp tasks; transformer-xl is mainly suitable for autoregressive tasks while the other three are equally suitable for autoregressive and bidirectional tasks. we will compare the performance of the other three models on various nlp tasks. the next section discusses pretraining and finetuning of the transformer models. for pretraining, we will discuss different approaches to warm start the model weights from existing pretrained models for short sequences @xcite . these approaches are versatile and make it possible to adapt most existing pretrained transformer models for short sequences into models that can process long sequences with a tiny pretraining cost. we will also demonstrate how to finetune such models for tasks such as question answering and classification. the following section is a practical use case on summarization. we will show how to start from the bart @xcite ) checkpoint, convert it into a model that can work with a long input that's tens of thousands of tokens long, then finetune it on a long-input summarization task. it will also discuss practical techniques necessary to run the model on current hardware, including memory optimization techniques such as gradient checkpointing @xcite and gradient accumulation. these are generic memory saving methods applicable to all neural models, and especially applicable in the long sequence setting. finally, the future work section will discuss open questions and future research directions like pretraining objectives that are better suited for long documents, encoder-decoder models with long output sequence, the balance between two-stage retrieval methods and single stage methods with long input, and how we think about long-sequence scaling for large models where the self-attention compute overhead reduces relative to feed-forward layers. relevance to acl the models we cover are generic machine learning tools, but we discuss them from the nlp perspective, and study their application to core nlp tasks like ie, qa, and text generation. these methods have the potential to improve tasks that are currently challenging like multi-document summarization, story generation, and long dialogues. it can also enable new applications that have not yet been considered. ## type of the tutorial this is a cutting-edge tutorial. the methods we discuss, especially the transformer-based and the graph-based methods, are active areas of research. ## outline this tutorial will be 3 hours long. 1. introduction (15 minutes long): this section will introduce the theme of the tutorial: why processing long sequence is important and why it is difficult. it will also introduce the nlp end-tasks that we will use throughout the tutorial. ## breadth we estimate 75% of the work covered will not be by the tutorial presenters. ## reading list reading the following papers is nice to have but not required for attendance. iz beltagy iz beltagy is a research scientist at ai2 focusing on language modeling, domain adaptation, and document-level understanding. his research has been recognized with a best paper honorary mention at acl 2020. he worked as a teaching assistant at the university of texas at austin teaching computer science courses. email: beltagy@allenai.org homepage: beltagy.net arman cohan arman cohan is a research scientist at ai2 focusing on representation learning and transfer learning methods, as well as nlp applications in scientific and health-related domains. his research has been recognized with a best paper award at emnlp 2017, an honorable mention at coling 2018, and harold n. glassman distinguished doctoral dissertation award in 2019. email: armanc@allenai.org homepage: armancohan.com hannaneh hajishirzi hannaneh hajishirzi is an assistant professor in the paul g. allen school of computer science & engineering at the university of washington and a research fellow at the allen institute for ai. her research spans different areas in nlp, focusing on developing machine learning algorithms that represent, comprehend, and reason about textual data at large scale. honors include the sloan fellowship, allen distinguished investigator award, intel rising star award, multiple best paper and honorable mention awards, and several industry research faculty awards. she has given previous tutorials at top nlp conferences. email: hannaneh@cs.washington.edu homepage: homes.cs.washington.edu/ ˜hannaneh/ sewon min sewon min is a ph.d. student in the paul g. allen school of computer science & engineering at the university of washington, advised by hannaneh hajishirzi and luke zettlemoyer. her research focuses on natural language understanding, question answering, and knowledge representation.