Document Level Hierarchical Transformer (2021)

Abstract: Generating long and coherent text is an important and challenging task encompassing many application areas such as summarization, document level machine translation and story generation. Despite the success in modeling intra-sentence coherence, existing long text generation models (e.g., BART and GPT-3) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the model to revise, replace, revoke or delete any part that has been generated by the model. In this paper, we present a novel semi-autoregressive document generation model capable of revising and editing the generated text. Building on recent models by (Gu et al., 2019; Xu and Carpuat, 2020) we propose document generation as a hierarchical Markov decision process with a two level hierarchy, where the high and low level editing programs. We train our model using imitation learning (Hussein et al., 2017) and introduce roll-in policy such that each policy learns on the output of applying the previous action. Experiments applying the proposed approach sheds various insights on the problems of long text generation using our model. We suggest various remedies such as using distilled dataset, designing better attention mechanisms and using autoregressive models as a low level program.

## introduction generating long and coherent text encompass various tasks such as summarization, story generation, document level machine translation and document level post editing. each task is characterised by modelling long range dependencies to make the document coherent as well as modelling a high level plot to make the document thematically consistent @xcite . this is challenging as the models need to plan content, while producing local words consistent with the global context in a timely manner. recent work on autoregressive generation models, such as gpt-3 and bart @xcite , have shown impressive performance in generating short fluent text with a maximum length ranging from 150 to 350 tokens @xcite @xcite . but applying the same model to generate longer passages of text (e.g., 1000 tokens) has resulted in syntactic and semantic errors throughout the document requiring extensive human curations @xcite . these massive language models are usually pre-trained using large corpora of generic text, and then fine-tuned with small domainspecific data. most of the time, the models are not publicly available to adapt to arbitrary desired domains. on the other hand, recent non-autoregressive approaches allow generation to be done within a much smaller number of decoding iterations @xcite @xcite . but due to its problems with modelling dependencies among the tokens, the approach still lags behind its autoregressive counterparts and has not yet been applied to long text generation @xcite . in both of these model families, the length of generated sequences is either fixed or monotonically increased as the decoding proceeds. this makes them incompatible with human-level intelligence where humans can revise and edit any part of their generated text. in this paper, we present a novel semiautoregressive document generation model capable of revising and editing the generated text. we build on recent models by @xcite , who framed generation as a markov decision process @xcite and showed that iteratively refining output sequences via insertions and repositions yields a fast and flexible generation process for machine trans-lation and automatic post editing task. we extend their model by proposing document generation as a hierarchical markov decision @xcite process with a two level hierarchy. the high level program produce actions a @xmath0, insert, update} which tries to capture global context and plan content while the low level program produce actions a @xmath1, insert} to generate local words in a consistent and timely manner. due to unavailability of large-scale data to train our model, we propose a noising process to simulate the error patterns observed in document level tasks such as redundancy of words, key information omission and disordered sentences. the noising process can be reversed by applying a set of high and low level actions to get back the original document. this serve as an efficient oracle to train our model using imitation learning @xcite . the rollin policy is defined such that each policy learns on the output of applying the previous action. ## related work previous work on long text generation has mostly focused on generating tokens up to three hundred words. these method usually employ the idea of planning a document before generating it @xcite @xcite . another line of work, focus on extending transformer architecture to model long sequences @xcite . recent work by @xcite used pre-train language models to progressively generate longer text greater than 300 tokens. our work differs from previous approaches as it allows editing the generated text while it is being written. previous work on non-monotonic generation and refinement @xcite @xcite has mostly focused on generating shorter text. our proposed approach, differs from prior works by extending non-monotonic generation towards longer texts. ## conclusion we present a hierarchical document generation model, that is capable of revising and editing its generated text thus bringing it closer to humanlevel intelligence. although results showed that our approach lags behind the baselines, it did shed light into various problems present in semiautoregressive models and long document generation. in the future, we will be incorporating these insights into our model to make it more robust.