Document Sub-structure in Neural Machine Translation (2020)

Abstract: Current approaches to machine translation (MT) either translate sentences in isolation, disregarding the context they appear in, or model context at the level of the full document, without a notion of any internal structure the document may have. In this work we consider the fact that documents are rarely homogeneous blocks of text, but rather consist of parts covering different topics. Some documents, such as biographies and encyclopedia entries, have highly predictable, regular structures in which sections are characterised by different topics. We draw inspiration from Louis and Webber (2014) who use this information to improve statistical MT and transfer their proposal into the framework of neural MT. We compare two different methods of including information about the topic of the section within which each sentence is found: one using side constraints and the other using a cache-based model. We create and release the data on which we run our experiments - parallel corpora for three language pairs (Chinese-English, French-English, Bulgarian-English) from Wikipedia biographies, which we extract automatically, preserving the boundaries of sections within the articles.

## introduction considerable progress has been made in machine translation (mt) thanks to the use of neural mt (nmt). while most nmt systems translate at the level of individual sentences, following similar practices in statistical mt (smt), there has been significant interest in recent years in using document context to improve translation @xcite @xcite . however the intermediate level of the internal structure of documents, particularly for documents with regular sub-structure, could also provide useful information to improve mt. documents are rarely without internal structure, and certain document types (e.g. biographies, scientific articles and encyclopedia entries) are characterised by a more well-defined and regular structure than others. in such documents, the structure is explicitly defined by sections associated with headings dealing with different aspects of the main subject. these sections, many of which can be found across multiple documents, are often associated with specific vocabularies or even grammatical patterns. figure 1 illustrates the classic structure of a wikipedia article, with section headings (e.g. "early life", "career", "personal life", etc.) that are likely to be found in other articles concerning high-profile people. this regularity in document sub-structure could be beneficial to the mt of such documents by providing additional information about the type of vocabulary used in different sections. exploiting document sub-structure in mt has previously been studied by @xcite for smt. they use topic models to integrate section information into a cache-based system and see improvements when applying their method to the translation of wikipedia biographies for @xmath0. however mt techniques have since changed considerably, and the quality of mt has improved with the introduction of nmt. the change in paradigm provides us with new methods of integrating external information. in this article, we draw inspiration from @xcite to explore the effect of using predictable docu-figure 1 : an extract of a wikipedia biography article, in which the document structure is summarised. ment structure to improve nmt. as in their work, we focus on the domain of wikipedia biographies, which are segmented into sections covering different aspects of the life of the person they are describing -their childhood, career, personal life, old age, etc. we automatically create datasets of parallel wikipedia biographies in three language pairs -french-english (fr-en), bulgarian-english (bg-en) and chinese-english (zh-en) -preserving document and document sub-structure information. we also collect monolingual wikipedia biography corpora in the four languages. 1 to test the usefulness of exploiting document substructure for nmt, we conduct experiments to compare methods us-ing topic information at the level of document sections. as in @xcite , we use topic modelling to model article sections. we compare two methods of integrating this information into nmt. the first uses side constraints @xcite and involves prepending topic information to source sentences. the second, adapted from @xcite , uses caches containing relevant vocabulary. it is similar to the approach used by @xcite but applied to nmt. we test these methods on three language directions (@xmath1, @xmath2, @xmath3. our main contributions can be summarised as follows: the remainder of the paper is organised as follows. section 2 presents relevant work on modelling document structure and background on the approaches we compare. section 3 describes the collection and processing of our datasets. in section 4 we present two methods of integrating document structure information. section 5 gives the experimental setup and results, and section 6 presents analysis of these results. finally, in section 7 we provide a conclusion and ideas for further research. ## related work the exploitation of document sub-structure has previously been studied in the context of smt by @xcite . the authors focus on the domain of wikipedia biographies, presenting domain-adapted mt models. as they discuss, the biography domain is interesting because it consists of documents with a regular structure. most biographies consist of sections that discuss topics such as their early life, career (which can fall into many categories based on the person), personal life and later life. to demonstrate the usefulness of document structure for mt, they compare models using document-wide context with models using context at the level of individual sections. they use two caches to pass information about a sentence's topic and its preceding context to the mt model. in their structured model they load the topic cache with words that are specific to the given section, as opposed to the whole document. they also clear and reload the caches at section boundaries instead of at document boundaries. they show that the structured model has an advantage over the model that treats documents as a whole, with topic information found to be particularly useful. while current work in nmt has not made use of document structure, there have been many efforts to supply nmt models with document-level context to improve translation @xcite @xcite . one way to integrate structural information into nmt is to adapt approaches to documentlevel nmt to consider section boundaries. of particular relevance is the work of @xcite , who supply contextual information using caches. their model involves two caches, a topic cache and a dynamic cache, which contain words that are important for the document-level context. the topic cache consists of words related to the document's topic, while the dynamic cache is updated to contain the translations of previous sentences within the same document and the current sentence up to the current time step. using both the topic and the dynamic cache provides the model with document-wide information ensuring consistency, as well as information from the preceding context which allows for better coherence. the topic information used in @xcite and @xcite is supplied using topic models that identify the most important words for any given topic. the method they use is latent dirichlet allocation (lda) @xcite . it is important to note that topic modelling can be done on entire documents as in @xcite or on sections within documents (as in this paper). in both cases, the text is treated as a "bag of words" and the important information comes from the frequency of the words in the segments. that is, topic modelling done on sections within documents will learn topics that are relevant to the text in the sections. furthermore, while topic models treat documents as a mixture of topics with different probabilities, for downstream tasks it can be useful to only consider one or several of the most probable topics. for instance, @xcite only consider the topic with the highest probability for each section of a document. ## datasets we follow @xcite in using wikipedia biographies to test whether section information can be useful for nmt. we agree that data from wikipedia is particularly illustrative of regularly structured documents as it contains separate sections with section headings. it is reasonable to assume that some sections within different biographies might share similar vocabulary, while other sections in the same documents may be very different. for instance, a biography of a composer and one of a basketball player may both contain information about their early years, but the rest of the documents may be different. therefore, providing more information about the recurring topic may be useful, as opposed to providing only document-level information which may be too general. we use a similar method to @xcite for data creation. however, we extend the process to cover three language pairs (fr-en, zh-en, bg-en), and considerably more data, foot_1 which is necessary as nmt models require more data to be trained effectively. data was collected and processed using the following steps, which are described in more detail below: ## integrating document sub-structure we compare two methods of integrating document structure information, both relying on the assumption that different sections of the text cover different topics. here, we consider different sections to be those that are delimited by headings (whether these are section, subsection or subsubsection headings), which creates a flat hierarchy of delimited areas of text. in both cases, we follow @xcite in using topic modelling to learn section topic representations, which we describe in section 4.1. the methods differ in how this learnt topic information is integrated. the first, described in section 4.2.1, uses side constraints @xcite to incorporate information about the section topic associated with each sentence. for the second method we use @xcite 's cache-based neural model to provide the model with topic information and previous context within the boundaries of the current section. while our first method is simple to implement and train, the second provides more fine-grained information about each section within an article. ## experiments we apply the two methods described in section 4 to the translation of three languages (fr, zh and bg) into english. we train and test the models using the wikipedia biography data previously described in section 3. ## analysis to gain more insight into the differences between the proposed models, we briefly analyse the cache model topics (section 6.1), and in section 6.2 provide a manual evaluation and some qualitative analysis of translations. 6 with the exception of the bg-orig. test set for which there is no significant difference between the two methods at the document level, which could be due to the fact that this test set is very small. ## conclusion and future work in this paper we propose two methods to transfer into the framework of @xcite 's idea of exploiting document structure when translating documents with regular and predictable structure. as in their work, we use topic modelling to model document sections, under the assumption that different sections within articles display different lexical properties, and compare the integration of this information using side constraints @xcite to a more complex approach using cache mechanisms, adapted from @xcite . we have created three parallel corpora of wikipedia biographies (for en-fr, en-bg and en-zh), structured into sections, as well as monolingual corpora for all four languages, all of which will be made freely available. our experiments using this data show that there are no consistent gains to be seen across all language directions for a particular model type, and using section-level information as opposed to document-level information does not systematically improve mt quality. these results, which are different from results found by @xcite for smt, suggest that while there are circumstances in which providing section (or document) topic information does help nmt through domain adaptation (shown by gains for certain subsets of the test set), in high-resource scenarios such as the ones tested here, this information is not systematically useful. a direction to look into in the future is how this type of information could help low-resource mt, as a way of injecting monolingual topic information in a more light-weight and efficient manner than most current techniques @xcite .