Recovering document annotations for sentence-level bitext (2024)

Abstract: In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets was less noticeable. Now, despite the emergence of long-sequence methods, we remain within a sentence-level paradigm and without data to adequately approach context-aware machine translation. Most large-scale datasets have been processed through a pipeline that discards document-level metadata. In this work, we reconstruct document-level information for three (ParaCrawl, News Commentary, and Europarl) large datasets in German, French, Spanish, Italian, Polish, and Portuguese (paired with English). We then introduce a document-level filtering technique as an alternative to traditional bitext filtering. We present this filtering with analysis to show that this method prefers context-consistent translations rather than those that may have been sentence-level machine translated. Last we train models on these longer contexts and demonstrate improvement in document-level translation without degradation of sentence-level translation. We release our dataset, ParaDocs, and resulting models as a resource to the community.

## introduction since the early days of statistical methods, machine translation has been centered within a sentencelevel paradigm. n-gram based approaches, which typically obey sentence-boundaries, were the predominant machine translation method and did not effectively use the wealth of information contained across contexts @xcite . later, newer neural techniques, such as transformers @xcite , became popular and have been shown to be effective at handling longer sequences @xcite @xcite . during this time, researchers have periodically considered contextaware machine translation in training methodologies @xcite , evaluation sets and metrics @xcite @xcite , and its unique ability to address discourse phenomena which are otherwise impossible to correctly translate without context @xcite . to present, little work has been done to address the most obvious hurdle: a lack of document-level training data. many cornerstone datasets were created by finding known sources of professionally translated documents @xcite @xcite . these parallel documents were then sentence-segmented and aligned with a document-based alignment technique. when more data was demanded, these datasets proved insufficient, so data curators moved towards global mining-treating web crawls as a bag of sentences and searching for the most similar sentence in the target language @xcite . this removes document order and makes document-reconstruction impossible. some datasets, such as paracrawl, exist as a sentence-level resource but have been constructed in a way more amenable to documentreconstruction. this work confronts this resource gap by providing document-level annotations for news commentary, europarl, and the unfiltered paracrawl @xcite data. our contributions are: art is trying to become science. die kunst versucht, wissenschaft zu sein. it deals with philosophy, sociology and psychology, and desires to be a pedagogical resource too. it wants to be political, yet maintain the charisma of the underground (that is, by being non-political). it wants to be a commodity. sie beschäftigt sich mit philosophie, soziologie, psychologie und möchte eine quelle für die pädagogik sein. die kunst will politisch sein, aber sich das charisma des unpolitischen undergrounds erhalten. sie will ware sein. ## related works three areas of research are relevant to this work: sentence-level bitext mining, document-level bitext mining, and context-aware machine translation. sentence-level mining is the current default for most of the largest parallel datasets. in the most recent wmt translation task @xcite , parallel training data (i.e., paracrawl or wikimatrix @xcite )) tends to be sentence-level. similarly, of the top ten corpora on opus @xcite @xcite , which make up over 93% of their entire collection, eight foot_1 are sentence-level @xcite and comprises over 72% of the data, with the exceptions being opensubtitles @xcite and dgt foot_2 which have rather specific domains. 4 many of these datasets are constructed via the global-mining technique, which indexes all sentences using some semantic-based hashing using laser @xcite and aligns based on similarity matches irrespective of document-boundaries. document-level mining is representative of early datasets where document-alignment was already assumed and sentence-alignment could be done via simple features such as sentence lengths @xcite . this technique was used to create original datasets such as europarl @xcite . other techniques are constrained by document order when aligning sentences @xcite @xcite . these have been used to create more recent corpora such as news commentary @xcite . paracrawl @xcite initially aligns documents; however, the preprocessing before the final release destroys any of this preserved document-structure. ccaligned (el-kishky et al., 2020b) has a middle-ground methodology which first aligns documents determined by both a combination of url pairs and document similarity, followed by a laser-based alignment. ccaligned releases data in rough document-order; however there are no annotations for document boundaries or other context labeling. recent work has also explored using paracrawl as a source for document-level data due to the original alignment being constrained by documents @xcite . to extract documents, they extract paragraphs by using a subset of the data for which wmt released aligned document-level data in german-english as a document-alignment benchmark task. this work diverges from this idea in considering the full paracrawl release which is a significantly larger portion of data and also expands the language set. context-aware machine translation concerns itself with any form of integrating additional context to translate. there are a handful of well-studied discourse phenomena, such as gendered-pronoun translation, which are impossible to translate without incorporating this information (wicks and @xcite @xcite @xcite . further, transformers are entirely capable of incorporating longer contexts as input @xcite . thus, the field of context-aware translation has either involved modifying the input framework to better incorporate the signal of context-features @xcite @xcite , or studying alternative uses of training data @xcite . the recent introduction of large language models (llms) has also had a swath of studies that show these llms are better-situated for documentlevel machine translation as they naturally have significantly large context-windows @xcite @xcite . llms provide a desirable path towards document-level translation; however with their novelty and training pipeline opacity, it is still unknown what types of data elicits this performance in order to replicate on different sets of languages or smaller scales. ## document annotation and reconstruction released bitext files rarely have document annotations necessary to create contextual inputs for context-aware training. in the case of europarl, news commentary, and paracrawl, document ids are released which point to the original monolingual files. we use these monolingual files to reconstruct the original parallel document structure. we construct annotations for each original aligned segment in the bitext. each segment has an individual source and target segment which receive separate annotations dependent on their original document. following the reconstruction process described in section 3.1, we define a pipeline (section 3.2) to extract contexts from this data. finally, we describe a novel document filtering method (section 3.3) that can be used to filter out low-quality documents instead of only low quality sentences. with each progression of these steps, one can whittle the dataset smaller while improving the relative quality of documents. all filtered splits, and the original unfiltered data, are publicly released to facilitate further research into context-aware machine translation, and document-based data selection and filtering methods. the sizes of each dataset with the increased filtering is displayed in ## source data to produce our dataset, we select source data from three large, publicly-available datasets that initially used document-level alignment. paracrawl @xcite prioritizes a high-recall alignment so the original unfiltered data is orders of magnitude larger than the official release, but has an inferior quality. for instance, paracrawl v9 en-de is approximately 278m lines of cleaned sentences in arbitrary order, but an additional "raw" file is available that contains nearly 10b of uncleaned sequential sentence alignments. 7 the official release has undergone the aforementioned sentence-level filtering (deduplication, similarity score filtering, etc) that removes the original context. fortunately, paracrawl releases an original unfiltered (raw) data version. this equates to each source-target pair, a document id, and a pointer to the original monolingual source. 8 news commentary @xcite is a smaller, albeit cleaner newswire dataset released annually for wmt training data. 9 the released versions do maintain document order, but documents are not labeled. europarl @xcite was produced from the proceedings of the european parliament which is obligatorily translated into a handful of european languages. europarl is typically n-way parallel which makes it ideal for machine translation despite the specific domain. the most recent version (v10) releases document ids; however, it is not available in all languages. we produce the alignment ourselves with the accompanying tools. ## experimental design in order to show the viability of constructing a dataset in such a fashion, we need to show two things: (1) a contextual model is at least as good at sentence-level translation quality as a sentencelevel model and (2) a contextual model outperforms sentence-level models when considering contextbased phenomena. ## results in general, we find that context-aware models trained with the paradocs data outperform any model trained without contextual inputs across evaluation metrics. further, we find that the document-level filtering method is able to improve performance even as data quantity diminishesindicating a higher quality of data selection. in ## analysis of document filtering document-level filtering is rare, and to our knowledge, has not been studied. we demonstrate that our slide-cometkiwi filtering improves contextual performance, but this may be an indication of better sentence-level translations and not better document-level translations. for a documentlevel filtering method to hold up against webcrawled data, it should not only filter out poor quality translations, but should also filter out contextinconsistent translations. by context-inconsistent, we refer to translations that were produced at the sentence-level. this is roughly synonymous with machine translated text in a web-crawl setting; however, there is no robust machine translation detection methodology to the best of our knowledge. we instead turn towards a proxy measure to determine if context-inconsistent translations are filtered out. after scoring each document as de-scribed in section 3.3, we can rank documents sequentially based off these slide-cometkiwi scores. we should find more context-consistent translations in the upper percentiles of this distribution and fewer at the lower-end. the ctxpro toolkit, which generated these evaluation sets, also identifies when there are ambiguous uses of gendered pronouns. as the toolkit is intended to be high precision, low recall, it is extremely likely to not identify ambiguity in pronouns when the translations are incorrect or inconsistent. thus, we can use the ctxpro toolkit as a proxy measure for context-consistent translations. we identify all examples of gendered-pronouns in our data according to the ctxpro toolkit, and investigate where slide-cometkiwi ranks them in relation to the remaining documents. we show percentages in as shown in ## conclusions research in contextual machine translation is hampered by the lack of document annotations on parallel data. we augment three large popular mt datasets (paracrawl, news commentary, and europarl) with this information, creating a document-level dataset, paradocs. we introduce a document-level filtering method to apply to this data in lieu of traditional context-destroying sentence filtering methods. simple, context-aware machine translation models trained on this data have shown to be better at machine translation in both general performance as measured by wmt test sets as well as targeted performance-measured by the ability to correctly translate discourse phenomena. we release the data as a resource to the community. 10