Does Sentence Segmentation Matter for Machine Translation? (2022)

Abstract: For the most part, NLP applications operate at the sentence level. Since sentences occur most naturally in documents, they must be extracted and segmented via the use of a segmenter, of which there are a handful of options. There has been some work evaluating the performance of segmenters on intrinsic metrics, that look at their ability to recover human-segmented sentence boundaries, but there has been no work looking at the effect of segmenters on downstream tasks. We ask the question, “does segmentation matter?” and attempt to answer it on the task of machine translation. We consider two settings: the application of segmenters to a black-box system whose training segmentation is mostly unknown, as well as the variation in performance when segmenters are applied to the training process, too. We find that the choice of segmenter largely does not matter, so long as its behavior is not one of extreme under- or over-segmentation. For such settings, we provide some qualitative analysis examining their harms, and point the way towards document-level processing.

## introduction contemporary machine translation assumes a sentence-level paradigm. however, data doesn't exist naturally at the sentence level, requiring the use of automatic segmenters to split the data at both training and inference time. training data is prepared with the use of sentence segmenters, 1 which are preprocessing steps that occur prior to alignment and bitext creation. at test time, deployed models also require the use of a segmenter. many times, for downloaded models, especially, this inference-time application must be made without knowing what segmenter was used to train the model, introducing a potential misalignment or discrepancy and resulting performance degradation. 1 sometimes called sentence breakers. sentence segmentation itself has received only a little attention in the research literature, although there has been a recent uptick @xcite . but to our knowledge, no work has been done investigating the effects of segmentation on machine translation. in fact, most research papers do not deal with the question at all, relying as they do on pre-segmented parallel data for both training and test time. this is a practical problem for deployment scenarios, where segmentation must be considered. it is also a deeper problem, since segmentation is ultimately a modeling decision that should be noted and made available with any published models, such as is done for other modeling decisions affecting input text, such as normalization, tokenization, and subword processing. to understand whether and to what extent segmentation matters, we ask a series of questions: (1) what segmenter is best used at inference time? (2) when training a model, how important is the choice of segmenter? we break down this last question into two settings: (i) the standard training procedure in which sentences from parallel documents are aligned @xcite , and (ii) more recent "mining" approaches, which use sentence representations to find sentence pairs without regard for document boundaries. ## evaluation our research questions address two scenarios. in the first, a researcher has downloaded a shared model and wishes to use it to translate new data. in many instances-perhaps most-the providers of the model have neither shared nor reported what segmenter they used. likely the model was trained on "standard" provided datasets such as those from wmt. we would like to have some understanding of the effect of different segmentations when we don't have control over the training segmentation. alternatively, we have a "glass box" model. in this setting, we are training the model, and have full flexibility over the choice of segmentation. by reconstructing the entire nmt pipeline with segmentation as the first step of dataset preprocessing, the researcher has complete control over the resulting model. this settings provides us with a more granular look at the effect of segmenter choice. ## experimental setup we focus on our investigation on english and german. we make this choice because this language pair has sufficient document-level information in datasets released by wmt. for many language pairs, datasets with true document pairs do not exist. a wider consideration of language pairs is not possible without further work to cultivate document-pair datasets. given document pairs in german and english, we extract sentences with a variety of segmenters and apply a typical document-based aligner to create bitext. each segmenter creates a unique training set which we use to train a neural machine translation model. traditional alignment methodologies assume true document pairs. a search through both the source and target assumes alignments will be found in roughly sentence order. vecalign @xcite is an example of one of these document-based aligners. this method has the benefit of being capable of recovering erroneous segmentation because over-segmented sequences will still be consecutive during the search. the growing field of bitext alignment has created new trends that search for sentence pairs outside the context of a document. one method of extracting sentences from all documents and searching globally for a sentence pair has created massive datasets such as ccmatrix @xcite , wiki-matrix @xcite , and ccaligned @xcite . to compare the effects of segmentation in conjunction with both alignment techniques, we train models on data produced from all segmenters using both a document-based alignment method and a global, context-less based aligner. ## segmentation at inference with a black box system in order to replicate a real-world use-case, we use an off-the-shelf pre-trained model. datasets used to train these models are reported, but for the mostpart, segmentation is unknown. we consider test sets in a variety of language pairs foot_7 for comprehensiveness. for model consistency, we chose a multilingual nmt model. we use the prism @xcite as a blackbox translation model, after applying the segmentation methods described in section 2, we translate with prism. preserve realign document prism prism prism original 27.1 27.5 28.9 always 28.7 29.1 30.5 ersatz 29.0 29.4 30.8 moses 29.0 29.5 30.8 punkt 29.0 29.4 30.8 spacy 28.7 29.3 30.6 we report results in ## segmentation at training with a glass box system segmentation occurs at an early stage in the nmt pipeline, so it is intuitive to think it could have a large effect: incorrect segmentations can lead to incorrect alignments; incorrect alignments lowers the quality of the training data; and low quality training data will produce worse models. in order to study the effects on training, we recreate the nmt pipeline by segmenting documents and aligning bitext to train models. we apply each segmenter to the training data resulting in a new unique set of "sentences" for each segmenter. we can then align these sentences to create a unique dataset. ## qualitative analysis hallucinations, or addition of content during translation, and deletions are common in neural machine translation. these models are no exception. qualitative analysis reveals two types of errors that are worth investigating further: 1) seemingly arbitrary deletion of content when the input is unsegmented 2) addition of content without a true signal in the source. we suspect the explanations for these behaviors are 1) a lack of many-sentence inputs occuring in training data and 2) incorrect segmentations leading to poorly aligned data. we display some examples in ## related works to the best of our knowledge, not much work has been done about the effects of segmentation on down stream tasks. @xcite investigates corpus-level noise and empirically links noise patterns to types of nmt hallucinations. other work has focused on the effects that punctuation has on neural language models @xcite . in online simultaneous speech segmentation, @xcite proposes an online sentence segmentation approach which improves downstream bleu scores. there is much more work pushing away from the sentence-level paradigm and encouraging document translation. @xcite has recently shown that modern neural architectures still achieve strong performance with longer, multisentence inputs. a spate of recent work has gone into better document evaluation metrics @xcite . document pairs contain the additional context needed to correctly translate certain discourse phenomenon such as coreference resolution and consistent lexical choices. further, mining documents instead of sentences circumvents the error propagation from using various segmentation methodologies during bitext mining. ## conclusion an nmt system trained on segmented data requires segmentation at inference; however, the exact method of segmentation at inference seems to have little quantitative effect. the larger impact of segmentation occurs during the creation of bitext. whether the effect stems from the quality of the produced sentence pairs or the limitations of different alignment methods cannot be determined based on these results. despite this, various segmentation and alignment method combinations create significantly different amounts of bitext to train models on-something that needs to be investigated further. the differences in the resulting data produce models that perform differently. lastly, we note that when models are trained on segmented data, they dramatically hallucinate at inference with unsegmented data by deleting long segments. by adding some amount of unsegmented data in the training data, this effect can be mitigated to recover upwards of 4 bleu points. together, we might conclude that avoiding segmentation is the path forward. when the segmentation and alignment techniques failed, half a million sentence pairs were sometimes lost or left unaligned. additionally, we see that less-segmented bitext produces models that are more robust to unsegmented data at inference. the biggest hurdle in training document-level models is the lack of sufficient document-level annotations. if true document pairs exist in larger web-scraped corpora, most of the original document structure (and informative context) has been removed via bitext filtering and deduplication. future work might explore potential solutions to mining document-level data, and circumvent these segmentation tools and their respective noise.