A Comparison of Different Punctuation Prediction Approaches in a Translation Context (2018)

Abstract: We test a series of techniques to predict punctuation and its effect on machine translation (MT) quality. Several techniques for punctuation prediction are compared: language modeling techniques, such as n-grams and long shortterm memories (LSTM), sequence labeling LSTMs (unidirectional and bidirectional), and monolingual phrase-based, hierarchical and neural MT. For actual translation, phrase-based, hierarchical and neural MT are investigated. We observe that for punctuation prediction, phrase-based statistical MT and neural MT reach similar results, and are best used as a preprocessing step which is followed by neural MT to perform the actual translation. Implicit punctuation insertion by a dedicated neural MT system, trained on unpunctuated source and punctuated target, yields similar results.

## introduction in speech translation, the first step often consists of automatic speech recognition (asr). most asr systems output an unsegmented stream of words, apart from some form of acoustic segmentation which splits a transcript into so-called utterances. translating this stream of words, using off-theshelf mt, results in a lower translation quality compared to translating punctuated input, as mt systems are usually trained on properly punctuated and segmented source and target text. end-to-end speech translation systems, that do not suffer from c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. we compare several techniques and approaches for punctuation prediction in a translation context, starting from an input that already contains the correct sentence boundaries. all techniques and approaches are trained on the same dataset, allowing us to fully attribute different results to the specific techniques and approaches used. thus, the main contribution of this paper is not introducing new methods for punctuation prediction, but a thorough comparison of methods previously used, since extensive comparisons are often lacking in related work. we compare three families of approaches for punctuation prediction: (1) language modeling, (2) sequence modeling, and (3) monolingual mt. these approaches are combined in three different architectures resulting in translated and punctuated output: (1) preprocessing adds punctuation before translating with a normal mt system, trained on punctuated source and punctuated target data; (2) implicit insertion adds punctuation during mt, which is trained on unpunctuated source and punctuated target data; and (3) postprocessing adds punctuation after mt, which is trained on unpunctuated source and unpunctuated target data. figure 1 shows these different strategies, together with the baseline strategy, in which the unpunctuated data is translated by a regular mt system trained on punctuated source and target data. ## related work in this section we discuss work that explicitly tries to predict punctuation marks like we do. we do not consider sentence boundary prediction. ## methodology we test several methods, keeping the data for training, tuning, and testing constant. section 3.1 describes the data, section 3.2 discusses the models for punctuation prediction and section 3.3 the bilingual translation models. finally section 3.4 explains how the quality of the punctuation prediction and translation is measured. ## conclusions and future work we set out to compare different approaches to punctuation prediction in the context of translation. we test several different architectures and methods for punctuation prediction as well as for mt, all trained on the exact same data sets, and evaluate the punctuation prediction quality as a monolingual phenomenon, as well as its effect on mt quality. while there is a clear deterioration of mt quality when working with unpunctuated input, this gap can be closed for 66% in the case of our best bilingual mt system, nmt, by applying monolingual mt as punctuation insertion, or by using a dedicated implicit insertion mt system. whether we use pre-or postprocessing did, in most cases, not result in a significant difference, indicating that the general punctuation prediction quality for dutch is similar to that of english. in future work, we would like to develop a similar experiment for segmentation prediction, and test the results on real speech signals in order to determine the usefulness of the results in a more realistic setting. a possible improvement would be to use nmt as punctuation prediction model, but constrain the word order with the help of the attention weights, thus combining the advantage of neural mt with the constraints on reordering of pbsmt.