Comprehensive Punctuation Restoration for E nglish and P olish (2021)

Abstract: Punctuation restoration is a fundamental requirement for the readability of text derived from Automatic Speech Recognition (ASR) systems. Most contemporary solutions are limited to predicting only a few of the most frequently occurring marks, such as periods, commas, and question marks - and only one per word. However, in written language, we deal with a much larger number of punctuation characters (such as parentheses, hyphens, etc.), and their combinations (like parenthesis followed by a dot). Such comprehensive punctuation cannot always be unambiguously reduced to a basic set of the most frequently occurring marks. In this work, we evaluate several methods in the comprehensive punctuation reconstruction task. We conduct experiments on parallel corpora of two different languages, English and Polish - languages with a relatively simple and complex morphology, respectively. We also investigate the influence of building a model on comprehensive punctuation on the quality of the basic punctuation restoration task

## introduction the task of restoring punctuation can be crucial for the readability of text derived from asr systems. as @xcite has shown, a lack of punctuation in transcription can have a greater negative impact on readability than a large number of word transcription errors. in recent years, punctuation prediction was most often approached as a token classification task @xcite , @xcite , @xcite ). in this context, the target labels are often reduced to only a few most frequently occurring marks, such as periods, commas, and question marks. however, in written language, we deal with a much larger number of characters (such as parentheses, hyphens, etc.). the usual approach is to try to reduce those punctuation marks into the basic set via role similarity (e.g., semicolon and exclamation marks are often reduced to periods) or discard them entirely @xcite , @xcite ). however, such a process always comes with a loss of information. furthermore, a word can end with more than one punctuation mark -for example, the end of parenthesis can coincide with the end of a sentence, resulting in the combination of these marks into '). '. predicting only a period in such a place would quite strongly violate the structure of the original statement (see in the following work, we explore the possibility of generating a manageable-sized set of labels directly from the dataset, based on the percentile of punctuation cases present in the set. we measure increased recall by using broader class sets and a potential cost in terms of precision. in addition, we test whether models trained on more narrowly defined classes will suffer (or gain) on a reduced, conventionally defined 4-class task. we conducted our research on a parallel corpus of polish and english -two languages with very different levels of morphological complexity @xcite . with this approach, we can directly compare a set of semantically identical and volumetrically very similar datasets and see how well our results generalize. we would be able to catch if a trend in our results was very specific to a single language. in summary, in this paper we made the following contributions: also, to the best of our knowledge, our proposal is the first publicly described research for restoring punctuation for the polish language. 1 2 related work the first approach to punctuation restoration (in the sense of restoring punctuation marks) has been proposed by @xcite . they introduced a model based on the markov chain, designed for restoring commas in the output of asr systems. in the field of deep learning, the punctuation restoration task is often approached with bidirectional recurrent neural networks. most often lstm and gru architectures are used. although lstm networks are often -computational performance aside -considered better than grus in the general case ( yang et al. (2020), weiss et al. (2018) ), it is reported in several papers that grus outperformed lstms in the punctuation restoration task @xcite , @xcite ). in @xcite authors explored the possibility of using bidirectional recurrent networks with attention for the punctuation restoration 1 there is the polish language mentioned as a part of a multilingual model in @xcite , however, the authors did not publish per-language results. on the estonian language. they provided their code foot_0 with the publication and we will be using it in our research as an example of a recurrent network model. lately, an interesting approach based on lstms was proposed by @xcite , where they tried to create a single model for restoring punctuation for 43 languages using language-independent bpe tokenization. they also included polish in the training set, however, authors did not publish per-language results. in recent years, large, pre-trained models based on transformer architecture @xcite seem to perform best on a number of nlp tasks, including punctuation restoration. perhaps the most comprehensive comparison of various transformer encoder models in the task of punctuation restoration is done by @xcite , where the authors compared a number of models based on different variants of pre-trained bert @xcite , roberta @xcite and albert @xcite encoders as a base of their model. they've shown that generally larger pre-trained models were better than the smaller ones in punctuation restoration and that between models of the same size, generally, roberta was better than both albert and bert. they also showed that xlm (cross-lingual models) variants of roberta were slightly worse than english-only ones. the authors of the paper published their code and we also used it in our research. in @xcite authors show that punctuation restoration can also benefit from multitask learning (pos tagging being the secondary task in their work). they trained a single bert-based model with 2 token classification heads -one for punctuation restoration and one for pos tagging. while the weights in the heads were separate in the task, the bert core was shared. they have shown that such a form of regularization can help in the punctuation restoration of unseen data. our take on the punctuation restoration task is inspired by the work of @xcite , where the authors used an approach with automatic generation of a set of labels from the data to approximate the capabilities of sequence to sequence models token classification. they did it in the context of the grammatical error correction task. to be able to research comparable corpora for different languages, similarly to @xcite , we used a parallel corpus from the europarl v7 dataset 3 . the corpus is extracted from the proceedings of the european parliament and translated into multiple languages. specifically, we use the parallel corpus of polish and english taken from proceedings from 01/2007 to 11/2011. the corpus is made up of 15.27m words (english) and 12.82m words (polish) divided into sentences, with each sentence on a separate line. as some of the lines are very short and contain e.g. only a single number, we removed all of the lines that had fewer than 4 words as a preprocessing step. then we divided the corpus randomly into training, validation, and test collection in the ratio 8/1/1 (line-wise). see the text preprocessing step consisted only of normalization of all whitespace characters (including newline) into a single space. the decision was motivated by the fact that whitespace is mostly connected with formatting rather than punctuation. in the specific case of the dataset we used, new lines were used to separate sentences. however, if the dataset was annotated in a way that whitespace formatting was meaningful (ie., using newlines or tabulations for paragraph splitting), this step could be skipped and attempts could be made to also reproduce subtle differences in whitespace. after preprocessing, the text was broken into tokens based on the occurrence of any nonalphanumeric character (including whitespace). each alphanumeric sequence was considered a single token, and each non-alphanumeric sequence following it was considered a label of that token. the set of all unique non-alphanumeric sequences was considered the largest possible set of punctuation labels for this specific dataset. the classes from the label set were then sorted by their frequency of occurrence in the text. obviously, in most cases, by far the most represented class was single whitespace (that made up 88.36% of all labels in the english dataset and 85.14% in polish). overall, we got 513 unique classes for the english version and 716 classes for polish one. in both cases, there was a long tail of underrepresented classes. such classes consisted mainly of combinations of rare marks (eg. ""=.") or very long strings of punctuation characters (e.g., " [.../...] ["). in the case of the english dataset, there were 330 classes with fewer than 5 occurrences and 186 classes with only one occurrence. in the case of polish, such a long tail was even longer, with 486 classes with less than 5 occurrences and as many as 287 with only one occurrence. as stated in the introduction, the goal of this work is to reproduce as much of the original punctuation as possible. because of that, in the test set no class reduction was done and all the original labels were put there in unchanged form (even if the class had only one occurrence in the test set and no occurrence in the training set). however, training a model on classes that had only a few samples would be impractical. because of that, the training and validation sets were reduced in such a way that we maximize class coverage. to do that, we created a set of label subsets according to a minimum number of classes to achieve a given percentile (not counting single whitespace). the percentiles with a corresponding number of classes are presented in as the last step, we created another version of the label set with the number of classes reduced to a commonly used quadruple of labels: comma, pe-riod, question, other. we used this version of the dataset to test whether training models on a larger, more narrowly defined set of labels would have negative effects relative to models trained on a small label set. to map the comprehensive label set into a simple label set we used inclusion criteria. i.e., if the character "." was present in the comprehensive label (e.g. "". "), it would be mapped into period class. if more than one base class were found in the comprehensive label, then the more frequently represented one would be chosen (the precedence order was a comma, period, question). if no base label was found, other class was assigned. this variation of label set will be referred to as "reduced" 4 experimental setup ## results for english example predictions (on an excerpt from the test set) from the best model for english (roberta) trained on a different number of training labels is shown in as expected, increasing the number of classes on which the model was trained increases the average recall (r). depending on the method, the increase over the method's native class list was between 12 and 14 percentage points. as for averaged precision (p), its clear decrease was observed only in the case of the bigru model. in models based on pre-trained berts, the highest precision was obtained with an increased number of classes. since the fluctuation of precision with increasing label set was relatively small compared to the gain on recall, the f1 metric in each case increased with the increasing number of classes. also, the less rigid mrs metric showed an average gain of about 14 points when using models trained on 99 th percentile. this number shows how much we would be losing when we would reduce the punctuation to the base set. ## results for polish the results for the polish language are presented in ## conclusion and future work in our work, we have shown that token classifier models are able to restore a much larger range of punctuation than it is done in most other reported researches. our experiments show that such an increase in coverage can be achieved without a drop in quality for key punctuation marks. we have also shown that this effect is not limited to english, and we have obtained very similar results in polisha language with much more complex morphology. additionally, the advantage of the approach with automatic generation of a set of labels from the data is that we are also able to predict the composition of punctuation marks. in further work, it would be of great benefit to investigate what effect reproducing a wide range of punctuation would have on text readability for people compared to reproducing only the basic characters. it would also be interesting to perform a comparative study of how token classifier models perform in the task of reproducing broad punctuation compared to sequence-tosequence models, such as bart @xcite or t5 @xcite for which such behavior would be natural. we also plan to take part in the poleval 2021 5 shared task, concerning punctuation restoration from read text in polish. in contrast to the problem analyzed here, the data sets will contain acoustic information, e.g. one that could allow determining the duration of gaps between words. appendix a: list of generated labels for english dataset