Punctuation Restoration in S panish Customer Support Transcripts using Transfer Learning (2022)

Abstract: Automatic Speech Recognition (ASR) systems typically produce unpunctuated transcripts that have poor readability. In addition, building a punctuation restoration system is challenging for low-resource languages, especially for domain-specific applications. In this paper, we propose a Spanish punctuation restoration system designed for a real-time customer support transcription service. To address the data sparsity of Spanish transcripts in the customer support domain, we introduce two transferlearning-based strategies: 1) domain adaptation using out-of-domain Spanish text data; 2) crosslingual transfer learning leveraging in-domain English transcript data. Our experiment results show that these strategies improve the accuracy of the Spanish punctuation restoration system.

## introduction automatic speech recognition (asr) systems play an increasingly important role in our daily lives, with a wide range of applications in different domains such as voice assistant, customer support and healthcare. however, asr systems usually generate an unpunctuated word stream as the output. unpunctuated speech transcripts are difficult to read and reduce overall comprehension @xcite . punctuation restoration is thus an important post-processing task on the output of asr systems to improve general transcript readability and facilitate human comprehension. punctuation restoration for transcripts of spanish-speaking customer support telephone dialogue is a non-trivial task. first, real-world human conversation transcripts have unique characteristics compared to common written text, e.g., filler words and false starts are common in spoken dialogue. moreover, further challenges arise when addressing noisy asr transcripts in a specific domain, as the lexical data distribution can be quite different compared to public spanish datasets. examples of spanish sentences from different sources are shown below: recent advances in transformer-based pretrained models have been proven successful in many nlp tasks across different languages. for spanish, available pre-trained resources include multilingual models such as multilingual bert (mbert) @xcite and xlm-roberta (xlm-r) @xcite , as well as monolingual models such as beto @xcite . however, large pre-trained models are trained on various written text sources such as wikipedia and commoncrawl @xcite , which are very distant from what we are trying to address in noisy asr transcripts in the customer support domain. while spanish is not usually considered a low-resource language in many nlp tasks, it is much more challenging to acquire sufficient training data in spanish for our domain-specific task, since most of the publicly-available spanish datasets do not come from natural human conversations, and have little coverage in the customer support domain. in addressing the challenge of in-domain data sparsity we make the following contributions: ## background punctuation restoration is the task of inserting appropriate punctuation marks in the appropriate position on the unpunctuated text input. a variety of approaches have been used for punctuation restoration, most of which are built and evaluated on one language: english. the use of classic machine learning models such as n-gram language model @xcite and conditional random fields @xcite are common in early studies. more recently, deep neural networks such as long short-term memory (lstm) @xcite and transformers @xcite have been adopted in @xcite and @xcite . punctuation conventions differ between spanish and english. namely, in addition to the equivalents of english and spanish periods, commas, terminating question marks and terminating exclamation marks, we must also account for the inverted question marks (¿) and inverted exclamation marks (¡) used to introduce these respective clauses in spanish. there has been limited work done in spanish punctuation restoration and in most cases spanish is covered as part of the multilingual training. @xcite proposed a multilingual lstm including the support for spanish. @xcite uses a transformerbased model with both lexical and acoustic inputs for spanish and basque. transfer learning has been widely studied and applied in nlp applications for low-resource languages @xcite . domain adaptation and cross-lingual learning both fall under the category of transductive transfer learning, where source and target share the same task but labeled data is only available in source @xcite . data selection is among the data-centric methods used in domain adaptation, which aims to select the best matching data for a new domain @xcite . @xcite uses data selection to improve english punctuation restoration with out-of-domain datasets. recent advances in multilingual language models such as mbert and xlm-r have shown great potential in crosslingual zero-shot learning, wherein a multilingual model can be trained on the target task in a highresource language, and afterwards applied to the unseen target languages by zero-shot learning @xcite . @xcite and @xcite demonstrate the effectiveness of mbert as a zero-shot cross-lingual transfer model in various nlp tasks, such as classification and natural language inference. ## future work when analysing the prediction errors, we found that many close_question classes are predicted as period by the model, as shown in ## conclusion for this study, we trained and tested a spanish punctuation restoration system for the customer support domain based on pre-trained transformer models. to address in-domain data sparsity in spanish, transfer learning approaches were applied in two directions: domain adaptation and cross-lingual transfer. we explored and fine-tuned three different pre-trained models with our transfer learning approaches for this task; our results demonstrate that the domain adaptation method improves the accuracy of all three pre-trained models. cross-lingual transfer with joint training of english and spanish datasets improves the performance of both multilingual pre-trained models. xlm-r substantially outperforms the monolingual beto after crosslingual transfer and achieves the best f1 score in our spanish punctuation restoration task.