Seed Words Based Data Selection for Language Model Adaptation (2021)

Abstract: We address the problem of language model customization in applications where the ASR component needs to manage domain-specific terminology; although current state-of-the-art speech recognition technology provides excellent results for generic domains, the adaptation to specialized dictionaries or glossaries is still an open issue. In this work we present an approach for automatically selecting sentences, from a text corpus, that match, both semantically and morphologically, a glossary of terms (words or composite words) furnished by the user. The final goal is to rapidly adapt the language model of an hybrid ASR system with a limited amount of in-domain text data in order to successfully cope with the linguistic domain at hand; the vocabulary of the baseline model is expanded and tailored, reducing the resulting OOV rate. Data selection strategies based on shallow morphological seeds and semantic similarity via word2vec are introduced and discussed; the experimental setting consists in a simultaneous interpreting scenario, where ASRs in three languages are designed to recognize the domainspecific terms (i.e. dentistry). Results using different metrics (OOV rate, WER, precision and recall) show the effectiveness of the proposed techniques.

## introduction in this paper we describe an approach to adapt the language models (lms) used in a system designed to give help to simultaneous interpreters. simultaneous interpreting is a very difficult task that requires a high cognitive effort especially to correctly translate parts of the source language that convey important pieces of information for the final users. these are: numerals, named entities and technical terms specific of each interpretation session. as an example, a study reported in @xcite claims that the error rate made by professional interpreters on the translation of numbers is, on average, equal to 40%. this demands for a technology, based on automatic speech recognition (asr), capable of detecting, in real time and with high accuracy, the important information (words or composite terms) of a speech to interpret and to provide it to a professional interpreter by means of a suitable interface. therefore, our goal is not to minimise the word error rate (wer) of an audio recording, as usual in asr applications, instead we aim to maximise the performance of the developed system, in terms of precision, recall and f-measure, over a set of "important" terms to recognise, as will be explained in section 4. to do this we experimented on a set of data properly labelled by human experts. it is worth to point out that this task is different from the usually known "keywords spotting" task, since we cannot assume to know in advance the terms to spot inside the audio stream but we can only start from some "seed" terms belonging to a glossary which is part of the experi-ence of each human interpreter. this demands for further processing modules that: a) extend, in some way, the given glossary including also "semantically" similar terms, as will be explained in section 2.2, in order to adapt both the dictionary and the language model (lm) employed in the asr system, and/or b) detect along an automatically generated transcription the pieces of information (i.e. numerals, named entities, etc) useful to the interpreter. actually, the asr system described below is part of a bigger system that integrates natural language processing (nlp) modules, dedicated to both named entity and numeral extraction, and a user interface specifically designed according to the requirements of professional interpreters. this system, named smarterp foot_1 , aims to support the simultaneous interpreters in various phases of their activities: the preparation of glossaries, automatic extraction and display of the "important" terms of an interpreting session, post-validation of new entries @xcite . related works. as previously mentioned spotting known words from audio recordings is a largely investigated task since the beginning of speech recognition technology (e.g. see works reported in @xcite ; @xcite ; @xcite . basically all these approaches used scores derived from acoustic log-likelihoods of recognised words to take a decision of keyword acceptance or rejection. more recently, with incoming of neural networks, technologies have begun to take hold based on deep neural networks @xcite , convolutional neural networks @xcite and recurrent neural networks @xcite to approach keyword spotting tasks. the last frontier is the usage of end-to-end neural architectures capable of modelling sequences of acoustic observations, such as the one described in @xcite or the sequence transformer network described in @xcite . the approach we use for enlarging the dictionary of the asr system and to adapt the corresponding language model to the application domain is to select and use from a given, possibly very large and general text corpus, the sentences that exhibit a certain "similarity" with the terms included in the glossaries furnished by the interpreters. similarly to the keyword spotting task, "term based similarity" represents a well investigated topic in the scientific community since many years. a survey of approaches can be found in the work reported in @xcite . also for this task the advent of neural network based models has allowed significant improvements both in the word representation, e.g. with the approaches described in @xcite , and in text similarity measures, e.g. as reported in @xcite ; @xcite . worth to notice is that in the asr system used for this work we do not search for new texts to adapt the lm, instead, as explained in section 2, we select the adaptation texts from the same corpus used to train the baseline lm. note also that our final goal is not that to extract the named entities from the asr transcripts -this task is accomplished by the nlp modules mentioned above -instead it consists in providing to the asr system a lm more suitable to help the human interpreter of a given event. also for asr system adaptation there is an enormous scientific literature, both related to language models and to acoustic models adaptation; here we only refer some recent papers: @xcite for lm adaptation and @xcite for a review of acoustic model adaptation approaches, especially related to neural models. ## automatic selection of texts usually a language model (lm) is trained over huge amounts of text data in a given language, e.g. italian. during the training phase, a fixed lexicon is selected -typically the n most frequent words in the text -and millions or billions of n-grams are stored to give some probability to any possible word sequence. this process allows to build a somehow generic lm, capable to represent the language observed in the text. however, interpreters often need to specialise their knowledge on a very specific topic, e.g. dentistry. in this case, they also have to quickly become experts in that particular field. we could say that they need to adapt their general knowledge to that field: this means that, before the event, they have to collect material about that topic, study it, prepare and memorise a glossary of very specific technical terms together with their translations. the same process holds for an asr system: it can perform in a satisfactory way in a general situation, but it may fail when encountering technical terms in a specific field. so, it has to be adapted, both in terms of lexicon (it may be necessary to add new terms to the known lexicon) and in terms of word statistics for the new terms. in the smarterp project we are going to explore different adaptation procedures and describe in this paper our preliminary work in this direction. at present, we hypothesise that an interpreter could provide some text and the asr system will be able to adapt to the corresponding topic in a short time (some hours on a medium computer). this short text could range from a few words to a quite large set of documents that identify that particular topic, depending on the expertise and the attitude of the interpreter. here are some possibilities: in a very near future, in smarterp a pool of interpreters will be engaged in simulations where they have to provide data that, in a complete automatic way (i.e. without the intervention of some language engineer), will adapt the asr system for a particular topic. in this work we are testing some tools and procedures in order to provide them some possible solutions, assuming that at least some small text (i.e. a glossary, or even a few words) will be available. from this small text we will derive some seed words that will be used, in turn, both to update the dictionary of the asr system and to select lm adaptation texts from the available training corpora (see ## asr systems the asr system is based on the popular kaldi toolkit @xcite , that provides optimised modules for hybrid architectures; the modules support arbitrary phonetic-context units, common feature transformation, gaussian mixture and neural acoustic models, n-gram language models and on-line decoding. ## description of smarterp multilingual benchmark as mentioned above, in smarterp we prepared benchmarks for the 3 languages of the project: english, italian, spanish. for each language, a number of internet videos having creative commons licence were selected, in order to reach at least 3 hours of material on a particular topic, dentistry. ## experiments and results since several approaches can be employed to obtain, enlarge and use the seed words (e.g. based on texts distance, texts semantic similarity, etc) we consider the following indicators that allow to measure their effectiveness on the benchmarks collected and manually transcribed within the smarterp project. the following experiments were carried out for each of the three languages: ## conclusions we described two different approaches for extending the dictionary of an asr system in order to detect important terms from technical speeches, namely dental reports, to be translated by simultaneous professional interpreters. the two approaches consist in extracting adaptation text from a huge set of text data, starting from some seed words. in the first one, seed words come from a given glossary. the second one is based on the application of a text similarity measure to an initial (very small) set of 5 seed words. after the application of the selection procedures we adapted the language models used in the asr system employed in a computer assisted interpretation (cai) system under development and we proved the effectiveness on the approaches in terms of different evaluation metrics.