Induced Inflection-Set Keyword Search in Speech (2020)

Abstract: We investigate the problem of searching for a lexeme-set in speech by searching for its inflectional variants. Experimental results indicate how lexeme-set search performance changes with the number of hypothesized inflections, while ablation experiments highlight the relative importance of different components in the lexeme-set search pipeline and the value of using curated inflectional paradigms. We provide a recipe and evaluation set for the community to use as an extrinsic measure of the performance of inflection generation approaches.

## introduction keyword search (kws) is the task of finding certain words or expressions of interest in a body of speech. kws is relevant to incident-response situations such as those modeled by lorelei @xcite and was a focus of the iarpa babel program. 1 in the event of a humanitarian crisis, processing speech to determine mentions of certain keywords can inform better decision making when time is critical. kws is typically framed as searching for instances of a keyword in lattices that result from speech recognition decoding, as this means search is not restricted to a potentially incorrect one-best transcription. however, existing work on kws assumes the relevant form of a keyword has been correctly specified. many concepts to be searched for in speech take different forms through inflection as a result of the language's morphosyntax. in most cases, distinctions between such inflections (e.g. kill, kills, killing, killed) are irrelevant to the problem of searching for the underlying concept of interest. producing such inflection sets manually is arduous, even for native speakers, yet curators of keyword lists may have to construct them crosslingually using bilingual dictionaries, which typically only contain canonical forms. compounding this issue are the limitations of existing language technology for most of the world's languages across the whole kws pipeline, including inflection generation, the language model (lm), the pronunciation lexicon, and the acoustic model. in this paper we explore the application of inflection generation to kws by searching for instances of a lexeme (see figure 1 ). to the best of our knowledge, this task has not been investigated before. using bengali and turkish as evaluation languages, we scale the number of inflections generated per lexeme-set to examine how the tradeoff between false positives and false negatives affects downstream kws. we additionally perform experiments that assume varying quality of inflection generation. our findings show that lexemeset kws yields promising results even when all inflections must be generated on the basis of a distantly supervised cross-lingual approach to training inflection tools, though we observe that having a curated set of inflectional paradigms is important for achieving good performance. these first results encourage future work for lexeme-set search in speech, and the use of kws as an extrinsic eval-uation of inflection generation tools. to this end, we make available to the community a lexeme-set kws pipeline with baseline models for inflection generation, grapheme-tophoneme conversion (g2p), multilingual acoustic modeling, and cross-lingual inflection generation, and a kws evaluation set built on a suitable intersection of unimorph inflection sets @xcite and the babel speech @xcite . the combination of these components serves as a novel downstream evaluation of inflection generation approaches, as well the other components in the pipeline. we make this recipe and evaluation set freely available online. 2 ## the lexeme-set kws pipeline the pipeline starts with a lexeme of interest from the evaluation set ( §2.1). inflections of the lexeme are generated using some generation tool or manual resource ( §2.2). these inflections are then converted to a phonemic representation ( §2.3) before being added to the lexicon used in speech recognition. kws is then performed ( §2.4) by decoding the speech and the model is scored on the how well it finds instanes of the lexeme. ## kws evaluation metrics we evaluate kws performance on a per lexemeset basis, rewarding the system when it finds any form of an evaluation lexeme, regardless of how it is inflected, while also penalizing failure to find any inflection. as an evaluation metric we use term weighted value (twv), a standard metric in kws devel-oped for the nist 2006 spoken term detection evaluation @xcite , which rewards joint maximization of recall with minimization of false positives. twv relies on a threshold parameter to determine what minimum level of confidence is required by the system in order to assert keyword findings. there are several variations of term weighted value (twv) that are different in the way the threshold is handled: actual (atwv), optimum (otwv), and supreme (stwv). atwv is the twv of the system given some global threshold (provided by the system) of confidence common to all keywords, and is the most common metric used to compare systems. otwv determines a per-keyword (in our case, per lexeme-set) threshold. for our purposes this is the most informative metric because it gives a better sense of how the atwv would be if system effectively normalized confidences across lexemes. improvements to twv may also potentially be made beyond what is represented by the otwv. some inflections are more likely than others, yet the thresholds for otwv are made at a per lexeme-set basis, not a per-inflection basis. improving how the system weights the likelihood of different inflections (either during inflection generation or in the lm probabilities) would likely substantially improve atwv. stwv is a recall-oriented version of twv that disregards the confidence of the terms and does not penalize false positives. it is thus similar to lattice recall and serves as a useful metric in system analysis for determining whether low atwv/otwv is due to large number of false positives or issues in effective speech word lattice decoding. ## experiments we conduct experiments to see how performance of kws relates to the number of inflections hypothesized by the cross-lingual distantlysupervised method described in section 2.2 (henceforth rnn+dtl), before comparing it to several alternative benchmark methods. as evaluation languages we used bengali and turkish, a subset of languages for which we have bibles and that also occur in unimorph. we observed similar trends and relative performance of methods for both languages languages so in the subsequent results we present the arithmetic mean of the results of bengali and turkish. ## conclusion we have presented an evaluation of lexeme-set kws. our results shed light on the relative impact of undergenerating and overgenerating inflected forms on kws, indicating that high recall can be achieved via an inflection method of cross-lingual distant supervision, but with the best all-round performance achieved by making use of unimorph. we release our evaluation set along with scripts to reuse our pipeline so that the community can explore lexeme-set kws as an extrinsic evaluation of inflection generation.