Play música alegre: A Large-Scale Empirical Analysis of Cross-Lingual Phenomena in Voice Assistant Interactions (2022)

Abstract: Cross-lingual phenomena are quite common in informal contexts like social media, where users are likely to mix their native language with English or other languages. However, few studies have focused so far on analyzing cross-lingual interactions in voice-assistant data, which present peculiar features in terms of sentence length, named entities, and use of spoken language. Also, little attention has been posed to European countries, where English is frequently used as a second language. In this paper, we present a large-scale empirical analysis of cross-lingual phenomena (code-mixing, linguistic borrowing, foreign named entities) in the interactions with a large-scale voice assistant in European countries. To do this, we first introduce a general, highly-scalable technique to generate synthetic mixed training data annotated with token-level language labels and we train two neural network models to predict them. We evaluate the models both on the synthetic dataset and on a real dataset of code-switched utterances, showing that the best performance is obtained by a character convolution based model. The results of the analysis highlight different behaviors between countries, having Italy with the highest ratio of cross-lingual utterances and Spain with a marked preference in keeping Spanish words. Our research, paired to the increase of the cross-lingual phenomena in time, motivates further research in developing multilingual Natural Language Understanding (NLU) models, which can naturally deal with cross-lingual interactions.

## introduction the interaction of different languages produces a variety of linguistic phenomena, the most prominent examples being code-switching and lexical borrowing. code-swiching (cs), or code-mixing foot_0 , refers to the alternation of languages within an utterance or a conversation @xcite , while linguistic borrowing occurs when a word is adopted from a language and integrated into another without translation. examples of these are: (i) "play música alegre" (ii) "bravo, that was a great performance", with the former being a case of code-switching and the latter exhibiting lexical borrowing. these phenomena are particularly frequent in bilingual countries, where the local language, called framelanguage, is influenced by a second language, which is instead called the mixing-language. this phenomenon is abstracted by the matrix language frame model @xcite in code-switching literature. common pairs of frame-mixing languages are for example spanglish (spanish-english) and hinglish (hindi-english). countries for which these phenomena happen usually undergo a broader influence which also permeates their culture, as it happens for example with american artistic production of cinema and music. as a side effect, utterances originated in the frame language are rich in foreign named entities, which contribute to their linguistic heterogeneity. voice assistants operating in these locales have to face a significant amount of foreign words while being in most cases trained on monolingual corpora, hence posing a severe threat to their performance. indeed, the growing interest in multi-lingual models @xcite @xcite and datasets @xcite ) may help mitigate the problem. we will use in the rest of the paper the term cross-lingual to denote utterances which contain one or more words from a mixing language while belonging to a frame language. these may be caused by any of the mentioned phenomena, i.e. code-switching, lexical borrowing and foreign named entities. a major challenge, both in improving the performances of multilingual models on cross-lingual data and in their overall evaluation, is the scarcity of cross-lingual datasets. nevertheless, while human annotation is already costly and timeconsuming in general, annotating cross-lingual data is made harder by the fact that bilingual annotators are needed for each pair of languages of interest; these may be especially hard to find for less common languages. in particular, while there has been some interest for different kinds of data (e.g. social media), voice assistant data, which is the focus of this paper, has been mostly ignored. although such datasets may be obtained by crowdsourcing, the process would be expensive and time-consuming. this reason leads to the necessity of a procedure to generate synthetic data over several language pairs while providing large-scale datasets. these can be used to train a learning model to infer crosslingual utterances. the trained model can finally be employed on voice assistants data to detect real cross-lingual utterances. our contribution is three-fold: (i) we propose in section 3 a scalable synthetic data generation technique to obtain challenging benchmarks which exhibit a significant ratio of cross-lingual influences. the method is language agnostic and here we employ it on four common european languages (german, french, italian, spanish) with english as mixing language. (ii) we compare the performance of different baselines in detecting cross-lingual utterances by solving the more fine-grained task of word level language identification. to validate the generation procedure, we test the models trained on the synthetic distribution over a benchmark dataset obtained through an extremely precise heuristic. (iii) finally, we analyze in section 6 the phenomenon of cross-lingual influence in a large set of cross-lingual utterances detected using our method on alexa user queries. ## related work code-switching has received significant interest both in the linguistic literature @xcite @xcite in natural language processing (nlp); @xcite provide a survey of code-switching in nlp. from a linguistic point of view, the two phenomena differ in the fact that the latter occurs in the lexicon, while code-switching mostly regards the utterance-construction level @xcite . despite the apparently different definitions, the two are not always clearly distinct from one another, and may be thought of as lying on a continuum @xcite . various efforts have been made to collect codeswitched annotated data over which to perform core nlp tasks, such as ner @xcite , pos @xcite and asr @xcite . nevertheless, most of the available resources have been gathered from twitter, and therefore do not resemble the distribution of data encountered by a voice assistant. few works exist on generating synthetic cs data: in @xcite , a synthetic dataset is obtained by applying linguistic theory-based rules, while in @xcite an encoder-decoder architecture is used for the generation. these approaches, however, focus on strict code-switching, while we aim to also encompass lexical borrowing and foreign named entities. the de-facto standard way to infer codeswitched utterances is to train models on the task of word-level language identification. again, existing datasets of code-switched text annotated with wordlevel language labels have been collected from twitter @xcite or facebook @xcite , leaving conversational data out of the scope. provided a word-level annotated dataset, any sequence-labeling algorithm can be employed to solve the task. approaches include conditional random fields @xcite , recurrent neural networks @xcite and transfer learning @xcite . ## models we describe in this section the proposed baselines, namely an ad hoc deterministic heuristic and two neural models. these will be trained over the synthetic datasets generated according to section 3 and used to infer real code-switched alexa utterances. we consider as baseline a dictionary-based heuristic parameterized by two thresholds t 1 and t 2 . the latter deterministically classifies an utterance as code mixed if at least t 1 % of the lemmatized words do not appear in the frame language vocabulary while appearing in the mixing language vocabulary and no more than t 2 % appear in the mixing vocabulary while not belonging to the frame vocabulary. despite its simplicity, the heuristic allows to arbitrarily trade-off recall and precision by manually tweaking the two parameters. we then propose two neural models, one character based and the other transformer based. the intuition behind the former is that character-level convolutions @xcite should be able to capture the distinguishing morphological features of the considered languages which are key to the task. in particular, given an input utterance, each word is split in characters and embedded via a trainable embedding layer to obtain @xmath0, where l is the maximum word length encountered in the data and @xmath1. the embedded word is then passed through a set of @xmath2, yielding a @xmath3, where o is given by (lk + 1). at this point, the maximum is taken along the axis on which the resulting feature maps are stacked, so to have a new word embedding tensor @xmath4. three different sets of filters of different kernel sizes are then passed over e, having sizes 3, 4 and 5 in our implementation. max pooling over time allows to obtain a fixed-dimension digest for each of the resulting maps, which can be concatenated to form a single tensor to be fed to a bidirectional lstm along with the rest of the utterance. the latter returns a dynamic representation of the word and its context, which is then mapped to the label space by a standard fully-connected layer. a visual overview of the architecture is given in fig. 1 . we will refer to this model as 'charbased'. the second proposed neural model leverages multilingual bert @xcite to obtain contextualized embeddings which are then fed to a standard sequence classification pipeline, as can be seen in fig. 2 . in details, each word is first tokenized and encoded by the mbert tokenizer and fed to a pretrained mbert model along with the whole utterance. the embedding is then provided by the last hidden state of the pretrained model. since the tokenizer is based on the wordpiece model @xcite , words are often split in subwords: the word 'microfono' for example would be split in 'micro' and '##fono'. to still obtain word-level predictions, the resulting embeddings are averaged. utterances are finally fed to a bidirectional lstm whose output is mapped to the label space again by a fully-connected layer. we will refer to this model as 'bertbased' in the rest of the paper. ## evaluation as can be seen in ## results object of this analysis are code-switched utterances detected from real alexa queries by a model trained on an artificial dataset generated according to section 3. a separate model was trained for each locale versus english, and the inference was made on real data coming from the corresponding locale. as can be seen in ## conclusions in this paper, we have presented a large-scale analysis of the cross-lingual phenomena encountered by voice assistants. we first have proposed an artificial data generation technique, then we have presented two neural models that can be trained on the synthetic data to infer real cross-lingual utterances. finally, we have employed the top-performing model to infer such utterances from real data. the fact that loanwords and foreign named entities cover most of the found cross-lingual utterances may indicate that code-switching is rare in voice assistants in the considered locales. this may be explained by the fact that users code-switch the most in colloquial situations, while their way of speaking when querying a voice assistant is constrained by its understanding capacity. nonetheless, multilingual models still have a great opportunity of transfer learning on the large amount of foreign named entities and loanwords that are present in the data. the results show that the use of english words in de, fr, it, es is strongly skewed on popular entities such as 'amazon', 'netflix', and 'youtube', and on specific loanwords such as 'timer', 'computer' and 'stop'. the use of these popular named entities is consistent across locales and the ratio of cross-lingual interactions is similar, except for es, where users tend to prefer spanish words to english loanwords. the analysis also shows that most of the mixing words are contained in the 'media & entertainment' domains and on named entities such as service names, media names, item names and dish names. as we have explained in section 3, the current generation technique does not aim to model the complex phenomenon of code-switching in a theoretically correct manner. the simplicity of the procedure nevertheless allows it to be repurposed to focus on the latter. an interesting future direction could be to limit the attention to code-switching in the data generation, so that a model trained on that data could be used to collect a code-switched dataset of voice assistant queries. given the low fpr exhibited by the model, the collected utterances represent an high-quality resource which could in future be used to train generative models to produce better synthetic data, which in turn can be used to train detectors in an iterative manner. from an architectural prospective, models tackling word-level language identification expressly designed to solve the task of cross-lingual or codeswitched detection could benefit from the utterancelevel information about their distribution in the dataset. this could encourage the design of a multiheaded model tackling both tasks in an end-to-end approach. finally, we aim to expand the set of considered languages to encompass other frame and mixing languages, for example considering hinglish in india. it might be particularly interesting to compare the obtained results for spanish with ones obtained over spanish spoken in the united states and in mexico, as they may involve more code-switching.