Methods of Automatic Matrix Language Determination for Code-Switched Speech (2024)

Abstract: Code-switching (CS) is the process of speakers interchanging between two or more languages which in the modern world becomes increasingly common. In order to better describe CS speech the Matrix Language Frame (MLF) theory introduces the concept of a Matrix Language, which is the language that provides the grammatical structure for a CS utterance. In this work the MLF theory was used to develop systems for Matrix Language Identity (MLID) determination. The MLID of English/Mandarin and English/Spanish CS text and speech was compared to acoustic language identity (LID), which is a typical way to identify a language in monolingual utterances. MLID predictors from audio show higher correlation with the textual principles than LID in all cases while also outperforming LID in an MLID recognition task based on F1 macro (60%) and correlation score (0.38). This novel approach has identified that non-English languages (Mandarin and Spanish) are preferred over the English language as the ML contrary to the monolingual choice of LID.

## introduction code-switching (cs) is the process of speakers switching between two or more languages in spoken or written language ( the morphemes as units within the mlf framework were first introduced by @xcite and were split into content and system morphemes. some common examples of system morphemes are quantifiers, possessives and tense/aspect determiners, while content morphemes include nouns, pronouns, adjectives, verbs and prepositions. matrix language identity (mlid) is the identity of the language providing the grammatical frame for the utterance and it can be defined for both monolingual and cs utterances. moreover, the existence of ml implies a certain token distribution following the system morpheme principle @xcite which is highlighted in further myers-scotton works in the 4-m model. overall, mlid provides insight into the grammatical properties of the utterance and a computational implementation of an mlid would be able to reduce the amount of manual annotation. in this paper three mlid systems for cs text and audio were implemented. mlf theory formulates the morpheme order principle and the system morpheme principle, which were implemented into three systems for mlid determination from text (p1.1, p1.2 and p2) and from audio (𝑀 𝐿𝐼 𝐷 𝑃1.1 , 𝑀 𝐿𝐼 𝐷 𝑃1.2 and 𝑀 𝐿𝐼 𝐷 𝑃2 ). an extensive correlation analysis and comparison of an mlid determination system and a traditional acoustic language identities (lid) were carried out. recognised mlid and lid from cs texts and audio were compared to ground truth ml annotation and the quality of ml recognition was measured in terms of f1 macro and matthew's correlation coefficient (mcc). to conclude the findings, the distributions of textual lids were compared to the textual mlid distributions of the cs data. the remainder of the paper is structured as follows. the next section reviews the relevant research presented previously on mlf and lid in cs text and speech. the third section provides a detailed description of the methods used. this is followed by a section on experiments, which provides information on datasets, detailed implementation, experiment descriptions as well as a discussion of results. conclusions summarise and complete the paper. simultaneously in the speech processing domain a common technique to separate languages in cs is lid. lid of a whole cs utterance may be performed when cs is regarded as a separate language (mary n j et al., 2020), in this case the component performs both lid and cs detection. a multi-lingual asr system with an utterance-wise lid component as an auxiliary task was tested for cs utterances in @xcite but the model was not able to generate cs text as a result. lid and language segmentation (lis) systems make decisions based on similarity to the data they were trained on @xcite and, to the best of our knowledge, no study was done to determine if pretrained lid/lis are able to predict a dominant language in a cs utterance. neither mlid nor lid were previously used for cs analysis. furthermore, ml determination principles were never fully implemented and compared. however, statistical methods were introduced before @xcite which can assess the nature of cs. among the statistical methods only the m-index (multilingual index) quantifies the representation of the languages in multilingual corpora. while the m-index is useful to learn about the balance of the token lids, it might be insufficient to learn about the utterance lid and mlid distributions. the above indicates that theoretical methods to identify ml from text exist but previously there was only one attempt to determine mlid which was not based on the two ml determination principles. furthermore, there are no existing mlid predictors from audio to the best of our knowledge. therefore the objective of this study is to advance technologies for multilingual understanding and analysis by describing the implementation, comparison and performance of automatic mlid predictors from cs audio and text based on the ml determination principles from the mlf theory. ## principles for ml determination the mlf theory principles mentioned above have to be implemented in order to compare the main language recognised by an lid system to the mlid. each of them provides estimates for mlid but is conditioned by different evidence in the utterance. the morpheme order principle is separated into two implementations of the 1st principle (p1.1 and p1.2). the implementations of the principle deduction as per mlf theory are described below. ## conclusion to the best of our knowledge this is the first work that precisely carries out the matrix language (ml) determination of a code-switched (cs) utterance based on the matrix language frame (mlf) theory and that compares matrix language identity (mlid) to acoustic language identity (lid). three methods for ml determination in text and audio are implemented using the ideas and the concepts of the mlf theory @xcite ). an extensive correlation analysis of the mlid systems from text and speech is carried out. a pretrained lid system 𝐿𝐼 𝐷 is adapted to the data by training a mapping function 𝐿𝐼 𝐷 𝑚𝑎 𝑝 , while also mapping functions 𝑀 𝐿𝐼 𝐷 𝑃1.1 , 𝑀 𝐿𝐼 𝐷 𝑃1.2 , 𝑀 𝐿𝐼 𝐷 𝑃2 for mlid are trained. 𝑀 𝐿𝐼 𝐷 consistently outperforms 𝐿𝐼 𝐷 for ml determination from audio based on matthew's correlation coefficient (mcc). comparing the results to the ground truth ml annotation shows that the trained 𝑀 𝐿𝐼 𝐷 𝑃1.2 and 𝑀 𝐿𝐼 𝐷 𝑃2 outperform 𝐿𝐼 𝐷 in terms of f1-macro and mcc respectively. finally, this approach reveals that despite english dominating as the utterance lid for the monolingual utterances, non-english (mandarin or spanish) languages set the grammatical frame for cs utterances. the proposed approaches can be used for accurate automatic analysis of cs text and audio. it can provide insight into the nature of cs for whole datasets but also separate speakers and even utterances. further work will explore the usefulness of the mlid implementations in natural language processing and automatic speech recognition (asr) applications, namely in language and dialogue modelling and also in end-to-end multitask asr the mlid component will be used as a part of the asr setup. additionally, further development of p2 is required where the system morphemes would be automatically determined from a given set of cs data rather than using a closed set of pos tags.