Comparing Grammatical Theories of Code-Mixing (2021)

Abstract: Code-mixed text generation systems have found applications in many downstream tasks, including speech recognition, translation and dialogue. A paradigm of these generation systems relies on well-defined grammatical theories of code-mixing, and there is a lack of comparison of these theories. We present a large-scale human evaluation of two popular grammatical theories, Matrix-Embedded Language (ML) and Equivalence Constraint (EC). We compare them against three heuristic-based models and quantitatively demonstrate the effectiveness of the two grammatical theories.

## introduction code-mixing is the phenomenon of mixing multiple languages within a single conversation. while widely observed in spoken language, mixing languages has also become commonplace in informal text conversations with the increasing use of social media @xcite . prior work has focused on various code-mixed natural language processing (nlp) tasks, including part-ofspeech (pos) tagging @xcite , sentiment analysis @xcite , machine translation @xcite and speech recognition @xcite . accurately modeling when and how to mix languages is critical to the above code-mixed nlp tasks. research efforts on this front have either relied on established grammatical theories @xcite or neural sequence-tosequence models @xcite to generate realistic code-mixed text. such models have found applications in speech recognition @xcite , translation @xcite and other downstream nlp tasks @xcite . these generation techniques have also been applied in human-machine dialogue @xcite to generate seemingly natural code-mixed responses. however, current literature lacks a thorough comparison of different code-mixed text generation models. such comparison is necessary to understand the relevance of these generation models to individual downstream nlp applications. in this work, we present one such comparison involving two linguistic theory-based code-mixed text generation systems, one based on equivalence constraint theory @xcite and the other on matrix-embedded language theory @xcite . specifically, we crowdsource human judgments on the understandability and naturalness of text generated by the two systems. our results show that grammatical models are considerably better than heuristic counterparts. additionally, we find the two grammatical models are equally preferable by our human judges. we evaluate the importance of individual constraints in each grammatical theory. in the following sections, we first briefly describe the grammatical models ( §2.1) and related heuristic models ( §2.2). in §3, we describe our human evaluation setup, and present the results in §4. ## generative models of code-mixing code-mixed text generation has applications in nlp tasks of speech recognition, human-machine dialogue, and translation. prior work has primarily focused on three grammatical theories, equivalence constraint (ec) @xcite , matrix-embedded language (ml) @xcite and functional head constraint (fhc) @xcite . each of these theories presents an account of the grammatical constraints of codemixing. @xcite presented computational models for ec and ml theories. such computational models allow for generation of grammatically correct code-mixed text @xcite @xcite . however, the resulting text may or figure 1 : an illustration of outputs from code-mixed text generation models. figure 1a shows the constituency parse for an example english sentence, along with its word-aligned spanish translation. figure 1b presents 5 example code-mixed sentences generated from the above english-spanish aligned pair. sentence #1 is allowed under all the grammatical and heuristic models, #2 is disallowed by ec model, #3 is disallowed by ml model, #4 is disallowed by ec, ml, aligned and dictionary models, #5 is disallowed under all the grammatical and heuristic models. words from embedded language are italicized in each example. ## evaluation setup to evaluate the effectiveness of the above described models of code-mixing ( §2), we first sample spanglish (english-spanish) sentences using selected criterion ( §3.1), and then collect human judgments on their understandability and naturalness ( §3.2). ## mturk results & discussion we published a total of 429 hits and obtained judgments from 59 unique crowd workers. in our analysis, we only consider the responses where the understandability score is strictly greater than one. 8 before analyzing the human judgments, we ensure that the human judge has marked the appropriate responses for the control data involving real-spanglish and non-spanglish sentences. the non-spanglish sentences are to be scored low on the understandability scale. naturalness being a subjective marker, there is no definite answer; therefore, we haven't pruned any of the hits based on the judgments on real-spanglish sentences. our main aim in this experiment was to gauge the importance of various sentence-level features in the code-mixed sentence generation process. to this end, we have compared the absolute human judgment scores across sentence pairs. we have designed our task as an absolute judgment task instead of pair-wise comparison for logistical reasons, as it allows for a large scale evaluation. we construct a matrix (m ) with sentence/tweet pairs (t i , t j ) along the rows and annotators (a k ) along the columns (equation 4 ). a i,k denotes the score of annotator a k for the sentence t i . we construct a partial order between t i and t j 7 see a.1 in appendix for the questions used in the test. 8 we acknowledge that understandability is a requirement for a code-mixed text generation model. but in the scope of this work, we focus primarily on the naturalness evaluation. ## conclusion & future work we present a large-scale qualitative comparison of different code-mixed generation systems. through the judgments of spanish-english bilingual speakers, we analyze the need for each constraint in the two grammatical theories, matrix-embedded, and equivalence constraint. we also contrast against simpler heuristic systems and found grammatical systems to be superior. however, we do not observe a clear winner between the two grammatical