Optimizing Code-Switching in Conversational Tutoring Systems: A Pedagogical Framework and Evaluation (2024)

Abstract: Large language models demonstrate remarkable proficiency in various tasks across multiple languages. However, their potential in code-switching remains underexplored, particularly in cultural and educational contexts. Code-switching or translanguaging plays a crucial role in bilingual education, facilitating comprehension and engagement among students with varied linguistic proficiencies. In this work, we present a pedagogy-inspired framework that introduces traditional classroom practices of code-switching to intelligent tutoring systems. Specifically, we develop fine-grained instructional strategies tailored to multilingual and educational needs. We conduct experiments involving both LLM-based evaluation and expert analysis to assess the effectiveness of translanguaging in tutoring dialogues. Our experimental results indicate that strategic code-switching can significantly enhance the learning experience. This work not only advances dialogic tutors in language learning, but also extends LLMs to better accommodate multilingual interaction.

## introduction large language models (llms) excel in diverse tasks and various languages @xcite . while their task-solving capabilities in monolingual scenarios are wellstudied @xcite , their potential in codeswitching -the practice of alternating languages within an utterance -is still less explored @xcite . in multilingual communications, people sometimes switch languages during the conversation to convey context-specific concepts and reinforce social connections. however, current llms aren't specifically trained for translanguaging scenarios, highlighting an emerging research interest in their ability to understand and utilize code-switching @xcite . code-switching is not only relevant in the realm of natural language processing but also has significant implications in educational settings @xcite . for instance, bilingual children often have an imbalanced exposure to their first languages (l1) at home, and show less proficiency in vocabulary, grammar, and sentence structures of the target language (l2) at school. as a result, they tend to switch between l1 and l2 in classrooms. accordingly, as shown in figure 1 , teachers apply codeswitching strategies to clarify instructions, introduce new concepts, provide necessary encouragement, and facilitate the learning process @xcite . these efforts are termed as "translanguaging" @xcite or "code-switching" in pedagogical theories, referring to the planned and sys-tematic use of two languages in the classroom, and it includes both intra-sentential and inter-sentential switching @xcite . while code-switching in tutoring has been studied extensively in the last decades @xcite @xcite , most previous works focus on traditional classrooms. on the other hand, while intelligent tutoring systems (itss) have shifted adaptive and personalized education from traditional classrooms to online learning, they are often limited to monolingual contexts and lack adaptability to mixed bilingual communication. in particular, one major challenge of improving code-switching of dialogue systems is the scarcity of data @xcite . due to the highly colloquial nature of translanguaging, existing resources for specific domains are limited, and collecting data at a large scale requires considerable annotation efforts. furthermore, previous studies evaluate multilingual models in general-purpose code-switching scenarios @xcite and they often simulate mixed generation by replacing words in parallel corpora. however, this approach fails to capture the complexity of effective pedagogical code-switching, which requires strategic and purposeful integration that considers learners' proficiency levels and educational objectives. thus, pedagogical code-switching should go beyond mere word swapping to facilitate comprehension of complex concepts and provide instructional scaffolding. in this work, we aim to improve the generated code-switching in conversational tutoring systems. we first gain insights from translanguaging theories and empirical dialogue studies to construct a pedagogical code-switching framework, where each dimension combines relevant scaffolding strategies to enhance language teaching through targeted translanguaging interventions. our framework contributes to language learning by facilitating vocabulary acquisition, grammatical understanding, and conversational fluency. it also supports content mastery through concept clarification and emotional support. to anchor a practical application, we conduct a case study on image description for language learning, and leveraging llms as tutoring agents. we utilize our proposed framework in both instructing llms for fine-grained code-switching generation and assessment. we conduct experiments on two representative translanguaging cases (e.g., chinese-english, korean-english), and deploy automated evaluation and qualitative analysis to assess the effectiveness of pedagogical code-switching generation in llm-based tutoring systems. experimental results indicate that state-of-the-art models are capable of tailoring scaffolding actions and codeswitching to learners' language proficiency levels and teaching content, and strategic code-switching can significantly enhance the learning experience. ## pedagogical code-switching we conceptualize "pedagogical code-switching" as a combination of two aspects: scaffolding and translanguaging, as shown in figure 3 . scaffolding is a dynamic intervention finely tuned to the learner's ongoing progress. the support given by the teacher during scaffolding strongly depends on the patterns of teacher-student interactions @xcite . therefore, the scaffolding abilities of tutors become the key criteria of effective tutoring systems. there are seven dimensions of scaffolding strategies to facilitate teaching and learning, which are (1) feeding back, (2) hints, (3) instructing, (4) explaining, (5) modeling, (6) questioning, (7) social-emotional support @xcite . on the other hand, the multifaceted functions of code-switching in teaching practices highlight its strategic use in pedagogical contexts. dicamilla and antón (2012) identified four major functions of code-switching in classroom discussions: 1) creating, discussing, and translating content, 2) negotiating grammatical, lexical, and stylistic choices, 3) planning, defining, and managing tasks, and 4) maintaining and developing interpersonal relationships. building on this classification, @xcite expanded the framework to include five major functions of code-switching in bilingual classrooms: [a] negotiating content, [b] clarifying language, [c] checking for or confirming understanding, [d] task management, [e] building relationships. these findings reveal how multiple ways that teachers could employ code-switching strategies to scaffold learning. language learning is one of the typical applications of code-switching in classrooms. the inherent nature of students' limited proficiency in the target language requires extensive scaffolding through the use of l1 for explaining, modeling, and providing instructions. based on scaffolding strategies and tigert et al. (2019)'s framework, we analyze dialogues between teachers and students in bilingual classrooms from previous work dicamilla and antón (2012); tigert et al. (2019); vaish and subhan @xcite . we build a pedagogical code-switching framework with six dimensions (as shown in figure 3 ): 1) introducing new concept (vocabulary), 2) clarifying comprehension, 3) correcting grammar, 4) checking for understanding, 5) task management, 6) emotional support. for example, we integrate scaffolding strategies (2) hints and (4) explaining to code-switching function [b] clarifying language. we rename it as introducing new concept and define it as "to explain the meaning, use, and definitions of focal vocabulary words." ## improving tutoring systems with pedagogical code-switching llms pave a new way to build dialogue tutors with less data supervision and higher customization @xcite . they can be shaped along desired dimensions to mimic human conversational characteristics such as tone and personality traits to deliver better user experience @xcite @xcite . we thus leverage llms as a tutoring agent for language learning, and adopt pedagogical code-switching based on our proposed framework. previous work shows that in task-specific settings, the general and coarse instruction may lead to inconsistent generation @xcite . therefore, we set up fine-grained instructions to improve and evaluate the llm-based tutoring systems. bi-lingual setting: the conversational tutoring system is designed for l2 learning (l1 is english), and particularly focuses on the image description task. in each tutoring session, the student is presented with a picture and asked to describe the incidents in l2 (e.g., chinese, spanish). from a syntactic perspective, languages can be classified into three structural patterns: 1) subject-verb-object (e.g., english, chinese); 2) verb-subject-object (e.g., arabic, irish); 3) subject-object-verb (e.g., japanese, korean). given that code-switching is a structured and syntax-related phenomenon in linguistics, in our study, we select two representative translanguaging combinations (chinese-english, korean-english) in our language learning case study, to represent the applicability and generalizability of code-switching in different language contexts. tutoring setting: teaching and improving students' l2 acquisition through image description is a dynamic and engaging approach. in the image description tasks, the learning objective is using target sentences to describe the given image that includes a particular place or setting, people or animals, items and actions, etc. the teacher uses scaffolding and code-switching strategies to guide students step by step until they can independently complete the image description task. we build a multi-agent communication environment following previous work @xcite @xcite . ## experimental setup in our preliminary study, open llms such as mistral-7b and llama-3-8b cannot follow the pedagogical instructions well, they tend to produce monolingual responses and fail to generate coherent tutoring dialogues. therefore, our experiments are conducted on two state-of-the-art representative models: gemini @xcite and gpt-4turbo @xcite . 1 following previous work @xcite , we adjust instructions to the chat template of each model. for tutoring dialogue generation, both teacher and student roles use the same model, and we feed the concatenated utterances for dialogue generation. we randomly sample 50 open-sourced cartoon images and use one sentence of image description as a learning target to generate 400 tutoring dialogues. the total utterance number is 9k. for quality assessment and data analysis, we conduct both llm-based automated evaluation and manual rating by human experts. automated evaluation: we adopt the llm-as-ajudge @xcite with gpt-4 to validate the effectiveness of pedagogical code-switching according to our proposed framework. given a dialogue, the evaluator is to predict the occurrence of each dimension (e.g., introduce new vocabulary, clarify comprehension, etc.). next, we assess whether code-switching is applied to these functions. in each dimension, one point will be added if the dialogue meets the criteria (as shown in appendix ## conclusion in this work, we combined scaffolding strategies and translanguaging functions to propose a pedagogical code-switching framework. in a theoryinspired practice, we developed fine-grained instructional strategies tailored to multilingual learners and bilingual education needs, and leveraged llms as the tutoring agent and automated evaluator. our experimental results revealed that stateof-the-art llms demonstrated reasonable codeswitching and pedagogical ability in bilingual learning contexts. moreover, we observed that englishcentric llms show imbalanced performance in scaffolding, translanguaging, and pedagogical abilities across different languages, and improving cross-lingual consistency can be one of the future work. aside from language learning, our proposed code-switching framework can also be extended to broader multilingual interactions.