CS -Sum: A Benchmark for Code-Switching Dialogue Summarization and the Limits of Large Language Models (2025)

Abstract: Code-switching (CS) poses a significant challenge for Large Language Models (LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue to English summarization. CS-Sum is the first benchmark for CS dialogue summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language pair. Evaluating ten LLMs, including open and closed-source models, we analyze performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA on synthetic data) approaches. Our findings show that though the scores on automated metrics are high, LLMs make subtle mistakes that alter the complete meaning of the dialogue. To this end, we introduce 3 most common type of errors that LLMs make when handling CS input. Error rates vary across CS pairs and LLMs, with some LLMs showing more frequent errors on certain language pairs, underscoring the need for specialized training on code-switched data.

## introduction code-switching (cs) is the practice of alternating between two or more languages within a single conversation or utterance. bilingual and multilingual speakers frequently engage in cs. this form of communication has also become prevalent in social media and is the norm in multilingual societies. large language models (llms) @xcite @xcite , are trained on huge volumes of data majority of which is written in english. despite the prevalence of cs in real-world communication, existing benchmarks @xcite @xcite fail to assess how well llms process cs dialogues. improved cs comprehension would enable llms to interpret multilingual figure 1 : an instance from the benchmark prompts more effectively, enhancing accessibility for cs speakers @xcite . summarization is an ideal task to evaluate cs understanding, as it requires both grasping the dialogue's main idea and generating a concise, coherent summary across languages. the only existing cs dialogue summarization dataset, gupshup @xcite , covers only one language pair (english-hindi). other cs benchmarks like lince @xcite and gluecos @xcite focus on tasks such as language identification and pos-tagging, which assess word-or sentence-level understanding, whereas summarization requires discourse-level comprehension, a much harder test of cs ability. to truly evaluate the ability of llms on cs, we propose the cs-sum benchmark, which contains 900-1300 cs dialogue-summary pairs for three language pairs: mandarin-english (en-zh), tamil-english (en-ta) and malay-english (en-ms). the cs dialogues were created by native speakers of the respective languages. instead of generating a dialogue from scratch, the speakers were asked to translate the english dialogues in test sets of dialogsum @xcite and sam-sum @xcite -two popular english dialogue summarization datasets -to cs dialogues in their respective languages. we evaluate 9 open source and 1 proprietary llms under few-shot, translate-summarize, lora @xcite and qlora @xcite on synthetic data and provide a detailed analysis on the struggles of current sota llms(<=9b). for lora and qlora fine-tuning, the training data was generated using gemini-2-flash @xcite , as it is a large-scale llm with a freeto-use api, making it a practical choice for our experiments. our qualitative analysis of the summaries generated by llms showed that tranditional summarization metrics like rouge @xcite , bertscore @xcite are not sufficient for measuring the quality of the summaries since the llms are capable of generating summaries that contain subtle errors that change the entire meaning of the cs dialogues. to this end, we identify the three most common errors (csl, mst, sma section 5) that the llms make when summarizing cs dialogues and suggest an llm-driven approach to analyse their summaries with respect to the errors. our main contributions are: ## related work while llms have demonstrated strong performance in many multilingual tasks, they still face significant challenges when dealing with cs. studies @xcite have shown that llms, including gpt-4 and gpt-3.5, struggle with generating code-switched text, often resulting in language collapse, where the model fails to mix languages properly, and in task-specific failures such as poor performance in summarization and machine translation for cs data. these issues arise because llms are typically trained on monolingual data and lack the capacity to handle the intricacies of cs discourse. additionally @xcite , their performance is inconsistent, particularly when translating low-resource language pairs, with models showing better results for high-resource languages but underperforming when confronted with languages that are underrepresented in training datasets. these findings highlight a big problem as people prefer to interact with llms in code-switch @xcite . recent benchmarks such as lince @xcite and gluecos @xcite have been instrumental in advancing the understanding of cs phenomena by focusing on tasks like lid, ner, and pos tagging. while these datasets have provided valuable insights, it's important to recognize that llms continue to encounter challenges when dealing with the complexities inherent in code-switched discourse. to truly evaluate the cs comprehension of llms, other benchmarks are necessary-ones that go beyond wordlevel tasks and assess deeper linguistic understanding. the first step in this direction is cs-sum, a benchmark that evaluates llms on a more complex task, such as summarization, which requires a nuanced comprehension of code-switched text. ## error analysis in this section, we present an in-depth analysis of the common error types exhibited by llms when summarizing cs dialogues. our qualitative analysis across the en-zh, en-ta, and en-ms language pairs revealed three major failure modes: figure 4 illustrates csl, demonstrating how the summary incorrectly emphasizes english segments and misinterprets speaker intentions. further examples across categories are provided in tables 9 , 10 , and 11. errors were identified automatically via gpt-4o, detailed further in appendix c. to quantify these errors and compare model behaviors, we report the percentage of summaries exhibiting each error type across 9 llms under both few-shot and lora settings ( scaling within a model family does not reliably reduce errors. comparing parameter-matched variants shows no monotonic gains: gemma-2-9b lowers csl on en-zh (77.61→59.70) yet raises speaker misattribution (sma) on en-ta by 7 pp over its 2b sibling, while qwen2.5-7b improves csl on en-ms but leaves mst virtually unchanged. these inconsistencies indicate that sheer scale is not a substitute for task-specific multilingual evaluation. en-ta exhibits consistently higher error rates across models. all evaluated models show their highest rates of csl and sma on en-ta dialogues (e.g., qwen2.5-7b: 91.81% csl, 57.31% sma). this suggests that certain language-specific properties, such as morphological complexity and syntactic divergence from english, may contribute to increased summarization difficulty in this setting. sea-oriented models excel in error analysis despite mediocre automatic scores. regional models such as seallm-7b and sea-lion-gemma-2-9b achieve the lowest csl and sma within their parameter class (e.g., 53.73% csl on en-zh), yet trail larger english-specific models models on rouge and bertscore. this divergence reinforces that conventional metrics do not capture cs comprehension, underscoring the need for better metrics. ## conclusion in this paper, we introduced cs-sum, the first cs dialogue-to-english summarization benchmark across three language pairs: en-ta, en-zh, and en-ms. our experiments revealed that llms perform inconsistently across cs pairs, with some open-source models outperforming sea-specific models, highlighting that cs data must be present in pretraining, not just added through fine-tuning. furthermore, our analysis identified three major error types-csl, mst, and sma-persisting across all models, demonstrating that current llms lack intrinsic cs comprehension. the quantitative results reinforce the severity of these failures, showing that fine-tuning alone does not fully resolve cs-specific challenges. to drive further research in multilingual nlp, we will release cs-sum along with our codebase and evaluation framework.