Next-Level C antonese-to- M andarin Translation: Fine-Tuning and Post-Processing with LLM s (2025)

Abstract: Large Language Models (LLMs) have improved performance across various natural language processing tasks. Despite these improvements, LLMs continue to face significant challenges, such as grammatical issues and code-switching to English, when applied to low-resource languages like Cantonese in Machine Translation (MT) scenarios. By addressing the unique linguistic and contextual challenges of Cantonese, we present a novel strategy to improve the understanding and translation capabilities of LLMs for Cantonese-to-Mandarin MT. Our strategy comprises three key components: (1) Syntax and Part-of-Speech (POS) fine-tuning, where we use the Universal Dependencies (UD) corpus to fine-tune LLM, focusing on the linguistic structures of Cantonese; (2) Specialized Cantonese to Mandarin sentence pairs, collected from diverse sources such as Cantonese grammar textbooks and manually translated sentences across various domains, to expose the model to a wide range of linguistic contexts; (3) Post-processing with additional LLMs, where we introduce additional LLMs to improve the initial translations, correcting Mandarin grammar and punctuation. Empirical evaluations on human-created test sets show that our proposed strategy improves translation performance and outperforms existing commercial translation models with at least 3 BLEU scores. Additionally, our strategy also benefits other LLMs and a reversed translation direction, demonstrating its generalization and effectiveness.

## introduction the rapid advancement of large language models (llms) has impacted various natural language processing (nlp) tasks, including machine translation (mt), where llms leverage extensive pretraining to capture a wide range of linguistic patterns and contextual information to improve translation quality @xcite . cantonese, a major chinese dialect spoken primarily in hong kong, macau, and parts of southern china, has unique linguistic characteristics that differ from standard mandarin @xcite . as figure 1 shows, lexical divergence represents a cardinal disparity between cantonese and mandarin, revealing not solely in lexical choice but also encompassing frequent usage, notably function words. at the syntactic level, although both dialects exhibit a broad alignment, they diverge markedly in specific facets. an example is the inversion of double object ordering, wherein mandarin adheres to a [human object] + [thing object] configuration, while cantonese reverses this to a [thing object] + [human object] construct @xcite . code-switching to english is a common phenomenon in cantonese, which is not typically observed in standard mandarin. however, the english used in these codeswitching instances often deviates from formal english grammar and instead follows cantonese grammatical structures in most cases @xcite . this unique form of code-switching often results in a blend of cantonese syntax and english vocabulary. these differences, including distinct phonology, syntax, and vocabulary, pose challenges for existing llms in mt scenarios, which are often trained on mandarin-centric datasets and lack the necessary linguistic knowledge to handle cantonese effectively @xcite @xcite . to address these challenges, we propose a novel strategy to improve the performance of llms in cantonese-to-mandarin mt. our strategy incorporates syntax and parts-of-speech (pos) fine-tuning, specialized cantonese-to-mandarin sentence pairs, and post-processing with additional llms within the translation pipeline. experimental results show that our strategy significantly improves llm performance in cantonese-to-mandarin translation tasks, yielding higher bleu scores on a humancreated test set. moreover, other llms also benefit from our proposed strategy to improve translation performance and are better able to handle reverse translation from mandarin to cantonese. our main contributions are as follows. existing studies on cantonese-to-mandarin mt primarily focus on translating from mandarin to cantonese. unlike widely studied language pairs 1 our dataset can be found at https: //huggingface.co/datasets/hkallen/ cantonese-chinese-parallel-corpus such as english-to-mandarin, cantonese is a lowresource language, and large-scale, high-quality parallel data for cantonese is limited. the scarcity has prompted the exploration of diverse methods for corpus construction and translation improvement. @xcite conduct parallel sentence mining to generate a substantial number of sentence pairs, significantly improving translation quality. additionally, @xcite compare different model architectures, tokenization schemes, and embedding structures to investigate the linguistic differences between mandarin and cantonese. @xcite propose a non-autoregressive mt model for mandarin to cantonese translation where it improves the intelligibility and naturalness of synthesised speech. in recent years, traditional neural mt models have increasingly turned to llms to handle cantonese sentences. @xcite perform a cantonmt pipeline using llms to process cantonese sentences and fine-tuning translation targets. @xcite ) discuss llm's factual generation, mathematical logic, complex reasoning, and general knowledge in cantonese and translation scenarios. @xcite propose a strategy called talen, where it shows how to translate source sentence to target sentence via cantonese syntax patterns. these studies highlight the growing importance of llms in advancing the state-ofthe-art in low-resource language mt, particularly for cantonese. however, most studies have concentrated on mandarin-to-cantonese translation, leaving the cantonese-to-mandarin direction underexplored. this directional bias is likely driven by practical needs, such as converting standard mandarin text into cantonese for use in regions where cantonese is spoken, including hong kong and guangdong province in china. despite the progress made in mandarin-to-cantonese translation, there are still gaps in the performance and discussion of cantonese-to-mandarin translation that need to be further investigated. ## methodology in this section, we provide detailed descriptions of each module in our cantonese-to-mandarin translation strategy. figure 2 shows the overall pipeline of the proposed translation strategy. ## what happens to other models? the experiments presented above demonstrate the effectiveness of our proposed strategy in improving cantonese-to-mandarin mt performance. however, a question remains: does this strategy also enhance the translation performance when applied to other llms? to investigate this, we conduct additional experiments using the latest state-of-the-art llms and traditional mt models to evaluate the generalizability of our strategy and its potential impact on the field of mt. these experiments involve fine-tuning multiple llms with the same lora configurations and datasets, comparing their performance on the same tasks. ## what happens to a reversed direction? to further validate the robustness of our proposed strategy, we extend our experiments to the reverse translation direction, from mandarin to cantonese. it allows us to examine whether the improvements observed in cantonese-to-mandarin translation are specific to that direction or generalize to the opposite direction as well. we aim to establish its broader applicability and reliability in real-world mt scenarios. this comprehensive validation not only enhances the credibility of our strategy but also contributes significantly to the broader field of mt, especially for low resource languages like cantonese. ## conclusion in this paper, we present a strategy to improve translation performance in low-resource language mt scenarios, focusing on cantonese-to-mandarin translation. our approach enables yi-1.5-34b to better understand cantonese sentence structures through syntax/pos fine-tuning. by leveraging a custom-compiled dataset and additional llms for post-processing, we significantly improve cantonese-to-mandarin translation perfor-mance, with bleu scores increasing by at least 5 points compared to current commercial mt engines. this strategy is effective not only for yi-1.5-34b but also for other llms, particularly smaller parameter models. furthermore, our experiments show that llms continue to benefit from this strategy in the reverse translation direction, achieving higher bleu scores than commercial mt engines and baseline versions of llms.