Developing and Utilizing a Large-Scale C antonese Dataset for Multi-Tasking in Large Language Models (2025)

Abstract: High-quality data resources play a crucial role in learning large language models (LLMs), particularly for low-resource languages like Cantonese. Despite having more than 85 million native speakers, Cantonese is still considered a low-resource language in the field of natural language processing (NLP) due to factors such as the dominance of Mandarin, lack of cohesion within the Cantonese-speaking community, diversity in character encoding and input methods, and the tendency of overseas Cantonese speakers to prefer using English. In addition, rich colloquial vocabulary of Cantonese, English loanwords, and code-switching characteristics add to the complexity of corpus collection and processing. To address these challenges, we collect Cantonese texts from a variety of sources, including open source corpora, Hong Kong-specific forums, Wikipedia, and Common Crawl data. We conduct rigorous data processing through language filtering, quality filtering, content filtering, and de-duplication steps, successfully constructing a high-quality Cantonese corpus of over 2 billion tokens for training large language models. We further refined the model through supervised fine-tuning (SFT) on curated Cantonese tasks, enhancing its ability to handle specific applications. Upon completion of the training, the model achieves state-of-the-art (SOTA) performance on four Cantonese benchmarks. After training on our dataset, the model also exhibits improved performance on other mainstream language tasks.

## introduction high-quality data resources are essential for the advancement of large language models @xcite , particularly for languages with limited digital resources, such as cantonese. although cantonese boasts over 85 million native speakers @xcite , predominantly located in southern china and among chinese communities worldwide, it remains classified as a low-resource language within the domain of nlp. this is mainly due to the dominance of mandarin, the lack of uniformity within the cantonese-speaking community, and the diversity of character encoding and input methods. in addition, overseas cantonese speakers tend to use english, which further hinders the development of cantonese in the nlp domain. the rich colloquial vocabulary of cantonese, its english loanwords, and the widespread phenomenon of code-switching make corpus collection and processing more complex. compared to modern standard chinese, there is a significant disparity between spoken and written cantonese; many colloquial expressions lack a standardized written form. furthermore, cantonese writing involves the conversion between traditional and simplified characters, as well as the use of unique cantonese characters and words @xcite . these factors increase the difficulty of text data normalization and processing. these challenges have led to a scarcity of high-quality cantonese corpora, limiting the performance enhancement of llms in the cantonese context. to address these issues, we collect diverse cantonese text data to construct a high-quality cantonese dataset. data sources include opensource corpora, hong kong-specific forums such as lihkg foot_0 , openrice foot_1 , the cantonese version of wikipedia foot_2 , and common crawl data foot_3 , etc. during the data collection process, we pay attention to the variations in cantonese usage across different regions and platforms. we utilize custom web crawlers and data extraction tools to efficiently gather large amounts of text from these sources. to ensure the quality and purity of the data, we establish and executed a stringent data processing workflow. first, we perform language filtering on figure 1 : overview of our work. we construct cantonese continuous pre-training and sft data, apply language and quality filters to the former, and derive the latter from it. the base model of yuetung is the qwen-2.5-7b model, which is trained on the yuedata. yuetung achieves sota performance on cantonese benchmarks, and its performance on mainstream language benchmarks not only did not decline but actually improved. the collected texts using language identification models to ensure only cantonese content is retained. next, we conduct quality filtering based on a series of heuristic rules to select high-quality texts. we also apply specialized classifiers to detect and remove harmful content, such as toxic language and sensitive information. in addition, techniques like minhash and locality sensitive hashing (lsh) @xcite are used for deduplication, ensuring the corpus's uniqueness and diversity. after continuous pre-training the model on this extensive corpus, we apply sft using additional cantonese datasets to further enhance its performance on downstream tasks. through these efforts, we successfully build a high-quality cantonese corpus containing over 2 billion tokens, laying a solid foundation for training large language models. in model performance evaluations, our model achieve industry-leading performance on four cantonese benchmark tests, accurately handling cantonese-specific vocabulary and expressions while generating fluent and natural text. notably, after training with our data, the model also demonstrate performance improvements on other mainstream language tasks, proving that high-quality cantonese data contributes to the overall performance enhancement of the model. ## results and analysis section a.2 for a more detailed analysis. ## conclusion in this study, we successfully developed and utilized a large-scale cantonese dataset (yuedata) specifically for training and testing large language model. by collecting over 2 billion tokens of cantonese text from multiple sources, we constructed a high-quality corpus and trained the yuetung model on this foundation. through rigorous data process-ing and refined training, the yuetung demonstrated excellent performance in four cantonese benchmark tests. this not only showcases the quality of the yuedata dataset but also validates the effectiveness of our data processing and training strategies.