C antonese Natural Language Processing in the Transformers Era (2024)

Abstract: Despite being spoken by a large population of speakers worldwide, Cantonese is under-resourced in terms of the data scale and diversity compared to other major languages. This limitation has excluded it from the current “pre-training and fine-tuning” paradigm that is dominated by Transformer architectures.In this paper, we provide a comprehensive review on the existing resources and methodologies for Cantonese Natural Language Processing, covering the recent progress in language understanding, text generation and development of language models.We finally discuss two aspects of the Cantonese language that could make it potentially challenging even for state-of-the-art architectures: colloquialism and multilinguality .

## introduction cantonese, or yue chinese, is a diaspora language with over 85 million speakers all over the world @xcite . 1 it is commonly used in colloquial scenarios (e.g., daily conversation and social media) but also in formal and written contexts, such as in the legislative council of the hong kong special administrative region, or in sections of special local interests in the newspapers, such social and entertainment, or in horse racing and betting information. otherwise standard chinese (scn) foot_1 , sometimes called putonghua (普 通话) or guoyu (國語), is generally favored in formal and written contexts @xcite . in terms of digital language support, mandarin chinese thrives with a mature natural language processing (nlp) environment. chinese nlp has a versatile and growing literature from major conferences, such as acl and coling. in contrast, as for digital language support cantonese is at the vital level, one level lower than thriving (cf. ethnologue) @xcite . in fact, cantonese is an rare exception as a main diaspora language, as most diaspora languages -including but not limited to arabic, chinese, english, french, hindi, japanese, korean, portuguese, spanish, etc.-have both a thriving digital language support and a strong nlp community, while cantonese does not @xcite . more specifically, while current nlp paradigms have been deeply changed by large-scale pretraining models based on transformer architectures, such as bert @xcite , roberta @xcite , xlnet @xcite , electra @xcite , @xcite and gpt-4 @xcite , which have achieved state-of-theart (sota) level of performance on several tasks. compared to the previous generation systems, the progress was particularly remarkable in task requiring fine-grained semantic understanding, such as textual entailment, question answering and causal reasoning @xcite . on the other hand, language technologies for cantonese have not yet benefited from this revolution @xcite . from this point of view, the number of publications in the acl anthology is emblematic (see figure 1 ): only 61 papers are related to "cantonese", compared to 9,756 papers for english, and 5,312 (4,919 + 393) for scn/mandarin. the history of publications in cantonese nlp, as in figure 1 , shows that the numbers of papers published yearly remains in single digit, although there is a moderate increasing trend (cf. figure 2 ). however, as an emergent language in nlp, it is surprising that only a small portion (17/61, 27.9%) introduces language resources, as shown by in light of these concerns, this paper presents a first overview of cantonese nlp, going through essential issues regarding this language's uniqueness, data scarcity, research progress, and major challenges. as a pilot study, we also present some preliminary analysis on cantonese data from social media and discuss the possible challenges. we found that, given the prominence of colloquial language and code-switching in the data, it is desirable that future models will be developed to properly deal with such phenomena. finally, we conclude our contribution by indicating some possible directions for future research. ## pilot study for cantonese in the previous sections, we have illustrated the general scarcity of resources in nlp for cantonese. we also mentioned that cantonese has a numerous and active social media community, and cantonese social media language provides an interesting example for analysis, as it can show the main challenges related to the automatic processing of this language. as we anticipated, colloquialism and multilinguality are primary obstacles to robust and effective processing. in the next sections, we present an analysis of the two phenomena in cantonese social media. ## conclusions in this paper, our goal is to present the status of the research on cantonese nlp, to describe the uniqueness of this language and to suggest possible solutions for addressing the current shortcoming, due to the lack of resources. indeed, most research on cantonese nlp has not translated into the release of useful models, corpora and benchmark datasets, which are often not publicly available or not up to date. a possible reason of this difficulty is the limited number of online sources of cantonese text with non-restrictive licenses @xcite , which does not leave too many options to researchers for putting together new benchmarks and for training large-scale models that are cantonese-specific. after reviewing the existing resources and methods, we analyzed the two main challenges that such data pose to automatic systems: the pervasive colloquialism and the multilinguality of cantonese text, which often leads to the simultaneous presence of multiple languages in the same message or post. as strategies to tackle the challenges of cantonese nlp, we could safely indicate data augmentation and crosslingual learning as two possible ways to go, in case the collection and balancing of large-scale cantonese corpora turn out to be too problematic. cantonese is one of the most pervasive diaspora languages with native speaking communities spread around the world and has a vibrant and multicultural online community, and unique features that deserve a special attention for computational modeling. with our contribution, we hope we will manage to stimulate a new interest around this language in the nlp community, and to encourage future studies that will be devoted to resource sharing and to the reproducibility of the research results on public benchmarks.