When C antonese NLP Meets Pre-training: Progress and Challenges (2022)

Abstract: Cantonese is an influential Chinese variant with a large population of speakers worldwide. However, it is under-resourced in terms of the data scale and diversity, excluding Cantonese Natural Language Processing (NLP) from the stateof-the-art (SOTA) “pre-training and fine-tuning” paradigm. This tutorial will start with a substantially review of the linguistics and NLP progress for shaping language specificity, resources, and methodologies. It will be followed by an introduction to the trendy transformerbased pre-training methods, which have been largely advancing the SOTA performance of a wide range of downstream NLP tasks in numerous majority languages (e.g., English and Chinese). Based on the above, we will present the main challenges for Cantonese NLP in relation to Cantonese language idiosyncrasies of colloquialism and multilingualism, followed by the future directions to line NLP for Cantonese and other low-resource languages up to the cutting-edge pre-training practice.

## tutorial description in our tutorial, there will be five parts (shown in figure 1 ), each presented by a tutorial presenter. the first part will be the overview of cantonese nlp research (part i) and the second exhibits the progress in language specify, resources, and methodologies (part ii). then, we will introduce the roles played by language pre-training in the sota nlp practice (part iii), based on which we further discuss the challenges to benefit cantonese nlp from the trendy "pre-training and fine-tuning" fashion (part iv). lastly, the potential solutions will be pointed out to shed light on the promising future direction of cantonese nlp (part v). the detailed content is shown in the following. part i: cantonese nlp overview (in 30 min). at the beginning, we will briefly introduce cantonese language and its related research in nlp. cantonese is a language from the chinese family with over 73 million speakers in the world @xcite . it is mostly used in colloquial scenarios (e.g., daily conversation and social media) and exhibits different vocabulary, grammar, and pronunciation compared to standard chinese (scn) foot_0 , which is mainly designed for formal writing @xcite . despite the substantial efforts in chinese natural language processing (nlp), most previous studies center around scn, where limited work attempts to explore how to process cantonese with the cuttingedge nlp techniques @xcite . modern nlp paradigms have been deeply revolutionized by large-scale pre-training models, e.g., bert @xcite and @xcite , which have achieved sota performance on many nlp tasks via fine-tuning. although the general nlp field is thriving, cantonese nlp has drawn limited attention so far, as demonstrated by the recent publications in the acl anthology -only 47 papers are related to "cantonese", compared to 7,018 papers for english, 2,355 for common chinese, and 323 for mandarin. this tutorial will present a roadmap lining cantonese nlp up to the sota practice based on pretraining. we will start with the previous progress made by linguistics and nlp researchers, followed by the major challenges caused by the language specificity, and end with the promising future directions to allow cantonese and other low resource languages to benefit from the advanced nlp techniques. the details will be covered in part ii-v. part ii: progress in language specificity, resources, and methodologies (in 40 min). cantonese (or yue) is the second most popular dialect among all chinese variants @xcite . formal writing exhibits no essential difference among the scn in various regions (kataoka and lee, 2008; yu, 2013), whereas in informal situations, colloquial cantonese diverges substantially from scn in phonology, orthography, lexicon, and grammar. in dealing with cantonese data, colloquialism and multilingualism are two fundamental challenges. unlike standard chinese, which character standardization can be dated back to the qin dynasty (221 bc), cantonese is mainly derived from pronunciation and may contain many colloquial features, such as non-standard spelling, local slang, neologisms. on the other hand, cantonese language historically evolved in multi-lingual environments and this is especially true for hk cantonese, as shown by the large inventory of english loanwords borrowed through phonetic transliteration. unlike scn, which benefits from abundant wellannotated textual resources, there is a chronic lack of digital resources for cantonese data. the existing resources are summarized into three categories: corpora, benchmarks, and expert resources. in general, using existing cantonese resources may be difficult for three reasons: (1) the data scale is relatively small (especially compared to scn); (2) the domain is usually specific and lacks diversity and generality; (3) many resources mix cantonese and scn which might confuse nlp models and hinder them from mastering 'authentic' cantonese. cantonese specific nlp methods are relatively less explored. the following presents a detailed review of cantonese nlp methods in (1) natural language understanding and (2) natural language generation. several language understanding tasks will be introduced, including word segmentation, spell checking, rumor detection, sentiment analysis, and dialogue slot filling. as for language generation, we will summarize previous studies on dialogue summarization, machine translation, etc. part iii: pre-training in sota nlp (in 40 min). the cutting-edge nlp takes advantages of the promising results achieved by the pre-training of language representations. a typical pre-trained and fine-tune scheme refers to pre-train a large model on massive unlabelled corpora by selfsupervised objectives, and fine-tune the model on downstream tasks with task-specific loss. such self-supervised objectives, e.g. masked langauge modeling (mlm) and next sentene prediction (nsp) @xcite enable the model to gain generalized language representations without human supervision. during fine-tuning stage, the pre-trained representations can be further used to learn a specific natural language understanding (nlu) task with small-scale annotations via incremental training. transformer @xcite @xcite is the most widely employed pre-trained architecture in nlp. the transformer encoder consumes the input text and project it into high-dimension vectors, which are feed into the transformer decoder to generate the output sequences. inspired by the transformer architecture, researchers explore the transformer encoder for nlu tasks and transformer decoder for natural language generation (nlg) tasks. since the transformer-based pre-training (devlin et al., 2019; liu et al., 2019) was introduced to the world, championing the leaderboards of many nlp benchmarks, the "pre-training and fine-tuning" paradigm has profoundly revolutionized the way we research nlp for most of the majority languages, such as english and chinese. nevertheless, the success of language pre-training is built upon the availability of rich language resources and large-scale textual corpora, hindering cantonese and other low-resource languages from gaining the benefit of pre-training. the following presents the challenging low-resource issue in cantonese. part iv: challenges from colloquialism and multilingualism (40 min). in cantonese nlp, colloquialism and multilingualism jointly present the low-resource challenges. cantonese is by nature colloquial, where using casual language would sparsify the context and require more data for contextual model training, making the low-resource issue serious. like other low-resource language processing @xcite , it is possible to gather large-scale data from social media. however, models might also compromise their performance using just noisy social media data. it not only requires effective data cleaning and augmentation, but also non-trivial model capabilities of capturing salient information in context with diverse quality. cantonese exhibits a code-switching convention with multiple languages. substantially richer context is hence required for nlp models to gain the multilingual understanding capabilities, despite the limitation to make it happen in the lowresource scenarios. although it is possible to transfer the knowledge gained in a similar language with rich resources (henceforth cross-lingual learning) @xcite @xcite , cantonese, as a vibrant language, absorbs the knowledge from numerous languages beyond scn and english. part v: future nlp directions for cantonese and other low-resource languages (30 min). data scarcity and limited methodology exploration are top issues for cantonese in benefiting deep semantics and general nlp tasks. to mitigate lowresource problems, data augmentation is an alternative to scale up the cantonese dataset for nlp model training. for example, we might employ heuristic rules @xcite , machine learning ( şahin and steedman, 2018) and information retrieval @xcite to automatically boost the data scales. in the augmentation process, we might need to learn how to distinguish scn from cantonese. though both are encoded in the chinese language system, the former dominates the chinese resources while the latter is a minority @xcite . cross-lingual learning might provide another be a promising alternative for the pre-training in low resource, which borrows knowledge from other languages @xcite @xcite . we may take advantage of sota pre-trained transformers to capture the general and specific language features for transfer learning @xcite . in addition, based on cantonese's phonological history, future work may consider injecting phonetic knowledge into language learning or developing multi-modal understanding across text and speech. ## type of the tutorial this is an introductory tutorial of cantonese nlp, where we draw nlp community's attention to look at the research of cantonese -a language with over 73 million speakers in the world @xcite while only has 47 papers in acl anthology related to it. the tutorial will present a roadmap going through the essential issues regarding language specificity, data scarcity, research progress, and major challenges for cantonese nlp to be benefited from the cutting-edge nlp paradigms based on language pre-training. ## target audience our tutorial is designed for the attendees of premier computational linguistics conferences, who preferably have interests and working experience in the processing of asian languages and low-resource languages. the audiences would better have the following prerequisites. ## reading list for trainees interested in reading important studies before the tutorial, we recommend the following: @xcite ; @xcite ; sachs and @xcite . vaswani et al. (2017); devlin et al. (2019); liu et al. (2019); brown et al. (2020); sun et al. (2019); liu et al. (2019); nguyen et al. (2020). ## tutorial presenters our tutorial will contain 5 parts and here we introduce the presenter for each of them. ## other information inclusion of others' work. this tutorial will survey the progress of cantonese nlp and language pretraining, which substantially contain others' work.