Using BERT for choosing classifiers in M andarin (2021)

Abstract: Choosing the most suitable classifier in a linguistic context is a well-known problem in the production of Mandarin and many other languages. The present paper proposes a solution based on BERT, compares this solution to previous neural and rule-based models, and argues that the BERT model performs particularly well on those difficult cases where the classifier adds information to the text.

## introduction the grammar of mandarin and certain other chinese languages requires that, in a number of syntactic positions, a noun must be preceded by a classifier word. classifiers often give a rough indication of the kind of entity denoted by the noun. for example, the classifier "只" (zhī) in the noun phrase (np) "一只狗" (yì zhī gǒu; a dog) indicates the noun "狗" (gǒu; dog) is an animal. it is worth noting that, in addition to mandarin, classifiers also play a critical role in a few other languages, especially the east asian languages, such as korean, japanese, and vietnamese @xcite . generally speaking, it is, in many ways, not unlike types in functional programming languages like haskell, which add to each function defined by the programmer a broad semantic categorisation of that function @xcite . mandarin contains a large number of classifiers, and although the choice of classifier is limited by the (head) noun with which the classifier is associated, this may still leave several options, which may sometimes produce a different meaning, e.g., although each of these cases involves classifier choice, the problem of choosing a classifier is likely to be more challenging in those cases, such as (b)-(d), where the classifier adds information, for example, in terms of politeness ((b), neutral vs. polite), number ((c), singular vs. plural), or quantity ((d), a cup vs. a can of coffee). this is perhaps clearest in the case of (d), where "杯" (bēi; cup) and "听" (tīng; can) indicate different containers, and consequently different quantities, of coffee; these classifiers are known as measure words, as opposed to the "pure" classifiers of (a)-(c). researchers have asked what determines the choice of classifier, constructing algorithms that predict what classifier suits a given discourse context. the most sophisticated model we are aware of is @xcite . ambitiously, these authors decided to deal with classifiers of all different types, also including measure words for instance, which are difficult to predict because they add information. they approached the problem as follows: given a sentence in which a classifier is yet to be realised, and the head noun is flagged, predict the missing classifier. for example, in the input: cl indicates where the missing classifier is and the h tag pair flags the head noun. the authors construct a large-scale classifier dataset, namely chineseclassifierdataset 1 (henceforth, ccd) by extracting and filtering data from publicly available chinese corpora. they did experiments on their ccd corpus with several baselines, including a rulebased system, two machine learning based system, and a lstm-based system @xcite ). an initial evaluation study indicated that the lstm achieved the best performance. our own work takes the same perspective as @xcite . but although the performance of the model of peinelt et al. is encouraging, it still leaves considerable room for improvement; in particular, the question comes up whether bert, with its superior ability to take context into account, might perform better. in addition, the model of peinelt et al. offers only limited insight, because it does not distinguish between different types of classifiers. in other words, the performance of the model may mask important differences between different types of classifier choice. a good way to address this limitation would be to make use of an existing categorisation of classifier types. but although linguists generally agree that "true" (or "sortal") classifiers should be distinguished from measure words @xcite , there exist subtle disagreements regarding exactly how these sub-types should be defined, and what further divisions between sub-types should be taken into account. subtypes are often described by example, without computationally implementable criteria or explicit lists of classifiers @xcite . to our knowledge, @xcite are the only ones to provide comprehensive lists of classifiers of various subtypes, and in what follows we will make use of these lists. in section 2, we introduce two different bertbased models, one of which uses word masking and one of which performs classification. in section 3, we report on our comprehensive evaluation experiments, in which we compare our bert-based models with each other and with several baselines, using the ccd dataset. ## discussion we conclude that (1) contextualised pre-trained models (i.e., bert and mlm) perform remarkably well on the task of choosing classifiers in mandarin, and fine-tuning helps improve the recall of choosing classifiers; (2) a simple rule-based system has respectable performance; (3) in terms of accuracy, a pre-trained masked language model (i.e., mlm) was able to select proper classifiers about equally well as the above rule-based system; (4) bert struggles to predict classifiers that add information (measurement, plurality, politeness). the last finding confirms our (linguistically well-established) expectation that some classifier occurrences cannot be predicted from their linguistic context alone since they themselves carry additional information. since the choice of classifier is not deterministic (e.g., consider the choice between "个" and "台" in example (a)), the type of corpus evaluation that was performed in this paper arguably does not "tell the whole story" regarding the quality of the different models. to remedy this issue, we plan two further experiments, each of which starts from the observation that the classifier that was chosen in a given linguistic context in the corpus will often not be the only felicitous choice. one experiment will focus on speakers. we will ask several participants to choose classifiers given a linguistic context. by comparing the outcomes of such an elicitation experiment with the ccd corpus, we will obtain a better understanding of the variations that exist between speakers and of the difficulty of the task that we have set our algorithms. by thus asking multiple participants to accomplish the same task as our algorithms, we will obtain a new corpus, in which each linguistic context is associated with a bag of (1 or more) possible classifiers. this new dataset will enable us to conduct a new, non-deterministic evaluation of the models. another additional experiment will have human readers judge the acceptability of each classifier choice that is made by a given model. reader experiments of this kind are a standard tool in judging the quality of decisions taken by a natural language generation algorithm (e.g. van der lee et al. ( 2019 )) and will give rise to a new set of analyses analogous to the ones in the present paper, which will give us a better understanding of the quality of the decisions that are taken by each model. in the future, we also plan to extend the models we tested in this study. for example, regarding the pre-trained language model, a promising candidate to investigate is ernie @xcite , which has proved to be more powerful in modelling mandarin chinese. regarding the unsupervised mlm setting, the following option would be worth trying: instead of choosing the most probable word type from the whole vocabulary, one could ask the model to output the most probable classifier from all classifiers.