Lessons from Computational Modelling of Reference Production in M andarin and E nglish (2020)

Abstract: Referring expression generation (REG) algorithms offer computational models of the production of referring expressions. In earlier work, a corpus of referring expressions (REs) in Mandarin was introduced. In the present paper, we annotate this corpus, evaluate classic REG algorithms on it, and compare the results with earlier results on the evaluation of REG for English referring expressions. Next, we offer an in-depth analysis of the corpus, focusing on issues that arise from the grammar of Mandarin. We discuss shortcomings of previous REG evaluations that came to light during our investigation and we highlight some surprising results. Perhaps most strikingly, we found a much higher proportion of under-specified expressions than previous studies had suggested, not just in Mandarin but in English as well.

## introduction referring expression generation (reg) originated as a sub-task of traditional natural language generation systems @xcite . the task is to generate expressions that help hearers to identify the referent that a speaker is thinking about. reg has important practical value in natural language generation @xcite , computer vision @xcite , and robotics @xcite . additionally, reg algorithms can be seen as models of human language use @xcite . in line with this second angle, and unlike reg studies which have started to use black-box neural network based models (e.g., @xcite and @xcite ), we focus on two aspects (cf., @xcite ): 1) designing and conducting controlled elicitation experiments, yielding corpora which are then used for analysing and evaluating reg algorithms to gain insight into linguistic phenomena, e.g., gre3d3 @xcite , tuna @xcite , @xcite , and map-task @xcite . 2) designing algorithms that mimic certain behaviours used by human beings, for example the maximisation of discriminatory power @xcite and/or the preferential use of cognitively "attractive" attributes @xcite ; see @xcite for discussion. the focus of these studies was mostly on indo-european languages, such as english, dutch @xcite and german @xcite . recently researchers have started to have a look at mandarin chinese @xcite , collecting a corpus of mandarin res, namely mtuna. so far, only a preliminary analysis has been performed on mtuna, and this analysis has focussed on issues of linguistic realisation @xcite : the res in the corpus have not yet been compared with those in other languages, and the performance of reg algorithms on the corpus has not been evaluated. to fill this gap, we provide a more detailed analysis of the use of mandarin res on the basis of the mtuna corpus. we annotated the mtuna corpus in line with the annotation scheme of tuna @xcite , after which we used this annotation to evaluate the classical reg algorithms and compared the results with those for the english etuna corpus. since it has been claimed that mandarin favours brevity over clarity -the idea that mandarin is "cooler" than these other languages @xcite -relying more on communicative context for disambiguation than western languages, we concentrated on the use of over-and under-specification. after all, if mandarin favours brevity over clarity to a greater extent than english and dutch, then one would expect to see less over-specification and more over-specification in mandarin. ## background the analysis reported in paper is based on the mtuna (for mandarin) and etuna (for english) corpus. we start by briefly introducing the tuna experiments in general, and we highlight some special features of mtuna together with its initial findings. ## method before we address the four research questions in section 3, we explain how we annotated the corpus. the annotated corpus is available at github.com/ a-quei/mtuna-annotated ## analysis before reporting results and analysis, we explain what datasets and algorithms were analysed, and how evaluation was performed. dataset. the sources of our dataset are the mtuna and etuna corpora. for mandarin, we used the whole mtuna dataset. for comparing between languages fairly, we only used res for scenes that were shared between mtuna and etuna; we call this set of shared scenes mtuna-ol. the original mtuna has 20 trials, with 10 trials for each domain. the mtuna-ol and etuna contains 13 trials, in which there are 7 and 6 trials from furniture and people domain respectively. (more details of which scene is used can be found in the appendix.) algorithms. we tested the classic reg algorithms, including: 1) the full brevity algorithm (fb @xcite ): an algorithm that finds the shortest re; 2) the greedy algorithm (gr @xcite ): an algorithm that iteratively selects properties that rule out a maximum number of distractors (i.e., a property that has the highest "discriminative power"); and 3) the incremental algorithm: an algorithm that makes use of a fixed "preference order" of attributes (ia @xcite .