The Third Multilingual Surface Realisation Shared Task ( SR ’20): Overview and Evaluation Results (2020)

Abstract: This paper presents results from the Third Shared Task on Multilingual Surface Realisation (SR’20) which was organised as part of the COLING’20 Workshop on Multilingual Surface Realisation. As in SR’18 and SR’19, the shared task comprised two tracks: (1) a Shallow Track where the inputs were full UD structures with word order information removed and tokens lemmatised; and (2) a Deep Track where additionally, functional words and morphological information were removed. Moreover, each track had two subtracks: (a) restricted-resource, where only the data provided or approved as part of a track could be used for training models, and (b) open-resource, where any data could be used. The Shallow Track was offered in 11 languages, whereas the Deep Track in 3 ones. Systems were evaluated using both automatic metrics and direct assessment by human evaluators in terms of Readability and Meaning Similarity to reference outputs. We present the evaluation results, along with descriptions of the SR’19 tracks, data and evaluation methods, as well as brief summaries of the participating systems. For full descriptions of the participating systems, please see the separate system reports elsewhere in this volume.

## introduction sr'20 is the fourth in a line of shared tasks focused on surface realisation, the name originally given to the last stage in the first-generation (pre-statistical and pre-neural) natural language generation (nlg) pipeline, mapping from semantic representations to fully realised surface word strings. when we ran the first surface realisation shared task in 2011 @xcite , it was to address a situation where there were many different approaches to sr but none of them were comparable. we developed a commonground input representation that different approaches could map their normal inputs to, making results directly comparable for the first time. most sr'11 systems (and all top performing ones) were statistical dependency realisers that did not make use of an explicit, pre-existing grammar. however, the question of how inputs to the realisers were going to be provided in an embedding system was left open. by the time we proposed the second sr task @xcite , universal dependencies (uds) had emerged as a convenient standard in parsing, with many associated data sets, that we were able to pick up and use as the common-ground representation. by now, the third, neural generation of nlg methods was beginning to dominate the field, and systems participating in sr'18 were all trained to map directly from the ud inputs to the surface strings by some form of neural method. the question of how inputs to the realisers were going to be supplied remained open; moreover, most current approaches to nlg no longer even distinguished a separate surface realisation stage. nevertheless, the community enthusiastically participated in sr'18 and sr'19 @xcite as we expanded tracks to 11 languages. this year, things look different again. there is much discussion in the field of how to control the vexed tendencies of neural generators to 'hallucinate' content @xcite , and how to instil some order and coherence over longer texts. multi-hop approaches are increasingly proposed to address such issues @xcite @xcite , and are beginning to look somewhat like the old nlg pipeline. in this context, surface realisation is very much back on the agenda, and the term is coming back into frequent use @xcite . our aim for future editions of the sr shared task is to test whether multi-hop gives better results overall than single-hop, but also to link up with content selection modules capable of supplying the inputs required by sr systems. for this year, our main objective is to explore the impact of restricted vs. unrestricted resources in system training, and cross-domain generalisability. we start below with an overview of the shared task and tracks (section 2), followed by descriptions of the participating systems (section 3), the data (4), evaluation methods (section 5), and results (section 6). the two main tracks are as follows: t1 shallow track: the inputs in this track are ud structures in which most of the word order information has been removed and tokens have been lemmatised. in other words, it starts from unordered dependency trees with lemmatised nodes that hold pos tags and morphological information as found in the original treebank annotations. the outputs are the fully realised sentences. the task in this track therefore amounts to determining the word order and inflecting words. a. restricted-resources subtrack (same as sr'19 track 1): teams built models trained on the provided t1 dataset(s), but use of external task-specific data was not permitted. however, teams were allowed to use external generic resources. for example, available parsers such as uuparser @xcite could be run to create a silver standard versions of provided datasets and use them as additional or alternative training material. also permitted was the use of generic publicly available off-the-shelf language models such as gpt-2 @xcite , @xcite , polyglot @xcite . alternatively, bert @xcite could be fine-tuned with publicly available datasets such as wikitext @xcite or the deepmind q&a dataset @xcite . ## participating systems there were two distinct sets of participating systems this year. firstly, there were the new participants who built systems specifically for sr'20 (each system briefly summarised in section 3.1). in addition, we asked the 2019 participants to run their systems on the new test sets, and 4 teams were able to do so (section 3.2), two of which also submitted new systems to sr'20. ## results in this section, we present evaluation results produced with all evaluation methods from the preceding section, for all test set outputs received from participants. the best scores are generally better this year compared to 2019, both according to automatically computed metrics and human assessments. by way of introduction, ## concluding remarks the 2020 edition of the sr shared task (sr'20) saw 5 teams submitting new systems and 4 teams submitting outputs for the new test sets using their 2019 systems. datasets, evaluation scripts, system outputs and more information about the shared task can be found on the genchal repository. 13 among the notable trends we can observe in evaluations this year are the following: (i) the best shallow track english systems appear to have closed the gap to human-written texts in terms of all evaluation measures; (ii) for the first time we have seen outputs for a non-english language (spanish) approach the quality of human-written reference texts; and (iii) allowing additional resources to be used in system building can make a very big difference to performance. further progress has also been made in sr'20 for deep track systems: the best deep track system performed equally well or better than most shallow track systems for both readability and meaning similarity. overall, the sr'20 results provide further evidence that generation from structured meaning representations can be done with impressive success by current neural methods. our aim for next year's edition of the shared task is to add linked tasks corresponding to an earlier stage in the generation process, asking for submissions which use the intermediate ud representations as well as submissions that by-pass them in order to compare which gives better results overall.