IMS ur R eal Too: IMS in the Surface Realization Shared Task 2020 (2020)

Abstract: We introduce the IMS contribution to the Surface Realization Shared Task 2020. The new system achieves substantial improvement over the state-of-the-art system from last year, mainly due to a better token representation and a better linearizer, as well as a simple ensembling approach. We also experiment with data augmentation, which brings some additional performance gain. The system is available at https://github.com/EggplantElf/IMSurReal .

## introduction this paper presents our submission to the surface realization shared task 2020 @xcite . as in the previous year, we participate in both the shallow and the deep track. additionally, we also participate in the new setting, where additional unlabeled data is permitted to train the models. since last year's sr'19 shared task @xcite , we have made some improvements over the winning system, mostly described in @xcite . in particular, we designed a new linearization module, where we cast the task of word ordering as a traveling salesman problem (tsp), and use the biaffine attention model @xcite to calculate the score. the new linearizer greatly reduces the decoding time and improves the performance. we also addressed a problem in the previous system which restricted the output to projective trees by employing a transition-based reordering system, which is inspired by transition-based non-projective parsing @xcite . inspired by the large improvement from data augmentation in @xcite , we utilize extra training data for all languages. we parse unlabeled sentences from news and wikipedia text, then convert the parse trees into the shared task format, and train our system on the augmented data. finally, we apply simple ensembling by majority voting using ten models to improve the stability of the system and further push towards the performance upper bound. ## surface realization system our system largely follows our submission from the previous year @xcite , with some major change in the linearization module. it consists of up to five modules working in a pipeline fashion. the first module is linearization, which orders the input dependency tree (shallow or deep alike) into a sequence of lemmata. the second module is reordering, which adjusts the sequence from the previous step such that the new sequence could be non-projective. the third module is function word completion (for the deep track only), which generates function words as the dependents of the content words from the previous step. the fourth module is morphological inflection, which turns the sequence of lemmata into inflected word forms. finally, the fifth module is contraction, which contracts several words as one token, e.g. from de el to del in spanish. for a more detailed description, we refer the readers to @xcite . in this section, we mainly describe the changes we made since last year's shared task. figure 1 gives an example of the linearization process (including the reordering), which is our main contribution in this work. we first divide the input dependency tree into subtrees, and use the linearization decoder to order each subtree separately, then combine the ordered subtrees as a full projective sentence respecting the orders in the subtrees. finally, we use a reordering system to adjust the sentence, and produce potentially non-projective sentences. ## evaluation this year, we are the only team to submit predictions on all the tracks and languages. the average bleu scores in the shallow and deep tracks for last year and this year's closed and open subtracks are listed in in the closed subtrack, our system improves over the previous year by 3.59 points in t1 and 2.68 points in t2. the improvement mainly comes from a better token representation, a better linearizer, and the model ensembling. in the open subtrack, the augmented data brings an additional gain of 0.49 in t1 2019 2020a 2020b and 1.48 in t2. when we consider only the subset of t1 treebanks that also appear in t2 (t1 ∩ t2), the gain of the additional data is only 0.35. it is clear that the additional training data are more beneficial for the t2 track, since the function word completion module requires more training data. however, the improvement of augmented data in our system is much smaller compared with the adapt team which only submitted results for english. this suggests that there is still room for improvement for applying data augmentation in our system. in the human evaluation, our systems (with or without data augmentation) come in second place for english in terms of meaning similarity and readability, and first for russian and spanish. ## conclusion in this paper, we describe our submission to the sr'20 shared task, which is based on our system in sr'19, with some improvements in the feature representation and the linearization module. additionally, we augment the training data with parsed news and wikipedia texts. the results show that the additional data can improve the performance to some extent, especially in the deep track, but not as prominent as in @xcite . more careful analysis and design are thus required for the augmentation as well as the training process in order to fully capitalize on the vast amount of unlabeled data.