Semantic Structural Decomposition for Neural Machine Translation (2020)

Abstract: Building on recent advances in semantic parsing and text simplification, we investigate the use of semantic splitting of the source sentence as preprocessing for machine translation. We experiment with a Transformer model and evaluate using large-scale crowd-sourcing experiments. Results show a significant increase in fluency on long sentences on an English-to- French setting with a training corpus of 5M sentence pairs, while retaining comparable adequacy. We also perform a manual analysis which explores the tradeoff between adequacy and fluency in the case where all sentence lengths are considered.

## introduction in this paper, we apply a semantic decomposition approach for neural machine translation (nmt) and demonstrate that it can tackle two of the main limitations of state-of-the-art nmt. the first is the translation of long sentences, which is a recurrent issue arising in nmt evaluation @xcite @xcite @xcite . the second limitation is that current research in nmt mostly focuses on translating single sentences to single sentences, and is evaluated accordingly. however, @xcite showed that using several sentences to translate a source sentence is sometimes the preferable option. therefore, the simplicity of the output could be an important quality marker for translation. in our model, each source sentence is split (or decomposed) into semantic units, namely scenes, building on the direct semantic splitting algorithm (dss; @xcite ) that uses the universal conceptual cognitive annotation (ucca; @xcite scheme for semantic representation. scenes are then translated separately and concatenated for generating the final translation output, which may consist of several sentences. our main experiments use the state-of-the-art transformer model @xcite in english-to-french settings. we also include experiments with other mt architectures and training set sizes, and evaluate our results using the crowdsourcing protocol of @xcite . we obtain a significant increase in fluency on sentences longer than 30 words on the newstest2014 test corpus for english-to-french translation, with a training corpus of 5m sentence pairs, without degrading adequacy. considering all sentence lengths, we observe a tradeoff between fluency and adequacy. we explore it using a manual analysis, suggesting that the decrease in adequacy is partly due to the loss of cohesion resulting from the splitting ( §6). we then proceed to investigate the case of simulated low-resource settings as well as the effect of other sentence splitting methods, including splitand-rephrase models @xcite . the latter yield considerably lower scores than the use of simple semantic rules, supporting the case for corpusindependent simplification rules. ## semantic decomposition ucca (universal cognitive conceptual annotation; abend and rappoport, 2013) is a semantic annotation scheme rooted in typological and cognitive linguistic theory @xcite . it aims to represent the main semantic phenomena in the text, abstracting away from syntax. formally, ucca structures are directed acyclic graphs whose nodes (or units) correspond either to the leaves of the graph or to several elements viewed as a single entity according to some semantic or cognitive consideration. a scene is ucca's notion of an event or a frame, and is a unit that corresponds to a movement, an action or a state which persists in time. every scene contains one main relation, which can be either a process or a state. scenes may contain one or more participants, interpreted in a broad sense to include locations and destinations. for example, the sentence "john went home" has a single scene whose process is "went". the two participants are "john" and "home". scenes can provide additional information about an established entity (elaborator scenes), commonly participles or relative clauses. for example, "(child) who went home" is an elaborator scene in "the child who went home is john". a scene may also be a participant in another scene. for example, "john went home" in the sentence: "he said john went home". in other cases, scenes are annotated as parallel scenes (h), which are flat structures and may include a linker (l), as in: "when l [he arrives] h , [he will call them] h ". for ucca parsing, we use tupa, a transitionbased parser @xcite (specifically, the tupa bilst m model). we build on the dss rule-based semantic splitting method @xcite , and use rule #1 which targets parallel scenes. we further explore the use of the additional kinds of scenes in section 7 for less conservative sentence splitting. in rule #1, parallel scenes of a given sentence are extracted, split into different sentences and concatenated according to the order of appearance. more formally, given a decomposition of a sentence s into parallel scenes sc 1 , sc 2 , • • • sc n (indexed by the order of the first token), we obtain the following rule, where "|" is the sentence delimiter: as ucca allows argument sharing between scenes, the rule may duplicate the same sub-span of s across sentences. for example, the rule will convert "he came back home and played the piano" into "he came back home"|"he played the piano.". using ucca-based sentence splitting in our model is motivated by the corpus-based analysis presented in @xcite where it is shown that a scene in english is generally translated to a scene in french. ## experimental setup corpora we experiment on the full english-french training data provided in the wmt setting @xcite , which corresponds to about 39m sentence pairs after cleaning. 2 we refer to this setting as the fulltrain setting. we also experiment on the lesstrain setting where less training data is involved by removing the large un corpus and the 10 9 french-english corpus from the training data, obtaining a new training corpus of about 5m sentence pairs. the development set is newstest 2013, that consists of 3000 sentences. the test set is newstest2014, consisting of 3003 sentences. systems to investigate the use of semantic structural decomposition for nmt, we propose a twostep method. first, the original sentence is split into several sentences the dss rule (see § 3), implemented with the ucca software. 3 . then, each of the obtained sentences is translated separately by the opennmt-py implementation of the transformer @xcite . 4 the translated sentences are concatenated to form the final output. we name the combined system transformer sem-split and compare it to the transformer baseline, where no splitting is performed. the pipeline architecture is summarized in figure 1 . the transformer is trained for 200k training steps, both in the fulltrain and the lesstrain settings. the development data was used for selecting the model with the highest accuracy (where perplexity was used in cases of ties). the system was evaluated on the development data every 10k steps. for comparison, we also implement our system in the case where the transformer is replaced by another nmt system, namely a two-layers lstm model and the moses phrase-based machine translation system @xcite . the neural model, also implemented with opennmt-py, is trained and validated in the same way as the transformer. for moses, the default model is used in a single setting (lesstrain) with mgiza word alignment, 5 and kenlm language model @xcite using the monolingual data provided in wmt 2014, and mert tuning on the development set. here too we compare the combined systems to baseline systems which do not perform decomposition. ## results the results in both fulltrain and lesstrain settings are presented in in terms of fluency, lesstrain transformer sem-split ranks first in this setting and significantly outperforms the corresponding baseline system (52.5 vs. 42.5, @xmath0. 7 for moses too, the use of semantic sentence splitting increases fluency (40.2 vs. 38.1), but not significantly. on the other hand, where splitting is used as preprocessing, adequacy scores decrease. in particular, lesstrain transformer baseline significantly outperforms the semsplit counterpart (47.5 vs. 39.8, @xmath1. for sentences longer than 30, semsplit transformer in the lesstrain setting significantly outperforms the baseline in terms of fluency (52.1 vs. 39.6, @xmath2.02), with only a non-significant (small) degradation in adequacy (41.7 vs. 40.1, @xmath3.46). ## manual analysis to further zoom in on the obtained adequacy scores, we decompose adequacy into two dimensions: preservation of semantic content in the level of scenes and the cohesion of the text (i.e., whether the different scenes are cohesively linked together). to do so, we manually annotate a sample of 150 sentences from the original test set with a similar proportion of sentences in different length categories as the original corpus, and assess the semantic preservation at the scene-level for each of the extracted scenes, as well as the sentence-level cohesion (see appendix b for the protocol). for lesstrain, we find that 66.2% of the scenes are deemed equally preserved by the semsplit and baseline systems. on the other hand, 20.9% of the scenes are better preserved by the baseline and 10.7% of the scenes are better preserved by the semsplit system. averaging over scenes that belong to the same sentence, we find that 68% of the sentences are either better preserved by semsplit or equally preserved. regarding cohesion, sem-split and the baseline have a comparable cohesion for 59% of the sentences. the baseline has a better cohesion for 36% of the sentences, while it is improved by semsplit in 5% of the cases. the analysis suggests that cohesion has a central role in the decrease (and the non-increase for long sentences) of the adequacy scores. therefore the tradeoff between adequacy and fluency observed when all sentence lengths are considered can be explained by a tradeoff between the cohesion and structural simplicity aspects of translation quality. the different aspects of the translation quality are further illustrated in ## additional experiments we first explore the performance of the proposed system in low-resource machine translation, by following the approach of hoang et al. ( 2018 ) and randomly select 1m and 100k sentence pairs from the entire english-french training set, defining the 1mtrain and 100ktrain settings respectively. tuning and testing remain as before. the resulted raw scores for the 1mtrain and 100ktrain settings are presented in appendix d, second, to further explore the sentence splitting component, we replicate our model, separating both parallel and embedded scenes before the (1) input: hamas has defended its use of tunnels in the fight against israel, stating that the aim was to capture israeli soldiers so they could be exchanged for palestinian prisoners. output: le hamas a défendu son utilisation de tunnels dans la lutte contre israël, affirmant que l'objectif était de capturer des soldats israéliens afin qu'ils puissent être échangés contre des prisonniers palestiniens. literal translation: hamas has defended its use of tunnels in the fight against israel, stating that the aim was to capture israeli soldiers so they could be exchanged for palestinian prisoners. ## conclusion this work investigates the application of semantic structural decomposition for nmt, proposing an intermediate way between sentence segmentation used in mt and ts preprocessing, where each of the semantic components is separately translated. using the transformer and large-scale crowd-sourcing evaluation, we obtain an increase in fluency on long sentences on an english-to-french setting without significantly lowering adequacy. we further observe increased fluency when evaluating on all the sentences, albeit at the cost of adequacy. future work concerns the recombination of the output sentences, inserting the linkage between them, so as not to lose semantic content.