The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation (2020)

Abstract: Does neural machine translation yield translations that are congenial with common sense? In this paper, we present a test suite to evaluate the commonsense reasoning capability of neural machine translation. The test suite consists of three test sets, covering lexical and contextless/contextual syntactic ambiguity that requires commonsense knowledge to resolve. We manually create 1,200 triples, each of which contain a source sentence and two contrastive translations, involving 7 different common sense types. Language models pretrained on large-scale corpora, such as BERT, GPT-2, achieve a commonsense reasoning accuracy of lower than 72% on target translations of this test suite. We conduct extensive experiments on the test suite to evaluate commonsense reasoning in neural machine translation and investigate factors that have impact on this capability. Our experiments and analyses demonstrate that neural machine translation performs poorly on commonsense reasoning of the three ambiguity types in terms of both reasoning accuracy ( 6 60.1%) and reasoning consistency (6 31%). We will release our test suite as a machine translation commonsense reasoning testbed to promote future work in this direction.

## introduction sixty years ago, the pioneering machine translation researcher and linguist bar-hillel published his well-known argument on the non-feasibility of general-purpose fully automatic high-quality machine translation (fahqt) due to the inevitable requirement of world knowledge to help machine translation to infer correct translations for ambiguous words or linguistic structures @xcite . the example that bar-hillel uses as an * equal contributions. modern natural language processing (nlp) has made tremendous progress, not only in building abundant resources to develop linguistic insights, but also in plenty of methodological practices. on the one hand, machine translation has been substantially advanced with large-scale parallel data and statistical models. recent results even suggest that the quality of machine-generated translations is approaching professional human translators @xcite . on the other hand, a wide variety of efforts have been conducted to either examine the commonsense reasoning capability of neural models in natural language understanding, establish commonsense reasoning challenges or enhance neural models in commonsense reasoning @xcite @xcite . comparing with bar-hillel's doubts and recent progress on machine translation and commonsense reasoning, it is natural for us to ask questions: do we solve the machine translation impasse related to commonsense reasoning? or particularly, are current neural machine translation models able to learn common sense? and if so, how much do they learn? does neural machine translation acquire sufficient commonsense knowledge and have strong ability in commonsense reasoning to generate human-level high-quality translations? methodological discussion on the feasibility of fahqt given the recent progress is far beyond the scope of this work. instead, we focus on empirically ana-(1) 这个 人 戴 的 表 走 了 3 分钟 。 the watch worn by this person went/walked for 3 minutes. the crocodile who ate the tourist/ate the tourist's crocodile. figure 1 : examples of the lexical ambiguity (1), contextless syntactic ambiguity (2) and contextual syntactic ambiguity (3). english translations in bold are correct while underlined translations are incorrect. in order to achieve this goal, we manually build a machine translation commonsense reasoning test suite on chinese-to-english translation with three types of commonsense-related ambiguities: lexical ambiguity, contextless and contextual syntactic ambiguity (see section 3.1 for more details). examples are shown in figure 1 . with this test suite, we thoroughly evaluate the commonsense reasoning ability of state-of-the-art neural machine translation models, e.g., lstm-and transformer-based nmt @xcite . we also conduct analyses on the commonsense reasoning capability according to commonsense knowledge types, sentence length and reasoning consistency and the size of training data. to the best of our knowledge, this is the first work to understand and measure the commonsense reasoning capability in neural machine translation. the contributions of this paper can be summarized as follows: ## related work we briefly review recent efforts related to commonsense reasoning in nlp. we refer readers to @xcite 's article for a thorough survey in this area. ## commonsense reasoning test suite for machine translation in this section, we discuss the design and construction of the test suite, including the rules and steps for building this test suite. ## test suite analysis we provide statistical analyses on the built test suite, which cover its size, distribution of knowledge types and the reasoning accuracy of pretrained language models on target translations of target translations of this test suite. ## experiments in this section, we conducted extensive experiments to evaluate the commonsense reasoning capability of state-of-the-art neural machine translation on the built test suite. ## conclusion in this paper, we have presented a test suite, including a lexical ambiguity test set and two syntactic ambiguity test sets, to evaluate the commonsense reasoning capability of state-of-the-art neural machine translation models. we elaborate the rules of building this test suite and conduct statistical analyses on it. our evaluation experiments and analyses on this test suite suggest that commonsense reasoning in modern machine translation models is still in its infant stage and that more efforts are to be expected to advance nmt in this direction.