Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge (2023)

Abstract: The ability of commonsense reasoning (CR) decides whether a neural machine translation (NMT) model can move beyond pattern recognition. Despite the rapid advancement of NMT and the use of pretraining to enhance NMT models, research on CR in NMT is still in its infancy, leaving much to be explored in terms of effectively training NMT models with high CR abilities and devising accurate automatic evaluation metrics. This paper presents a comprehensive study aimed at expanding the understanding of CR in NMT.For the training, we confirm the effectiveness of incorporating pretrained knowledge into NMT models and subsequently utilizing these models as robust testbeds for investigating CR in NMT. For the evaluation, we propose a novel entity-aware evaluation method that takes into account both the NMT candidate and important entities in the candidate, which is more aligned with human judgement. Based on the strong testbed and evaluation methods, we identify challenges in training NMT models with high CR abilities and suggest directions for further unlabeled data utilization and model design. We hope that our methods and findings will contribute to advancing the research of CR in NMT. Source data, code and scripts are freely available at https://github.com/YutongWang1216/CR-NMT .

## introduction commonsense reasoning (cr; @xcite is the ability to understand and navigate the world using basic knowledge and understanding that is shared by most people. in the context of neural machine translation (nmt; @xcite @xcite , cr is important because it allows the model to move beyond simply recognizing patterns in the data and instead make more informed, nuanced translations. recent studies have witnessed the the hunter is hunting . ## commonsense reasoning in pt-nmt this section aims to answer the question of whether the incorporation of pretrained knowledge can improve cr ability of nmt models. ## improving automatic evaluation of commonsense reasoning based on the strong testbed of pt-nmt, in this section, we re-examine the existing automatic evaluation methods, and further enhance the evaluation. ## entity-aware bertscore in this section, we will further investigate bertscore and propose a novel method for enhancing its correlation to human judgement by introducing the commonsense entity in cr of nmt. ## challenge in commonsense reasoning the above results sufficiently validate the positive impact of cr ability brought by pretraining. in this section, we prob the use of unlabeled data to improve nmt performance, with a focus on determining their impact on cr ability and identifying potential areas for improvement. ## conculsion this paper expands on the understanding of commonsense reasoning in nmt. we confirm the superior commonsense reasoning ability of pre-training enhanced nmt models through both automatic and human evaluations. we introduce a novel entityaware evaluation metric that takes into account the nmt candidates to alleviate the limitations of existing metrics. based on the enhanced evaluation metric, we identify the challenges and potential research directions for further enhancing the commonsense reasoning ability of nmt, including the further enhancement of the encoder module and utilization of target monolingual data.