BUT - FIT at S em E val-2020 Task 4: Multilingual Commonsense (2020)

Abstract: We participated in all three subtasks. In subtasks A and B, our submissions are based on pretrained language representation models (namely ALBERT) and data augmentation. We experimented with solving the task for another language, Czech, by means of multilingual models and machine translated dataset, or translated model inputs. We show that with a strong machine translation system, our system can be used in another language with a small accuracy loss. In subtask C, our submission, which is based on pretrained sequence-to-sequence model (BART), ranked 1st in BLEU score ranking, however, we show that the correlation between BLEU and human evaluation, in which our submission ended up 4th, is low. We analyse the metrics used in the evaluation and we propose an additional score based on model from subtask B, which correlates well with our manual ranking, as well as reranking method based on the same principle. We performed an error and dataset analysis for all subtasks and we present our findings.

## introduction commonsense knowledge is collection of facts, that an average human being is expected to know and be able to reason with these facts. it includes the ability to asses physical qualities, behavior and purpose of inanimate objects, animals and people. in other words, it is an implicit knowledge, something we feel we do not have to explain. since this topic is among the most important challenges in artificial intelligence, there is a large number of publications regarding commonsense -see for example books by @xcite or @xcite . the goal of semeval 2020 task 4 -commonsense validation and explanation @xcite ) was to asses how current nlp approaches compare with humans in terms of such knowledge. three subtasks were devised to determine the ability of the participating systems to reason within commonsense knowledge in english. in the first subtask, the goal was to differentiate between statements that make sense and those that do not. in the second and third subtask, only nonsensical statements were considered, and the assignment was to explain why are the statements against common sense. in subtask number two, the systems were presented with three possible choices for the explanation, while in the third subtask, the systems were expected to generate the explanation from scratch. a more detailed description of the tasks is provided in the task paper. our submissions are based on recent pretrained models, namely albert @xcite and bart @xcite . we experimented with data augmentation using round trip translations of the datasets and also with machine translation of either the queries, or the whole dataset into another language (czech). our code is available in a github repository at https://github.com/cepin19/semeval2020_ task4 . ## results and analysis in this section, the results of models described earlier are presented, along with data and error analysis. ## related work classical approach to machine commonsense reasoning is to use knowledge graphs and symbol manipulation to infer an answer to a commonsense question, or to find out if a statement agrees with common sense. one of the largest general purpose knowledge bases is conceptnet by @xcite . with the rise of pretrained language representation models (lrm), it is being questioned whether knowledge bases and exact inference are necessary for commonsense understanding, or the lrms already have commonsense capabilities embedded into them. a work by @xcite combines both approaches, integrating relevant parts of a knowledge graph into an input of bert-like model. authors of currently the largest pretrained language model, gpt-3 (brown et al., 2020), provide a comprehensive analysis of the reasoning capabilities of the model on various tasks. however, whether the pretrained models really perform reasoning, or rather they only match patterns present in the training data, remains an open question. winograd schema challenge @xcite and winogrande @xcite are benchmarks for natural language understanding and commonsense reasoning, considered an alternative to the turing test. the task is to disambiguate a pronoun, which has an antecedent in previous statement, based on common sense. a relaxed version of the challenge is a part of glue @xcite natural language understanding dataset. commonsenseqa @xcite is a multiple-choice dataset for commonsense question answering, consisting of questions that are difficult for current natural language processing models and require prior knowledge. ## conclusions we have demonstrated that pretrained language representation models are able to obtain high accuracy on datasets for subtasks a and b. we experimented with solving subtask a in the czech language by means of machine translating data and training multilingual models. we have shown that using an english model and simply translating the input sentences from czech to english yields superior performance compared to multilingual models, even though this approach is prone to error propagation. in subtask c, we used pretrained sequence-to-sequence and language models. we demonstrated that bleu score does not correlate well with human evaluation in this subtask, and we propose a perplexity metric based on the subtask b dataset, which agrees more with our manual ranking. finally, we show that reranking hypotheses of subtask c systems by our subtask b model improves the manual ranking score.