S amsung Research P oland ( SRPOL ) at S em E val-2022 Task 9: Hybrid Question Answering Using Semantic Roles (2022)

Abstract: In this work we present an overview of our winning system for the R2VQ - Competence-based Multimodal Question Answering task, with the final exact match score of 92.53%.The task is structured as question-answer pairs, querying how well a system is capable of competence-based comprehension of recipes. We propose a hybrid of a rule-based system, Question Answering Transformer, and a neural classifier for N/A answers recognition. The rule-based system focuses on intent identification, data extraction and response generation.

## introduction the goal of the task 1 was to develop a system applying existing knowledge to new situations, demonstrating a kind of understanding of a real-world domain. the competition presents a qa 2 challenge requiring linguistic and cognitive competencies that humans have while speaking and reasoning @xcite . the task dataset contains questions belonging to "question families" based on clevr @xcite , reflecting specific reasoning competences. these families were explicitly marked as 19 categories, the last one having no answer (n/a), but direct reference to these categories was prohibited by the task requirements. the cooking recipes included in the dataset were provided with exceptionally extensive annotations containing semantic information. the authors applied crl and span-based srl using verbatlas 1 https://competitions.codalab.org/ competitions/34056 (access apr 28th, 2022). the dataset was split into training (26,526), validation (3,829) and test set (3,442 questions). at the competition evaluation stage, the answers to the latter were not revealed, but the annotations were retained. our source code is available on github foot_0 . ## related work in the recent years, deep learning systems trained on large datasets began to outperform humans and other algorithms in the whole qa discipline. challenges presented in works such as squad @xcite , ms marco @xcite , coqa @xcite , multilingual mlqa @xcite and others popularized various machine learning models for extractive qa. meanwhile, visual and multimodal qa contests started to appear, e.g. vqa @xcite or audio-visual scene-aware dialog @xcite . they require understanding of images, natural language and their mutual relations to produce answers. one should not overlook qa systems applying srl annotations used in advanced answer and question generation, such as @xcite . qa systems were proposed for open domains as well as specific ones, including medicine, education, tourism, weather forecasting, etc. one of the most popular yet challenging topics for qa is cooking. the system in @xcite required preparing a cooking-related ontology, categorizing questions and extracting potential answers. @xcite focused on a knowledge graph used to answer a range of questions related to healthy diet. furthermore, food recognition could be perceived as a part or a preliminary step for cooking qa. @xcite and @xcite seek for deep learning classifiers to properly identify food from real images. ## results our end-to-end hybrid system reached em scores between 80% and 100% per question category and the official result overall amounted to 92.53%. details can be found in our pipeline starts with the semantic-based system. if no answer is returned, the rc system is used if its confidence threshold exceeds 98%. this fallback mechanism produced any significant improvement only for the location (srl) category. we additionally consider the n/a classifier result: if it exceeds the 99% certainty threshold, the n/a answer is returned. this operation enhances the results for the event ordering category. adding the dnn systems in such a way leads to a 0.145 pp result increase. in the post-evaluation phase we made further improvements, mainly in the rule-based system, and we ultimately reached the 92.969% em result. human annotators reached notably low em score of 52%. it is mainly due to the fact that the exact match metric leaves no room for human creativity. a manual review of the semantic validity of the responses gave us 73% alignment with the gold answers. this is discussed further in section 4.2. manual analysis revealed that only some elements of the images associated with the recipes relate directly to recipe content. this was also mentioned by task organizers. only 62 from 500 analyzed pictures were considered helpful in answering questions by our human evaluators. they also reported that they were often assigned to a different recipe step. for these reasons we further disregarded the images and focused only on the textual part of the data. ## conclusion and future work our main contribution is the hybrid system for the cooking-related qa. while we are satisfied with the result, the ~7% error rate still leaves some room for improvement. the most challenging task for our system was the correct intent identification. this is visible in the fairly low results in the method category. it may relate to four different intents, and we did not always distinguish them properly. other problematic aspects were counting actions and objects and generating answers that contain all required items in the right order. these issues solely contribute to as much as 5.5 out of 7 pp constituting the whole the error rate. the obvious question left unanswered is the possibility of srl/crl annotation automation, also for other competence domains. this is a missing component in a full end-to-end application of our solution.