S em E val-2022 Task 9: R 2 VQ – Competence-based Multimodal Question Answering (2022)

Abstract: In this task, we identify a challenge that is reflective of linguistic and cognitive competencies that humans have when speaking and reasoning. Particularly, given the intuition that textual and visual information mutually inform each other for semantic reasoning, we formulate a Competence-based Question Answering challenge, designed to involve rich semantic annotation and aligned text-video objects. The task is to answer questions from a collection of cooking recipes and videos, where each question belongs to a “question family” reflecting a specific reasoning competence. The data and task result is publicly available.

## introduction one of the fundamental goals of artificial intelligence (ai) has been to create systems that interact with human users fluently and intelligently, by demonstrating inferencing and reasoning capabilities that would be expected of a human partner. this includes a growing interest in posing larger challenges to end-to-end systems employing architectures with deep neural networks (dnns) @xcite @xcite . here we argue that we should start focusing on linguistic competencies, and not just on question answering (qa) skills or "challenge checklisting". there are some moves in this direction already @xcite , but there is still no generally accepted distinction in current natural language processing (nlp) between challenge-based tasks and competence-based performance @xcite . analogous to human cognitive competencies, there is both a methodological and modeling advantage to focusing a system's performance on competence-based learning rather than a narrowly defined task or challenge checklist. first we define competence-based knowledge, and then the questions that can be generated from such knowledge. while @xcite 's distinction between competence and performance has long been debated in linguistics, the term competence-based has been applied to a number of different concepts in both the science of learning and educational communities @xcite @xcite @xcite . the common core to both is a concept capturing a coherent set of abilities that an individual has in a specific domain @xcite . here we focus on lexical competence as deployed in both single and multiple sentence composition @xcite @xcite . a competence-based question will query competence-based knowledge structures. for this task, lexical competence will involve the following: it is clearly the case that these two phenomena require non-extractive qa capabilities of some sort. we describe our dataset, recipe-to-video questions (r2vq), and summarize the procedures implemented by task participants for answering such questions in the remainder of the paper. ## related work nlp challenges have helped drive progress in the field recently. these challenges in part have been framed as specific tasks, and advances are largely driven by leaderboards on benchmark datasets or model comparison on individual datasets. common benchmarks such as glue @xcite and superglue @xcite have been used widely. they contain several language understanding tasks such as winograd natural language inference (wnli) @xcite as an inference task, and winograd schema challenge (wsc) @xcite as a coreference resolution task. a survey @xcite showed the recent trend to measure various machine reasoning capabilities using different designs of qa tasks. while all the tasks aim to advance the research towards corresponding nlp challenges, whether these reflect human competencies remains a question, especially in recent years with the success of transformers @xcite @xcite . many top-ranked nlp models that have shown better performance than humans on benchmarks may have come from overfitting to the dataset rather than addressing the challenge @xcite . current pre-training paradigms may also tune models towards capturing merely statistical patterns, so datasets should be designed to align the model's ability with human expectations (linzen, 2020). @xcite found that most of the questions from common qa and reading comprehension datasets can be correctly answered by models without complex reasoning. recent work has been trying to identify and evaluate the tasks that are reflective of human linguistic and reasoning competencies. for example, @xcite proposed a semantic parsing dataset that evaluates the human-like compositional generalization of models. ribeiro et al. (2020) designed three test types that can be used to test various linguistic capabilities of nlp models. more closely related to our work, qa-srl @xcite use predicate-argument structure to represent qa pairs. synqg (dhole and manning, 2021) and roleq @xcite try to incorporate existing semantic annotations to generate comprehension questions. ## task description we formulate the task as competence-based qa, designed to involve rich semantic annotation and aligned text-video objects. the goal of this task is to answer questions from a collection of cooking recipes and images. each question belongs to a "question family" that characterizes a specific reasoning competence to be tested. these competencies include abilities such as spatial and temporal reasoning, semantic role assignment, and object recipe title: appelkoek passage: peel and cut apples into eighths (wedges). sift together flour, baking powder and salt with 4 tablespoons of the sugar. cut in butter. combine egg and milk and add to flour mixture. turn batter into greased 8 inch square cake pan. press apple wedges partly into batter. combine remaining 2 tbsp sugar and cinnamon. sprinkle over apple. bake at 425 degf for 25 to 30 minutes. implicit how do you cut apples into wedges? -by using a knife elision what should be sprinkled over apple wedges? -cinnamon sugar loc. change where was the batter when you press apple wedges? -in the pan obj. lifespan what's in the appelkoek? -apples srl-time for how long should you bake appelkoek? -20 to 35 minutes srl-value how do you bake appelkoek? -bake at 425 degf the question families are defined as follows: the textual component of our dataset consists of a collection of english language recipes sourced from two open-source recipe wikis, recipe fan-dom foot_2 and foodista foot_3 , and is labeled according to three distinct annotation layers: (i) cooking role labeling (crl), (ii) semantic role labeling (srl), and (iii) aligned key frames image triples taken from creative commons cooking videos downloaded from youtube. compared to text of news or narratives, procedural text such as recipes and user manuals tend to be task-oriented, and the main content is split into steps that describe small goals to accomplish the final task. we believe such texts are a good fit for our task, as it involves the understanding of how to reach the goal locally for each step, as well as how each step contributes to the final task globally. further, the step-wise progression inherent in the goal-oriented narrative contributes both an interpretative dynamics as well as contextualized elision of arguments. ## participation we discuss the baseline system and the systems from participants in this section. ## discussion in this paper we have described the new task of competence-based multimodal question answering. in this task, we extended the traditional question answering by providing text-visual aligned data as the context, and asking questions that reflects reasoning competences over the question context. to create the dataset for our task, we proposed and applied a rich annotation of semantic role labels, cooking role labels and aligned video key frames to a set of cooking recipes. a criticism of the approach we adopted to create annotated dataset is that the video key frames are not well aligned with the text, thus making it difficult to include those into the modeling training. although with the full awareness of this, video annotation and alignment remains a very difficult task. copyright issues also make it challenging for us to get enough video sources to work with. future work to improve the key frame annotation may include utilizing entity recognition so that more ac-curate alignment to text can be made. we will also consider reusing the key frames and adding static images to represent similar events from different recipes to increase the coverage of annotation. another criticism of the data is the semantic ambiguity and loose definition of certain questions. for example, the same how-to question can have multiple reasonable answers, but only one is considered as the gold answer. although this is the semantic ambiguity as it is, we intend to improve it by replacing the question phrase "how to ..." to more specific phrase like "what tool ..." based on the answer it is inquiring about. an analysis of the systems that participated in our task showed the major improvement over the evaluation scores is achieved by making the hidden information appear on the surface. in general, two approaches are proven to be useful for this purpose by the participating systems. one is to train an endto-end system to generate text that contains crl-srl annotation, so that the hidden information is expressed explicitly in the generated text. then an extractive qa system can be adopted to identify text spans as answers. the second approach involves rules and heuristics to identify question intents, and get auxiliary knowledge. intent identification can help classify questions into different categories. each question category is associated with a rather fixed set of answer templates and possible entity types to be filled in. auxiliary knowledge is generated by associating specific entities with their co-referred mentions or result ingredients (e.g. "small balls" to "flour mixture"). the analysis of the results from participating systems also reveals some interesting characteristics about the dataset and is useful for future task design. despite the error rate of the top-performing systems such as srpol and itnlp&qmul is only 8%, the cardinality questions and how-to questions solely contribute the majority of the errors. as it is mentioned above, the innate ambiguity of how-to questions makes it difficult for both humans and systems to get a single correct answer. the poor performance on cardinality questions shows that the "counting reasoning" remains a big challenge to current transformer-based systems. in the r2vq dataset specifically, the mentions of the entity involved a cardinality question can scatter over the whole recipe, which requires a larger context to answer such questions. due to nature of "constant ingredient transformation" in cooking recipes, the mentions of the same entity could vary in our definition. for example, in the appelkoek recipe ( the human benchmark created by the srpol team provides useful insights on our future qa task design. they asked six linguists to answer 2, 000 questions selected randomly from the validation set. by examining the manual annotation on the questions, they found that although 73% of the annotated qa pairs have the same meaning as the gold answers, the em score is quite low. this reveals the fact that traditional qa metrics that focus on string match might be too strict in our task. for example, from the analysis of the human benchmark, for the question what's in the mixture?, both the gold answer the egg and mixture and the human answer the butter, sugar, tangerine zest, vanilla, baking powder, salt and egg can be considered correct. other metrics like bertscore @xcite might be a good compliment to account for the syntactic and semantic variance between the model inference and the gold answer. ## conclusion in this paper we described semeval-2022 task 9: r2vq -competence-based multimodal question answering. the task is to answer questions from a collection of cooking recipes and videos, where each question belongs to a "question family" reflecting a specific reasoning competence. we developed a new dataset of cooking recipes with rich annotation for cooking roles, semantic roles and aligned video key frames. we collected 8 result submissions and analyzed the participating systems by highlighting and summarizing their findings to help future research pertaining the topic of our task.