HIT & QMUL at S em E val-2022 Task 9: Label-Enclosed Generative Question Answering ( LEG - QA ) (2022)

Abstract: This paper presents the second place system for the R2VQ: competence-based multimodal question answering shared task. The purpose of this task is to involve semantic&cooking roles and text-images objects when querying how well a system understands the procedure of a recipe. This task is approached with text-to-text generative model based on transformer architecture. As a result, the model can well generalise to soft constrained and other competence-based question answering problem. We propose label enclosed input method which help the model achieve significant improvement from 65.34 (baseline) to 91.3. In addition to describing the submitted system, the impact of model architecture and label selection are investigated along with remarks regarding error analysis. Finally, future works are presented.

## introduction the objective of text-and-image multimodal question answering (qa) is to jointly leverage both textual and visual information to mutually inform each other for semantic reasoning @xcite . a semeval 2022 shared task titled competence-based multimodal question answering (r2vq) focuses on this task. in the r2vq task, participants were invited to develop qa models to resolve questions associated with procedural recipe instructions. the corpus provided for the task is made of recipes, which include rich semantic annotations and where textual instructions are aligned with images illustrating them. the authors proposed to address a set of 18 question families, for which participants could develop and evaluate their own proposed solutions. ## task and system description building on the transformer architecture @xcite , we use t5 an encoder-decoder model @xcite implemented using hugging face foot_0 . we chose t5 given its reasonably good general language learning abilities, and provided that the downstream task of r2vq covers a diverse set of task types that are also shared by t5. intuitively, a task-agnostic model like t5 would be expected to perform well on r2vq. we further adapt the t5 model to the task by altering the input to include semantic and cooking role labels enclosed in the textual recipe instructions. ## experiments since different formats of enclosing special tokens can result in considerably different scores, we compare a range of experiment settings to evaluate their performance. for more detail on the different enclosing methods tested, see we conduct a series of experiments to answer the following questions: q1: which way of enclosing special tokens performs best? is there any big difference between them, and any measurable explanation for the gap? q2: which label combos achieve the best accuracy? q3: how significant is the difference between results generated by size-equivalent models?