PINGAN _ AI at S em E val-2022 Task 9: Recipe knowledge enhanced model applied in Competence-based Multimodal Question Answering (2022)

Abstract: This paper describes our system used in the SemEval-2022 Task 09: R2VQ - Competence-based Multimodal Question Answering. We propose a knowledge-enhanced model for predicting answer in QA task, this model use BERT as the backbone. We adopted two knowledge-enhanced methods in this model: the knowledge auxiliary text method and the knowledge embedding method. We also design an answer extraction task pipeline, which contains an extraction-based model, an automatic keyword labeling module, and an answer generation module. Our system ranked 3rd in task 9 and achieved an exact match score of 78.21 and a word-level F1 score of 82.62.

## introduction in this paper, we discuss an approach to the question answering (qa) task for semeval-2022 task9 @xcite . this task is structured as question answering pairs, querying how well a system understands the semantics of recipes derived from a collection of english cooking recipes and videos, which involve rich semantic annotation and aligned text-video objects. in this task, a large proportion of answers can't be derived directly from the original recipe text and the information of these answers is hidden in annotated knowledge data. so we adopted two knowledge-enhanced methods in this model: the knowledge auxiliary text method and the knowledge embedding method. the knowledge auxiliary text method incorporates the hidden-roles knowledge and co-reference knowledge in generating auxiliary text. the knowledge embedding method encodes u-pos knowledge and entity knowledge into knowledge embedding. a key evaluation measure in this task is the exact match score. because the extraction-based model is more robust than the generative-based model, we design an answer extraction task pipeline. the pipeline contains an extraction-based model, an automatic keyword labeling module, an answer restructures module. in the training phase, we locate keywords of answers in recipe text and provide training data by labeling the keywords. in the prediction phase, we collect output keywords of the model and generate answer text by the keywords. our system ranked 3rd in task 9 and achieved an exact match score of 78.21 and a word-level f1 score of 82.62. we make our code publicly available on github foot_0 . ## related work question answering (qa) @xcite described the span extraction task in mrc (machine reading comprehension) and @xcite described the extraction-based question answering task. bert @xcite displayed a general extraction-based approach for the qa task. knowledge enhanced colake @xcite , ernie 3.0 @xcite and k-bert @xcite shown the method of constructing a new context by using knowledge information. know-bert @xcite used the knowledge embedding method and zhang et al. make some improvements. prompt in generating knowledge auxiliary text, we are inspired by prompt @xcite learning. we focus on prompt engineering in this paper. @xcite rewrite the context by replacing phrase. @xcite use a unidirectional lm to score the prompt patterns. @xcite use the t5 model to generate a template. ## system overview in this paper, we discuss an approach to the question answering (qa) task for semeval-2022 task9. we divided the samples into 18 categories after analyzing the patterns of qa pairs. we predict answers of 6 categories by rule-based method and 12 categories by deep-learning model. we chose bert @xcite as the backbone of our model and chose the answer extraction method for this qa task. knowledge information is significant in this task, so we adopted two knowledge-enhanced methods: the knowledge auxiliary text method and the knowledge embedding method. knowledge auxiliary text method generating expanded text that contains original text and knowledge auxiliary text. the knowledge auxiliary text incorporates the hidden roles and co-reference knowledge. knowledge embedding method encoding u-pos (universal-part-of-speech) knowledge and entity knowledge into knowledge embedding. ## results we ranked 3rd in the competition of semeval-2022 task 9. our model's word-level f1 score is 82.62 and the exact match score is 78.21. in our system, parts of questions got answers by rule module while others by deep-learning model module, which is mentioned in section 3. in this paper, we mainly focus on the deep-learning model module and word-level f1 score. the oblation experiment is about knowledge enhanced method. the baseline of our original model is 61 of squad word-level f1 score, without knowledge auxiliary text method and knowledge embedding method. its corresponding model is bert and its input is the original text of the recipes without any annotation of srl and crl. then we adopt the methods of knowledge enhancement including knowledge auxiliary text and knowledge embedding. we got a squad word-level f1 score of 88.78 when we used the knowledge auxiliary text method and an f1 score of 89.23 when using both of the two knowledge enhanced methods. the ablation experiment results are shown in the record of knowledge auxiliary text method is shown in the first line (so as the first section) of the second part of the best score for the second part is 89.23, its corresponding parameters are: adding knowledge embedding in the embedding layer, using both entity knowledge and upos knowledge. the third part of some conclusions could be drawn from ## conclusion we present two knowledge-enhanced methods in this model: the knowledge auxiliary text method and the knowledge embedding method. we design an answer extraction task pipeline to accommodate semeval-2022 task 09. future work will involve incorporating the video data and predicting the an-1277 swer of all pattern categories by model.