P ro MQA : Question Answering Dataset for Multimodal Procedural Activity Understanding (2025)

Abstract: Multimodal systems have great potential to assist humans in procedural activities, where people follow instructions to achieve their goals. Despite diverse application scenarios, systems are typically evaluated on traditional classification tasks, e.g., action recognition or temporal action localization. In this paper, we present a novel evaluation dataset, ProMQA, to measure the advancement of systems in application-oriented scenarios. ProMQA consists of 401 multimodal procedural QA pairs on user recording of procedural activities, i.e., cooking, coupled with their corresponding instruction. For QA annotation, we take a cost-effective human-LLM collaborative approach, where the existing annotation is augmented with LLM-generated QA pairs that are later verified by humans. We then provide the benchmark results to set the baseline performance on ProMQA. Our experiment reveals a significant gap between human performance and that of current systems, including competitive proprietary multimodal models. We hope our dataset sheds light on new aspects of models’ multimodal understanding capabilities.

## introduction procedures are human knowledge of experience that enables one to obtain an expected outcome without much trial and error. yet, following procedures (i.e., a set of instructions), itself requires skills such as, in cooking @xcite , assembly @xcite , or surgery @xcite , among others. in supporting such user activities, current evolving multimodal foundation models like @xcite and claude 3.5 sonnet @xcite have great potential by monitoring the situation through the perception of a user's wearable device. despite such diverse application scenarios, existing studies typically provide traditional, but less practical evaluation testbeds. to support an applicationoriented evaluation, we present a novel multimodal question-answering (qa) dataset for understanding procedural activity, produced by our cost-effective human-llm collaborative approach. when supporting procedural activities, an assistant should comprehend information from multiple sources: 1) actual process from their perception; 2) each step and the overall flow from instructions. for instance, in cooking, answering "what is the next step now?" requires an assistant to recognize which steps have been completed until "now" from its video recording and identify what else/next should be done from its recipe. assuming recipes are typically written in text, assistants receive multimodal information of how one did it as video and how one should do it as text. prior work has explored the task in a text-only, unimodal setting, where a user verbalizes all of their actions @xcite . however, it is not ideal in practice as a beginning cook might give misleading explanations that cannot be corrected by a system without raw information (video) about the actual process. figure 1 illustrates how one receives cooking support from a system in a reactive manner. tailoring toward such a practical scenario, we formulate our task as qa so that multimodal capabilities can be evaluated directly on the downstream task ( §2.1). in contrast, prior work traditionally tackles visual action understanding as action recognition and temporal action segmentation @xcite @xcite . we argue that these tasks are suboptimal to evaluate procedural activity assistants as they are subtasks of such procedural activity support. in this work, we present a novel dataset, promqa (procedural multimodal question answering), to evaluate models' capabilities of understanding procedural activities in multimodal settings ( §2). our work is motivated by the fact that a recipe 1) fill a mug with milk 2) microwave 2 mins 3) add 2 pieces of chocolate 4) add 1/5 tsp. cinnamon 5) mix the contents 6) heat the mug did i heat it for the correct amount of time? you microwaved it for 2 mins, instead of just 1 min. what should i do next? your next step is to add cinnamon. did i miss any step? yes, you missed adding sugar. [missing step] missed a recipe step (3-1): step performed before adding 1 tsp. sugar. [timing error] at a recipe step (2): 2 mins, instead of 1 min start (1) fill a mug with milk (2) microwave 1 min (3-1) add 1/5 tsp. cinnamon (4) mix the contents (5) heat the mug end (3-2) add 1 tsp. sugar (3-3) add 2 pieces of chocolate performing steps ## annotation: generate-then-verify in this work, we take a human-llm collaborative approach to annotate qa pairs: llms generate qa pairs with lower cost, then humans verify them to ensure quality. we hypothesize that llms can substantially generate valid questions when given sufficient information, inspired by synthetic data generation @xcite . specifically, we leverage existing annotations of action and error labels to form textual prompts. we note that, as our annotation framework is llm ag-nostic, it can plug and play llms, and importantly, it can benefit from ongoing llms' improvement. ## benchmarking on our promqa, we provide the baseline results of existing models to facilitate the development of a user-support system for procedural activities. considering that our task contains natural language answers, we employ an llm-based metric to evaluate the performance of the baselines. ## self-preference bias analysis prior studies report that llms may introduce selfpreference bias: "an llm favors its own outputs over texts from other llms and humans." @xcite this can be a critical issue when llms play multiple roles, as in our experiment, i.e., use llm-generated qas to evaluate llms with llm-as-a-judge. to justify the fairness of promqa as a benchmark dataset, we investigate: (1) generator-predictor self-preference bias, where the generator's outputs harbor styles or characteristics that make it easier for the model to answer, and (2) predictor-evaluator self-preference predictor generator evaluator gpt-4o gemini gpt-4o gemini gpt-4o 22.8 36.4 27.5 31.4 gemini 14.0 12.7 21.6 24.5 ## conclusion in this paper, we propose a human-llm collaborative approach, generate-then-verify, and develop a novel evaluation dataset, promqa, for multimodal procedural activity understanding. promqa consists of 401 qa pairs that require understanding both instructions and videos to derive answers, queried by questions. we also provide the baseline performance of existing models, showing that there is still a large gap in performance between humans and machines, even the competitive proprietary multimodal models. we believe that promqa can shed light on the new aspect of multimodal capabilities to facilitate model development.