Combining Tradition with Modernness: Exploring Event Representations in Vision-and-Language Models for Visual Goal-Step Inference (2023)

Abstract: Procedural knowledge understanding (PKU) underlies the ability to infer goal-step relations. The task of Visual Goal–Step Inference addresses this ability in the multimodal domain. It requires to identify images that represent the steps towards achieving a textually expressed goal. The best existing methods encode texts and images either with independent encoders, or with object-level multimodal encoders using blackbox transformers. This stands in contrast to early, linguistically inspired methods for event representations, which focus on capturing the most crucial information, namely actions and the participants, to learn stereotypical event sequences and hence procedural knowledge. In this work, we study various methods and their effects on PKU of injecting the early shallow event representations to nowadays multimodal deep learning-based models. We find that the early, linguistically inspired methods for representing event knowledge does contribute to understand procedures in combination with modern vision-and-language models. In the future, we are going to explore more complex structure of events and study how to exploit it on top of large language models.

## introduction procedural knowledge understanding (pku) implies reasoning about how to complete a task or achieve a goal @xcite . while previous works focus on plain texts @xcite @xcite @xcite , recent studies extend the task to the visual-linguistic domain. they ground procedural everyday tasks in the visual world, as a step towards situated procedural understanding in the real world. @xcite propose a novel pku task that utilizes both textual and visual information by selecting an image conditioned on a sentence which 1 the code is available at https://github.com/ st143575/exploring-event-in-vgsi . describes a high-level goal (illustrated in figure 1 , cf. section 2.2). their experimental results show that there is still a large gap to human performance on this task. while @xcite represent goal descriptions by their neural embeddings, earlier approaches to representing procedural knowledge or stereotypical event sequences (i.e., goals and steps; cf. scripts, @xcite , in contrast, focus on capturing the most essential information of events, namely the actions and their main participants @xcite . in this work, we explore different ways to inject these linguistically inspired representations to the recent powerful deep learning approaches, and study their contribution to multimodal pku. specifically, we investigate the relational event representation @xcite and the multi-argument event representation @xcite due to their simple but condensed structure holding the most crucial information such as the action and the main participants in the main clause. we also evaluate different approaches to encode and inject such event knowledge to the model used by @xcite , while also taking the contextual information into account. we conduct our experiments from three perspectives. first, we explore two approaches for event knowledge injection: (1) event replaces the sentence describing the event by the two aforementioned event representations; the main contributions of this paper are: (1) comparison between two approaches for linguistically-inspired event knowledge injection for the task of multimodal procedural knowledge learning; (2) comparison of three levels of linguistic information in the text embedding; (3) investigation of local and contextualised event embeddings; we find that appending the multi-argument event representation to the input sentence with @xmath0, and taking the average of the last 4 hidden layers of clip's text encoder is the best way to encode and inject event knowledge to a deep learning model. specifically, first encoding the full sentence and then extracting and averaging the word-level embeddings of the components of the event representation can use the contextual information in the sentence outside the event itself. ## vgsi: visual goal-step inference task task definition. @xcite define vgsi as a 4-way multiple choice problem. as shown in the example in figure 1 , given a textual goal g and four images i i , @xmath1, 2, 3, 4} representing four candidate steps, the task is to select the image that represents a correct step towards accomplishing g. in this paper, we additionally explore a stricter definition of vgsi, where the task is to select the respective correct image of all steps that are necessary to reach the goal g. ## results tables 3 and 5 give the most important results. the full results can be found in appendix 9.2. ## qualitative analysis we provide a qualitative analysis on the semantic gap between the ground-truth and the predicted images. figure 2 shows part of an example of the model's predictions for the goal how to stop twitching in your sleep? in this example, four out of ten steps are incorrectly predicted. for step 5, the textual input for training @xmath2. 
 exercise every day. the model selects image (e) which depicts a hand holding a heart. the model may associate "twitching" with the heart in the image, but fails to infer the rela-tion between "twitching" and the jogging people in the correct image (a). thus, the model may not learn causal relationships between the goal and the step image, such as "jogging can improve people's health condition and thus stop twitching in the sleep". for step 7 with the textual @xmath3. 
 eat plenty of magnesium., the model selects image (f) illustrating a person sitting at a laptop. possible reasons could be: (1) the action "eat" is usually performed by humans, but the correct image only describes some food, which the model misses to associate with "eat"; and (2) the phrase "plenty of magnesium" may mislead the model to select the wrong image with a laptop, which is associated more with magnesium than vegetables. hence, the model may only learn knowledge about simple, superficial properties of the objects in images, and may lack more complex commonsense knowledge about the relations between objects, such as "laptop is not edible" or "human cannot take magnesium by eating laptops". for step 8, the input @xmath4. 
 adjust what you consume before bed. the model selects the image showing a lady with a hat being pointed to by an arrow. this again indicated that the model's decision heavily relies on the verb. furthermore, it also suggests that the model has limited capability of identifying the affordances of the objects in the image and associating them with the goal. for step 10 with the @xmath5. 
 address potential vitamin deficiencies., the model again (a) step 5: 
 exercise every day. (b) step 7: 
 eat plenty of magnesium. (c) step 8: 
 adjust what you consume before bed. (d) step 10: 
 address potential vitamin deficiencies. (e) step 5: 
 be gentle. (f) step 7: 
 search online for job postings. (g) step 8: 
 put on a sun hat to protect your hair and keep you cool. (h) step 10: 
 start to learn about and change any patterns in your daily life that may act as triggers or contribute to your loved one's destructive behavior. seems to not capture causal relationships such as "vitamin deficiency can lead to twitching in sleep", but to base its inference on shallow object features such as "a man opens the door and wakes the sleeping woman up". in conclusion, our observations indicate that the model's decision highly depends on shallow features in the image and their alignment to the verbs and nouns in the text, while its effectiveness is impaired by its limited understanding of deeper semantics and causal relationships between the goal and the step images. ## conclusions in this paper, we investigate two linguisticallyinspired event knowledge injection approaches for the visual goal-step inference (vgsi) task. we experimentally compare three levels of linguistic information in the text embedding produced by stateof-the-art neural deep learning models. furthermore, we also compare event embeddings which encode only the information of the event components themselves with contextualised event embeddings which include information about the overall sentence syntactically not belonging to the arguments forming an event representation itself. last but not least, we assess different representations for the implicit subject of instructional sentences. we find that the early, linguistically inspired methods for representing event knowledge do contribute to understand procedures in combination with modern v&l models.