M ULTIMEDIA
G ENERATIVE
S CRIPT
L EARNING
FOR
T ASK P LANNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities in daily life.
An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps.
Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the ﬁrst benchmark containing 5,652 tasks and 79,089 steps with descriptive images.
This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps.
We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective.
We deﬁne metrics to evaluate both generation quality and inductive quality.
Experiment results demonstrate that our approach signiﬁcantly outperforms strong baselines 1 .
I NTRODUCTION
Robots rely on understanding the present real-world state and predicting the subsequent steps to better assist humans in daily stereotypical tasks such as meal preparation and gardening (Ruth
Anita Shirley et al., 2021; Liu et al., 2022).
As an example, Robohow (Beetz et al., 2016) uses articles from WikiHow 2 to assist robots in everyday tasks in human working and living environments.
The problem, however, is that not all daily tasks are well documented.
Thus, generating a sequence of steps that lead to a given goal (i.e., goal-oriented generative script learning)
(Lyu et al., 2021; Huang et al., 2022) has a fundamental importance in allowing robots to perform unseen tasks by understanding the patterns in previously observed similar tasks.
Despite this, previous goal-oriented generative script learning focuses solely on text (Lyu et al., 2021; Huang et al., 2022), which is commonly affected by reporting bias (Gordon & Van Durme, 2013), since important details may be omitted in the source text.
However, such information is often implicitly contained in images.
For example, in Figure 1, the image of Step 1 illustrates the items needed to make a bracelet , which is not mentioned in the text but helps predict the action of threading beads as a future step.
Existing multimedia script learning work seeks to bridge this cross-media gap, but the task settings are multi-choice selection
(Yang et al., 2021b) or ordering (Wu et al., 2022), which require candidate steps as input so it is not a practical setting for real-life robots.
To address these problems, we propose a new task, Multimedia Generative Script Learning (Figure 1), that requires systems to generate future steps based on the goal and previous steps with visual scenes depicting their states.
Speciﬁcally, given the goal and previous step history in the form of natural language sentences paired with descriptive images, the model should automatically generate the natural language instruction for the next step.
A good script has three hallmarks:
[Figure]
(1) Visual-State Trackable : it records the historical visual scenes and recognizes signiﬁcant changes that impact future steps.
We call it multimedia challenge .
To address this challenge, we focus on salient differences in visual scenes, and propose a novel selective multimedia encoder .
Rather than learning directly from the visual details of each object, we ﬁrst leverage an image captioner as an abstract summary of the image about global interactions among multiple objects.
We then introduce a selection gate to focus on the selected captions and steps closely related to the future step.
For example, the second caption “a child’s hand with a measuring tape on it” in Figure 1 can be ﬁltered out by the selection gate because it is not closely related to the future steps.
(2) Inductive : it transfers knowledge from a previously observed task to similar unseen tasks.
We call it induction challenge .
To induce procedural knowledge from previously observed tasks, we propose a retrieval augmented decoder to obtain relevant steps to guide the subsequent step generation.
As an example, the future step in Figure 1 closely resembles the scripts used in previous retrieved steps about threading items , thus enabling script knowledge to be transferred to an unseen task.
(3) Diverse : it displays distinct information at each step.
We call it diversity challenge .
Existing pre-trained transformer-based language models such as T5 (Raffel et al., 2020), BART (Lewis et al., 2020a), and GPT-2 (Radford et al., 2019) tend to generate repeated or highly similar future steps as shown in Figure 1.
Therefore, we introduce a novel diversity-oriented contrastive learning objective to control all subsequent steps to convey different information.
We treat all other steps in the given input and retrieved steps in other tasks similar to the given input as hard negatives.
To evaluate task performance, in addition to traditional generation-based metrics, we propose a new multimodal-retrieval based metric to capture cross-modal semantic similarity.
While the model design can be applied to any domain of interest, we experiment the model on two domains Gardening and Crafts , where task planning has not been well researched.
Automatic evaluation shows that our generated step predictions are close to the human written ground truth.
Human evaluation further conﬁrms that our diversity-oriented contrastive learning objective leads to diverse and correct steps.
The contributions are threefold: (1) We propose the ﬁrst multimedia goal-oriented generative script learning task to record historical steps in both text and images.
We also release a new benchmark from WikiHow, featuring 5,652 tasks and 79,089 steps with descriptive images.
(2) We propose a novel approach to produce visually trackable , inductive , and diverse scripts through a selective multimedia encoder, a retrieval augmented decoder, and a diversity-oriented contrastive learning objective.
(3) We propose a new multimodal-retrieval based metric to evaluate the cross-modal semantic similarity and the inductive ability by checking factual correctness.
M ETHOD 2. P ROBLEM F ORMULATION
We propose a new multimedia generative script learning task as follows.
Given an activity goal G , an optional subgoal M that speciﬁes the concrete needs, and the previous multimedia step history
S e l ec t i ve
M u l t i m e d i a
[Figure]
od
[Figure]
cut
[Figure]
[Figure] [Figure]
Caption 1 :
BLIP
Step 5 :
Caption 5 : a pair of scissors and a measuring tape.
BLIP
H i s t o r i ca ll y
R e l eva n t
S t e p
R e t r i eva l
[Figure]
[Figure]
R e t r i eva lA ug
[Figure]
n t
[Figure]
d
[Figure]
ec od
[Figure]
Historically
[Figure]
Entropy
[Figure]
Diversity
[Figure]
OrientedContrastive Loss
[Figure]
H n = { ( S 1 , V 1 ) , ..., ( S n , V n ) } with length h , a model is expected to predict the next possible step
S n +1 , where S i ∈{ 1 ,...,n +1 } are sequences of words, and V i ∈{ 1 ,...,n } are images.
2. M ODEL A RCHITECTURE
The overall framework is illustrated in Figure 2.
Given the activity goal G , optional subgoal M , and multimedia step history H n , we ﬁrst use an image captioner to map each input image into a precise caption and produce the caption-enhanced step history ˆ H n .
Then we propose a selective multimedia encoder by extending the BART encoder with a gated fusion layer to learn contextualized representations for the step history.
After that, a retrieval module retrieves historically relevant steps from the training corpus and encodes them with a retrieved step encoder .
Finally, we introduce a retrieval-augmented decoder , which enhances the BART decoder with a retrieval gate fusion layer to fuse the representations of the input step history and retrieved steps to generate the next step.
The entire model is trained by our proposed diversity-oriented contrastive loss and cross-entropy loss.
2. S ELECTIVE M ULTIMEDIA E NCODER
Image Encoding Compared to step descriptions which focus more on action description, captions provide more visual environment/object information such as beads in Step 1 from Figure 2.
Because we are more concerned with the overall semantics of the salient objects in the image rather than the details of every object, we adopt image captioners to encode visual features and track visual state changes.
For instance, while multiple objects are present in Step 3 in Figure 1, the ﬁnger object can be ignored in the third step as it does not represent the key information conveyed by the image.
Speciﬁcally, we use the state-of-the-art image captioner BLIP (Li et al., 2022), which is pretrained on a large-scale vision-and-language corpus with 129M images to generate a caption C i for each image
V i in the input step history H n .
After that, we obtain the caption-enhanced step history ˆ
H n = { ( S 1 , C 1 ) , ..., ( S n , C n ) } , where C i is the caption of the image V i in step i .
Selective Multimedia Encoding To help the encoder capture the activity goal and subgoal information, we concatenate goal G and optional subgoal M to serve as the ﬁrst sequence in the history X
=
[ G, M ] .
For the subsequent steps in the history, we concatenate each step and caption as X 2 i − 1 =
S i and X 2 i
= C i .
We prepend a learnable [ CLS ] token to the sequence as a contextualized vector to summarize the step history.
The entire text sequence is then represented as
X = { [ CLS ] , X 0 , X 1 , ..., X 2 n } .
We pass the text sequence X into a BART encoder to get the contextualized hidden representation H
= { h 0 , ..., h 2 nL X 2 n } =
Enc( X ) .
We denote
H X j = { h
j 1 , ..., h jL Xj } as the hidden states for sequence X j , where L X j is the length of sequence X j .
Since the input sequence contains steps or captions that are not directly relevant to the future step, we need to mask those sentences based on the step/caption representations.
For instance, in Figure 2, the step description for Step 1 is vague and needs to be masked.
We treat the representation of the [ CLS ] token, h 0 , as the contextualized representation of the entire step history and use it to compute a mask that ﬁlters out the irrelevant step/caption information.
Speciﬁcally, we use h 0 as query and
H X j as both the key and value to compute Multi-Headed Attention ( MultiHead ) (Vaswani et al., 2017) for each sequence hidden states H X j :
[Equation]
where ˆh X j is the weighted representation for text sequence X j .
Then, for each sequence X j , we can calculate the mask probability as:
[Equation]
where W α is a learnable parameter.
Similar to Sengupta et al.
(2021), we update the hidden states for each sequence X j as
[Equation]
where emb [ MASK ] is the embedding of the [ MASK ] token.
The ﬁnal hidden state sequences from the selective multimedia encoder are therefore ¯H
=
[ h 0 ; ¯H 1 ; ... ; ¯H 2 n ] .
2. S TEP R ETRIEVAL A UGMENTATION
Historically Relevant Step Retrieval
In addition to the caption-enhanced step history, ˆ H n , we retrieve historically relevant steps R n +
= { R 1 , ..., R k } from the training tasks, where k is the number of retrieved relevant steps.
We ﬁrst use SentenceBERT (Reimers & Gurevych, 2019) to encode all steps.
We then retrieve k steps from the training corpus, which have the top-k highest cosine similarity to the previous step S n 3 from the representation given by SentenceBERT.
Finally, we consider the immediate next step for each of those k steps as potential relevant steps R n +1 .
For instance, because Step 5 in Figure 2 is similar to pull the thread out in the training corpus, we choose its immediate next step thread the bobbin as a historically relevant step.
Retrieved Step Encoder For historically relevant steps R n +
= { R 1 , ..., R k } with length
L R 1 , ..., L R k , we apply the BART encoder to get hidden states H R =
{ H R 1 ; .... ; H R k } .
Similarly, we use h 0 in multimedia encoder as the query and H R i as both the key and value to compute a multi-headed attention for each sequence hidden states:
ˆ h R i =
MultiHead( h 0 , H R
i , H R i ) , where ˆh R i is the weighted representation for step sequence R i .
Similarly, we can calculate the mask probability as: β j = σ ( W β
[ h 0 ; ˆh R j ]) , where W β is a learnable parameter.
We then update the hidden states for each sequence R j as ¯H R i =
β j · emb [ MASK ] + (1 − β j ) H R i .
The ﬁnal hidden state sequences from historically relevant step encoder is ¯H R
= [ ¯H R 1 ; ... ; ¯H R k ] .
2. R ETRIEVAL -A UGMENTED D ECODER
In the decoder, we compute the probability P (cid:16) s q, ( n +1) | s <q, ( n +1) , ˆ H , G, M (cid:17) for the q -th token.
Our retrieval-augmented decoder is similar to Liu et al.
(2021), which aims to capture historically relevant steps related to the next step based on previous decoder hidden states.
Given z lq which is the hidden state of s q, ( n +1) in layer l , we ﬁrst use a multi-head cross-attention to fuse the hidden states from the retrieved steps ¯H R : z (cid:48) ql = MultiHead( z lq , ¯H R , ¯H R ) .
We also append a gating mechanism to control the knowledge from the retrieved steps and previous hidden states as:
[Equation]
where W γ is a learnable parameter and LN( ∗ ) is the layer norm function.
Finally, the fused hidden states in the top layer are used to compute the generation probability.
We supervise the next step generation using the standard cross-entropy loss as follows:
[Equation]
2. D IVERSITY -O RIENTED C ONTRASTIVE L EARNING
In the experiment, we observe that the model tends to keep generating similar future steps in a row given the beginning steps as input or just paraphrase the input steps.
Therefore, we propose a contrastive learning-based loss to encourage the model to return diverse step prediction results.
Negative Sampling Sequence-to-sequence generation models suffer from the “exposure bias” problem (Dhingra et al., 2016; An et al., 2022) because they are trained by teacher forcing .
Contrastive loss provides an additional sequence level loss which can help models increase the diversity of the output steps.
We adopt two types of negative sampling strategies to discourage the model from paraphrasing the previous step as the future step: self-negatives (Wang et al., 2022) where we consider the input steps as negative samples and retrieved-negatives where we consider the retrieved steps from training corpus which are similar to the input step as negative samples.
Diversity-Oriented Contrastive Loss Since the model needs to distinguish between the ground truth and those negative samples, we design a novel diversity-oriented contrastive loss.
Speciﬁcally, given an input sequence ˆ H , G, M , the ground truth next step S n +1 , and a set of K negative samples { S 1 n +1 , S 2 n +1 , ..., S Kn +1 } , we aim to maximize the probability of classifying the positive sample correctly with the InfoNCE loss (Oord et al., 2018):
[Equation]
(6) where ¯H
+ and ¯H − k are decoder hidden states from the positive and k -th negative samples, W y is a learnable parameter, τ is the temperature, and Avg( ∗ ) denotes the average pooling function.
2. T RAINING O BJECTIVE
We jointly optimize the cross-entropy loss and our proposed diversity-oriented contrastive loss: L =
L gen + λ L cl , where λ is a hyperparameter that controls the weight of the contrastive loss
[Table]
E VALUATION B ENCHMARK 3. D ATASET
[Table]
Using articles from Gardening and
Crafts categories as case studies
[Table]
we create a new dataset based on the English WikiHow dump (2021/05).
There are typically three levels of hierarchy in a WikiHow article: goals which describe the overall task, subgoals which represent intermediate process to accomplish a goal , and steps which are speciﬁc actions to complete a subgoal .
For each WikiHow article, we collect step-image pairs as well as their goals and methods 4 .
We split the whole dataset based on the task categories.
Therefore, the validation and test sets contain tasks that are not included in the training set.
Table 1 shows the detailed data statistics.
3. E VALUATION M ETRICS
Generation Quality Evaluation Following common practice in text generation, we ﬁrst evaluate our model with BLEU (Papineni et al., 2002), ROUGE (Lin, 2004)
[Table]
and METEOR (Denkowski &
Lavie, 2014) scores to examine the content overlap between generated steps and ground truth.
[Table] [Table]
Inductive Quality Evaluation In order to determine whether the inferred subsequent steps are factually correct, we further evaluate the models with BARTScore (Yuan et al., 2021) and the semantic similarity score (Thakur et al., 2021).
The semantic similarity score uses a cross-encoder pretrained on STSBenchmark (Cer et al., 2017) to calculate the semantic similarity between two sentences.
[Table]
In addition to evaluating whether the generated step matches the next step, we also check whether the generated step matches any subsequent step.
This enables the model to earn credit if it generates a step that appears in the future.
We propose a Multimodal-Retrieval based metric : for each generated step, we use it as a query to search all corresponding stepimage pairs under the same subgoal/goal from the testing set.
We then compute HIT@1 for results that fall into ground-truth future stepimage pairs.
Similar to Section 2.4, we use
SBERT (Reimers & Gurevych, 2019) to rank the most similar steps under the same subgoal to get Text@1 (T@1).
To compute Image@1 (I@1), we use CLIP (Radford et al., 2021) to rank the most similar images under the same subgoal.
If the top-1 retrieval results appear in the subsequent steps, we consider it as a HIT.
The retrieval-based metric captures normalized semantic similarity concerning all related steps under certain subgoals.
The CLIP-based retrieval metric also enables the evaluation of the cross-modality semantic similarity.
Additional details of our evaluation setup
E XPERIMENTS 4. B ASELINES
We ﬁrst compare our model with (1) state-of-the-art pretrained text-only generation models to examine the results without tracking visual states, including GPT-2 (Radford et al., 2019), T5 (Raffel et al., 2020), and BART (Lewis et al., 2020a).
We then compare our model with the (2) retrieval baselines including naive retrieval baseline which directly uses retrieved historically relevant sentences as discussed in Section 2.4, and retrieval BART which takes in the concatenation of the retrieved historically relevant sentences with the original text input.
We also include (3) multi-modal generation baselines that can take image embedding instead of captions as input, which is equivalent to CLIP-BART (Sung et al., 2022).
The CLIP-BART has a similar backbone as VL-BART (Cho et al., 2021) but instead replacing the Faster R-CNN (Ren et al., 2015) with ViT-B/32 CLIP encoder (Radford et al., 2021) which has better image-text alignment.
Additionally, we compare our model with a state-of-the-art script learning model: GPT2-SIF Sancheti & Rudinger (2022) ﬁnetuned on our dataset.
Finally, we include the variances of our model as (4) baselines for ablation .
The hyperparameters and training details are presented in the Appendix A, B.
4. A UTOMATIC E VALUATION
[Table]
As shown in Table 2 and
[Table]
our model outperforms baselines.
Since our task is open-ended and we are testing on unseen activities
[Table]
our generated sentences usually contain paraphrases.
Therefore, the BLEU scores, which rely on the exact word n -grams match (Goldberg, 2018), are not high.
In particular, because our ground truth only has an average length of 11 which contains less 4 -grams than the text in other tasks, our BLEU4 is lower than other text generation tasks.
The substantial gap between CLIP-BART and BART or BART with caption indicates that captions usually carry more straightforward information than images and the current multimodal encoders still cannot perfectly embed text and images into the same semantic space.
Meanwhile, the low performance of the retrieval baselines shows that simple retrieval methods are not sufﬁcient to predict accurate next steps.
[Table]
Among our model variants, adding selective encoding leads to a further performance increase, demonstrating that selective encoding helps the model concentrate on the content in step history that is most related to future steps.
The superior performance on BARTScore and semantic similarity of the retrieval-augmented model indicate the effectiveness of the guidance from historical relevant steps.
Our contrastive learning model achieves larger gains compared to baselines with respect to BLEU and METEOR, suggesting that our contrastive loss helps the model generate results similar to the ground truth.
Automatic Evaluation with Future Steps We evaluate whether the predicted step is related to any future steps.
Our contrastive learning model outperforms other ablations signiﬁcantly on text retrieval for the Gardening domain, as shown in Table 4.
These results imply that the contrastive learning objective encourages the model to generate more informative future steps.
The decrease in n-gram overlap between input step history and step predictions (Table 5) suggests that the contrastive learning objective also decreases the model’s paraphrasing tendency.
Interestingly, the performance decreases when adding the retrieval augmentation to the model because the retrieval model introduces additional information related to the step history, which makes the model generate results similar to previous steps (Table 5).
[Table]
Automatic Evaluation on Diversity
To evaluate the diversity between generated steps in the test sets
[Table]
we employ two diversity metrics: selfBLEU (Zhu et al., 2018)
(Table 6) and unique n -grams (Fedus et al., 2018)
(Table 7).
The self-BLEU evaluates whether a model produces similar n -grams in different samples by measuring the similarity between one sentence and the rest in the test set.
The retrieval model achieves the best results for the Gardening domain because it acquires additional knowledge from the retrieved steps and thus diversiﬁes the output.
The contrastive learning model achieves the best self-BLEU for 3,4 grams for the Crafts domain, implying our model’s effectiveness.
The unique n -grams calculate the percentage of distinct n -grams.
It considers the repetition of n-grams within a generated step and across samples.
The contrastive learning model achieves the highest distinct scores for 3,4 grams for both domains, which further indicates the effectiveness of our diversity-based contrastive loss in generating more diverse steps.
4. H UMAN E VALUATION
[Table]
Since script learning is an openended task that is inherently difﬁcult for automatic metrics to measure the correctness of generated scripts (Huang et al., 2022), we further conduct a human evaluation.
We hire four proﬁcient English speakers as human annotators to independently rank the generation results from 1 (best) to 5 (worst) for: (1) next step correctness which measures whether the generated results match the next step; (2) future steps correctness measuring whether the generated results match any of the future steps; (3) diversity which measures the diversity of generated results under the same subgoal; (4) executability which checks the generated results repeat or conﬂict with step history.
We randomly select ten subgoals, including 41 and 44 generated steps from the test set for Gardening and Crafts separately.
The human evaluation results 5 are shown in Table 8.
Our contrastive learning model achieves the best performance over all metrics on two datasets.
By adding each component of our model, we observe a consistent trend in correctness to ground truth.
However, we also observe that scores for selective encoding decrease because the output space with selective encoding is more constrained than the BART baseline, and the length of our generated sequence is not very long.
4. D ISCUSSIONS
Impact of Selective Multimedia
Encoder
The caption input helps the model understand the general step descriptions better.
For example, given the activity “cure azaleas of leaf gall” , the step text only shows a generic instruction: “rule out other diseases” .
However, the BLIP captioner generates “a green leaf with white dots on it” which helps the model generate “remove the leaf gall from the shrub” instead of “keep your shrub healthy” .
The selective gate successfully ﬁlters out unrelated steps which are not directly related to the current subgoal.
For example, in Figure 1, the selective gate successfully ignores Step 2 and generates “thread the beads onto the thread” .
Impact of Retrieval Augmentation
The retrieved steps provide relevant knowledge from similar tasks: given the subgoal “ﬁnding or growing roses” because the retrieved sentence mentioned “fertilizer” and “mulch” , the model successfully generates “fertilize your roses” .
Additionally, the model also beneﬁts from retrieval augmentation with an analogy, e.g., the model generates “know when to harvest” given the retrieved step “plant the bulbs when you get them” .
Impact of Contrastive Learning In addition to the improvement in diversity from the previous section, we observe that contrastive learning helps the model generate results closer to ground truth compared to other baselines.
For example, it generates “pick creeping charlie plants from the ground” , similar to ground truth “pick your creeping charlie leaves” .
The addition of contrastive learning also helps our model generates instructions with more details than other baselines by stating “place the plant in the hole and cover it with soil” instead of “place the plant in the hole” .
R ELATED W ORK
Previous script learning tasks fall into two forms: selective and generative.
The ﬁrst category focuses on modeling the script interactions given a list of candidates.
The selective script learning tasks include multi-choice goal step inference/ordering (Zhou et al., 2019; Zhang et al., 2020), script retrieval (Lyu et al., 2021;
Zhou et al., 2022), action anticipation (Damen et al., 2018; 2021), procedure segmentation Richard et al.
; Ghoddoosian et al.
; Zhou et al.
(2018), multi-choice visual goal-step inference (Yang et al., 2021b), multimedia procedure planning (Zhao et al., 2022), multimedia step ordering (Zellers et al., 2021; Wu et al., 2022), instructional video retrieval (Yang et al., 2021a), and step classiﬁcation (Lin et al., 2022).
Despite promising results, their performance heavily relies on the given candidates, making them difﬁcult to generalize for unseen activities.
The second category is text-based generative script learning (Tandon et al., 2020; Lyu et al., 2021; Huang et al., 2022; Sancheti & Rudinger, 2022).
However, this is the ﬁrst work to provide a multimedia goal-oriented generative script learning along with a new multimodal-retrieval based metric.
Different from Sener & Yao (2019), which uses a video as input to generate the next step, our generative script learning task uses step image-text pairs as input.
Unlike previous multimedia script learning frameworks with a multimedia encoder to capture visual and textual information, we use a captioner to convert images into captions summarizing the important objects in images.
To handle irrelevant sentences in the input, instead of using a token-level gating mechanism that only depends on the token itself (Sengupta et al., 2021), we introduce a sentence (step/caption) level gating mechanism whose gates depend on global context and weighted sentence representations.
Our work is also related to retrieval-augmented text generation models (Liu et al., 2021; Lewis et al., 2020b).
However, instead of retrieving knowledge from an external corpus, we use steps from similar tasks in training data to guide the generation process.
Moreover, we introduce a new contrastive learning loss to increase diversity.
Previous contrastive learning-based text generation methods usually use negative samples constructed by sequence manipulation (Cao & Wang, 2021;
Hu et al., 2022) or perturbation (Lee et al., 2021).
Inspired by Wang et al.
(2022) which uses selfnegatives for knowledge graph completion and the fact that the generation output tends to repeat the input, we extend self-negatives for sequence-to-sequence contrastive learning.
We also retrieve similar steps from the training set as additional hard negatives.
C ONCLUSION
We propose a novel Multimedia Generative Script Learning task with the ﬁrst benchmark featuring step and descriptive image pairs to generate subsequent steps given historical states in both text and vision modalities.
We build a new script learning framework consisting of a selective multimedia encoder, a retrieval-augmented decoder, and a diversity-oriented contrastive learning objective to generate the next steps.
We deﬁne a new multimodal-retrieval based metric which can be used for multimedia script learning tasks.
Automatic and human evaluation results demonstrate consistent performance improvements.
Additional external knowledge will be added in the future.
Limitations and broader impact are in the Appendix.
A H YPERPARAMETERS
Our model is built based on the Huggingface framework (Wolf et al., 2020) 6 .
We choose topretrieved historically relevant steps as input for our retrieval model.
For the gardening domain, we choose 5 negative samples for each step during contrastive learning.
Speciﬁcally, 4 self-negative samples are randomly chosen from the title, method, and step history input, including steps and captions.
The remaining 1 retrieved negative samples are randomly chosen from top- 20 most similar steps retrieved from the training set based on the last step.
For the crafts domain, we choose 5 selfnegative samples and 5 retrieved negative samples.
For the training objectives, we set τ as 1 for contrastive loss and λ as 0.5 based on validation performance.
We optimize our model by AdamW (Loshchilov & Hutter, 2019) with Cosine Annealing Warm Restarts schedule (Loshchilov & Hutter, 2017).
Our learning rate is 1 × 10 − 5 with (cid:15) = 1 × 10 − 6 for gardening domain and 2 × 10 − 5 with (cid:15) = 1 × 10 − 6 for crafts domain.
The number of warm-up steps is 2000 .
The batch size is set tofor both domains, and the maximum training epoch is set as 30 with 10 patience.
During decoding, we use beam-search to generate our results with a beam size of 5.
B T RAINING DETAILS
[Table]
We use BART-base from Huggingface Wolf et al.
(2020) for our method and baselines.
We normalize all our input sentences into lower case.
We add 5 special tokens for BART-base model including < title > , < method > , < step > , < caption > , < template > , and < cls > .
We prepend < title > to goal, < method > to subgoal, < step > to text step, < caption > to step caption, < template > to retrieved step, and ¡cls¿ to the beginning of step history input.
We truncate our step, caption, goal, and subgoal to 30 tokens and target step to 40 tokens.
We only choose the closest 10 step-caption pairs.
We use BLIP (Li et al., 2022) 7 pretrained with 129M images including including COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), Conceptual Captions (Sharma et al., 2018), Conceptual 12M (Changpinyo et al., 2021), SBU (Ordonez et al., 2011), and
LAION (Schuhmann et al., 2021).
We use all − mpnet − base − v2 from SentenceBert (Reimers & Gurevych, 2019) which performs best in semantic search to retrieve similar steps.
[Table]
We train out model with NVIDIA
A6000 GPUs with 48G memory with full precision.
We choose our best model based on validation score with
BLEU-4 (Papineni et al., 2002) and
ROUGE (Lin, 2004).
The best validation scores for our contrastive learning model are BLEU-4 is 2 .and
ROUGE-L is 15 .
24 for the gardening domain and BLEU-4 is 4 .and
ROUGE-L is 20 .
25 for the crafts domain.
The average training time for each model is 2 to 4 hours.
Table 9 shows the number of the parameters for each model.
C E VALUATION M ETRICS
We use BLEU (Papineni et al., 2002),
ROUGE (Lin, 2004), and METEOR (Denkowski & Lavie, 2014) from Microsoft COCO Caption Evaluation package 8 .
We use ofﬁcial implementation of BARTScore (Yuan et al., 2021) 9 .
We use cross − encoder / stsb − roberta − large which performs best on STSBenchmark (Cer et al., 2017) to compute semantic similarity score from Augmented SBERT (Thakur et al., 2021).
For multimodal retrieval based metric, we use the best sentence embedding model: from SentenceBert (Reimers & Gurevych, 2019) for text retrieval, and the best language-image pretraining model ViT − L / 14 @ 336px from CLIP (Radford et al., 2021) for image retrieval.
D P REDICTION FOR DIFFERENT HISTORY LENGTH
[Figure]
R T S c o r e
[Figure]
In Figure 3a and Figure 3b, we show the averaged BARTScore and semantic similarity scores of our contrastive learning models in the next step prediction task over different step history lengths.
In both ﬁgures, we observe that the results with eight step-caption pairs obtain the highest scores.
We analyze the reasons as follows.
For the instances that contain less than eight history steps, increasing the step history introduces more information than noise from the step text and corresponding captions.
However, as the step length grows even larger, the additional step-caption pairs introduce more noise than information relevant to the future step.
Empirically, the eight-step length achieves an optimal balance between noise and relevant information.
Another potential reason is related to the number of instances.
In table 10, we see an clear decline in the number of instances because of our dataset construction strategy.
Therefore, the model cannot generalize over long history input.
E D ATASET C OLLECTIONS
[Table]
We crawled the English WikiHow website from Jan 2021 to May 2021.
We extract all articles from the crawled website dump in the Gardening and Crafts categories.
Each article contains a unique activity.
We use
BeautifulSoup
(Richardson, 2007) to parse the article and obtain JSON ﬁles.
Each JSON ﬁle contains a gardening activity.
For each gardening activity, we remove those steps without paired images or steps whose images do not exist in the dump.
Then, we use a regular expression to remove the URLs in the steps.
We remove those steps that are too short (less than two words) or contain no values.
Finally, we remove the activity containing only one step in each subgoal.
F H UMAN EVALUATION DETAILS
We measures inter-annotator agreement with Krippendorff- α scores (Krippendorff, 2018).
The results are in Table 11.Table
12 shows the annotation examples.
Because we do not have a virtual environment to execute those steps, we do not have a good inter-annotator agreement on the executability
[Table] [Table]
water your roses adequately Step 4 :
Immediate Next Step 1: amend the soil with fertilizer ...
2: apply a thick layer of mulch to keep moisture in the soil.
Historical Relevant Step:
SentenceBERT
Training Set
SentenceBERT
Consine Similarity (a) Details for historical relevant step retrieval harvest roses finding or growing roses
Goal:
Subgoal: find areas where roses grow water your roses adequately identify roses correctly ...
two pink and red roses with green leaves a man standing in front of a bush of roses
Step 1 :
Caption 1 :
Step 2 :
Caption 2 :
Step 4 :
Caption 4 : a person using a garden hose to clean the ground
Historical Relevant Step Retrieval
Diversity-Oriented
Contrastive Loss fertilize your roses Next Step Prediction: 1: void watering the leaves and blooms.
2: create a watering regimen for your roses once they are established.
Retrieve-negatives:
Self-negatives: (b) Details for diversity-oriented contrastive loss
G A DDITIONAL MODEL ARCHITECTURE
Figure 4 shows additional details for our framework.
The immediate next step refers to the step right after the previous given steps.
H I MPACT OF H ISTORICAL R ELEVANT S TEPS
We analyze the relation between the quality of the retrieved historical relevant steps and the quality of the model predictions.
The quality of retrieved steps and model predictions are evaluated by the semantic similarity score, which measures the embedding space similarity between a given text and the ground-truth next step.
The Pearson’s correlation between the semantic scores of historical relevant steps and the semantic scores of model predictions is 0.39 with a p < 0 .
We also illustrate their relation in Figure 5.
The results suggest that the performance of our model is positively correlated with the relevance of the retrieved historical steps.
I L IMITATIONS I. L IMITATIONS OF D ATA C OLLECTION
Regarding data collection, we crawled the English WikiHow website from Jan 2021 to May 2021.
The number of available activities is limited by the data we crawled from WikiHow.
We currently only choose Gardening and Crafts categories as a case studies.
Because we focus on multimedia
[Figure]
S i m il a r i t
y b e t w ee n P r e d i c t i o n a n d G r o un d T r u t h image-step pairs, we remove steps that are not attached to any illustrative images. We also observe that a small portion of activities in the dataset do not follow chronological order.
We plan to expand our model to other categories written in other languages.
I. L IMITATIONS OF S YSTEM P ERFORMANCE
The model might generate incorrect nouns because of the occurrence of patterns (e.g. “refrigerate the slane for up to 1 year” instead of “refrigerate the purslane for up to 1 year” ).
In addition, our model sometimes tends to generate generic step description because of insufﬁcient input information, e.g., given the last step “lay the t-shirt out on a clean, ﬂat surface.” , the model generates “cut the shirt out” which is vague compared to ground thruth “carefully cut around the sleeve” .
Moreover, the pretrained model might focus more on language modeling instead of inherent logic: for the activity of “make paint can planters” , after “removing the label” from the paint can, the
BART+CAP generates “read the label” .
In addition, there is still a small chance that the model generates the same output for various similar inputs.
Because we rely on image captions and retrieval results for step prediction, the upper bound of our generation quality is limited by the performance of the image caption module and the sentence retrieval module.
Our framework also needs to improve on imbalanced topics in the dataset.
For example, the dataset contains more activities about tree for the gardening domain compared to other gardening-related plants.
Because our multimedia generative script learning is a new task, we cannot compare our model with other established state-of-the-art models.
Moreover, because WikiHow is a crowd-sourcing website, some everyday activities might have better human annotations than the remaining activities.
We plan to include a ﬁne-grained human written step prediction as an upper bound to address this issue.
I. L IMITATIONS OF E VALUATION
The automatic metrics we chose, including BLEU Papineni et al. (2002), ROUGE Lin (2004), METEOR Denkowski & Lavie (2014), BARTScore Yuan et al.
(2021), self-BLEU Zhu et al.
(2018), and unique n -grams Fedus et al.
(2018), might not be the best metrics to evaluate our results.
Some other metrics such as semantic similarity and multimodal-retrieval based metric are based on pretrained model including Augmented SBERT Thakur et al.
(2021), SentenceBert Reimers &
Gurevych (2019) and CLIP Radford et al. (2021).
Those metrics might not align with human judgment and might be biased towards pretrained datasets.
While we complement it with human evaluation, we only focus on relevance to ground truth and diversity.
Although we found ﬂuency is not an issue, it is likely we still need to cover all aspects of generation results.
J E THICS AND B ROADER I MPACT
The type of multimedia scripts learning framework we have designed in this paper is limited to
WikiHow articles, and they might not be applicable to other scenarios.
J. U SAGE R EQUIREMENT
Our multimedia script learning framework provides investigative leads for multimedia script prediction.
Therefore, it is not intended to be used for any activity related to any human subjects.
Instead, our system aims to generate step predictions with unseen activities similar to those in the training set.
Accordingly, domain experts might use this tool as an assistant to write more constructive instructional scripts that would be too time-consuming for a human to create from scratch.
Experts can also use this system to improve writing instruction by adding missing instructions.
However, our system does not perform fact-checking or incorporate any external knowledge, which we leave as future work.
The IRB board should ﬁrst approve human subjects who follow instructions generated by our system.
J. D ATA C OLLECTION
We collect data by crawling the raw ofﬁcial English WikiHow website, which is under AttributionNoncommercial-Share Alike 3.0 Creative Commons License 10 .
We ensure that our data collection procedure follows the Terms of Use located at https://www.wikihow.com/wikiHow:
Terms-of-Use .
Therefore our dataset can only be used for non-commercial purposes.
As mentioned in Section 4.3, we perform human evaluation.
All annotators involved in the human evaluation are voluntary participants and receive a fair wage.
K S AMPLE O UTPUT
[Table]
[Figure] [Figure]
[Figure] [Figure]