Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
AffiliationAddress email
Abstract
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specificcaptioning, question answering, and future event prediction.
Existing few-shotvideo-language learners focus exclusively on the encoder, resulting in the absenceof a video-to-text decoder to handle generative tasks.
Video captioners have beenpretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting.
We propose VidIL , a few-shot Vid eo-language Learner via I mage and L anguagemodels, which demonstrates strong performance on few-shot video-to-text taskswithout the necessity of pretraining or finetuning on any video datasets.
We use theimage-language models to translate the video content into frame captions, object,attribute, and event phrases, and compose them into a temporal structure template.
We then instruct a language model, with a prompt containing a few in-contextexamples, to generate a target output from the composed content.
The flexibility ofprompting allows the model to capture any form of text input, such as automaticspeech recognition (ASR) transcripts.
Our experiments demonstrate the powerof language models in understanding videos on a wide variety of video-languagetasks, including video captioning, video question answering, video caption retrieval,and video future event prediction.
Especially, on video future event prediction,our few-shot model significantly outperforms state-of-the-art supervised modelstrained on large-scale video datasets.
Code and processed data will be publiclyavailable for research purposes.
Introduction
One major gap between artificial intelligence and human intelligence lies in their abilities to generalizeand perform well on new tasks with limited annotations.
Recent advances in large-scale pre-trainedgenerative language models [42, 6, 68, 24] have shown promising few-shot capabilities [69, 40, 60]in understanding natural language.
However, few-shot video-language understanding is still in itsinfancy.
A particular limitation of most recent video-language pretraining frameworks [28, 21, 58,65, 64, 25, 61] is that they are encoder-only, which means they do not have the ability to generatetext from videos for purposes such as captioning [59, 54], question answering [57], and futureprediction [23].
Meanwhile, unified video-language models [33, 46] that are capable of languagedecoding still rely heavily on finetuning using a large number of manually annotated video-textpairs, therefore cannot adapt quickly to unseen tasks.
Few-shot video-to-text decoding is challengingbecause the natural language supervision for learning video-language representation is typically basedon subtitles and automatic speech recognition (ASR) transcripts [36, 65], which differ significantlyfrom downstream tasks in terms of distribution and may have poor semantic alignment across vision and text modalities
We propose to address this problem by harnessing the few-shot power of large-scale language models,such as GPT-3 [6] and InstructGPT [37].
Our inspiration is derived from the fact that humans areexcellent visual storytellers [15], with the ability to piece together a coherent story from a fewisolated images.
To mimic this, we propose VidIL , a few-shot Vid eo-language Learner via I mageand
L anguage models, to use image models to provide information about the visual content in thevideo (as well as optionally use ASR to represent speech),
and then we instruct language models tointerpret a video-based summary, answer, or other target output for diverse video-language tasks.
[Figure]
Videos contain rich semantics and temporal content at multiple granularities.
Similar to staticimages, videos depict objects, attributes, andevents.
However, the sequence of frames further conveys object state changes, actions, andevents.
For example, in Figure 1, the frame captions describe static visual features such as “aperson holding a green object in hand” for thefirst frame.
In contrast, the video clip could becorrectly captioned as "a woman makes realistic looking leaves and flowers for a cake" , orrepresented as a collection of the objects andevents that occur at different timestamps in thevideo clip, such as cutting mat and flowered design .
Hence, to inform video-level descriptionand queries, we need to represent all of this information and its temporal ordering.
To address these requirements, we propose to decompose a video into three levels: the video output,frame captions, and visual tokens (including object, event, attribute).
One major benefit from thishierarchical video representation is that we can separate the visual and temporal dimensions of a video.
We leverage well-trained image-language foundational models at lower levels to collect salient visualfeatures from the sparsely sampled frames.
Specifically, we leverage pretrained image-languagecontrastive model CLIP [41] to perform visual tokenization, based on the similarity score betweenframes and tokens of objects, events and attributes.
The tokenization is done under the guidanceof semantics role labeling [14], which provides us with candidate events with involved objects andrelated attributes.
Next, in order to capture the overall semantics at the frame level, we employ thepretrained image captioner in image-language model BLIP [26] to obtain frame captions.
We theninstruct pretrained language models using in-context learning [37, 13, 48, 45] to interpret visualtokens and frame captions into the target text content.
In detail, we temporally order visual tokensand frame captions using specially designed prompts such as “First...
Then...
Finally” , to instruct thepretrained language model to track the changes of objects, events, attributes and frame semanticsalong the temporal dimension.
Without pretraining or finetuning on any video datasets, we show that our approach outperformsboth video-language and image-language state-of-the-art baselines on few-shot video captioning andquestion answering tasks.
Moreover, on video-language event prediction, our approach significantlyoutperform fully-supervised models while using only 10 labeled examples.
We further demonstratethat our generative model can benefit broader video-language understanding tasks, such as text-videoretrieval, via pseudo label generation.
Additionally, we show that our model is highly flexible inadding new modalities, such as ASR transcripts.
Related Work
2.1 Image-Language Models and Their Applications on Video-Language Tasks
Large-scale image-language pretraining models optimize image-text matching through contrastivelearning such as CLIP [41] or by learning image-text alignments [62, 27, 55, 63, 32, 49, 8, 29, 70,67, 17, 18, 16].
Recently, BLIP [26] proposes an image captioner pretrained with filtering in orderto achieve optimal image-text alignment and has shown promising performance on various imagelanguage tasks.
However, video-language pretraining [25, 33, 28, 35, 3, 1, 39] is still hindered by noisy and domain-specific video datasets [22, 36].
Naturally, researchers start to explore transferringthe rich knowledge from image models to videos.
Different from the traditional way of representingvideos by 3D dense features [12], recent work [21, 25] proves that sparse sampling is an effectiveway to represent videos, which facilitates applying pre-trained image-language models to videolanguage tasks [34, 11].
Specifically, the image-language model BLIP [26] sets new state-of-the-arton zero-shot retrieval-style video-language tasks, such as video retrieval and video question answering.
However, for generation-style tasks such as domain-specific video captioning, video-language model
UniVL [33] still leads the performance but highly rely on fine-tuning.
In this work, we extend the ideaof leveraging image-language models to a wide variety of video-to-text generation tasks.
We furtherconnect image-language models (ILM) with language models (LM) which empowers our model withstrong generalization ability.
We show that the knowledge from both image-language pretraining andlanguage-only pretraining can benefit video-language understanding in various aspects.
2.2 Unifying Multi-Modal Tasks with Language Models
Connecting different modalities with a unified representation has been paid much attention torecently.
Text-only generation models, such as T5 [43], has been extended to vision-language tasksby text generation conditioned on visual features [9, 50, 47, 72, 52].
In order to fully leveragethe generalization power from pretained language models, [60] represents images using text in afully symbolic way.
[31] includes more modalities such as video and audio, but requires annotatedvideo-text data to jointly training the language model with the video and audio tokenizer.
In this work,we propose a temporal-aware hierarchical representation for describing a video textually.
To ourknowledge, we are the first work to leverage prompting a frozen language model for tackling few-shotvideo-language tasks with a unified textual representation.
Concurrent work Socratic [66] uses azero-shot language-based world-state history to represent long videos with given time stamps, whileour model can quickly adapt to different video and text distributions with few examples.
Furthermore,we show that by injecting temporal markers to the prompt we can make a pre-trained language modelunderstand fine-grained temporal dynamics in video events.
Compared with the concurrent work
Flamingo [2], which requires dedicated vision-language post-pretraining, our framework does notrequire to pretrain on any videos.
Our framework is simple and highly modulated where all thecomponents are publicly available.
Additionally, our framework is more flexible on adding newmodalities, e.g., automatic speech recognition, without the need for complex redesigning.
Method
We propose a hierarchical video representation framework which decomposes a video into threelevels, i.e., visual token level , frame level and video level .
The motivation is to separate the spatialand temporal dimension of a video in order to leverage image-language and language-only foundationmodels, such as CLIP [41] and GPT-3 [6].
All three levels use a unified textual representation whichenables us to leverage the powerful few-shot ability from pretrained language models.
3.1 Frame Level: Image Captioning
Following [21] we first perform sparse sampling to obtain several video frames.
Unless otherwisespecified, we sample 4 frames for frame level and 8 frames for visual token level.
We then feed eachframe into a pre-trained image-language model to obtain frame level captions.
An example can befound in the blue part of Figure 2.
In our experiments, we use BLIP [26], a recent image-languageframework containing both image-grounded encoder and decoder, for generating frame captions.
Wefollow [26] to do both captioning and filtering on each frame.
However, as mentioned in Section 1,videos contain rich semantics and temporal contents at multiple granularities.
It is not enough togenerate video-level target text such as video captions solely based on frame captions.
Thus, wefurther perform visual tokenization for each frame to capture features at a finer granularity.
3.2 Visual Token Level: Structure-Aware Visual Tokenization
At this level, we aim to extract the textual representations of salient visual token types, such asobjects, events and attributes.
We found that pre-defined classes for classification, such as those in
ImageNet [10], are far from enough for covering the rich semantics in open-domain videos.
Thus,
[Figure] [Figure]
At visual token level, we extract salient objects, events, attributes for each sampled frame.
At frame level, we perform image captioning and filtering.
And at video level, we construct the video representation by aggregating the visual tokens, frame captions and other text modalities such as ASR , using a few-shot temporal-aware prompt.
We then feed the prompt to a pre-trained language model together with task-specific instructions to generate target text for a variety of video-language tasks.
instead of using classification-based methods for visual tokenization as in previous work [31, 60],we adopt a retrieval-based visual tokenization approach by leveraging pre-trained contrastive imagelanguage models.
Given a visual token vocabulary which contains all candidate object, event, andattribute text phrases, we compute the image embedding of a frame and the text embeddings of thecandidate visual tokens using a contrastive multi-modal encoder, CLIP [41].
We then select top 5visual tokens per frame based on the cosine similarity of the image and text embeddings.
An exampleof the extracted object tokens can be found in the green part of Figure 2.
Unlike in images where objects and attributes already cover most visual features, events are moreinformative in videos.
In order to discover events from video frames, we construct our own eventvocabulary by extracting event structures from Visual Genome [19] synsets 1 using Semantic Role
Labeling .
Specifically, we first select the phrases that contains at least one verb and one argumentas events .
Then we remove highly similar events based on their sentence similarity using SentenceBERT [44] embeddings.
For object vocabulary, we adopt OpenImage [20] full classes ( ∼ 20k), insteadof using the visually groundable subset ( ∼ 600) as in concurrent work [66].
We found that using largebut noisy vocabulary is more effective than using small but clean vocabulary in our retrieval-basedsetting with CLIP.
For attribute vocabulary, we adopt visual genome attribute synset.
In Section 4.6,we provide ablation study on the impact of different types of visual tokens.
The statistics of visualtoken vocabulary can be found in Appendix.
Level: Temporal-Aware Few-
Prompting
Once we obtain the textual representation from frame level and visual token level, the final step isto put the pieces together to generate a video level target text.
The goal is to build a model thatcan be quickly adapted to any video-to-text generation task with only a few examples.
To this
[Figure]
Figure end, we propose to leverage large-scale pre-trained language models, such as GPT-3 [6], with atemporal-aware few-shot prompt.
As shown in Figure 2, our framework can be readily applied tovarious video-to-text generation tasks, such as video captioning and video question answering, with ashared prompt template.
The proposed prompting strategy enables a language model to attend to thelower level visual information as well as taking into account the temporal ordering.
Here, we use the video captioning task depicted in Figure 2 to illustrate the details.
The few-shotprompt consists of three parts: instruction , few-shot context , and task query .
The instruction isa concise description of the generation task, e.g., "Generate a video caption based on theobjects, events, attributes and frame captions.
Example:" , which is proved to beeffective in zero-shot and few-shot settings [6, 56].
The few-shot context contains the selectedin-context examples as well as the test video instance.
Each video instance is represented by theaggregated visual tokens 2 , e.g., "Objects:
First, bath toy.
Then,..." , the frame captions, e.g., "Frame Captions:
First, a toddler playing in a bathtub filled withtoys.
Then,..." , and the ASR inputs if available, e.g., "Subtitle:<ASR Transcript>" .
Finally, the task query is a task-specific suffix indicating the target text format, e.g. "Video
Caption:" .
For in-context examples (omitted here for simplicity), the task query is followed byground truth annotation, while for the test instance, the generation starts at the end of the task query.
Formally, we denote the instruction line as t , few-shot context as c , the task query as q , and the targettext as y , where y =
( y 1 , y 2 , ..., y L ) .
The generation of the next target token y l can be modeled as:
[Equation]
183 In order to capture the temporal dynamics between frames and visual tokens, we further propose toinject temporal markers to the prompt.
As shown in the few-shot context in Figure 2, each visualtoken and frame caption is prefixed with a natural language phrase indicating its temporal ordering,e.g., "First," , "Then," , and "Finally," .
We found adding the temporal marker can make thelanguage model condition on not only the literal but also the temporal information of the context.
Weshow an example in Figure 3, where we compare our temporal-aware prompt with a static prompt onvideo captioning using InstructGPT.
Again, the in-context examples are omitted here, which can befound in Appendix.
In this example, the only difference between the two context is the ordering ofthe visual tokens and the frame captions.
For the context on the left, where "sun moving" appearsbefore "night sky" , we are expected to see a story talking about sunset , while for the context
onthe
right, we are expected to see sunrise .
We can see the static prompt generates captions aboutsunset for both contexts, while the temporal-aware prompt can capture the temporal ordering correctlyand generate sunrise for the context on the right.
Experiments
4.1 Experimental Setup
To comprehensively evaluate our model, we show results on four video-language understandingtasks in few-shot settings: video captioning, video question answering (QA), video-language event prediction, and text-video retrieval.
We compare our approach with state-of-the-art approaches onfive benchmarks, i.e, MSR-VTT [59], MSVD [7], VaTeX [54], YouCook2 [71], and VLEP [23].
Details about implementation and in-context example selection are included in the Appendix.
4.2 Few-shot Video Captioning
We report BLEU-4 [38], ROUGE-L [30], METEOR [5], and CIDEr [51] scores on three video captioning benchmarks covering both open-domain (MSR-VTT, VaTeX) and domain-specific (YouCook2)videos.
We compare with both state-of-the-art video captioner (UniVL [33]) and image captioner(BLIP [26]).
In order to implement the BLIP baseline for few-shot video captioning, we extend theapproach used for text-video retrieval evaluation in [26] to video-language training.
Specifically,we concatenate the visual features of sampled frames and then feed them into the image-groundedtext-encoder to compute the language modeling loss.
This is equivalent to stitching the sampledframes into a large image and then feeding it to BLIP for image captioning.
We found that this simpleapproach results in very strong baselines.
[Table] As shown in Table [Table] existing methods have strong bias on certain datasets.
For example, UniVLperforms well on YouCook2 but fails on MSR-VTT and VaTeX, while BLIP performs the opposite.
This is because UniVL is pretrained on HowTo100M which favors instructional videos, i.e.,
YouCook2, while BLIP is pre-training on image-caption pairs which favors description-style captions,i.e., MSR-VTT and VaTeX. On the contrary, our model performs competitively on both open-domainand instructional videos, and significantly outperforms the baselines in the average CIDEr scoreacross all three benchmarks.
This indicates that by leveraging language models, we can maintainstrong few-shot ability regardless of the video domain and or the target caption distribution.
As discussed in Section 1, video captions describes the content in various semantic levels.
The
N-gram based metric may not fairly reflect the models’ performance in capturing the video-captionalignment.
We further verifies this hypothesis in Section 4.5.
Thus, in addition to the automaticmetrics, we include qualitative examples illustrated in Figure 4.
More examples are in Appendix.
Additionally, for most existing methods and also concurrent work, e.g., Flamingo [2], adding a newmodality often requires a dedicated model redesign and or retraining.
However, the nature of ourframework, where we use a unified textual representation for each level, making it highly flexible forincorporating new modalities.
As shown in row 6 in Table, our model can effectively utilize extrainformation from ASR to obtain significantly better few-shot performance on certain datasets such as
YouCook2.
[Figure]
VTT
[Figure]
YouCook
[Figure]
VaTex
[Figure]
4.3 Few-shot Video Question Answering
We compare the test accuracy of our approach with few-shot pretrained BLIP, BLIP V QA [26], andconcurrent work Flamingo [2] on two video question answering benchmarks, MSR-VTT_QA and
MSVD_QA.
BLIP V QA represents finetuned BLIP on VQA [4] dataset, which is the previous SOTAon zero/few-shot video question answering.
In order to have fairer comparison with BLIP V QA , wereduce the shot number to 5 and report the average accuracy on three sets of randomly selected5-shot examples.
As shown in Table 2, our method outperforms previous SOTA by a large margin.
Comparing with concurrent work Flamingo, which is post-pretrained on a large number of video-textdata, our model is training-free and did not observe any video data.
However, with only imagelanguage and language-only knowledge, our 5-shot model is able to beat 8-shot Flamingo-3B andachieve on-par performance with 4-shot Flamingo-80B
[Table]
[Table]
[Table]
4.4 Few-shot Video-Language Event Prediction
In this section, we show that our model not only can answer questions about the video visual featuresbut also answering "What is more likely to happen next?".
Given a video with associated subtitletranscript as premise, the video-language event prediction (VLEP) task is to predict the most likelyfuture event.
The original VLEP [23] paper formulates the problem as a binary classification problemwhere the model will be chosen from two possible future event candidates.
Instead, we formulate thisproblem as another video-to-text generation problem to fit into our framework.
Figure 5 depicts anexample with the same format as in Figure 2.
Similar to the evaluation setting in QA, the generatedfree-form text will first be mapped to one of the two candidate answers using SentenceBert [44], andthen calculate the accuracy.
In Table 3, we report accuracy on the hidden test set of VLEP [23].
Toour surprise, our 10-shot model beats state-of-the-art fully-supervised baseline, i.e., MERLOT [65], by a large margin ( ∼ 4% ).
This is showing that our model has strong few-shot ability not only invideo-language understanding but also in prediction.
Since event prediction tasks rely heavily ontemporal ordering, we show that with the proposed temporal-aware prompting, language models canbe guided to capture temporal dynamics between historical and future events.
[Figure]
[Table]
4.5 Semi-supervised Text-Video Retrieval
In addition to video-to-text generation tasks, we show that a broader range of video-language taskscan benefit from our few-shot video captioner from a data perspective.
Here, we consider a lowbudget semi-supervised setting where we only have a few labeled video-caption pairs and a largeamount of unlabeled videos.
The idea is to leverage our video captioner to generate pseudo labels fortraining any given vision-language models.
As a case study, we evaluate on two text-video retrievalbenchmarks, i.e., MSR-VTT and
VaTeX.
We use greedy decoding to generate pseudo caption foreach video in the training set.
We then train an identical base model, i.e., BLIP, using different pseudolabeled data as well as ground truth annotations.
We report Recall @ 1 and 5 for both video-to-textand text-to-video retrieval.
Table 4 shows that through training on our pseudo labels, we can achievesignificant improvements compared with zero-shot BLIP.
We also show that the performance gainis not simply a result of training on more data, since finetuning on the pseudo labels generated byother baselines (UniVL, BLIP) is less effective and can even hurt the performance.
Furthermore, on
MSR-VTT Recall @
we can even achieve comparable performance against BLIP model finetunedon full ground truth annotations.
Another interesting observation is that, compared with the video captioning results in Table 1, wefound that the gain of our model over baselines on text-video retrieval is more visible than oncaptioning.
A key factor in performing well on text-video retrieval tasks is to learn a good video-textmulti-modal alignment.
This result shows that our pseudo labels capture richer video-text alignmentthat can benefit the retrieval-style downstream task.
The N-gram based generation metrics, e.g.,
BLEU, may not be able to fully reflect the alignment information, due to the variety of semanticlevels in video captions.
Furthermore, from a data perspective, our video captioner can be viewedas a data augmentation tool which is capable of generating or augmenting any open-domain videolanguage pretraining datasets with minimal human effort.
As a result, we can potentially improvingvideo-language pretraining by constructing a cleaner and more diverse video-text corpus.
4.6 Ablation Studies
We perform comprehensive ablation studies on our few-shot prompt including the impact of differentvideo representation, the number of shots and the in-context selection.
All the ablation results are [Table] [Table] evaluated on MSVD_QA validation set, we report the mean and standard deviation of each settingon three sets of randomly sampled shots.
For the cases with in-context example selection, wefurther select 5 examples as in-context examples from the sampled shots, while for the cases withoutin-context selection, all shots will be feed into the prompt.
In Table 5, we show adding visual tokensconsistently improves not only the model accuracy but also the model variance.
A lower standarddeviation indicates that the model is less sensitive to the few-shot sampling.
To further demonstrate the impact of the additional temporal dimension of videos, we performtwo ablation on the "Frame+Object+Event+Attribute" setting.
First, we reduce the number offrame captions and visual tokens to be one 3 for each video.
We found that the performance dropssignificantly compared with using the default four frames, which indicates the model’s ability toincorporate information from multiple timestamps.
Further, we found that fine-grained temporalmodeling is rarely required for performing well on current video-language benchmarks.
As shown inthe ablation result where we reverse the order of all visual tokens and frame captions, the performancedecreased only marginally, which indicates that current benchmarks may not be sufficient in reflectingthe benefits from better temporal ordering.
In Table 6, we first show that, with the same context length, namely, 5 in-context examples, in-contextexample selection significantly increases the performance as well as the robustness.
At 10 -shot, and20 -shot, directly fitting more shots into the prompt results in better performance.
In-context selectionachieves slightly lower performance but with significantly better efficiency due to shorter context.
Interestingly, at 30 -shot, in-context selection with 5 examples outperforms directly adding all 30shots into the prompt.
This is showing that in-context selection can help the model utilize a largernumber noisy video examples.
Nevertheless, we still observe that the benefit of adding more shotssaturated at around 20 to 30 shots, even if with in-context selection.
we view this as a remainingchallenging on how to make language models benefit from longer contexts.
Conclusions, Limitations and Future Work
This paper proposes VidIL, a few-shot Vid eo-language Learner via I mage and L anguage models.
It demonstrates the strong ability of large-scale language models on performing video-to-text taskswhen frame features are provided as unified text representations using image-language models.
Wepropose a temporal order aware prompt by decomposing videos into a hierarchical structure, which isable to plug in multiple levels of frame features, along with speech transcripts.
Without pretrainingon videos, our model outperforms vision-language models learned from large-scale video datasetson a variety of few-shot tasks, such as domain-specific captioning, question answering, and futureevent prediction.
One limitation of using unified textual representation is that we might lose low-levelvisual features which can be essential for some specific tasks, such as fine-grained spatial visualquestion answering.
We also observe that current video-language benchmarks rarely require explicittemporal tracking on the frames and visual tokens.
Future work will focus on leveraging large-scalelanguage models for learning script knowledge from long videos where temporal
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 and supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Section 4.6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] see Section 4.6.
(b) Did you mention the license of the assets?
[Yes] see the citations in Section 4.6.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] see the citations in Section 4.6.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] see the citations in Section 4.6.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]