Multi-dataset Training of Transformers for Robust Action Recognition
Anonymous Author(s)
AffiliationAddress
Abstract
We study on robust feature representations that can generalize on multiple datasetsfor action recognition using transformers.
Although we have witnessed greatprogress of action recognition in the past decade, it remains challenging yet
valuablehow to train a single model that can perform well across multiple datasets.
Herewe propose a novel multi-dataset training paradigm, MultiTrain , with the designof two new loss terms, namely informative loss and projection loss, aiming to learnrobust representations for action recognition.
We verify the effectiveness of ourmethod on five challenging datasets, Kinetics-400, Kinetics-700, Moments-in-Time,
Activitynet and Something-something-v2 datasets.
Extensive experimental resultsshow that our method can consistently improve the state-of-the-art performance.
We will release our code and models.
Introduction
Human vision can recognize video actions efficiently despite the variations of scenes and domains.
Convolutional neural networks (CNNs) [40, 41, 6, 36, 16] fully utilize the power of modern computational devices and employ spatial-temporal filters to recognize actions, which outperform traditionalmodels such as oriented filtering in space time (HOG3D) [25].
However, due to the high variations inspace-time, the state-of-the-art of action recognition is still far from being satisfactory, compared withthe success of 2D CNNs in image recognition [21].
Recently, vision transformers like ViT [12], MViT[14] that are based on the self-attention [44] mechanism are proposed to tackle the problems of imageand video recognition.
Instead of modeling pixels as CNNs, transformers apply attentions on top ofvisual tokens.
The inductive bias of translation invariance in CNNs makes it require less training datathan pure-attention-transformers in general.
However, transformer has the advantage that it can betterharness the parallel processing units of modern computing devices such as GPUs and TPUs, makingit more computationally efficient than CNNs.
We have seen a rapid growth in video datasets [23] inrecent years, which would make up for the shortcomings of data-hungry transformers.
The videodata has not only grown in quantity from hundreds to millions of videos [34] but also evolved fromsimple actions such as handshaking to complicated daily activities from the Kinetics-700 dataset [7].
Meanwhile, transformers combined with low-level convolutional operations have been proposed [14]to further improve the original design.
Due to the data-hungry nature of transformers, most transformer-based models for action recognitionrequires large-scale pre-training with image datasets such as ImageNet-21K [11] and JFT-3B [50] toachieve good performance.
This pre-training and fine-tuning training paradigm is time-consumingand it is not parameter-efficient, meaning that for each action dataset, a new model need to be trainedend-to-end.
Different from large image datasets such as ImageNet-21K that covers a wide rangeof object classes, currently the most diverse action dataset, Kinetics-700, only contains 700 classes.
Each action dataset may be also limited to a certain topic or camera views.
For example, Moments- in-Time [34] only contains short actions that happen in 3 seconds and Something-Something-v2 [20]focuses on close-up camera view of person-object interactions.
These dataset biases might hindersmodels trained on a single dataset to generalize and be used in a practical way.
These challenges inaction datasets make learning a general-purpose action model difficult.
An ideal model should beable to cover a wide range of action classes meanwhile keeping the computation cost low.
However,simply combining all these datasets to train a joint model does not lead to good performance [30].
In previous work [51], the authors have shown the benefit of training a joint model using multipleaction datasets but their method requires large-scale image datasets such as ImageNet-21K [11] and
JFT-3B [50], which is not available to the research community.
In this paper, we propose a general training paradigm for Multi -dataset Train ing of robust actionrecognition models, MultiTrain .
Our method is designed to learn robust and informative featurerepresentations in a principled way, using the informative loss for regularization.
We do not assumethe availability of large-scale image dataset pre-training (although one can certainly start with).
Sincethere are intrinsic relations between different classes across different action datasets (See Fig.
1 forexamples of similar classes from two datasets), we propose a projection loss to mine such relationssuch that the whole network is trained to avoid over-fitting to certain dataset biases.
Finally, allproposed loss terms are weighted using learned parameters, so no hyper-parameter tuning is needed.
Our empirical findings as shown in Table 1 indicate that our robust training method can consistentlyimprove model backbone performance across multiple datasets.
We show that our model can achievecompetitive results compared to state-of-the-art methods, even without large-scale image datasetpre-training, and with lower computational cost.
The main contributions of this paper are three-fold:
• To our knowledge, this is the first work to introduce informative representation regularization
into multi-dataset training for action recognition.
• We propose an effective approach to mine intrinsic class relations in multi-dataset trainingby introducing the projection loss.
• Our method requires negligible computation overhead during training and no additional computation during inference to the backbone network.
Extensive experiments on various
datasets suggest our method can consistently improve performance
Related Work
CNNs and Vision Transformers.
CNNs work as the standard backbones throughout computer visiontasks for image and video.
Various effective convolutional neural architectures have been raised toimprove the precision and efficiency ( e.g. , VGG [37], ResNet [21] and DenseNet [22]).
Although
CNNs are still the primary models for computer vision, the Vision Transformers have alreadyshown their enormous potential.
Vision Transformer (ViT [12]) directly applies the architecture of
Transformer on image classification and get encouraging performance.
ViT and its variants ( e.g. ,
ViViT [2], TimesFormer [4], MViT [14], Swin [32], MTV [46]) achieve outstanding results in bothimage and video processing in recent years.
Action Recognition/Classification.
The research of action recognition has advanced with both newdatasets and new models.
The modern benchmarks for action recognition is the Kinetics dataset[23].
The Kinetics dataset proposes a large benchmark with more categories and more videos ( e.g. ,400 categories 160,000 clips in [23] and 700 categories in [7]) as a harder benchmark compared toprevious datasets like UCF-101 [39].
The Moments-in-Time [34] (MiT) dataset provides a millionshort video clips that covers 305 action categories.
Note that it is impossible for Kinetics and MiTdatasets to cover all the possible actions in all possible scales.
For example, surveillance actions aremissing in the two datasets.
Many new approaches [42, 52, 31, 17, 47] have been carried out onthese datasets, of which the SlowFast network [17] and MViT [14] obtain promising performance.
We can see the trend of action recognition in the last two decades is to collect larger datasets ( e.g. ,
Kinetics) and build models with a larger capacity.
Multi-dataset Co-Training.
Previously, multi-dataset co-training has been explored in the imagedomain such as detection [54, 45] and segmentation [27].
Several works [35, 8, 38] were proposed tocombine multiple video datasets for training.
Larger datasets often deliver better results.
Combiningmultiple datasets to boost data size, and improve the final performance [19], and the simultaneous useof multiple datasets is also likely to alleviate the damaging impact of dataset bias.
OmniSource [13]utilizes web images as part of the training dataset to expand the diversity of the training data to reducedataset bias.
VATT [1] uses additional multi-modal data for self-supversied pretraining and finetuneson downstream datasets.
CoVeR [51] combines image and video training even during the finetuningstage and reports significant performance boost compared to single-dataset training.
PolyViT [30]further extends to training with image, video and audio datasets using different sampling procedures. In this paper, we propose a simple yet effective way (no multi-stage training, no complex dataset-wisesampling and hyper-parameter tuning) for multi-action-dataset training, without the use of any imageor additional data from other modality. Video Domain Generalization (VDG).
Our work is also related but different from video domaingeneralization [53].
The key distinction is that our goal is to train a single model on multiple relatedtasks (multiple action datasets) such that the model performs well on the same set of tasks, whereas
VDG aims to generalize a model to unseen out-of-distribution target domain [48, 8–10, 43]. Thesemodels still suffer from problem of parameter-inefficiency, meaning that separate models are neededfor different target datasets.
Method
Our method is built upon the backbone of the Improved Multi-scale Vision Transformers(MViTv2)
Note that our approach works with any action recognition backbones.
Givenvideos from multiple datasets during training, the model backbone takes the video frames and produces feature embeddings for each video.
The same number of Multi-layer Perceptron (MLP) as thedatasets are constructed as model heads to predict action classes for each dataset.
To facilitate robustcross-dataset training, we propose two loss terms, namely, the informative loss and projection loss .
The informative loss aims to maximize the embeddings’ representative power.
The projection loss,with the help of multiple cross-dataset projection layers, guides the model to learn intrinsic relationsbetween classes of different dataset, hence the model heads can be trained jointly.
See Fig.
1 for anoverview of our framework.
In this section, we first briefly describe the MViTv2 backbone design,and then present our proposed robust cross-dataset training paradigm.
3.1 The MViTv2 Backbone
Our model is based on the improved multi-scale vision transformers (MViTv2) [14, 29], which learnsa hierarchy from dense (in space) and simple (in channels) to coarse and complex features.
Theseries of work of vision transformers [12] (ViTs) follows the basic self-attention architecture [44]originally proposed for machine translation.
The key component of the MViTv1 model [14] is the
Multi Head Pooling Attention (MHPA), which pools the sequence of latent tensors to reduce thespatial or temporal dimension of the feature representations.
In MViTv2 [29], a residual connection in MHPA for the pooled query tensor and a decomposed relative position embedding 1 are added.
Inthis paper, we use 3D convolution as the pooling operation.
Please refer to supplemental material fora visualization of the MViTv2 block.
Each MViTv2 block consists of a multi-head pooling attentionlayer (MHPA) and a multi-layer perceptron (MLP), and the residual connections are built in eachlayer.
The feature of each MViTv2 block is computed by:
[Equation]
where X is the input tensor to each block.
Multiple MViTv2 blocks are grouped into stages to reducethe spatial dimension while increase the channel dimension.
The full backbone architecture is listedin supplementary material.
Classification head. For the action recognition problem, the model produces C-class classificationlogits by first averaging the feature tensor from the last stage along the spatial-temporal dimensions(we do not use the [CLASS] token in our transformer implementation), denoted as z ∈ R d .
Alinear classification layer is then applied on the averaged feature tensor to produce the final output,y =
W out z ∈ R C .
Multi-dataset training paradigm.
In general, to facilitate multi-dataset training of K datasets,the same number of classification heads are appended to the feature embeddings.
The k -th datasetclassification output is defined as Y k
= h k ( Z ; W k ) ∈ R B × C , where h k could be a linear layer or a
MLP and W k is the layer parameter.
3.2 MultiTrain : Robust Multi-dataset Training
Our training process fully leverages different action recognition datasets by enforcing an informativeloss to maximize the expressiveness of the feature embedding and a projection loss for each datasetthat mines the intrinsic relations between classes across other datasets.
We then use uncertainty toweight different loss terms without the need for any hyper-parameters.
Informative loss.
Inspired by the recently proposed VICReg [3] and Barlow Twins [49] methodfor self-supervised learning in image recognition, we propose to utilize an informative loss functionwith two terms, variance and covariance , to maximize the expressiveness of each variable of theembedding.
This loss is applied to each mini-batch, without the need for batch-wise nor featurewise normalization.
Given the feature embeddings of the mini-batch, Z ∈ R B × d , an expander(implemented as a two-layer MLP) maps the representations into an embedding space for theinformative loss to be computed, denoted as Z ′ ∈ R B × d .
The variance loss is computed using ahinge function and the standard deviation of each dimension of the embeddings by:
[Equation]
Where : is a tensor slicing operation that extracts all elements from a dimension, and ¯ Z ′ :
j is themean over the mini-batch for j -th dimension.
ϵ is a small scalar preventing numerical instabilities.
With random sampling videos across multiple datasets for each batch, this criterion encourages thevariance of each dimension in the embedding to be close to 1, preventing embedding collapse [49].
The covariance loss c ( Z ′ ) is defined as:
[Equation]
, where [Equation] [Equation] Inspired by VICReg [3] and Barlow Twins [49], we first compute the covariance matrix of the featureembeddings in the batch, C ( Z ′ ) , and then define the covariance term L c as the sum of the squaredoff-diagonal
Projection Loss.
In previous works [51, 30], the intrinsic relations between classes from acrossdifferent datasets have been mostly ignored during training.
We believe that samples in one dataset can be utilized to train the classification head of other datasets.
As shown in Fig.
1, the “Cleanand jerk” video sample from Kinetics can be considered as a positive sample for “Weightlifting” in
Moments-in-Time as well (but not vice versa).
Based on this intuition, we propose to add a directedprojection layer for each pair of datasets for the model to learn such intrinsic relations.
One can alsoinitialize the projection using prior knowledge but it is out-of-scope for this paper.
Given the outputfrom the k -th dataset classification output, the projected classification output is defined as:
[Equation]
4) where C k is the number of classes for the k -th dataset and W projik is the learned directed classprojection weights from i -th to k -th dataset.
In this paper we only consider a linear projectionfunction.
We then use the ground truth labels of the k -th dataset to compute standarad cross-entropyloss:
[Equation]
) where ˆ Y k,c is the ground truth label for the c -th class from the k -th dataset.
Training.
We jointly optimize the informative loss and the projection loss during multi-datasettraining.
To avoid tuning loss weights of different terms, we borrow the weighting scheme frommulti-task learning [24] and define the overall objective function as:
[Equation]
6) where σ is a vector of learnable parameters of size K (the number of datasets) for each projectionloss term.
This avoids the need to manually tune loss weights for different datasets.
Experiments
In this section, to demonstrate the efficacy of our training framework, we experiment on five action recognition datasets, including Kinetics-400 [23], Something-Something-v2 [20], Momentsin-Time [34], Activitynet [5] and Kinetics-700 [7].
The action recognition task is defined to be aclassification task given a trimmed video clip.
Unlike previous works [30, 51], we do not initialize ourmodel using ImageNet [11] since it consumes more computation.
Please refer to the supplementarymaterial for detailed comparison between train from scratch recipe and from ImageNet.
In theexperiments, we aim to showcase that our method can achieve significant performance improvementwith minimal computation overhead compared to baselins.
4.1 Experimental Setup
Datasets.
We evaluate our method on five datasets.
Kinetics-400 [23] (K400) consists of about 240Ktraining videos and 20K validation videos in 400 human action classes.
The videos are about 10seconds long.
Kinetics-700 [7] (K700) extends the action classes to 700 with 545K training and35K validation videos.
The Something-Something-v2 (SSv2) [20] dataset contains person-objectinteractions, which emphasizes temporal modeling.
SSv2 includes 168K videos for training and 24Kvideos for evaluation on 174 action classes.
The Moments-in-Time (MiT) dataset is one of the largestaction dataset with 727K training and 30k validation videos.
MiT videos are mostly short 3-secondclips.
The ActivityNet dataset [5] (ActNet) originally contains untrimmed videos with temporalannotations of 200 action classes.
We cut the annotated segments of the videos into 10-second longclips and split the dataset into 107K training and 16K testing.
Following previous works [17, 51], wefollow the standard dataset split and report top-1/top-5 classification accuracy on the test split for alldatasets.
We conduct two sets of experiments, namely K400 + MiT + SSv2 + ActNet and K700 + MiT + SSv2 + ActNet.
Implementation.
Our backbone model utilizes MViTv2 as described in Section 3.1.
Our modelsare trained from scratch with random initialization, without using any pre-training (same as in [17]and different from previous works [51, 30] that require large-scale image dataset pre-training like [Table] [Table] [Table]
[Table]
[Table] [Table] VATT [1]* [Table]
+
[Table]
+ Downstream
[Table]
[Table]
[Table]
[Table]
CoVER [51]
IN-21K + K + SSv2 +MiT 83.1 / - 41.3 / - 64.2 / - -
PolyViT [30]
IN-1K + K + MiT + [Audio] + [Image]
[Table]
[Table]
[Table]
[Table]
MultiTrain
K400 + SSv +MiT + ActNet
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
MultiTrain (312p)
K400 + SSv +MiT + ActNet 83.2 / 96.
ImageNet-21K [11] or JFT-3B [50]).
We follow standard dataset splits as previous works [29, 17, 46].
See more details in the supplementary material.
Baselines.
PolyViT [30] utilizes multi-task learning on image, video and audio datasets to improvevision transformer performance.
The backbone they used are based on ViT-ViViT [2].
Similarly,
VATT [1] utilizes additional multi-modal data for self-supversied pretraining and finetunes ondownstream datasets.
The backbone network is based on ViT [12].
CoVER [51] is a recentlyproposed co-training method that includes training with images and videos simultaneously.
Theirmodel backbone is based on TimeSFormer [4].
We also compare our method with other recent modelstrained using large-scale image datasets.
See Table 1 and Table 2 for the full list.
4.2 Main Results
We summarize our method performance in Table 1 and Table 2.
We train our model jointly on MiT,
SSv2, ActNet and two version of the Kinetics datasets.
[Table] We first compare our method with the original MViTv2 backbone in Table 1. “MViTv2 w/ abs. pos.
”means MViTv2 model with absolute positional embedding, which is taken from Table A.6 of MViTv2paper [29] and it is (almost) the same as our model implementation.
We can not achieve the sameaccuracy with the same recipe as MViTv2, which may be due to differences in the Kinetics dataset(missing some videos, etc.
See supplementary material for full dataset statistics).
PolyViT [30] istrained jointly with multiple image, audio and video datasets.
We list the larger ones.
We train ourbaseline model on the training set of each dataset to investigate the baseline performance.
As we see,after adding robust joint training proposed in this paper, performance on each dataset has increasedby 2.1%, 3.1%, 1.9% and 5.9% on K400, MiT, SSv2, ActivityNet, respectively in terms of top-1accuracy.
Note that our method achieves such improvement withtout large-scale image pre-trainingand additional inference computational cost.
We then compare our method with state-of-the-art on these datasets.
We train a higher resolutionmodel with larger spatial inputs (312p) and achieves better performance compared to recent multidataset training methods, CoVER [51] and PolyVit [30], on Kinetics-400, and significantly better on
MiT and SSv2, as shown in Table 1.
Note that our model does not use any image training datasets,and our model computation cost is only a fraction of the baselines.
We also show that our performanceboost does not come from the additional training dataset of ActivityNet in Table 3.
Our method also achieves competitive results compared to state-of-the-art models trained with largescale image dataset (ImageNet-21K [11]).
Compared to a recent method, MTV-B [46], our method isable to achieve significantly better top-1 accuracy across Kinetics-400, MiT, SSv2 by 0.8%, 1.4%,0.8%, respectively, at half of the computation cost and without large-scale pre-training.
Note thatour model is parameter-efficient, while multiple MTV-B models need to be trained and tested onthese datasets separately.
Our method can achieve better performance with a deeper base backbone orlarger resolution inputs but we have not tested due to limitation of computation resources.
We then compare our method on the Kinetics-700, MiT, SSv2 and ActivityNet training with baselines.
Our parameter-efficient model can achieve better performance than MTV-B [46] at one-fifth ofthe computation cost.
With a larger resolution model at 312p, we achieves significantly betterperformance than the baseline across Kinetics-400, MiT, SSv2 by 2.2%, 4.9%, 3.4%, respectively. [Table]
4.3 Ablation Experiments
[Table] In this section [Table] we perform ablation studies on the K400 set.
To understand how action models canbenefit from our training method, we explore the following questions (results are shown in Table 3):
Does our proposed robust loss help?
We compare our model training with vanilla multi-datasettraining, where multiple classification heads are attached to the same backbone and the model istrained simply with cross-entropy loss.
The vanilla model is trained from a K400 checkpoint as ours.
As shown in Table 3, we try training the vanilla model with both the same training schedule as oursand a 4x longer schedule.
As we see, there is a significant gap between the overall performance of thevanilla model and ours, validating the efficacy of our proposed method.
Also, longer training scheduledoes not lead to better performance on some datasets, including SSv2, suggesting vanilla multidataset training is unstable.
In terms of performance on ActivityNet, we observe that both trainingmethods achieve good results, which might be because ActivityNet classes are highly overlappedwith Kinetics-400 (65 out of 200).
How important is the informative loss?
We then experiment with removing the informativeloss (Section 3.2) during multi-dataset training.
It seems that the feature embedding of the modelcollapse and the model is not trained at all.
We further investigate why “w/o informative Loss”completely fails but “Vanilla” seems to work by running an experiment of "w/o informative Loss& w/o projection add", which means we remove the projected logits addition in Eq.
and directlycompute classification loss on the projected logits.
Therefore we can consider this run as addingadditional projection branches to the vanilla architecture.
The results are slightly better than "Vanilla"on K400 and much better on MiT/SSv2.
It indicates that adding projected logits to the original branchwithout informative loss would prevent the model from converging (the total loss does not go down).
How important is the projection loss?
We then experiment with removing the projection heads(Section 3.2) during multi-dataset training.
The model is trained with the original cross-entropy lossand the informative loss.
As shown in Table 3, the performance on MiT and SSv2 suffers by a largemargin, indicating that the projection design helps boost training by better utilizing multi-datasetinformation.
What does the cross-dataset projection layer learn?
We analyze the cross-dataset projectionweights of the K400/312p model and list top 5 concepts for each pair of datasets in Table 4.
We maketwo observations.
First, the top projections are visually similar actions, which confirms our intuitionthat there are intrinsic relations in the datasets that the model can mine to improve performance.
Forexample, “bending metal” in K400 and “bending” in MIT, “parkour” in K400 and “Capoeira” in
[Table] Activitynet. Interestingly, “Wiping something off of something” in SSv2 and “cleaning windows” in K400.
Second, the action with the same name may not have the highest weights.
In “mit to kinetics”,the “sneezing” action ranks 5th in the projection weights, suggesting that there might be discrepanciesof the same concept in different datasets.
These observations are interesting and one may comparethe learned weights with textual semantic relations (like those in ConceptNet).
We leave this to futurework.
Does the additional ActivityNet data help?
In previous methods like CoVER and PolyViT,the ActivityNet dataset has not been used.
In this experiment, we investigate the important ofthe ActivityNet dataset by removing it from the training set.
From Table 3, we can see that theperformance across all datasets drop by a small margin, indicating our superior results compared to
CoVER (see Table 1 and Table 2) come from the proposed robust training paradigm rather than theadditional data.
4.4 Discussion
By multi-dataset training transformers on various datasets, we obtain competitive results on multipleaction datasets, without large-scale image datasets pre-training.
Our method, MultiTrain , is parameterefficient and does not require hyper-parameter tuning.
Current limitations of our experiments are thatwe have not tried co-training with image datasets such as ImageNet-21K [11].
Hence we do not knowhow much performance gain that would entail.
We plan to explore this in future work.
In addition,we have not tried training larger model with FLOPs on par with state-of-the-art or other backbonearchitectures (e.g. CNNs) due to limitation of our computational resources.
Hence we are not surehow our algorithm would behave with these models.
We have not explicitly explored how temporalmodeling could benefit from multi-action-dataset training, which we leave for future work.
Althoughour model is trained on multiple datasets, potential dataset biases can still cause negative societalimpact in real-world deployment, as the datasets we have do not fully represent all aspects of humanactions.
Conclusion
In this paper, we present MultiTrain , a robust multi-dataset training approach that maximizesinformation content of representation and learns intrinsic relations between individual datasets.
Ourmethod can train parameter-efficient models that perform well across multiple datasets.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] We plan to
open-source our code upon paper acceptance.
We have provided detailed implementa-
tion instructions in the main text and in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 and the supplemental material.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[No] We follow standard data splits and evaluation protocol as
previous works.
See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material.4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] We used open-source (Apache-2.
licensed) MViTv2 code and standard public datasets
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Section 4.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]