SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Anonymous Author(s)
AffiliationAddress email
Abstract
Recently, significant progress has been made in masked image modeling to catchup to masked language modeling.
However, unlike words in NLP, the lackof semantic decomposition of images still makes masked autoencoding (MAE)different between vision and language.
In this paper, we explore a potential visualanalogue of words, i.e., semantic parts, and we integrate semantic information intothe training process of MAE by proposing a Semantic-Guided Masking strategy.
Compared to widely adopted random masking, our masking strategy can graduallyguide the network to learn various information, i.e., from intra-part patterns tointer-part relations.
In particular, we achieve this in two steps. 1) Semantic partlearning: we design a self-supervised part learning method to obtain semanticparts by leveraging and refining the multi-head attention of a ViT-based encoder.2) Semantic-guided MAE (SemMAE) training: we design a masking strategythat varies from masking a portion of patches in each part to masking a portionof (whole) parts in an image.
Extensive experiments on various vision tasksshow that SemMAE can learn better image representation by integrating semanticinformation.
In particular, SemMAE achieves 84.5% fine-tuning accuracy on
ImageNet-1k, which outperforms the vanilla MAE by 1.4%.
In the semanticsegmentation and fine-grained recognition tasks, SemMAE also brings significantimprovements and yields the state-of-the-art performance.
Introduction
Together with transformers, masked language modeling (MLM) has revolutionized the field ofself-supervised learning (SSL) in natural language processing (NLP), which enables training ofgeneralizable NLP models containing over one hundred billion parameters (5).
The concept of MLMis quite intuitive, i.e., a portion of the data is removed and a model is trained to predict the removedcontent.
Recently, significant progress has been made in masked image modeling to catch up tomasked language modeling, where the masking mechanism is a key factor.
Context encoder (28), aninpainting-based masked image modeling (MIM) pioneer, proposes to use a random and fix-shaped mask; SiT (2) and BEiT (3) use random “blockwise” masks, where patches in the local neighbourhood are masked together (also called GMML: group mask model learning); MAE (19) randomly masks out 75% patches of an image.
Actually, masking mechanisms define the specific pretext task, i e., what kind of information is to be exploited and what kind of information is to be predicted.
Thus of tokens to mask and proposes an attention-guided mask strategy to ) takes one step further to “learn to mask” by
Although promising performance has been achieved, there is still a large gap for masked autoencoding(MAE) between vision and language due to different signal natures.
A sentence can be semanticallydecomposed into words, while the semantic decomposition of an image is not trivial to be obtained.
To find a visual analogue of words, we investigate part-based image representation.
Specifically, thereal world is composed of objects, which consist of different parts.
Therefore, part-based imagerepresentation is a fundamental image representation method that fits the inherent properties ofobjects (8; 15; 16; 18; 20).
For example, part-based Pictorial Stracture (16) dominated the imagerepresentation field for several years in the early days of computer vision, and Deformable Part Model(DPM) (15) was also a milestone in image recognition and detection.
Moreover, GLOM (20) arguesthat the hierarchical representation with five levels (i.e., the lowest level, sub-part level, part level,object level, and scene level) would be a powerful image representation method in the future.
To thisend, we argue that semantic parts would be a potential visual analogue of words .
With such visualanalogue, more controllable hints can be built up to guide the learning of MAE, thus high-level visualrepresentations can be well learned.
In this paper, we first propose a self-supervised semantic part learning method to obtain semantic partsfor each image.
Our insight is that the spatial information to reconstruct an image is highly correlatedto the position of semantic parts.
In particular, our part learning model consists of a ViT-based encodertogether with an attention module that generates a class token and multiple attention maps, and a
StyleGAN-based decoder that reconstructs the original image.
The attention maps are optimized toprovide spatial information, and the class token is integrated into the decoder via AdaIN to providetexture information.
We find that the optimized attention maps can indicate part positions, and weconduct an argmax operation to obtain part segmentation maps.
After that, we study how semanticparts can facilitate the learning of MAE.
We design a masking strategy that varies from masking aportion of patches in each part to masking a portion of (whole) parts in an image.
Such a design cangradually guide the network to learn various information, i.e., from intra-part patterns to inter-partrelations.
Extensive experiments on various vision tasks (e.g., linear probing, fine-tuning, semanticsegmentation, and fine-grained recognition) show that SemMAE can learn better image representationby integrating semantics.
Our contributions include 1) designing a self-supervised semantic part learning method that cangenerate promising semantic parts on multi-class datasets, i.e., ImageNet, and 2) verifying thatsemantic parts can facilitate the learning of MAE by proposing a semantic-guided masking strategy.
While more importantly, we hope our attempts can provide insights for the community to study thevisual analogue of words and unified vision and language modeling.
Related work
Semantic part learning.
Part-based image representation is a fundamental image representationmethod that fits the inherent properties of objects (8; 15; 16; 18; 20).
However, due to the tremendouscost of labeling parts, there are still no large-scale datasets containing part labels.
Thus previousworks are mainly two-fold, i.e., unsupervised/weakly-supervised part learning and few-shot partsegmentation.
Unsupervised/weakly-supervised part learning methods (10; 22; 39) propose to minepart information by leveraging spatial priors, the semantics of convolutional channels, or designingcontrastive proxy tasks.
Few-shot part segmentation methods (4; 29; 38) mainly learn an additionalclassifier over pre-trained features that are trained by GAN, self-supervised contrastive learning,or denoising diffusion probabilistic modeling.
Although promising results have been obtained,these models are designed to deal with fine-grained datasets, where all images belong to a singlesuper-class (e.g., birds, cars, or human faces).
It is much more challenging to solve the problemof unsupervised part learning on multi-class datasets such as ImageNet.
With the development of
ViT and self-supervised learning (SSL), some recent works show a potential solution.
In particular,
DINO (6) and iBOT (42) have observed intuitive semantics in the ViT trained by their SSL methods,where the multi-head attention maps can somehow indicate different semantic parts of an object.
Inspired by these works, we design a reconstruction-based method to further refine the attention mapslearned by iBOT to obtain semantic parts on the ImageNet dataset.
Masked image modeling.
Inspired by the success of Masked Language Modeling (MLM) (5; 12) inpre-training of the NLP field, Masked Image Modeling (MIM) has been proposed recently and exhibitspromising potential for visual pre-training (3; 7; 19; 30).
Existing works mainly study the problemof MIM from two directions, i.e., regression targets and masking strategies.
In terms of regressiontargets, BeiT (3), mc-BEiT (24), and PeCo (13) adopt tokens produced by VQ-VAE (32) or itsvariants.
MaskFeat (35) studies a broad spectrum of feature types and proposes to regress Histogramsof Oriented Gradients (HOG) features of the masked content.
MAE (19) and SimMIM (37) arguethat predicting RGB values of raw pixels by direct regression performs no worse than the patchclassification approaches with complex designs.
In this paper, we follow MAE (19) to adopt the mostsimple and intuitive raw pixels regression.
In terms of masking strategies, SiT (2), MC-SSL0.0 (1)and BeiT (3) use a block-wise masking strategy, where a block of neighbouring tokens arrangedspatially are masked.
MAE (19) and SimMIM (37) use random masking with a large masked patchsize or a large proportion of masked patches.
MST (25) and AttMask (21) propose to use attentionmaps to guide the masking strategy, where the former proposes to mask the nonessential regions topreserve crucial patches while the latter proposes to learn image representations with challengingtasks by masking the most attended tokens.
Moreover, ADIOS (30) proposes to learn an optimalmask by adversarial training.
Compared to these works, our SemMAE takes one step further andexplicitly learn semantic parts to build reasonable hints for masked image modeling.
Figure 1 is anillustration of different masking strategies.
[Figure]
Semantic-guided masked autoencoders
[Figure]
We propose a Semantic-guided Masked Autoencoder (SemMAE) for self-supervised imagerepresentation learning with mask image modeling.
The framework of SemMAE is shown in
Figure 2, which consists of two key components, i.e., Semantic Part Learning (A) and SemanticGuided Masking (B).
First, given an image in Figure 2 (a), we extract the class token in Figure 2(b) and patch tokens in Figure 2 (c) by an iBOT-pretrained ViT. After that, we learn an embeddingover the class token to obtain part tokens in Figure 2 (d).
We calculate the correlation of each parttoken to patch tokens to obtain attention maps in Figure 2 (e), whose texture information is furtherremoved by a large-kernel blur operation.
The attention maps are optimized by a diversity constraintand a reconstruction task where the attention maps and the class token are fed into a StyleGAN-baseddecoder to control the spatial and texture information of the reconstructed image, respectively.
Finally,we conduct argmax over the attention maps to obtain part segmentation maps in Figure 2 (f) and usedthem to guide the mask generation for MAE.
Specifically, we design a masking strategy that variesfrom masking a portion of patches in each part to masking a portion of (whole) parts in an image.
Such a design can gradually guide the network to learn various information, i.e., from intra-partpatterns to inter-part relations.
[Figure]
W
C
N (e) attention maps
A da I N
Ld (a) original image
Lr reconstruction task diversity constraint
[Figure]
iteration # iteration #i iteration #n (B) Semantic-Guided Masking
V i T - ba s ed en c ode r
V i T - ba s ed de c ode r (a) original image guide
[Figure]
ViT
[Figure]
H C embed blur conv
[Figure]
3.1 Semantic part learning
In this subsection we introduce our self-supervised semantic part learning method.
Previousunsupervised/weakly-supervised part learning methods are mainly designed to deal with singleclass datasets (i e., fine-grained datasets where images belong to the same superclass).
Few methodsare able to solve this problem under a multi-class scenario (e g., ImageNet).
While some recentworks (i.e., DINO (6) and iBOT (42)) on ViT-based self-supervised learning show that the multi-headattention maps in their model can somehow indicate different semantic parts of an object.
In thiswork, we take advantage of semantics learned in iBOT and design a reconstruction task together witha diversity constraint to refine and obtain semantic parts.
In particular, given an image I , we first use an iBOT-pretrained ViT to extract its features, i.e., aclass token F c ∈ R C × 1 and patch tokens F ∈ R C × HW .
Then, we embed the class token into Npart tokens F p ∈ R C × N .
The main idea of such embedding is to re-weight feature channels of theclass token.
As shown in previous methods (39), feature channels may be corresponding to specificsemantics and channel re-weighting can group channels with similar semantics together to obtainsemantic part features.
Thus we can obtain part tokens by:
[Equation]
where i ∈ [1 , 2 , ..., N ] , F ( i ) p ∈ R C × 1 is the i th column vector of F p ∈ R C × N , ◦ indicates hadamardproduct, W ( i ) c1 and W ( i ) c2 are embedding weights, tanh( · ) and sigmoid( · ) are activation functions.
After that, we calculate the correlation of each part token to the patch token in each position, thus wecan obtain attention maps, i.e., the possibility of a semantic part to appear in each position:
[Equation]
where M ∈ R N × HW denotes N attention maps, ⊗ indicates correlation function, which isimplemented by softmax( F T p W T p WF ) in our work.
W p and W are embedding matrixes.
To learn such multi-attention maps (i.e., to optimize the parameters in Equation 1 and Equation 2),we propose a reconstruction task.
Our insight is that the spatial information to reconstruct an imageis highly correlated to the position of semantic parts.
Thus, we adopt a StyleGAN-based decoderto reconstruct the original image based on the spatial information from the attention maps and thetexture information from the class token.
To ensure the attention maps learn spatial information,
we
1) remove texture information from the attention maps by conducting a large-kernel blur operationand 2) further feed the blurred attention maps to stacked convolutional layers.
To integrate the textureinformation from the class token into the decoder, we use Adaptive Instance Normalization (AdaIN)operation, which is widely used to integrate texture/style information:
[Equation]
where each feature channel [conv( M )] i is normalized separately, and then scaled and biasedusing the corresponding scalar components from the embedded class token F c .
F d denotes theconvolutional feature in the decoder, W s and W b are embedding weights, [ · ] i denotes the i th featurechannel, conv( · ) denotes convolutional layers, µ ( · ) and σ ( · ) calculate the mean and variance values,respectively.
The reconstructed image ˆ I can be obtained by stacking convolutional can AdaIN layers:
[Equation]
We use the Mean squared error (MSE) loss function to optimize such reconstruction task:
[Equation]
Moreover, to obtain diverse multiple attention maps, we follow previous work (40) and add a diversityconstraint over attention maps:
[Equation]
where attention maps are optimized to be different from each other, m i and m j denotes the i th andj th attention map, respectively.
The overall objective function can be denoted by:
[Equation]
where λ is the loss weight.
3.2 Semantic-guided masking
After finished semantic part learning, we move to the next stage, i.e., semantic-guided MAE training.
Our informed masking strategy is based on the part information learned in Subsection 3.1.
Specifically,we can obtain multiple attention maps by Equation 2, where each attention map
m ∈
R
H
× W indicatesthe possibility of the corresponding semantic part appearing in H × W positions.
Thus we conductargmax( · ) operation over attention maps to obtain part segmentation, where each patch is classifiedinto a particular semantic part.
The patches in the same semantic part compose a visual analogueof words, which are semantically meaningful.
To leverage such visual analogue of words for MAEtraining, a most intuitive way is to mask a portion of semantic parts and learn to predict the maskedsemantic parts by other parts.
However, due to the learned semantic parts being coarse-grained (e.g.,6 parts for each image), we experimentally find that such a masking strategy makes the task too hardto effectively learn image representations.
To this end, we propose an easy-to-hard reconstruction task, which can provide reasonable hints (i.e.,visible patches) for the model to predict the masked patches during the training process of the MAE.
Specifically, at the beginning of the training process, we mask a portion of patches in each part, thusthe masked patches can be predicted based on the visual patches that belong to the same semanticpart.
Such a design can facilitate the models to learn intra-part patterns.
After that, we graduallymask all patches belonging to some parts and a portion of patches belong to the remaining parts.
Finally, we mask all patches belonging to a portion of parts and predict the remaining patches belongto the other parts, where inter-part relations or visual reasoning ability can be learned.
Algorithm 1 shows the details to obtain the number of masked patches for each semantic part.
First,we define two masking settings, i.e., 1) mask a portion of patches in each part and 2) random select some parts to mask (the whole part).
The number of masked patches for each semantic part can becalculated for these two settings.
After that, we introduce an interpolation hyper-parameter α .
Asmall α means the first setting dominates the masking strategy, and vice versa.
α is adjusted basedon training iterations and keeps increasing during the training process.
Finally, we random mask acertain number of patches based on the calculated masking number.
Experiments
4.1 Experiment setup
Semantic part learning.
As introduced in Section 3.1, we use ViT-small (14) as our part learningencoder, which is pre-trained by a self-supervised method iBOT (42).
We follow iBOT (42) to learn6 semantic parts for each image, as the head number of the multi-head attention in ViT-Small is 6.
The size of the blur kernel is experimentally set to be 7, and the loss weight λ in Equation 7 is set tobe 0.03.
The experiment is performed on ImageNet-1k (11) dataset.
The parameters of the ViT-basedencoder are fixed, and we only optimize the attention module and the StyleGAN-based decoder.
Ourmodel converges fast, which only takes 2 hours on one A100 GPU card.
Semantic-guided MAE training.
We follow MAE (19) and adopt an encoder-decoder structure toperform MIM.
Our method is general for ViT backbones, while most experiments are conductedwith a relatively small version, i.e., the original ViT-Base (14), due to the limitation of computationresources.
We follow the most comment setting to optimize our model by AdamW (27) with alearning rate of 2.4e-3.
The batch size is set to be 4096, and the weight decay is set to be 0.05.
We use a cosine learning rate strategy (26) with warmup (17).
The warmup number is set to be 40epochs, and we pre-train our model for 800 epochs.
For data augmentation, we only employ randomhorizontal flipping in our pre-training stage.
The hyper-parameter γ in Algorithm 1 is experimentallyset to be 2.
Our model is trained on 16 A-100 GPUs for 3 days, and more details can be found in ourcode, which is in the supplemental material and will be made publicly released. [Table] [Table]
[Table]
guided
[Table]
The effectiveness of integrating semantic information.
We conduct experiments under two differentsettings (i.e., with a patch size of 16 × 16 and 8 × 8) to verify the effectiveness of integrating semanticinformation for training MAE.
The results in Table 1 show that integrating semantic information canbring 1.3% and 1.9% accuracy gains for linear probing, respectively.
As we use 8 × 8 patch size tolearn semantic parts, the coarse-grained patch (i.e., large patch size) in the pre-training stage wouldcause imprecise part segment and suppresses the benefits of semantic parts.
To further study theimpact of patch size for masked image modeling, we conduct fine-tuning experiments in Table 2.
It can be observed that in SimMIM and original MAE, a larger patch size performs better; whilein our SemMAE, more precise semantic parts with 8 × 8 patch size can significantly improve theperformance.
Thus in the following experiments, we adopt 8 × 8 patch size for SemMAE.
It is notablethat although using a smaller patch size, our parameters and computational cost do not increaseduring pre-training and linear probing as only 1/4 patches in each image are used.
Specifically, weleverage the learned attentions maps to remove 3/4 patches that are most likely to be the background.
A detailed study on masking strategies.
Once obtained semantic parts, a most intuitive way toleverage such visual analogue of words for MAE training is to mask a portion of semantic parts andmake the model to predict the removed content.
However, due to the learned semantic parts beingcoarse-grained (e.g., 6 parts for each image), we experimentally find that such a masking strategymakes the task too hard to effectively learn image representations.
The results can be found in Table 3,where masking 75% parts cause 13.9% performance drops compared to random masking.
Moreover,it can be observed that masking 75% patches per part achieves comparable results with randommasking.
The self-supervised learning task of masking 75% patches per part would encourage themodel to learn local contexts/intra-part patterns, and masking 75% parts would encourage the modelto learn inter-part relations.
Interestingly, we find that the former task can enable the model to furtherlearn better image representation in the latter task.
The results in Table 3 show that our proposedadaptive masking strategy (i.e., varying from masking 75% patches per part to masking 75% partsgradually) with γ = 2 yields the best performance.
[Figure]
[Table]
4.3 Semantic part learning
We first evaluate the effectiveness of our proposed semantic part learning method.
Both qualitativeand quantitative experiments are conducted.
Note that most of the previous part learning models aredesigned for single-class datasets and cannot be effectively applied to ImageNet.
iBOT (42) is not [Figure] ours proposed for part learning, while the multi-head attention in their model achieves the state-of-the-artpart learning performance on ImageNet.
Figure 4 shows the qualitative comparison of our modeland iBOT (42), and it can be observed that our model can generate more complete semantic partsegmentation maps where different parts and the background are better separated with less noise.
Moreover, as there is no part segmentation ground truth, we conduct quantitatively evaluation in anindirect way by training SemMAE and analyzing ImageNet classification performance.
The resultsin Table 4 show that the semantic parts obtained by iBOT are not able to benefit the learning of MAE,while our semantic part learning methods can generate more precise part segmentation maps, whichare vital to learning a better image representation. Semantic parts for masking Baseline (w/o parts) iBOT-initialized parts Our learned parts Linear
[Table]
4.4 Compared with other methods on ImageNet
Linear probing and fine-tuning on ImageNet-1K classification dataset is the most common setting toevaluate SSL methods.
We collect all competitive methods that report their results on ImageNet-1Kdataset.
For example, we do not include the related work MST (25) and ADIOS (30) as they evaluatetheir model on other benchmarks.
Table 5 shows the comparison of our model and previous modelsin terms of linear probing and fine-tuning.
For a fair comparison, all experiments adopt the sameinput size, i.e., 224 × 224 unless specified otherwise.
Compared with “training from scratch”,our SemMAE can significantly improve the performance for both linear probing and fine-tuning.
For linear probing, our SemMAE outperforms the most competitive MIM-based methods by 0.8%even with fewer training epochs.
For fine-tuning, our SemMAE achieves 84.5% top-1 classificationaccuracy, outperforming SimMIM(37) and MAE(19) by 0.9% and 0.7% respectively.
Moreover, our
SemMAE can also surpass previous contrastive learning-based methods(6; 9) for fine-tuning.
4.5 Downstream tasks
Fine-grained image classification.
Table 6 shows our results of transfer learning on fine-graineddatasets.
Our model can surpass the most competitive MAE (19) with a clear margin, i.e., 0.3%, 0.6%,and 0.2% on the iNaturalists(33), CUB-Bird(34), and Stanford-Cars(23) dataset, respectively.
Theseresults show the promising transfer ability of our SemMAE for downstream classification tasks.
[Table]
Semantic segmentation.
Semantic segmentation aims to assign a label to each pixel of the inputimage.
We evaluate our SemMAE on the widely used semantic segmentation dataset ADE20K (41),which contains 25K images and 150 semantic categories.
We follow the most common setting to usethe task layer in UPerNet (36) and fine-tune the pre-trained ViT-Base model.
We use the standardsetting that pre-train a ViT-Base model with a patch size of 16 × 16 and fine-tunes 160K iterationswith a batch size of 16.
Such an experiment can validate the transfer ability of our SemMAE forsemantic segmentation.
As shown in Table 7, SeMAE surpasses MAE by 0.2 (46.3 vs. 46.1)
mIoUand outperforms the supervised pre-train model by 1.0 mIoU. These results show the promisingtransfer ability of our SemMAE for dense prediction visual tasks.
[Table] [Table]
Conclusion
In this paper, we study the visual analogue of words and propose a semantic-guided maskedautoencoder model to reduce the gap between masked language modeling and masked imagemodeling.
Our proposed self-supervised semantic part learning method can generate promisingsemantic parts on ImageNet and we show that the learned semantic parts can facilitate the learningof MAE.
Unlike the main-stream random masking strategy, our semantic-guided mask strategy caneffectively integrate semantic information in the pre-training process.
Extensive experiments withsuperior results show the effectiveness of our SemMAE.
Limitations : due to the lack of part segmentation labels, the semantic part in our work is kind ofcoarse (e.g., 6 parts per image), making it not an ideal visual analogue of words yet.
Moreover, usinga small patch size increases the computational cost in the fine-tuning stage.
In the future, we will 1)investigate finer-grained semantic parts (e.g., 20-30 parts per image) by few-shot part segmentationand 2) replace the widely obtained patch-based tokenization with part-based tokenization to furtherreduce the gap between vision and language modeling.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 5(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] No theoretical results. (b) Did you include complete proofs of all theoretical results?
[N/A] No theoretical results.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the mainexperimental results (either in the supplemental material or as a URL)?
[Yes] Inthe supplemental material
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 Experiment Setup.
(c) Did you report error bars (e g., with respect to the random seed after runningexperiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4 Experiment Setup.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes] In our code.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]