REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering
Abstract
This paper revisits visual representation in knowledge-based visual question answering (VQA) and demonstrates that using regional information in a better waycan significantly improve the performance.
While visual representation is extensively studied in traditional VQA, it is under-explored in knowledge-based
VQA even though these two tasks share the common spirit, i.e., rely on visualinput to answer the question.
Specifically, we observe that in most state-of-the-artknowledge-based VQA methods: 1) visual features are extracted either from thewhole image or in a sliding window manner for retrieving knowledge, and theimportant relationship within/among object regions is neglected; 2) visual featuresare not well utilized in the final answering model, which is counter-intuitive tosome extent.
Based on these observations, we propose a new knowledge-based
VQA method REVIVE , which tries to utilize the explicit information of objectregions not only in the knowledge retrieval stage but also in the answering model.
The key motivation is that object regions and inherent relationship are importantfor knowledge-based VQA.
We perform extensive experiments on the standard
OK-VQA dataset and achieve new state-of-the-art performance, i.e. , 58.0% accuracy, surpassing previous state-of-the-art method by a large margin ( +3.6% ).
Wealso conduct detailed analysis and show the necessity of regional information indifferent framework components for knowledge-based VQA.
Introduction
Many vision-based decision making processes in our daily life go beyond perception and recognition.
For example, if we see a salad bow in the deli bar, our decision on whether to buy it does notonly depend on what is in the bow, but also the calories in each of the item.
This motivates theknowledge-based Visual Question Answering (VQA) task [18] which extends traditional VQA task[1] to solve more complex problems, i e. , where commonsense knowledge is required to answer theopen-domain questions.
By definition knowledge-based VQA takes three different information sources to predict the answer: input visual information (image), input question and the external knowledge.
While existing research on knowledge-based VQA mainly focuses on improving the incorporation of external knowledge, this paper focuses on improving the object-centric visual representation and presents a comprehensive empirical study to demonstrate that visual features matter in this task
However existing state- -the-art ( ) methods [32, 5] have not fully utilized it
On the one hand they either the whole image or a sliding window on the image to retrieve the external knowledge
On the other hand they only visual information for knowledge retrieval but ignore it the final answering model
In other words they fuse only the retrieved knowledge and the question as a pure natural language processing (NLP) model to obtain the answer, a typical method[5] is illustrated in Figure 1 (b).
In this paper, we revisit visual representation in knowledge-based VQA, and argue that the informationof object regions and their relationship should be considered and used in a dedicated way.
Theunderlying motivation is shown in Figure 1 (a), which demonstrates that understanding the objectsand their relationship is necessary.
To this end, we propose REVIVE to better utilize RE gional VI sual
Representation for knowledge-based V isual qu E stion answering.
It not only exploits the detailedregional information for better knowledge retrieval, but also fuses the regional visual representationinto the final answering model.
Specifically, we first use the object detector GLIP [12] to locate theobject, and then use the cropped region proposals to retrieve different types of external knowledge.
Finally, we integrate the knowledge together with the regional visual features into a unified transformerbased answering model for final answer generation.
We perform extensive experiments on the OK-VQA dataset [18], and the proposed REVIVE achievesthe SOTA performance of 58.0% accuracy, a 3.6% absolute improvement from the results of previous
SOTA method [5].
We summarize our contribution as follows:
(a) We systematically explore how to better exploit the visual feature to retrieve knowledge.
The empirical results suggest the region-based approach performs the best, compared to
whole image-based and sliding window-based approaches.
(b) We integrate the regional visual representation, retrieved external and implicit knowledge
into a transformer-based question answering model, which can effectively leverage the three
information sources for solving knowledge-based VQA.
(c) Our proposed REVIVE achieves the state-of-the-art performance on OK-VQA dataset, i.e. ,58.0% accuracy, surpassing the previous methods by a large
margin.
Related Work
Knowledge-Based VQA.
Knowledge-based VQA [18] aims to predict answers for general questionsby leveraging external knowledge beyond image content.
Early works [29, 28] introduce externalknowledge to solve visual question answering (VQA) tasks.
OK-VQA dataset [18] is the first largescale dataset with questions that need be answered using external knowledge instead of a providedfixed knowledge base [28].
Recent studies [29, 28, 20, 19, 35, 31, 17, 32, 5] integrate differentknowledge from various external knowledge resources, e.g. , ConceptNet [23], Wikipedia [26], etc ,for solving knowledge-based VQA.
Later, PICa [32] regards large language models, e.g., GPT-3[2] as an implicit knowledge source and employs it [2] to get answer prediction based on textualprompts.
Inspired by the recent success of knowledge-retrieved methods [7, 6] that leverage externalknowledge retrieval with language generative models for open-domain question answering, KAT[5] exploits the FiD reader [7] to perform knowledge reasoning over retrieved implicit and explicit knowledge.
Our work instead emphasizes revisiting the visual representation for knowledge retrieval,i.e. , resorting to regional visual representation.
In addition, we propose to incorporate object-centricregional visual representation together with retrieved knowledge in the answer generative model.
Vision-Language Models.
Recent years have witnessed the rapid development of vision-languagemodels [25, 8, 33, 24, 13, 34, 27, 30].
Those works usually first pre-train a neural network on alarge-scale image-text dataset and then finetune the models for solving specific vision-languagetasks.
Among them, VinVL [34] aims to learn the object-centric representation.
CLIP [21] pre-trainsthe models with large-scale text-image pairs by contrastive learning.
GLIP [33] reformulates thepre-training process by unifying object detection and phrase grounding.
Our method uses the threemodels as sub-modules to identify object-centric regions and retrieve knowledge for knowledge-based
VQA task.
Proposed Method
Knowledge-based VQA task [18] seeks to answer questions based on external knowledge beyondimages.
Specifically, let us denote a knowledge-based VQA dataset as D =
{ ( I i , Q i , A i ) } Ni =1 , where
I i , Q i and A
i denote the input image, question and answer of the i -th sample respectively, and Nis the number of total samples.
Given the dataset, the goal is to train a model with parameter θ togenerate the answer
A i with input
I i and Q i .
In this section, we introduce our method REVIVE .
Figure 2 shows an overview of the method.
Weleverage the detected regions of the input image to obtain the object-centric region features andretrieve explicit knowledge, meanwhile, we prompt GPT-3 [2] by regional tags, question and contextto retrieve implicit knowledge.
After that, the regional visual features, retrieved knowledge, and thetext prompt consists of regional tags, question and context will then be fused into a encoder-decodermodule to generate the answer.
We explain more details in Section 3.1, 3.2 and 3.3.
3.1 Regional Feature Extraction Module
Given an image I , we first adopt a object detector to give us the positions of region proposals,
[Equation]
where B = { b j } Mj =1 is the set of bouding boxes, M is the number of detected boxes, and D ( · ) is theobject detector.
Here, we adopt D ( · ) as the visual grounding model GLIP [12].
We use the text prompt “ Detect:person , bicycle , car , ... , toothbrush ", which contains all object categories of MSCOCOdataset [14].
In this way, the model can provide us with all bounding boxes associated with thosecategories.
After we get the bounding boxes B of interested objects from GLIP, We crop the image I accordingto B to obtain region proposals R = { r j } Mj =1 .
We then extract the object-centric visual features from the proposals: v j = E ( r j ) , where v j ∈ R S is the visual embedding of the j -th proposal, S is theembedding dimension and E ( · ) represents the image encoder.
Inspired by the strong transferringcapability of recent contrastively trained vision-language models, we adopt the visual encoder of
CLIP [21] as our image encoder E ( · ) .
We use the encoding of [CLS] token as the final embedding.
To understand the relationship between/among the objects, we find it also important to introduce theposition information B along with its regional visual features.
In addition to the embeddings, explicitly obtaining the description of each region proposal in thetextual format is also helpful for knowledge retrieval.
For the contrastively trained vision-languagemodels, the training loss explicitly encourages inner product between the image embedding and thetext embedding to be larger if the image and the text are well-aligned.
Therefore, such a model iscapable of selecting the tags that describe the image from a set of customized tags ¯ T by computingthe inner product.
Denote the language encoder of CLIP as T ( · ) .
Given a set of tags ¯ T = { t i } N 1 i =1 ,
N 1 is the number of total tags, we compute the inner product between the region proposals and alltags, and adopt the tags with the top- P similarities as the description of the region proposals,
[Equation]
where ⟨· , ·⟩ is the inner product, P denotes the number of the obtained regional tags and H means theretrieved regional tags.
In complement to the localized textual description H , we adopt a caption model to explicitly describethe relationships between the major objects and provide more context,
[Equation]
(3) where C ( · ) is the caption model.
For example, in Figure 2, the context “ Two brown dogsfighting over a red frisbee ” provides us with the essential relationships between theobjects, e.g., fighting over a red frisbee.
Here, we adopt Vinvl [34] as the caption model C ( · ) .
In summary, we extract regional visual and positional information as { v j } Mj =1 and { b j } Mj =1 , andtextual descriptions for the objects and the relationship between the objects as H and c .
In the nextsection, we will elaborate on how we use these regional information sources to retrieve externalknowledge.
3.2 Object-Centric Knowledge Retrieval Module
Inspired by KAT [5], we consider both the explicit knowledge and implicit knowledge.
3.2.1 Explicit Knowledge
Since the questions from knowledge-based VQA [18] are general and open-ended, introducingexternal knowledge is important for model to generate accurate answers by providing extra andcomplementary knowledge beyond visual contents of input images.
External Knowledge Base.
We construct a external knowledge base Q by constructing a subset from Wikidata [26] following KAT [5].
Specifically, we extract 8 commonly appeared categories, i.e. , Role , Point of interest , Tool , Vehicle , Animal , Clothing , Company ,
Sport , to form the subset Q .
Each item in Q consists of an entity and a corresponding description,e.g. , one entity and its description can be “ pegboard ” and “ board wall covering withregularly-spaced holes for insertion of pegs or hooks " respectively.
Regional Knowledge Retrieval.
As mentioned earlier, vision-language models like CLIP arecapable of selecting the most relevant text from a set of texts.
We reformat the entries in knowledgebase Q as “{entity} is a {description}", and denote the reformatted text set as T .
We retrieve thetop-
K most relevant knowledge entries among all the regional proposals as explicit knowledge E ,
[Equation]
where K denotes the number of retrieved explicit knowledge samples.
In our implementation, we use
FAISS [9] to speed up the computation of Equation (2) and (4).
3.2.2 Implicit Knowledge
Large language models, e.g. , GPT-3 [2], not only excel in many language tasks, but also memorizelots of commonsense knowledge from its training corpus [32].
Therefore, we exploit GPT-3 [2] asour implicit knowledge base by reformulating the task as open-domain question answering.
Context-Aware Prompt with Regional Descriptions.
We design the textual prompt based onquestion Q , caption c , and regional tags H .
Specifically, we adopt the prompt X to be “ context:{caption} + {tags}.
question: {question}”.
In this way, the language model is also supplementedwith regional visual information.
Implicit Knowledge Retrieval.
Finally, we query GPT-3 model [2] which takes the reformulatedprompt X as input, and obtain predictive answer.
Since some of the questions may have ambiguity,we follow the prompt tuning procedure of PICa [32] and get answer candidates { o u } Uu =1 .
In additionto answer prediction, we also aim for acquiring corresponding explanation e u from GPT-3 model toobtain more context information.
To be more specific, the corresponding explanation is acquired byfeeding the text prompt “{question} {answer candidate}.
This is because " into GPT-3.
Notethat “{question}" and “{answer candidate} are input question Q and GPT-3’s answer o u for image Irespectively.
The final retrieved implicit knowledge can be denoted as I =
{ ( o u , e u ) } Uu =1 .
3.3 Encoder-Decoder Module
Once we’ve retrieved the explicit and implicit knowledge and the regional information, we utilize the
FiD network structure [7] to encode and decode retrieved knowledge and regional information.
Knowledge Encoder.
For the explicit knowledge, we reformat the input text as “ entity: {entity}description: {description}”, where the entity and the description is from the entries in theretrieved explicit knowledge E .
We denote this text as h k , where k = 1 , · · · , K .
For implicit knowledge, we adopt input format as “ candidate: {answer} evidence: {explanation}”, where answer is the retrieved answer o u and explanation is e u .
Here, u = 1 , · · · , U , where Uis the number of answers provided by GPT-3.
We denote the input text as s u .
We then encode the knowledge in textual format by the FiD’s encoder [25], which is denoted as F e , [Equation]
) in which α k ∈ R D , β u ∈ R D and D means the embedding dimension.
Visual Encoder.
We introduce a visual encoder for the visual embeddings { v j } Mj =1 and positionalcoordinates { b j } Mj =1 .
We feed v j and b j into two different fully connected layers, stack the outputsinto a sequence of vectors, and then feed them into a transformer encoder F v ,
[Equation]
where f ∈ R (2 M ) × D , FC 1 ( · ) and FC 2 ( · ) are two different fully-connected layers, Concat ( · ) is theconcatenation operation along a new dimension.
Context-aware Question Encoder.
To better leverage the context information, we replace the inputquestion Q by the context-aware prompt X , we then encode it by the same transformer encoder F e ( · ) ,
[Equation]
where q ∈ R D and q means encoded context-aware question.
Generative Decoder.
We have obtained the knowledge encoding { α k } Kk =1 and { β u } Uu =1 , visualencoding f , and context-aware question encoding q .
Note that as the outputs of the encoder F e , theyare all sequences of vectors.
We then concatenate these vectors along the first dimension, and feedthem into the FiD’s decoder F d ,
[Equation]
where y means the generated answer.
The cross entropy loss function is adopted to train the model,
[Equation]
in which L is the length of the ground truth answer text, ¯ y ℓ is ground truth text at the position ℓ and θis the model parameters.
Model Ensemble.
To generate more accurate answers, one promising method is to leverage multipletrained models, i.e. , model ensemble.
In our experiments, we just train three models whose initializedseeds are different, and then the most frequent result among the generated results from these threemodels is selected as final answer prediction for each sample.
3.4 Relationship to Existing Works
Motivated by KAT [5], the proposed REVIVE also retrieves two types of knowledge, i.e. , implicit andexplicit knowledge, for solving knowledge-based VQA.
However, we pay more attention to revisitingthe visual representation for retrieving knowledge.
In addition, we integrate the visual representationof object regions with retrieved knowledge in the answer generative model.
The pipeline differencesbetween KAT [5] and our method can be explained in Figure 1.
There’re two works [31, 17] that leverage visual regions for knowledge-based VQA as well.
However,
MAVEx [31] considers object regions as a kind of knowledge without using their visual representationto retrieve other knowledge, KRISP [17] utilizes object regions to learn implicit knowledge by atransformer-based model and retrieve external knowledge by the text symbols of these regions, whileour proposed REVIVE explores how to better leverage visual representation to retrieve knowledgeand integrate their visual features with retrieved knowledge into the answering model.
Experiments
4.1 Experimental Setup
Dataset.
OK-VQA dataset [18] is selected for evaluation, which is currently the largest knowledgebased VQA dataset.
OK-VQA dataset includes 14055 questions associated with 14031 images from
MSCOCO dataset [14].
Its questions cover a variety of knowledge categories, and are annotatedby Amazon Mechanical Turkers.
The training and testing split consist of 9009 and 5046 samplesrespectively.
Each data sample is made up of one question, one corresponding image and 10 groundtruth answers.
To construct the general domain tag set ¯ T , we collect the most frequently searched400K queries in Bing Search as the tags.
Pre-processing
We utilize the pre-trained visual grounding model GLIP [12] to detect objectcentric region proposals by using its default prompt “ Detect: person , bicycle , car , ... ,toothbrush ", which contains all object categories of MS-COCO dataset [14].
The captions ofimages are obtained by the pre-trained Vinvl-Large model [34].
For explicit knowledge and regionaltag retrieval, we choose CLIP model (ViT-B/16 variant)
In our experiments, we adopt U , K , Mand P as 5, 40, 36 and 30 respectively.
Note that the models of CLIP, GLIP, Vinvl and GPT-3 are allfrozen during usage.
Implementation Details.
We use 4 × NVIDIA
V100 32Gb to train models for 10K steps, with abatch size of 8.
The learning rate is 8e − 5 and AdamW [15] is chosen as optimizer.
The warm-upsteps are 1K and the trained models are evaluated every 500 steps.
We initialize our model with thepre-trained T5 model [22], i.e. , T5-large, following KAT [5].
The encoder F v in Equation (6) consistsof 9 transformer layers [25].
Note that we evaluate the prediction results after normalization, and thenormalization process mainly includes removing articles, punctuation and duplicated whitespace andlowercasing [3, 10].
Evaluation Metric.
In our experiments, we choose the soft accuracy of VQAv2 [1] as evaluationmetric for comparison.
4.2 Comparison with State-of-the-art Methods
As shown in Table 1, we can see that previous works ( e.g. , KRISP [17], Visual Retriever-Reader [16]and MAVEx [31]) achieve similar performances, about 38.4% to 39.4% accuracy.
Until recently, PICa[32] is the first one that exploits the pre-trained language model GPT-3 [2] as knowledge base forknowledge-based VQA task and KAT [5] further introduces Wikidata [26] as an external knowledgeresource, these two works obtain significant performances compared with previous ones.
[Table]
The proposed REVIVE can outperform all existing methods by large margins.
Specifically, even usingthe same knowledge resources ( i
[Table]
e. , Wikidata [26] and GPT-3 [2]), our single model can achieve56.6% accuracy versus previous state-of-the-art method KAT’s 53.1% accuracy [Table] when using modelensemble, our method can achieve 58.0% accuracy compared with KAT’s 54.4% accuracy.
Theseresults demonstrate the effectiveness of the proposed approach.
4.3 Ablation Study
Next, we conduct extensive ablation studies on the single model to figure out the influence of eachcomponent of REVIVE .
Effect of Region Proposal Number.
We perform the ablation study to figure out the effect of usingdifferent region proposal numbers.
The results are displayed in Table 2.
It can be observed thatwhen the region proposal number is 36, the model achieves optimal performance.
We conjecturethat when the number of region proposals is too large, there are some meaningless and noisy regionproposals, while if the number of region proposals is too small, many essential object-centric regionsare ignored, which both hurt the model’s performance.
Different Knowledge Retrieval Methods.
The way of utilizing visual representation for retrievingknowledge plays an important role in knowledge-based VQA.
We show the results of using threekinds of knowledge retrieval methods, i.e. , image-based, sliding window-based and region-based, in
Table 5.
Note that sliding window-based approach follows KAT [5].
Specifically, we first resizes inputimages to 384 × 384 and then crop the images with a sliding window whose size is 256 × 256 andstride size is 128.
We can observe that the proposed region-based approach achieves best performanceand surpasses sliding window-based method by 1.8% points, which can validate the effectiveness ofexploiting region-based visual representation for retrieving knowledge.
Effect of Regional Tag Number.
In order to introduce more semantics into contexts, we propose toadd region-aware descriptions ( i.e. , regional tags) behind given contexts.
We report the results ofusing different regional tag number for text prompt X in Table 3.
The results show that when thenumber of regional tags is 30, it achieves optimal performances.
In fact, when the number of regionaltags is too large, we’ll retrieve relatively irrelevant object tags, sacrificing the model’s performance.
Effect of Positional Coordinates.
In addition to incorporating visual representation of object-centricregion proposals into the model, we also adopt the position information ( i.e. , positional coordinates).
The results of whether using positional coordinates are reported in Table 4.
Introducing regionalcoordinates can improve the performance by 0.8% points.
[Table]
[Table]
[Table]
[Table]
Effect of Each Component.
In order to better show the effect of each component of REVIVE , wereport the experimental results in Table 6.
Note that we choose regional-based manner to retrievedifferent categories of knowledge.
We can observe that introduced components can consistentlyimprove the model’s performance.
Especially, the explicit knowledge and the object-centric regionfeatures can bring 1.6% and 1.4% points improvement respectively, feeding context-aware questions,which can be denoted as prompt “ context: {caption}.
question: {question}”, into the answergenerative model can have 0.5% points gain, further introducing regional descriptions ( i
[Table]
e. ., regionaltags) into contexts,
i
[Table]
e. , prompt X , can have 0.7% points improvement, and model ensemble canbring 1.4% performance gain. These results can validate the efficiencies of proposed components.
[Table]
✓ ✓ 56.6 ✓ ✓ ✓ ✓ ✓ ✓ 58.
4.4 Quantitative Result Analysis
Finally, we present the quantitative results and provide analysis for error cases, so that we can have aclear insight into the proposed approach.
Visualizing Results.
The success cases of our approach are shown in Figure 3.
We can observethat our approach can accurately retrieve implicit and explicit knowledge, which corresponds to thedetected object regions, and deal with the relationship among these object areas.
For example, in theleft example of Figure 3, our method can handle the spatial relationship and recognize the referredobject ( i.e. , the right mouse), thus generating the correct answer, in the right example, our methodcan retrieve important knowledge ( e.g. , brazilian terrier) to answer the breed of the referred dog.
Failure Cases Analysis.
We showcase the failure examples in Figure 4.
As shown in the leftexample, even though the prediction result Cabin doesn’t appear in the ground truth answers, thegenerated answer of our approach is still reasonable for such scenario.
For the right example, ourpredicted result is wrong due to the difficulty of answering such a general question.
From figure 4, wecan also observe that our method can generate useful object-centric regions and accurately retrievecorresponding knowledge, especially explicit knowledge, which can demonstrate the potential of theproposed method.
Conclusion
In this paper, we propose a method called REVIVE , which revisits regional visual representationfor knowledge-based VQA task. Specifically, we conduct comprehensive experiments to show theeffects of different components in the proposed approach, which can demonstrate the efficiency of theproposed region-based knowledge retrieval method.
In addition, REVIVE incorporates object-centricregional visual features and two kinds of knowledge, i.e. , implicit and explicit knowledge, into theanswer generative model for prediction. Our method achieves the state-of-the-art performance on OK-VQA dataset.  Limitations and Broader Impact
The quality of constructed Wikidata subset and designed textual prompt can influence final retrievedknowledge.
In addition, the detector for obtaining region proposals also affect retrieved knowledgeand visual features, all these factors affect the models’ performances.
This paper proposes a novel approach REVIVE for knowledge-based VQA.
REVIVE can help modelsto efficiently use visual and language information sources to answer open-domain questions.
It canalso generalize to real-life products, e.g. , dialogue robot.
However, the failure cases of REVIVE willbe negative to the society when using it as the educational technique.
There may also exist certainforms of bias, i.e. , the model may predict biased answers if the training data is overwhelmed withcertain scenarios.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]