Additive MIL: Intrinsic Interpretability for Pathology
Anonymous Author(s)
AffiliationAddress email
Abstract
Multiple Instance Learning ( MIL ) has been widely applied in pathology towardssolving critical problems such as automating cancer diagnosis and grading, predicting patient prognosis, and therapy response.
Deploying these models in a clinicalsetting requires careful inspection of these black boxes during development anddeployment to identify failures and maintain physician trust.
In this work, wepropose a simple formulation of MIL models, which enables interpretability whilemaintaining similar predictive performance.
Our Additive MIL models enablespatial credit assignment such that the contribution of each region in the image canbe exactly computed and visualized.
We show that our spatial credit assignmentcoincides with regions used by pathologists during diagnosis and improves uponclassical attention heatmaps from attention MIL models.
We show that any existing
MIL model can be made additive with a simple change in function composition.
We also show how these models can debug model failures, identify spurious features, and highlight class-wise regions of interest, enabling their use in high-stakesenvironments such as clinical decision-making.
Introduction
Histopathology is the study and diagnosis of disease by microscopic inspection of tissue.
Histologicexamination of tissue samples plays a key role in both clinical diagnosis and drug development.
It isregarded as medicine’s ground truth for various diseases and is important in evaluating disease severity,measuring treatment effects and biomarker scoring [37].
A differentiating feature of digitized tissueslides or whole slide images ( WSI ) is their extremely large size, often billions of pixels per image.
Inaddition to being large, WSI s are extremely information dense, with each image containing thousandsof cells and detailed tissue regions that make manual analysis of these images challenging.
Thisinformation richness makes pathology an excellent application for machine learning and indeedthere has been tremendous progress in recent years in applying machine learning to pathology data[38, 8, 18, 41, 11, 6, 13, 12].
The most important applications of ML in digital pathology involve predicting patient’s clinicalcharacteristics from a WSI image.
Models need to be able to make predictions about the entire slideinvolving all the patient tissue available; we refer to these predictions as "slide-level".
To overcomethe challenges presented by the size of these images, previous methods have used smaller handengineered representations, built from biological primitives in tissue such as cellular compositionand structures [11].
Another common way to overcome the challenges presented by the size of
WSI s is to break the slide into thousands of small patches, train a model with these patches topredict the slide-label, and then use a secondary model to learn an aggregation function from patchrepresentations to slide-level label [8].
Both methods are not trained in an end-to-end manner andsuffer from sub-optimal performance.
The second method also suffers from an incorrect assumptionthat each patch from a slide has the same label as the overall slide [16].
Multiple Instance Learning [28] is a weakly supervised learning technique which attempts to learn amapping from a set of instances (called a bag) to a single label associated with the whole bag.
MILcan be applied to pathology by treating patches from slides as instances which form a bag and aslide-level label is associated with each bag to learn a bag predictor.
This circumvents the need tocollect patch-level labels and allows end-to-end training from a WSI .
The MIL assumption that at leastone patch among the set of patches is associated with the target label works well for many biologicalproblems.
For example, the MIL assumption holds for the task of cancer diagnosis; a sufficiently largebag of instances or patches from a cancerous slide will contain at least one cancerous patch whereas abenign slide will never contain a cancerous patch.
In recent years, attention based pooling of patcheshas been shown to be successful for MIL problems [16].
Using neural networks with attention MILhas become the standard for end-to-end pathology models as it provides a powerful, yet efficientgradient based method to learn a slide-to-label mapping.
In addition to superior performance, thesemodels encode some level of spatial interpretability within the model through visualization of highlyattended regions.
The sensitive nature of the medical imaging domain requires deployed machine learning models to beinterpretable for multiple reasons.
First, it is critical that models do not learn spurious shortcuts overtrue signal [10, 32] and can be debugged if such failure modes exist.
Interpretability and explainabilitymethods have been shown to help identify some of these data and model deficiencies [14, 25, 3, 35].
Secondly, for algorithms in medical decision-making, accountability and rigorous validation precedesadoption [24, 17].
Interpretable models can be easier to validate and thus build trust.
Specifically,users can verify that model predictions are generated using biologically concordant features that aresupported by scientific evidence and are similar to the those identified by human experts.
Thirdly,use-cases involving a human expert such as decision-support require the algorithm to give a visualcue which highlights the regions to be examined more carefully.
In these applications, a predictedscore is insufficient and needs to be complemented with a highlighted visual region associated withthe model’s prediction [7].
For ML models in pathology, spatial credit assignment can be defined as attributing model predictionsto specific spatial regions in the slide.
Various post-hoc interpretability techniques like gradient basedmethods [30] and Local Interpretable Model-agnostic Explanation ( LIME ) [39] have been used to thisend.
However, gradient based methods which try to construct model-dependent saliency maps areoften insensitive to the model or the data [19, 1, 4].
This makes these post-hoc methods unreliable forspatial attribution as they provide poor localization and do not reflect the model’s predictions [14].
Model-agnostic methods like Shapley values or LIME involve intractable computations for largeimage data and thus need approximations like locally fitting explanations to model predictions, whichcan lead to incorrect attribution [29].
Applying Attention MIL [16] in weakly supervised problemsin pathology leads to learning of the attention scores for each patch.
These scores can be used as aproxy for patch importance, thus helping in spatial credit assignment.
This way of interpreting MILmodels has been used commonly in the literature to create spatial heatmaps, image overlays thatindicate credit assignment, for free without applying any post-hoc technique [42, 25, 22, 33, 21].
Theattention values that scale patch feature representations have a non-linear relationship to the finalprediction, making their visual interpretation inexact and incomplete.
We discuss these issues indetail in the next section.
To resolve these issues, we propose a simple formulation of MIL which induces intrinsically interpretable heatmaps.
We refer to this model as Additive MIL .
It allows for exact decomposition of amodel prediction in terms of spatial regions of the input.
These models are inspired by Generalized
Additive Models ( GAM s) [15] and Neural Additive Models ( NAM s) [2], but instead of being appliedto arbitrary features, they are grounded as patch instances in the MIL formulation which allows exactcredit assignment for each patch in a bag.
Specifically, we achieve this by constraining the spaceof predictor functions (the classification or regression head at the final layer) in the MIL setup to beadditive in terms of instances.
Therefore, the exact contribution of each patch or instance in a bag canbe traced back from the final predictions.
We show that these additive scores reflect the true marginalcontribution of each patch to a prediction and can be visualized as a heatmap on a slide for variousapplications like model debugging, validating model performance, and identifying spurious features.
We also show that these benefits are achieved without any loss of predictive performance even thoughthe predictor function is now fixed to be additive.
This is critical as the accuracy-interpretabilitytrade-off has been an active area of research and has deep implications for applications in medicalimaging.
Trading off performance for interpretability might make sense for improving validation and in turn adoption of ML tools, however it raises ethical questions about deploying sub-optimal models[31, 23, 9].
Furthermore, since our work is orthogonal to previous advancements in MIL modeling,we show that previous methods can be made additive by a simple switching of function compositionat the last layer of the model, making it applicable to all MIL models where instance-level creditassignment is important.
MIL Models & Interpretability
[Figure]
2.1 Interpretability in Attention MIL
[Figure]
An attention MIL model [16] can be seen as a 3-part model consisting of a featurizer ( f ) , typicallya deep CNN, an attention module ( m ) which induces a soft attention over the N patches and isused to scale each patch feature, and a predictor ( p ) which takes the attended patch representations,aggregates them using a permutation invariant function like sum pooling over the N patches, andthen outputs a prediction.
This MIL model g ( x ) is given as:
[Equation]
α i = softmax i ( ψ m
[Equation]
x
[Equation] [Equation]
[Equation]
where ψ m and ψ p are MLPs with non-linear activation functions.
The attention scores α i learned by the model can be treated as patch importance scores and have beenused previously to interpret MIL models [42, 25, 33, 21].
However, there are four issues in doingspatial attribution using these attention scores.
For example, consider the task of classifying a slideinto benign, suspicious or malignant:
(a) Since the attention weights are used to scale the patch features used for the prediction task, ahigh attention weight only means that the patch might be needed for the prediction downstream.
Therefore, a high attention score for a patch can be a necessary but not sufficient condition for
attributing a prediction to that patch.
Similarly, patches with low attention can be important for the
downstream prediction since the attention scores are related
non-linearly to the final classification
or regression layer.
For example, in a malignant slide, non-tumor regions might get highlighted by
the attention scores since they need to be represented at the final classification layer to providediscriminative signal.
However, this does not imply malignant prediction should be attributed to
non-malignant regions, nor that these regions would be useful to guide a human expert.
(b) The patch’s contribution to the final prediction can be either positive (excitatory) or negative
(inhibitory), however attention scores do not distinguish between the two.
A patch might be
providing strong negative evidence for a class but will be highlighted in the same way as a positivepatch.
For example, benign mimics of cancer are regions which visually look like cancer, but are
normal benign tissue [36].
These regions are useful for the model to provide negative evidence for
the presence of cancer and thus might have high attention scores.
While attending to these regions
may be useful to the model, they may complicate human interpretation of resulting heatmaps.
(c) Attention scores do not provide any information about the class-wise importance of a patch, but
only that a patch was weighted by a certain magnitude for generating the prediction.
In the case
of multiclass classification
, this becomes problematic as a high attention scores on a patch can
mean that it might be useful for any of the multiple classes.
Different regions in the slide might
be contributing to different classes which are indistinguishable in an attention heatmap.
For
example, if a patch has high attention weight for benign-suspicious-malignant classification, it
can be interpreted as being important for any one or more of the classes.
This makes the attentionscores ineffective for verifying the role of individual patches for a slide-level prediction.
(d) Using attention scores to assess patch contribution ignores patch interactions at the classification
stage.
For example, two different tumor patches might have moderate attention scores, but when
taken together in the final classification layer, they might jointly provide strong and sufficient
information for the slide being malignant.
Thus, computing marginal patch contributions for a bag
needs to be done at the classification layer and not the attention layer since attention scores do not
capture patch interactions and thus can under or over estimate contributions to the final prediction.
These difficulties in interpreting attention MIL heatmaps motivate the formulation of a traceablepredictor function, where model predictions can be exactly specified in terms of patch contributions(both positive and negative) for each class.
[Figure]
C l a ss e s
2.2 Additive MIL Models
We first define the desiderata for a visual interpretability method for MIL models:
The method should be intrinsic to the model and not be a post-hoc method.
This prevents incorrectassumptions about the model and does not require post-hoc modeling.
It also prevents manypitfalls of traditional saliency methods [19, 1].
Attribution in the MIL setup should be in terms of instances only.
For pathology, this means thatthe prediction should be attributed to individual patches.
This constraint enables expression ofbag predictions in terms of marginal instance contributions.
Should reflect model’s sensitivity and invariances by reliably mirroring its functioning [20].4.
Should distinguish between excitatory and inhibitory patch contributions.
Should also provideper-class contributions for classification problems.
To enable the desired instance-level credit assignment in MIL , we re-frame the final predictor to bean additive function of individual instances.
This translates to a simple switching of the functioncomposition in Equation 3:
[Equation]
Making this change results in the final predictor only being able to implement patch-additive functionson top of arbitrarily complex patch representations.
This provides both complexity of the learnedrepresentations as well as a traceable patch contribution for a given prediction which solves thespatial credit assignment problem.
ψ p ( m i ( x )) is the class-wise contribution for patch i in the bag.
At inference, ψ produces a R C × N for a classification problem where C is the number of classesand N is the number of patches in a bag.
Thus, we get a class-wise score for each patch, whichwhen summed gives the final logits for the prediction problem.
These scores can be visualized byconstructing a heatmap from the visual representation of patch-wise contributions for each class.
Thesign of the patch contribution decides whether the patch is excitatory or inhibitory towards each classsince positive values add to the final logit while negative values bring down the final class logit.
Inthe next section, we prove that the instance contribution obtained from an Additive MIL model isexactly equivalent to the actual marginal contribution of that patch to a model’s prediction.
2.3 Proof of equivalence between Additive MIL and Shapley Values
We highlight the spatial credit assignment properties of an Additive MIL model by proving itsequivalence to Shapley values.
Shapley value [34] is game theoretic concept used for calculatingthe optimal marginal contribution of each player in a n-player coalition game with a given totalpayoff.
In machine learning, it’s used to interpret models predictions by decomposing them in terms of their marginal feature contributions.
For most practical models, the computational complexitygrows exponentially, thus requiring approximations [26].
The Additive MIL model, g is defined for a MIL problem with instances x
i as input:
[Equation]
where α i is the attention weight for the i th instance, f is the function encoding each instance into afeature representation, and ψ p is predictor function which maps instance representation to the modeloutput (e.g. logits for classification models).
Theorem 1: The marginal instance contribution from an Additive MIL model, g ( x i ) is proportionalto the Shapley value of that instance, ϕ i .
[Equation]
Consequence: Additive MIL scores ensure optimal credit assignment across instances of an MIL bag.
Thus each bag-level prediction in MIL can be exactly decomposed into marginal instance contributionsgiven by Additive MIL scores and provide model interpretability.
Proof: The interpretation for the value function V S is taken from [26] where it’s defined as theexpected value of the model given a specific input set x ∗ S . [Equation] Since the conditional expectation is for the case where only the set S is known, rewriting the equationin the form of integrals and breaking it down by set S and its complement ¯ S gives: [Equation]
[Equation] Equation 9 uses the model definition from equation 5 to express the function g into its linearly additivecomponents over all instances which are either in set S or in ¯ S . Similarly, we can write the valuefunction when the i th index is included in S by removing it from set ¯ S and adding it to S : [Equation]
[Equation]
Since the second term here is the expected value of the model output, we can put this back in equation6 to get an equivalence between the Shapley value and the instance contribution from an Additive
MIL model.
[Equation]
2.4 Features of Additive MIL Models
Exact marginal patch contribution towards a prediction .
Additive MIL models provide exactpatch contribution scores which are additively related to the prediction.
This additive coupling of themodel and the interpretability method makes the spatial scores precisely mirror the invariances andthe sensitivities of the model, thus making them intrinsically interpretable.
Class-wise contributions .
Additive MIL models allow decomposing the patch contributions andattributing them to individual classes in a classification problem.
This allows us to not just assign theprediction to a region, but to also see which class it contributes to specifically.
This is helpful in caseswhere signal for multiple classes exist within the same slide.
Distinction between excitatory and inhibitory contributions .
Additive MIL models allow for bothpositive and negative contributions from a patch.
This can help distinguish between areas which areimportant because they provide evidence for the prediction and those which provide evidence against.
Experiments & Results
3.1 Experimental Setup & Datasets
We perform various experiments to show the benefits of using Additive MIL models for interpretabilityin pathology problems.
Concretely, we show the following results:
• Additive MIL models provide intrinsic spatial interpretability without any loss of predictive performance as compared to more expressive, non-additive models.
• Any pooling-based MIL model can be made additive by reformulating the predictor function andleads to predictive results similar to the original model. • Additive MIL heatmaps yield better alignment with region-annotations from an expert pathologistthan Attention MIL heatmaps.
• Additive MIL heatmaps provide more granular information like class-wise spatial assignment andexcitatory & inhibitory patches which is missing in attention heatmaps.
This can be useful forapplications like model debugging.
We consider 3 different datasets and 2 different predictions for our experiments.
The first problem isthe prediction of cancer subtypes in non-small cell lung carcinoma ( NSCLC ) & renal cell carcinoma( RCC ), both of which use the TCGA dataset [40].
The second problem is the detection of metastasisin breast cancer using the Camelyon16 dataset [5].
TCGA RCC contains 966 whole slide images( WSI s) with three histologic subtypes - KICH (chromophobe RCC ), KIRC (clear cell RCC ) and KIRP(papillary RCC ).
TCGA NSCLC has 1002 WSI s, with 538 slides belonging to subtype LUAD (Lung
Adenocarcinoma) and 464 to LUSC (Lung Squamous Cell Carcinoma).
{+Camelyon16 contains 
+}
WSI s for training and 129 for testing with a total of 159 malignant slides and 237 benign slides.
3.2 Implementation Details
Both TCGA datasets were split into 60/15/25 (train/val/test) as done previously [33] while ensuringno data leakage at a case level.
For Camelyon16, we use the original splits provided with the dataset.
For training the models, a bag size of 48 - 1600 patches and batch size of 16 - 64 was experimentedwith and the best one chosen using cross-validation.
The patches were sampled from non-backgroundregions for all datasets at a resolution of 1 microns per pixel without any overlap between adjacentpatches.
An ImageNet pre-trained Shufflenet [27] was used as the feature extractor and the entiremodel was trained with ADAM optimizer and a learning rate of 1 e- 4 .
For inference, multiple bagpredictions were aggregated using a majority vote to get the final slide-level prediction.
AUROC (areaunder the receiver operating curve) scores were generated using the proportion of bags predicting themajority label as the class assignment probability.
For TCGA - RCC , we compute macro average of1-vs-rest AUROC across the 3 classes.
The attention scores were obtained by directly taking the rawoutputs for each patch from the attention module.
For additive patch contributions, the patch-wiseclass contributions were taken and converted to a bounded patch contribution value using a sigmoidfunction.
Both the attention and additive patch-wise scores were used for generating a heatmap as anoverlay on the slide with Attention MIL heatmaps having a single value per patch and Additive MILheatmaps having C values per patch where C is the number of classes.
All training and inferenceruns were done on Quadro RTX 8000 , and it takes 3 to 4 hours to train the model with four GPU s. [Table]
3.3 Experimental Results3.3.1 Predictive Performance of Additive MIL Models & Applicability to Previous Methods
We compare Additive MIL models with existing techniques in terms of predictive performance on3 different datasets.
We implement the standard Attention MIL model ( ABMIL ) and a transformerbased MIL model, Trans MIL which is the state-of-the-art on these three datasets (for comparisonof Trans MIL with other methods, refer [33]).
Table 1 shows how Additive MIL models achievecomparable or superior performance to the standard Attention MIL model.
In the case of improvedperformance, we hypothesize that the additive constraint regularizes the model and limits overfittingin comparison to previous approaches.
This is particularly relevant to pathology datasets that oftenhave less than one thousand slides.
The results in the table also demonstrate how previous techniqueslike Trans MIL can be made additive by switching the function composition of the classifier layeras done in Equation 3 and 4.
This property is general, thus any high performing MIL method canbe converted to an Additive MIL model.
Implementing the additive formulation gives nearly allthe benefits of modeling complexity from previous methods, while enabling spatial interpretabilitywithout any loss of predictive performance.
3.3.2 Region-level Alignment of Additive Heatmaps with Expert Pathologist
Here we compare heatmaps obtained through Additive MIL models with attention heatmaps andevaluate both against region-level annotations from an expert pathologist.
We use the Camelyon16dataset where the goal is to classify the slide as benign or malignant.
Since the cancer foci are verylocalized and often occupy less than 1% of the slide, the task of generating localized cancer heatmapsin a weakly supervised setup is very challenging.
We obtained exhaustive segmentation annotationsfor cancer regions from a board-certified pathologist on the Camleyon16 test set.
We trained an
Additive MIL model and generated both the traditional attention heatmaps using the patch-levelattention scores and Additive MIL heatmaps using the patch contributions.
We compare the patchlevel precision-recall curves at different thresholds of the heatmap.
Note that this comparison controlsfor model performance as both heatmaps are generated from the same model.
Figure 2 shows thiscomparison.
At low thresholds, nearly all patches are highlighted and we see a high recall and lowprecision for both methods.
As we increase this threshold we see higher precision and lower recall.
The Additive MIL heatmaps ( AUPRC 0 . 42 ) highlighted cancer regions more precisely and sensitivelythan traditional attention heatmaps ( AUPRC 0 . 36 ), which detect more false-positives (Figure 2).
If wechoose the best operating point of both the curves, we find that the best F1 score for the attentionheatmap is 0 .
43 as compared to 0 .
47 from the Additive heatmap.
These experiments demonstrate thesuperior performance of Additive MIL heatmaps in localizing areas of interest. 3.3.3 Faithful Representation of Patch-level Contributions to Slide-level Predictions
Attention heatmaps are often used to signal regions of interest in a slide, however as argued in
Section 2.1, it is not straightforward to draw conclusion regarding the importance and contribution of‘attended’ areas towards the model prediction.
Additive MIL guarantees that each patch’s contributionis linear and thus faithfully represents its marginal contribution toward the slide-level prediction.
Thisproperty is shown in figure 3, where the linear relationship of Additive MIL (top row) is clear.
Incontrast, when considering the attention scores [Figure] [Figure] X
[Figure]
 R I
[Figure]
D W F
[Figure]
 O H Y H O
 /RJ L W V
[Figure]
3.3.4 Qualitative Assessment of Multi-Class & Excitatory-Inhibitory Heatmaps
We highlight the benefits of having class-wise excitatory-inhibitory contributions for each spatialregion in a slide.
Figure 4 shows zoomed-in regions of two slides from TCGA RCC & NSCLC .
In theseexamples, the attention heatmaps do highlight tissue regions predictive of the cancer subtype butdon’t provide information about the association of patches to classes.
In contrast, the Additive MILheatmaps show precisely how each patch contributes to each class, and in turn the final prediction.
Figure 5 shows the information about excitatory & inhibitory patches for different classes.
The
Additive MIL heatmaps for each class are visualized by the same colorbar where red denotes excitatorypatches & blue denotes inhibitory ones.
The RCC WSI is labeled as KIRC , but the selected regioncontains two subtypes, namely KIRC & small regions of KIRP , as evident from the raw slide.
The
Additive MIL heatmaps accurately show bottom right region being excitatory for KIRC , but inhibitoryfor the other two whereas the top left region is only excitatory for KIRP and inhibitory for twoother two.
All patches are correctly inhibitory for KICH .
Such granularity in heatmaps is helpfulin understanding how the model arrives at a prediction and can prove to be useful for practitionersbuilding the models as well as physicians using them. [Figure] [Figure] [Figure]
3.3.5 Model Debugging Using Additive MIL Heatmaps
The ability of Additive MIL models to perfectly reflect model predictions at a patch-level can beuseful in model debugging.
Here, we show examples of the cases where Additive MIL heatmapsidentify reasons for model failures during our experiments.
Figure 6 shows two such cases.
Theseheatmaps not only provide interpretability to MIL models but can also aid in validating specifichypothesis during model debugging.
3.3.6 Limitations
Since the interpretation of Additive MIL models is based on model predictions that are reformulatedto be interpretable, the interpretability method is inherently coupled with the model.
This is desirablesince the heatmaps now exactly track model predictions, but this coupling also potentially limits theflexibility of models and heatmaps.
For example, since the patch contributions are tied to the model,one can only generate heatmaps with patches at the same resolution as the model training.
Anotherlimitation is the reduction of model expressivity introduced by the additive constraint.
In this study,we did not find a practical example of this limitation, however, it may exist in other datasets.
Conclusion & Broader Impact
We propose a simple reformulation of the popular Attention MIL setup for pathology that makesmodels intrinsically interpretable through an additive function.
Our approach enables exact spatialcredit assignment where the final prediction of the model can be attributed to individual contributionsof each patch in a pathology slide.
These models provide spatial interpretability without any lossof predictive performance and can be used for various applications like model debugging andhighlighting regions-of-interest in a decision-support setting.
This high fidelity interpretability willbe critical in building trust for these models when deployed in medical decision-making.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experimentsmultiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e.g., type of GPU s,internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]