SoftCore: Unsupervised Anomaly Detection with Noisy Data
Abstract
Although mainstream unsupervised anomaly detection (AD) algorithms performwell in academic datasets, their performance is limited in practical application dueto the ideal experimental setting of clean training data.
Training with noisy datais an inevitable problem in real-world anomaly detection but is seldom discussed.
This paper considers label-level noise in sensory anomaly detection for the firsttime.
To solve this problem, we proposed a memory-based unsupervised ADmethod, SoftCore, which efficiently denoises the data at the patch level.
Noisediscriminators are utilized to generate outlier scores for patch-level noise elimination before coreset construction.
The scores are then stored in the memorybank to soften the anomaly detection boundary.
Compared with existing methods, SoftCore maintains a strong modeling ability of normal data and alleviates theoverconfidence problem in coreset.
Comprehensive experiments in various noisescenes demonstrate that SoftCore outperforms the state-of-the-art AD methods onthe MVTec AD benchmark, and is comparable to those methods under the settingwithout noise.
Introduction
Detecting anomalies by only nominal images without annotation is an appealing topic, especiallyin industrial applications where defects can be extremely tiny and hard to collect.
Unsupervisedanomaly detection (UAD) is proposed to solve this problem and has been largely explored.
Recentmethods[1; 2; 3; 4] usually model the AD problem as a one-class learning problem where a cleannomial training set is provided to extract representative nominal features.
To determine whether asample differs from the standard dataset, most previous unsupervised AD methods have to measurethe distance between the test sample and the standard dataset distribution.
For instance, Defardet al.
[2] described the CNN features of standard data with a multivariate Gaussian distribution andlearned the Gaussian parameters.
Roth et al.
[3] established a memory bank using the CNN featuresof standard data and utilized the nearest neighbor search to find the distance between the test sampleand the normal data.
Even though recent methods have achieved excellent performance, they all rely on the clean training set to extract nominal characteristics for later comparison with anomalous features.
If the standard normal dataset is polluted with noisy data, i e., the abnormal samples, the estimated boundary will be unreliable and the classification for abnormal data will have low accuracy
In general, current unsupervised AD methods are not designed for and are not robust to noisy data
However in real-world practice it’s inevitable that there are noises that sneak into the standard normal dataset especially for industrial manufacturing where a large number of products are produced daily
This noise usually comes from inherent data shift or human misjudgment.
Meanwhile, no existingunsupervised AD method can be directly applied to solve the problem of abnormal detection withnoisy data.
As in Fig.
1, noisy samples easily misinform AD algorithms, so algorithms misclassifysimilar anomaly samples in the test set and generate wrong locations.
In this paper, we first propose a more practical problem setting, namely the abnormal detection withnoisy data, which is a more valuable scenario but seldom investigated.
Our solution is inspired byone of the recent state-of-the-art methods, PatchCore [3].
PatchCore proposed a method to subsamplethe original CNN features of the standard normal dataset with the nearest searching and establisha smaller coreset as a memory bank.
However, the coreset selection and classification process arevulnerable to polluted data.
As shown in Fig.
3, the performance of PatchCore decreases sharplywith the increasing noise rate.
In this regard, we propose a patch-level selection strategy to wipeoff the noisy image patch of noisy samples.
Compared to conventional sample-level denoising,the normal image patches of a noise sample are exploited in coreset.
Meanwhile, the denoisingalgorithm assigns an outlier score factor to each patch to be selected into coreset.
Based on thepatch-level denoising, we propose a novel AD algorithm with better noise robustness named SoftCore.
Considering noisy samples are hard to be removed completely, SoftCore utilizes the local outlierfactor to re-weight the coreset examples, which makes the coreset become a soft core.
Patch-leveldenoising and re-weighting the coreset samples are proved effective in revising misaligned knowledgeand alleviating the overconfidence in coreset in inference.
Extensive experiments in various noisescenes demonstrate that SoftCore outperforms the state-of-the-art AD methods on MVTec Anomaly Detection (MVTecAD) [5] benchmark.
Our main contributions are summarized as follows: 1) To the best of our knowledge, we are the firstto study the image sensory anomaly detection with noisy data, which is a more practical setting butseldom investigated.
We propose a patch-level denoising strategy for coreset memory bank, whichlargely improves the data usage rate compared to conventional sample-level denoising.
We proposea novel SoftCore method to classify normal and abnormal samples based on the proposed denoisingstrategy, which assigns a outlier score factor to each patch in the coreset.
SoftCore utilizes the localoutlier factor to re-weight each patch in the coreset.
Re-weighting the coreset samples alleviates theoverconfidence problem on coreset.
We set a baseline for unsupervised AD with noisy data, whichoutperforms most of existing unsupervised AD methods under the setting of noisy data, as well ascomparable to these methods under the setting without noise.
Related Work2.1 Learning with Noisy Labels
Noisy label recognition is becoming an emerging topic for supervised learning but has rarely beenexplored in unsupervised anomaly detection.
For classification, some research [6; 7] propose to filter noisy pseudo-labeled data with a high confidence threshold.
Li et al.
[8] selects noisy-labeleddata with a mixture model and trains in a semi-supervised manner.
Kong et al.
[9] relabel harmfultraining samples.
For object detection, multi-augmentation [10], teacher-student [11], or contrastivelearning [12] are adopted to alleviate noise with the help of the expert model’s knowledge.
However,current noisy label recognition methods all rely on labeled data to co-rectify noisy data.
In comparison,we target to improve the model’s noise robustness in an unsupervised manner without introducinglabor annotations.
2.2 Unsupervised Anomaly Detection
Training with agent tasks , also known as self-supervised learning, is a viable solution when thereis no category and shape information of anomalies.
Sheynin et al.
[13] employ transformationssuch as horizontal flip, shift, rotation, and gray-scale change after a multi-scale generative model toenhance the representation learning.
However, Li et al.
[14] mention that naively applying existingself-supervised tasks is sub-optimal for detecting local defects.
So they propose a novelty agenttask named CutPaste, which simulates an abnormal sample by clipping a patch of a standard imageand pasting it back at a random location.
Similarity, DRAEM [15] synthesizes anomalies through
Perlin Noise.
Nevertheless, the inevitable discrepancy between the synthetic anomaly and the realanomaly disturbs the criteria of the model and limits the generalization performance.
The gap betweenanomalies is usually larger than that between anomaly and normal.
This is why AD methods deceivedby some noisy samples can still work well when handling other kinds of anomalies.
Knowledge distillation is used in anomaly detection in an ingenious way.
It is from a theory [16]that the representations of unusual patches are different between a pretrained teacher model and astudent model, which tried its best to simulate teacher output in the training stage.
Based on thistheory, Salehi et al.
[17] propose that considering multiple intermediate outputs in distillation andusing a smaller student network lead to a better result.
This improvement shows that restricting thegeneralization ability of the student model helps distinguish anomalies.
Reverse distillation [18] usesa reverse flow which avoids the confusion caused by the same filters and prevents the propagation ofanomaly perturbation to the student model.
However, too long a training stage limits its usage.
Feature modeling here specifically refers to the direct estimation of the output features of theextractor, including distribution estimation [2], reversible transformation [19; 20] and storage operations [1; 3].
PaDiM [2] utilize multivariate Gaussian distributions to estimate the patch embedding ofnominal data.
In the inference stage, the embedding of irregular patches will be out of distribution.
Itis a simple but efficient method, but Gaussian distribution is inadequate for more complex data cases.
So to enhance the estimation of density, DifferNet [19] and CFLOW [20] leverage the reversiblenormalizing flows based on multi-scale representation.
Hou et al.
[1] proposed that the granularityof division on feature maps is closely related to the reconstruction capability of the model for bothnormal and abnormal samples.
So a multi-scale block-wise memory bank is embedded into an autoencoder network as a model of past data.
PatchCore [3] is a more explicit but valuable memory-basedmethod, which stores the sub-sampled patch features in the memory bank and calculates the nearestneighbor distance between the test feature and the coreset as an anomaly score.
Although PatchCoreis outperformance in the typical setting, it is overconfident in the training set, which leads to poornoise robustness.
Except above, there are other kinds of related methods in AD, such as Pre-trained
Model Adaption [21] and Image Reconstruction [4; 22].
The Proposed Method
3.1 Overview
Unsupervised anomaly detection aims to automatically classify and segment defect areas by onlytraining on nominal images.
During the training process, the implicit label information is utilizedsince we assume that all training images are nominal.
In this paper, we focus on the situation wherethis assumption does not always hold.
N × H × W × C ...
Random Linear Projections
N × h* × w* × c*
N
N w h c* ......... h* × w* positions ... Top are removed N × h* × w* Coreset sampling ( Farthest Point Sampling )
Training samples
[Figure]
Feature Extractor
Pretrained
[Figure]
Dimension Reduction
Patch features
Patch feature set at the same position ...
c*
N ... c*
N
Noise discriminator LOF or Gaussian Nearest or
Patch-level noise discrimination at each position
Patch-level denoising among all positions
[Figure]
Neighbor
[Figure]
N × h*
[Figure]
[Figure]
The target of image-level denoising is to find X noise from X , where X
= { x i : i ∈ (1 , ..., N ) , x i ∈ 117 R C × H × W } denotes training images (channels C , height H , width W ), X nominal and X noise denotesnominal and noisy images in training set respectively.
Patch-based unsupervised anomaly methods,such as [3], have three main processes: feature extraction, coreset selection with memory bankconstruction, and anomaly detection.
One of the important assumptions in [3] is that the training setonly contains nominal images, and the coreset should have full coverage of all training data.
Duringthe test, An incoming image will directly search in the memory bank for similar features, and theanomaly score is the dissimilarity with the nearest patches.
Unfortunately, in real-world applications,noise always exists.
The searching process may collapse if the assumed clean full coverage memorybank contains noise.
Therefore, we propose SoftCore which filters noisy data by a noise discriminatorbefore coreset construction and softens the searching process for down-weighting the hard unfilterednoisy samples.
Following convention in [3], we use ϕ i ∈ R c ∗ × h ∗ × w ∗ as the feature map (channels c ∗ , height h ∗ ,width w ∗ ) of image x i ∈ X , ϕ i ( h, w ) ∈
R c ∗ as the patch at ( h, w ) on the aggregated feature mapwith dimension c .
3.2 Noise Discriminative Coreset Selection
With increasing training images, the online memory bank can become exceedingly large and infeasiblefor inference and GPU/CPU storage.
[3] uses min-max facility location to select coreset withoutconsidering noisy samples during selection.
Therefore, we propose three noise reduction methods forcoreset selection to fill in the blanks of noise unsupervised anomaly detection.
3.2.1 Nearest Neighbor
We set Nearest neighbor distance as our baseline [23] where a large distance means an outlier.
Withthe assumption that the amount of noisy samples X noise is much less than clean samples X nominal ,the neighbor is defined as the feature memory of the corresponding position of each patch instead of the whole coreset as in PatchCore.
Given a batch of images, ϕ ∈ R b ∗ × c ∗ × h ∗ × w ∗ represents batchfeature where b ∗ means batchsize.
ϕ i ( h, w ) ∈ R c ∗ is patch feature at position ( h, w ) on the featuremap of image i with dimension c .
Each patch’s nearest neighbor distance W nni is defined as:
[Equation]
We first calculate patch distance, then take the minimum among batch dimensions (neighbor) andkeep top τ patches sorted by W nn .
This method can discriminate apparent outliers but suffer fromuneven distribution of different clusters, where some clusters can have large inter-distance and leadto mistakenly threshed as noisy data.
To treat all clusters equally, we propose another multi-variate Gaussian method to calculate the cleanness score without the interference of different clusters’densities.
3.2.2 Multi-Variate Gaussian
With Gaussian’s normalizing effect, all clean images’ characteristics can be treated equally.
To apply
Gaussian distribution on image characteristics dynamically, we calculate the inlier probabilities onthe batch dimension for each patch ϕ
i ( h, w ) , similar to 3.2.1.
The multi-variate Gaussian distribution 153 N ( µ h,w , Σ h,w ) can be formulated that µ h,w is the batch mean of ϕ
i ( h, w ) and sample covarianceΣ h,w is:
[Equation]
where the regularization term ϵI makes (cid:80) h,w full rank and invertible [2].
Finally, with the estimatedmulti-variate Gaussian distribution N ( µ h,w , (cid:80) h,w ) , Mahalanobis distance is calculated as the noisymagnitude W mvgi ( h, w ) of each patch:
[Equation]
High Mahalanobis distance means high anomalous score, and we only keep top τ patches sortedby W mvgi ( h, w ) .
Even though Gaussian distribution normalizes and captures the essence of imagecharacteristics, small feature clusters may be overwhelmed by large feature clusters.
In the scenarioof a prominent feature cluster and a small cluster in a batch, the small cluster may be out of 1-,
2- or3- σ of calculated N ( µ h,w , (cid:80) h,w ) and erroneously classified as outliers.
After analyzing the abovetwo methods, we need a method that can: 1.
treat all image characteristics equally; 2.
treat large andsmall clusters equally; 3.
high dimension calculation applicable.
3.2.3 Local Outlier Factor (LOF)
LOF[24] is a local-density-based outlier detector used mainly on E-commerce for criminal activitydetection.
Inspired by LOF, we can solve above mentioned three questions in 3.2.2: 1.
Calculating therelative density of each cluster can normalize different density clusters; 2.
Using local k-distance[24]as a metric to alleviate the overwhelming effect of large clusters; 3.
Modeling distance as normalizedfeature distance d ( ϕ i ( A ) , ϕ i ( B ))
=
ϕ
i ( h, w ) ∗ ϕ
i ( h, w ) T can be used on high dimensional patchfeatures.
Therefore, the k-distance-based absolute local reachability density lrd i ( h, w ) is firstcalculated as:
[Equation] [Equation]
(5) where A/B are two locations ( h, w ) on feature map, dist k
( ϕ i ( B )) is the k-distance of B and B is
A ’s top-k nearest neighbor.
With local rechability density of each patch, the overwhelming effect oflarge clusters is largely reduced.
To normalize local density to relative density for treating all clustersequally, the relative density W LOFi of image i is defined below:
[Equation]
[Equation]
where A = ( h, w ) : h ∈ h ∗ , w ∈ w ∗ .
W LOFi ( A ) is the relative density of the neighbors over patch’sown[24], and represents as a patch’s the confidence of inlier.
We also keep top τ patches sortedby W LOFi .
Our experiments found that all three noise reduction methods above are helpful in datapre-selection before coreset construction, while LOF provides the best performance.
However, aftervisualization of our cleaned training set, we found that hard noisy samples, which are similar tonominal samples, are still hidden in the dataset.
To further alleviate the effect of noisy data, wepropose a soft re-weighting method that can down-weight noisy samples by anomalous level.
3.3 Anomaly Detection based on SoftCore
Besides the construction of the Coreset, outlier factors of all the selected patches are stored as softweights in the memory bank.
With the denoised patch-level memory bank M as shown in figure 2,the image-level anomaly score s ∈ R can be calculated for a test sample x i ∈ X test by nearestneighbor searching at patch level.
Denoting the collection of patch features of a test sample as
P ( x i ) =
P i,j ( ϕ j ( x i )) , for each patch p i,j ∈ P x
i the nearest neighbour searching can be formulatedas the following equation:
[Equation]
After nearest searching, pairs of test patch and its corresponding nearest neighbor in M can beachieved as ( p, m ∗ ) .
For each patch p i,j ∈ P x i , the patch-level anomaly score is calculated bys ij = W m ∗ i,j ∥ p i,j − m ∗ i,j ∥ 2 .
The image-level anomaly score is attained by finding the largest softweights re-weighted patch-level anomaly score:
[Equation]
Different from PatchCore [3] which directly considers patches equally, SoftCore softens anomalyscores by noisy level from noise discriminater.
The soft weights, i.e., local outlier factors, haveconsidered the local relationship around the nearest node.
Thus, a similar effect can be achieved as
PatchCore
but with more noise robustness and fewer searches.
According to the image-level anomalyscore, a sample is classified into a normal sample or abnormal sample.
Experiments
4.1 Experimental Details
Dataset.
Following [3], our experiments are mainly conducted on the MVTec Anomaly Detectionbenchmark[5].
MVTecAD contains 15 categories with 3629 training images and 1725 test imagesin total.
Since each category of MVTecAD is divided into nominal-only images and a test set withboth nominal and anomalous samples, to create a noisy training set, we sample anomalous imagesrandomly from the test set and mix them with the existing training images.
The sampled anomalousimages are removed from the test set in test time while the other images remain untouched.
In thissetting( No overlap ), the injected anomalous samples will not be evaluated, which is more likelythe case in the real application.
However, we still construct a different setting( Overlap ) where theinjected anomalous samples are also in the test set to demonstrate the risk that defects with similarappearance will severely exacerbate the performance of an anomaly detector trained with noisy data.
By controlling the proportion of negative samples being injected into the train set, we obtain severalnew datasets with different noise ratios dubbed MVTecAD-noise- n , where n refers to the ratio ofnoise.
Evaluation Metrics.
We report both image-level and pixel-level AUROC for each category in
MVTecAD and average them to get average image/pixel level AUROC.
In order to represent noiserobustness, the performance gaps between noise-free data and noisy data are also displayed.
Implementation Details.
We test three SOTA AD algorithms, PatchCore [3], PaDim [2] and
CFLOW [20] in noise scene and follow their main settings.
In the absence of specific instructions,
[Table]
screw
[Table]
toothbrush
[Table]
zipper
[Table]
Average
[Table] [Table]
the backbone of feature extractor is Wide-ResNet50 and the coreset sampling ratio of PatchCore and
SoftCore is 0.1.
For MVTecAD images, we only use 256 × 256 resolution and center crops theminto 224 × 224 along with a normalization.
No other data augmentation is applied since it requiresprior knowledge of the class, such as whether there is rotation in this class.
We train a separate modelfor each MVTec class.
The threshold τ in SoftCore and the LOF-K are set to 0.15 and 6 for allnoisy scenarios and classes.
The effects of hyperparameters are studied in the ablation study.
Allour experiments are run on Nvidia V100 GPU, costing roughly one minute for each category in
MVTecAD.
4.2 Anomaly Detection Performance with Noise
As indicated in Table 1 and Table 2, when 10% of anomalous samples are added to corrupt the trainset, all existing methods have different extend of performance decrease, although not disastrouslyunder No overlap setting.
Compared to other methods, the proposed SoftCore exhibits much strongerrobustness against noisy data both in terms of anomaly detection and localization, no matter whichnoise discriminator is used.
Among three variants of SoftCore, SoftCore-lof achieves the best overallperformance with the highest accuracy and strongest robustness.
Interestingly, under the Overlapsetting, PaDiM[2], CFLOW[20] and SoftCore-gaussian show significantly less performance drop than
PatchCore, which indicates that modeling feature as Gaussian distribution does help denoising.
Whilemodeling feature distribution at each spatial location as a single Gaussian distribution can’t handlemisaligned images, such as screw class in MVTecAD, which explains the poor performance on theseclasses(see screw row).
On the other hand, PatchCore’s greedy-sampling strategy is a double-edgedsword with higher feature space coverage and higher sensitivity to noise.
That’s why using randomsampling in PatchCore is more robust with compromised performance(see PatchCore 1%-Random column).
SoftCore-nearest does a slightly better job in the misaligned cases.
However, it doesn’t takefeature distribution into account, which leads to inferior performance.
In order to explore how different methods behave with the increasing noise level, experiments arefurther performed on MVTecAD-noise- { 0 ∼ 0 . 15 } .
The results are shown in Figure 3.
Besides thedefault noise setting( No overlap ), we also demonstrate the Overlap setting where the anomaloustraining images are included in the test set.
Under the No overlap setting, as the noise ratio increases,
PatchCore shows a pixel-level AUROC drop up to 3.7%.
The performance decreases as the noise ratiorises.
On the contrary, although the default performance is slightly poor than PatchCore(about 0.006and 0 decrease in image-level and pixel-level AUROC), the proposed SoftCore-lof deteriorates muchslower, which demonstrates better denoising ability.
As for SoftCore-nearest and SoftCore-gaussian,they are also more robust, however, with worse base performance(see Figure 3 at noise ratio=0).
Thevisualization of the coreset in Figure 5 also shows that random sampling avoids sampling the outlierbut can not model normal adequately.
Being consistent with the discussion above, under the Overlapsetting, PatchCore’s performance is getting worse and worse catastrophically(up to 40% AUROCdrop in both image and pixel level) as more noises are added.
This is expectable since PatchCoreuses a greedy strategy for coreset sampling, which favors outlier in feature space.
SoftCore-lofconsistently outperforms other methods with no significant performance drop as the noise level goesup.
The experimental results indicate that the risk is hidden by the fact that defects in MVTecADhave very different appearances.
In this case, even if some anomalous features are added mistakenlyto the coreset, they are unlikely to be retrieved during test time.
However, the risk still exists andwill be triggered when similar defects show up at test time.
More experiment details can be found in
Appendix.
[Figure]
[Table]
[Table]
4.3 Ablation Study
4.3.1 Effectiveness of the Proposed Modules
We validated the effectiveness of two proposed modules noise discriminator and soft weightby removing them from the pipeline.
As shown in Table 3, the noise discriminator significantlyimproves the noise robustness in terms of pixel-level AUROC.
Among three decision choices of noisediscriminator, LOF achieved the best balance between robustness and capacity, resulting in the most
I m ag e
L e v e l
AUR O C
Threshold
Overlap
No overlap 0.
P i x e l L e v e l
AUR O C
Threshold
Overlap
No overlap performance boost under all settings. We further analyzed the intermediate results by visualizingthe sampled coreset of different methods, which shows that SoftCore-LOF sampled much feweranomalous features than the baseline(see Figure 5).
Soft weight is used alongside noise discriminatorto further improve the final results.
We only observed minor improvement for using Soft weight in
SoftCore-Nearest.
We suspect that the other two kinds of noise discriminators are already robustagainst noise data.
[Table]
4.3.2 Parameter Selection
[Table]
To explore the impact of two parameters ( LOF-k and threshold τ ) on the final performance, weperform parameters searching on our method.
As in Table 4, our method achieves better performancewhen LOF-k is greater than 5, which suggests that our method is not sensitive to LOF-k , as long as itis not too small or too large.
If LOF-k is too small, it fails to estimate the local density accuratelybecause too few neighbors are considered.
On the contrary, a large LOF-k may lead to undesirablecross-clusters connection that can not capture real data distribution.
Threshold τ refers to the ratio of eliminated patch features when building coreset.
Figure 4 indicatesan increasing trend of AUROC as threshold τ increases under Overlap setting, which is expectedsince a higher threshold means a more aggressive denoising strategy.
Under Overlap setting, themistakenly sampled features are the direct reason for the drastic performance drop.
Therefore moreaggressive denoising improves the result significantly.
However, under No Overlap setting, theeffect of the noisy feature is less prominent.
Although the best LOF-k and threshold τ are changedaccording to the class and noise level, we simply use fixed values, 6 and 0.15, in all situations.
Conclusions
This paper emphasizes the practical value of investigating noisy data problems in unsupervised AD.
Introducing a novel noisy setting on the previous task, we test the performance of existing methodsand SoftCore.
For existing methods, despite no adaptation to noisy settings, some of them have aslight performance decrease in some scenes.
However, the performance decrease could be moresignificant and catastrophic for other methods or in other scenes.
For the proposed SoftCore, althoughperformance degrades slightly compared with the SOTA result in the no-noise situation.
It showsconsistent performance in all noise settings, which outperforms other methods.
Industrial inspection systems are an important computer vision application that requires good robustness.
The noise injected into the training set break with the naive assumption that the trainingsamples were normal.
Noise also gives the model an early exposure to the distribution of anomalies.
The unsupervised AD with noisy data needs more research in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] In the Experiments and Conclu- sions sections, we state that our method improve robustness at cost of slight performance
decrease for some categories.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?
[N/A] [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/
[Table]
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation
[N/
[Table]