ElasticMVS: Learning elastic part representation for self-supervised multi-view stereopsis
Abstract
Self-supervised multi-view stereopsis (MVS) attracts increasing attention for learning dense surface predictions from only a set of images without onerous groundtruth 3D training data for supervision.
However, existing methods highly rely onthe local photometric consistency, which fail to identify accurately dense correspondence in broad textureless or reflectant areas.
In this paper, we show thatgeometric correlations such as surface connectedness and occlusion boundariesparsed from images could serve as a reliable guidance for pixel-wise multi-viewcorrespondences.
With this insight, we present a novel elastic part representation,which encodes physically-connected part segmentations with elastically-varyingscales, shapes and boundaries.
Meanwhile, a self-supervised MVS frameworknamely ElasticMVS is proposed to learn the representation and estimate per-viewdepth following a part-aware propagation and evaluation scheme.
Specifically, thepixel-wise part representation is trained by a contrastive learning-based strategy,which increases the representation compactness in geometrically concentrated areasand contrasts otherwise.
ElasticMVS iteratively optimizes a part-level consistencyloss and a surface smoothness loss, based on a set of depth hypotheses propagated from the geometrically concentrated parts.
Extensive evaluations convey thesuperiority of ElasticMVS in the reconstruction completeness and accuracy, aswell as the efficiency and scalability.
Particularly, for the challenging large-scalereconstruction benchmark, ElasticMVS demonstrates significant performance gainover both the supervised and self-supervised approaches.
Introduction
Given a set of images with known camera parameters, multi-view stereopsis (MVS) aims to reconstruct the dense and accurate geometry of the scene.
Existing learning-based approaches (Yaoet al., 2018), typically being cast as an end-to-end depth regression task, outperform the traditionalgeometry-based ones (Schönberger et al., 2016).
However, their generalization ability in open environments is severely hindered by the availability of the onerous laser-scanned 3D training data.
Self-supervised MVS (Khot et al., 2019; Dai et al., 2019) lifts such limitation by leveraging themulti-view photometric consistency in place of the supervisory signals.
Unfortunately, the pixel-levelphotometric regularization is susceptible to textureless patterns and illumination variations, leadingto incomplete and inaccurate reconstructions, especially in outdoor environments.
In this paper, we tackle the above challenges with a novel image parsing paradigm.
Instead oftreating images as projections of 3D scenes, we show that the inherent geometric correlations,such as surface connectedness, smoothness and boundaries, could be inferred from images andserve as a reliable guidance for pixel-wise multi-view correspondence estimation.
The mainstreamsolutions of feature-based regularizations introduce either hand-crafted structures such as lines and planes from superpixels (Romanoni and Matteucci, 2019; Bódis-Szomorú et al., 2014; Micusíkand Kosecka, 2010), or semantic features pre-trained from limited geometric-unrelated labels (Xuet al., 2021; Hung et al., 2019).
Different from those, we aim to build the correlation betweenthe latent space of the representation and the scene space of the surface in a self-supervised way.
With this insight, we present a novel elastic part representation which encodes physically-connectedpart segmentations with elastically-varying scales, shapes and boundaries.
Unlike the traditionalsegmentation networks with a fixed number of labels, our pixel-wise part representation directlyreflects the surface correlation, i.e., the more similar the two representations are, the higher probabilitythey belong to the same physical surface part.
Meanwhile, a self-supervised MVS framework namely
ElasticMVS is proposed to learn the representation and estimate per-view depth iteratively followinga part-aware propagation and evaluation scheme.
Benefiting from the elastic part representation, the inherent geometric correlations are effectivelyexploited by a patchmatch-based propagation strategy (Galliani et al., 2015).
Specifically, in eachiteration, depth is estimated by selecting the best depth hypotheses propagated only from the geometrically concentrated part.
Besides, the pixel-level photometric consistency loss is enhanced bythe constraints of feature-level correspondence and surface smoothness, which provide part-awarematching cues to guide a more robust depth prediction in textureless and reflectant areas.
To train theelastic part representation, a contrastive learning-based strategy is implemented by increasing therepresentation compactness in geometrically concentrated areas and contrasts otherwise.
Withoutthe ground-truth 3D information, these geometrically concentrated areas are identified by the depthmap generated from the initialized ElasticMVS without the part representation, which owns the sameworkflow as the traditional geometry-based stereopsis (Schönberger et al., 2016).
A noise-robusttraining loss is implemented to get rid of undesirable identifications.
Experiments on ‘DTU’ and ‘Tanks and Temples (T&T)’
datasets demonstrate that the proposed
ElasticMVS significantly outperforms all state-of-the-art self-supervised learning-based methods.
Remarkably, in the challenging large-scale reconstruction benchmark (T&T advanced), ElasticMVSshows dramatic performance improvements w.r.t.
both the supervised and self-supervised approaches.
Besides, the systematic evaluations indicate that our elastic part representation is superior in modelingelastically-varied parts, while being robust and lightweight in training and inference.
Related work2.1 Multi-view Stereopsis (MVS)
The major works in multi-view stereopsis focus on the view-wise depth map estimation (Tola et al.
,2012; Furukawa and Ponce, 2010) and then fuse together to form the point cloud(Zach, 2008; Goesele et al., 2007).
Among these methods, patchmatch-based depth optimization methods(Galliani et al.
,2015; Schönberger et al., 2016; Xu and Tao, 2019), initialized from the randomly sampled depthhypotheses, show great run-time and memory efficiency in depth map estimation.
Some learningbased methods (Wang et al., 2021a; Lee et al., 2021) introduce an iterative coarse-to-fine optimizationin an end-to-end trainable architecture to improve each of the core steps in patchmatch.
However,these algorithms suffer from incomplete and noisy depth predictions in textureless and reflectantareas.
Currently, learning methods based on 3D cost volume regularization demonstrate state-of-the-artresults in several benchmarks.
These methods optimizes the 3D geometry by 3D CNN, unprojectingfrom 2D images or features into the 3D volume (Ji et al., 2017, 2020; Kendall et al., 2017; Yao et al.
,2018; Kar et al., 2017; Yao et al., 2019;
Yi et al., 2019; Chen et al., 2019; Gu et al., 2019; Chenget al., 2020).
However, these supervised methods crucially rely on ground-truth 3D data, which arehard to generalize in wild open environments.
To realize self-supervised MVS without the onerousground-truth 3D training data, (Dai et al., 2019; Huang et al., 2021;
Khot et al., 2019; Zhang et al.
Xu et al., 2021; Liu et al., 2019; Niemeyer et al., 2020) replace the supervision signal withthe unsupervised rendering loss or cross-view feature consistency loss.
However, reprojection undermultiple views is highly sensitive to environmental illumination, which is hard to generalize in highlyvariable scenarios.
2.2 Part Representation
Recently, learning representations from the image in a self-supervised training strategy has gainedgrowing popularity in 2D vision tasks, which use contrastive learning (Wu et al., 2018;
Chen et al.
,2020b,a; Caron et al., 2020) to deal with the lack of ground-truth labels.
They use InfoNCE (Van den
Oord et al., 2018) to generate a unique embedding for each picture in a batch.
Wang et al.
(2021b) further leverages a dense projection head to generalize such embedding from image-level to patch-level,which is more suitable for performing dense prediction tasks sucn as semantic segmentations.
Hunget al.
; Wang et al.
(2020) assigns the equivariance constraint and the semantic consistency regularization on the learned part response representation
(Zhou et al., 2016) to predict part segmentationsthat are consistent across different instances or camera poses.
Xu et al.
(2021) incorporates learnedsemantic features into multi-view stereopsis pipeline to boost the reconstruction ability.
However,previous works only concentrate on generating semantic features and using it forward to performdown-stream tasks.
Instead, we utilize the rough depth map as supervision to the semantic learningprocess, which forms a natural integration of photometric and geometric information to provideelastic part representations.
Method
We estimate the depth maps of a given reference image x and the corresponding M unstructuredsource images with known camera calibrations.
The final point cloud reconstruction is obtained byfusing all of them.
The key ingredient of our framework is a novel elastic part representation , whichencodes sufficient geometric details to predict a piecewise-smooth depth map.
We first formallydefine our part-aware representation in Sec. 3.1.
Based upon the learned representation, we introducetwo improvements on depth hypotheses propagation and evaluation for part-aware patchmatch in Sec. 3.2.
We present the self-supervised training scheme in Sec.
3.3 and implementation details in Sec. 3.4.
3.1 Elastic Part Representation
Suppose that ( ˜ d, ˜ n ) ∈
R H
× W × 4 are the underlying depth and normal map of the image x , wefirst define the geometry-aware part segmentation S p related to each pixel location p ∈ R 2 ,which is geometrically concentrated inside: S p ≜ { q ∈ R 2 | Γ { ( ˜ d p , ˜ n p ); ( ˜ d q , ˜ n q ) } ≤ ϵ } , whereΓ { ( ˜ d p , ˜ n p ); ( ˜ d q , ˜ n q ) } ≜ | ([ q, ˜ d q ] − [ p, ˜ d p ]) ·
˜ n p | is the 3D point-to-plane distance (Tateno et al., 2015)between two pixels.
Equipped with the part segmentation S p for each pixel p , we propose to learn a per-pixel elastic partrepresentation z p ∈ R D for each p ∈ R 2 , such that z p is compact in the geometric concentrated part
S p and contrast otherwise.
Specifically, we use the Soft-Nearest Neighbor Loss (Salakhutdinov and
Hinton, 2007) to formulate such:
[Equation]
where z p ∗ ∈ z is a feature on pixel local p ∗ , ⟨· , ·⟩ is the dot product, and τ is the temperature.
Finally, we use a ConvNet, F Θ ( · ) , to model the generation of the elastic part representation, i.e., forthe reference image x ∈ R H
× W × 3 , F Θ ( x ) ∈
R H
× W × D denotes the collection of per-pixel elasticpart representations.
3.2 Part-aware Matching
We build our part-aware matching module on top of the seminal work of patchmatch (Galliani et al.,2015).
In particular, we incorporate our elastic part representation and propose the following twonovel improvements: 1.
Propagation: we propagate hypotheses to neighbors with our learned elasticpart representation; 2.
Evaluation: we evaluate and choose the best hypotheses based on photometricconsistency, feature-level correspondence, and geometrically smoothness.
The resulting module isdifferentiable, and the architecture of part-aware patchmatch is illustrated in Fig.
Propagation.
The key idea of the part-aware propagation is to gather hypotheses from the nearbypixels of the same physical surface part.
As illustrated in Fig.
3, instead of propagating hypothesesnaively from a static set of neighbors (Galliani et al., 2015), our depth hypotheses are sampled fromthe nearby pixels which own similar features of the learned elastic part representation.
Since the depth value is unreliable in textureless and highly reflectant areas, a confidence mapc ∈ [0 , 1] H × W is generated to pick the reliable candidates from noise, which is a combination of thephotometric consistency and the geometric consistency (Schönberger et al., 2016).
Please refer tothe supplementary material for a more detailed explanation.
For each pixel p in the depth map to beupdated, K p patch hypotheses are propagated from the propagation candidates T p , which is a set ofnearby pixels q whose feature z q is close to the feature z p with high confidence c q :
[Equation]
According to the property of elastic part representation in Eq. 1, the generated depth hypotheses from
T p would stay close to the underlying surface.
Evaluation.
Beyond the photometric consistency loss, we propose two novel losses for hypothesisevaluation – Part-aware correspondence loss and Part smoothness loss.
Part-aware correspondence loss evaluates the feature similarity of the elastic part representationbetween the reference view and the warped source views.
Given in homogeneous coordinates anddepth hypothesis d , we obtain the warped source feature z
[ i ] p i ( d ) of the i-th source view via differentiable bilinear interpolation.
Therefore, given the features z [0] p , z
[ i ] p i ( d ) of the elastic part representation inthe reference and source view respectively, the part-aware correspondence at depth d p and normaldirection n p is given by:
[Equation]
where ρ ( · ) is the NCC value (Galliani et al., 2015) implemented in the traditional patchmatchalgorithm.
Part smoothness loss aims to augment the local patchmatch evaluation with a non-local piecewisesmoothness term corresponding to the same physical surface part.
Inspired by the work of Locally
Optimal Projection (LOP) (Lipman et al., 2007) for surface reconstruction from point clouds, the partsmoothness loss is formulated using an L1 median loss, which is robust to the outliers and gives apiecewise second-order surface approximation in each surface part (Lipman et al., 2007).
More specifically, the depth map d
∈ R H
× W × 1 is firstly transferred to the point set e ∈ R
H
× W × 3in the scene space given the camera matrix M : e = { e p ∗ ∈ R
| [ e ⊤ p ∗ , 1] ⊤ =
M − 1 [ p ∗⊤ , d p ∗ , 1] ⊤ } .
Then we define our part smoothness loss as the sum of the Euclidean distance to the points frompropagation candidates in T p
(Eq. 2):
[Equation]
where ω q = exp {− c p − α n ⟨ n p , n q ⟩} is a weight correlated with the confidence c q and the normalsimilarity.
Putting all the components together, as illustrated in Fig.
2, the depth map initialization consistsof K p random values within the depth range of interest.
For subsequent iterations, K p hypothesesare propagated based on the depth values of the former one.
To deliver a more diverse set ofhypotheses than just using propagation, another K r hypotheses are sampled by adding a smallrandom perturbation to the previous estimation.
In each iteration, the optimal depth and normalare chosen as: ( d opt p , n opt p ) = [Equation] [Equation]
[Equation] + [Equation] g ( d
[Equation] p , n [Equation] p | z ) (cid:9) .
Empirically, 5iterations are sufficient to obtain an accurate piecewise smoothness depth map.
3.3 Network Training
The elastic part representation z , as described in Sec. 3.1, is compact in the area of a smooth surfacepart.
We now describe how to train the ConvNet F Θ ( x ) using the self-supervised contrastive learningstrategy without knowing the ground-truth depth and normal information.
The core of contrastive learning is to construct a set of positive and negative samples, in whichthe representation in the positive samples stay close to each other while the negative ones are farapart.
Before training, a depth map d and a normal map n of each reference image x are generatedby the initialized ElasticMVS without the part representation, which is similar to the traditionalpatchmatch-based algorithm, e.g., Gipuma (Galliani et al., 2015).
For each pixel p with high confidence ( c p ≥ ξ ), we construct a set of pixel candidates ˆ S p for positivesamples, in which each of them is geometrically concentrated to p with high confidence:
[Equation]
Therefore, the dense contrastive loss for self-supervised training is defined as:
[Equation]
This loss function encourages representation compactness in parts with close surface distance andseparation otherwise, i.e., the distance in the representation space naturally reflects the distance in the3D scene space.
Due to the memory limitation, it is impossible to densely sample all positive andnegative samples from the whole image in Eq.
Therefore, we randomly select N c points from allsamples in ˆ S p , using a probability distribution inversely proportional to the distance away from p .
To obtain spatially concentrated representation, we utilize a spatial concentration loss that encouragesall the pixel embeddings to become isotropically isolated:
[Equation]
which prompts the weighted average of the sampled neighboring points q to be close to the point p .
Therefore, the pixel locations far away from the sampled point will be given low weights, leading toa spatially concentrated representation.
3.4 Implementation Details
The elastic part representation is trained with a Feature Pyramid Network , which shares thesame structure as in PatchmatchNet (Wang et al., 2021a).
To train the network, we use the SGD (Bottou, 2010) with an initial learning rate of 1 . [-
-]0 × 10 − 4 .
The threshold ϵ for defining S p is set to 0 .
5 in the DTU dataset and 0 .
01 in the T&T dataset.
We fix the temperature number τ = 0 .
5 and the confidence threshold ξ = 0 .
7 during training and propagation.
We sample N c = 8 points for each pixel.
During the propagation, we select K p = 8 hypotheses forpropagation and K r = 4 for perturbation.
We choose η = 0 .
1 in Eq.
2 to find nearby pixels.
During the reconstruction stage, we set respectively α s = 0 .
1 and α g = 0 .
5 in Eq.
3 and 4.
All the training and reconstruction are conducted on a single NVIDIA GTX 3090 graphics card. [Table]
Experiment
4.1 Benchmarks
DTU dataset (Aanæs et al., 2016) is an MVS dataset consisting of different indoorobjects scanned at 49 camera positions under 7 different lighting conditions.
We usethe original image size ( 1600 × 1200 ) asinput and follow the same evaluation protocol as (Aanæs et al., 2016) to computethe accuracy and the completeness of thefinal reconstructions.
The overall distancemetric is defined as the mean of the twometrics.
Quantitative results with three types of reconstruction methods are shown in Table 1,including geometry-based, supervised andself-supervised MVS methods.
Ourmethod outperforms all the self-supervised methods including SurRF (Zhang et al.,2021) and JDACS (Xu et al., 2021) in accuracy and overall distance metrics [Table] and achieves the second-best performance in completeness .
It isremarkable that ElasticMVS is competitive compared to state-of-the-art supervised methods, e
[Table]
g., ouroverall distance metric is the second best among all methods
[Table]
[Table]
Tanks and Temples (Knapitsch et al., 2017)dataset contains two benchmarks ( intermediateand advanced ) on outdoor real-world sceneswith complex geometry.
We use the originalimage size (1920 × 1080) as the input.
Weevaluate the results with F-score (Knapitschet al., 2017), the higher value of which indicatesthe better reconstruction.
We summarize thequantitative results in Table 2.
On the intermediate benchmark, ElasticMVS achieves stateof-the-art performance among all supervisedand self-supervised MVS methods.
Especially,our method outperforms all the self-supervised
MVS methods by a significant margin.
On themore challenging T&T advanced benchmark,which contains large-scale real-world sceneswith more than 300 camera views for each scene,
ElasticMVS even compares favourably with thesupervised methods.
The tremendous performance improvement in this benchmark demonstrates the effectiveness and robustness of
the proposed
ElasticMVS w.r.t complex categories and geometries.
We also compare our reconstructions with various baselines qualitatively in Fig.
ElasticMVSproduces much more complete predictions compared with COLMAP (Schönberger et al.,
2016),while generating much less noise than existing self-supervised methods (M3VSNet (Huang et al.,2021) and JDACS (Xu et al., 2021)).
Besides, we achieve even surpassing performances upon
4.2 Ablation Study
In this section, we provide detailed ablation study and analysis of the key components of ElasticMVS,including the part-aware patchmatch and the elastic part representation. [Table] [Table] [Table]
[Table]
[Table]
Part-aware patchmatch.
To investigate the effectsof the key components in the part-aware patchmatchduring inference, we conduct a comparison with respect to the part-aware propagation and evaluationdescribed in Sec. 3.2.
Our part-aware patchmatch iscompared with three variants: 1.
Propagation withoutelastic Part Representation (w/o PR), which employsfixed sampled locations for propagation similar to Gipuma (Galliani et al., 2015);
Evaluation
), which only em- the photometric consistency which simply employs an L1-median loss from all neighborhoods, similar to the LOP (Lipman et al. ,2007) for point cloud de-noising.
As shown in Table 3, our part-aware propagation and evaluation modules both play important roles inboosting the overall performance across different datasets.
Qualitative results are shown in Fig.
Implementations without the elastic part representation (a) and the part-aware correspondence (b)fail to recover the dense surface in textureless regions.
Although applying a naive surface de-noisingand completion method (c) helps to fill in some of the holes in textureless areas, it still suffers fromlow accuracy and noisy predictions due to the lack of part awareness.
Overall, the strong geometriccoherence revealed by the learned elastic sampling strategy helps ElasticMVS gather more accurateand complete depth maps in both small complex surfaces and large flattened areas.
[Table]
Training loss of elastic part representation.
Nowwe study how different training losses described in
Sec. 3.3 affect the performance of the representationand further the depth map predictions.
Our elasticpart representation is compared with two variants,which are trained without confidence map (w/o Con.)in Eq.
6 and without spatial concentration loss (w/o
Spa.)
in Eq. 7.
We test on the DTU dataset.
We first measure thecompactness of the learned part representation asdescribed in Eq.
1 using the ground truth depth map.
We also compare the predicted depth map error using these representations, which is evaluated by the
L1 distance from the predicted depth map to the ground-truth.
As shown in Table 4, our elastic partrepresentation is the most compact one (with lowest contrastive loss).
Visualizations of the representation are demonstrated in Fig. 6, shown by a heat map of the representation distance ∥
z q − z p ∥ from a pixel location p to all other pixels q in the whole image.
As expected, our elastic part representation successfully recognizes the part onthe same physical surface, resulting inproper surface boundary identificationswith elastically-varying scales and shapes.
Contrastingly, the representation trainedwithout the confidence map (w/o Con.)fails to identify a complete surface part dueto the noisy training samples.
Meanwhile,the representations trained without spatialconcentration loss (w/o Spa.) suffer fromoutliers far from the surface part, demonstrating the effectiveness of spatial regularization.
4.3 Discussion
Memory.
In Table 5, we compare the GPUmemory consumption and the run-time during inference with two memory-efficientlearning-based methods.
We choose scan 9 and a fixed reference view downsampled to the scaleof 640 × 800 in the DTU dataset in this study.
As shown in Table 5, ElasticMVS is more memoryefficient compared with two learning-based MVS backbones.
Notice that the memory and runningtime are adaptively changed with the hypothesis number and the iteration number, since propagationwith more hypotheses will converge with fewer iterations, but take a larger memory consumption.
Overall, the experiment demonstrates the superiority memory efficiency of ElasticMVS.
[Table]
Limitation and Future work.
It takes along time in ElasticMVS to calculate thepatch similarity for each pixel, since thetotal computational complexity to evaluatethe photo consistency from M source images of size H × W with K hypothesesis O ( M × H × W × K × l 2 ) , where l isthe patch size.
Empirically, l = 11 , whichis impossible to give a forward and backward pass from a single graphics card atonce.
As shown in Table 5, the running time of ElasticMVS is not competitive compared with thecost-volume-based methods.
One solution is to replace the l × l patch with the learned featurein Patchmatch.
However, the accuracy of the learned photometric consistency in self-supervisedstereopsis is still far from the traditional one.
In future work, we plan to design a lightweight networkarchitecture to represent not only the part correlation but also the pixel-wise wrapped image patches,jointly learned from a self-supervised learning-based strategy, which can meet the demand for morechallenging tasks such as real-time MVS and dynamic MVS.
Conclusion
In this paper, we present a novel elastic part representation that encodes physically connected partsegmentations with elastically-varying scales, shapes and boundaries.
We build the correlationbetween the embedding space and the scene space to represent the surface connectedness andboundaries behind the image.
A self-supervised MVS framework, ElasticMVS, is proposed tolearn the representation and estimate per-view depth following a part-aware patchmatch-basedalgorithm.
Benefiting from the inherent geometric correlations learned from the huge amount ofimages, ElasticMVS delivers state-of-the-art performance in both reconstruction completeness andaccuracy, and extraordinary generalization ability in the challenging large-scale scenes. Checklist The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 4.3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] See in the
supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See in Sec.
3.4 and in the supplemental material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See in the supplemental material.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See in Sec.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]