Concurrent 3D super resolution on intensity and segmentation maps improves detection of structural effects in neurodegenerative disease
Abstract
We propose a new perceptual super resolution (PSR) method for 3D neuroimagingand evaluate its performance in detecting brain changes due to neurodegenerativedisease.
The method, concurrent super resolution and segmentation (CSRS), istrained on volumetric brain data to consistently upsample both an image intensitychannel and associated segmentation labels.
The simultaneous nature of the methodimproves not only the resolution of the images but also the resolution of associatedsegmentations thereby making the approach directly applicable to existing labeleddatasets.
One challenge to real world evaluation of SR methods such as CSRSis the lack of high resolution ground truth in the target application data: clinicalneuroimages.
We therefore evaluate CSRS effectiveness in an adjacent, clinicallyrelevant signal detection problem: quantifying cross-sectional and longitudinalchange across a set of phenotypically heterogeneous but related disorders thatexhibit known and differentiable patterns of brain atrophy.
We contrast several 3D
PSR loss functions in this paradigm and show that CSRS consistently increases theability to detect regional atrophy both longitudinally and cross-sectionally in eachof ﬁve related diseases.
Introduction
Magnetic resonance image (MRI) datasets capturing in vivo longitudinal change in the human brainare currently available at unprecedented scale.
These data allow us to quantify the complex etiologyof neurodegenerative disease during life.
A fundamental problem in quantifying brain disordersfrom imaging is that many anatomical structures are small in comparison to image resolution.
Thisis caused by not only limited image resolution but also the potentially convoluted shape of thetargeted anatomy [1].
Thinner, more oblate and/or curved structures undergo more distortion dueto sampling-related aliasing in comparison to larger, more spherical structures.
These distortionscan limit detection power in the context of either clinical trials and/or at the level of patient speciﬁcmedicine [2, 3].
These results also show that, based on ﬁrst principles, many disease relevantanatomical structures in the brain, in particular cortical regions, mid-brain regions and hippocampalsubﬁelds, should be quantiﬁed at higher resolutions (e.g. ≈ 0.5mm 3 or smaller rather than themore commonly available ≈ 1mm 3 ).
The need for increased resolution is only heightened whenconsidering aging and neurodegeneration where some brain structures may lose half or more of theirpre-disease onset volume or thickness.
Perceptual super resolution (PSR) for 2D RGB imagery consistently demonstrates the ability toestimate more “realistic” looking upsampled data in comparison to traditional linear or nearestneighbor interpolants [4].
While many competitive methods are available, the deep back projectionnetwork (DBPN) [5] performed consistently in several competitions including NTIRE 2018 and
[6], AIM 2019 [7] and PIRM 2018 [8]).
These large challenges compared dozens of methods withrespect to a variety of both perceptual and reconstruction metrics at different levels of upsamplingand noise.
Can the 2D RGB performance advantages of methods like the DBPN translate to improvements inthe 3D quantiﬁcation of brain regions as seen in MRI?
If so, then PSR for 3D neuroimaging promisesto improve quantiﬁcation by better resolving the brain’s internal structures and tissue boundaries.
While traditional evaluations of PSR focus on reconstruction error and perceptual impression, thesemeasurements do not provide clinically relevant evidence of PSR’s value in quantiﬁcation.
One barrierto evaluating PSR’s impact on clinically relevant outcomes (segmentation volumes) is that groundtruth segmentations do not exist at the super-resolved scale.
To address this concern, [9] simulated lowresolution magnetic resonance images (MRI) of the brain from high-resolution (HR) images obtainedfrom the Human Connectome Project [10, 11].
They then applied a very deep super resolution(VDSR) model to the simulated data and the high-resolution data and compared the accuracy of anautomated cortical segmentation method.
This careful evaluation study demonstrated that corticalsegmentation on the VDSR images closely approximated the HR data.
However, relatively fewdetails are provided about the training of this model and associated loss functions.
Furthermore, itremains unclear whether these improvements in reconstruction error would translate to the detectionof population-level effects in real world data particularly in the aging populations that are the targetof the majority of interventional trials for the brain.
A more recent effort in volumetric PSR for medical images [12] proposed SOUP-GAN: Superresolution Optimized Using Perceptual-tuned Generative Adversarial Network (GAN).
SOUP-GANadopts transfer learning from 2D VGG19 to 3D as proposed in [13] to produce a pseudo-volumetricperceptual metric [14].
Shan et al.
used this metric to denoise low-dose computed tomography(CT) images and showed its effectiveness at preserving small anatomical structures.
Similarly, the
SOUP-GAN effort demonstrates that the pseudo-3D perceptual metric improves both PSNR and
SSIM as well as shows visually appealing upsampling for a variety of medical imaging modalities.
That is, the surprising utility (in 2D) of VGG weights as a feature space [15] appears to at leastpartially transfer to PSR in 3D medical imaging.
The current research provides perhaps the ﬁrst broadly scoped, real world evaluation of MRI PSR forquantiﬁcation of neurodegenerative disease.
Moreover, we demonstrate that a regression network(ResNet) that predicts T1w image quality can yield a directly useful perceptual feature space thatperforms competitively with pseudo-3D VGG19 features.
We build these contributions upon thebackbone of a set of methods that we call concurrent super resolution and segmentation (CSRS) thatextends the proven 2D DBPN to 3D and also includes extra output channel(s) enabling segmentationmaps to be upsampled concurrently.
We use this framework to test the impact of different lossfunctions on a set of domain-speciﬁc, clinically relevant segmentation measurements related tobrain atrophy.
Speciﬁcally, we evaluate CSRS on the quantiﬁcation of frontotemporal disorders [3]from publicly available longitudinal T1-weighted (T1w) neuroimaging (i.e. MRI).
Of the severalcombinations of losses that we evaluate, the best model improves not only segmentation performance(when ground truth is available) but also detection power across all our related disorders: structuralchanges in behavioral variant frontotemporal dementia (bvFTD), semantic variant primary progressiveaphasia (svPPA), nonﬂuent/agrammatic PPA (naPPA), progressive supranuclear palsy (PSP) andcorticobasal syndrome (CBS) each of which impacts known networks in the brain.
CSRS with a newperceptual loss based on a shallow ResNet layer performs as well or better than VGG-based modelsin this test of the practical usefulness of PSR.
The primary contributions of this work include:
• new PSR that upsamples multi-label segmentations at the same time as intensity;
• a new real world evaluation paradigm for PSR in neuroimaging;
• comparison of three perceptual loss functions for PSR, two of which are new;
• demonstration that loss choice impacts detection power in natural history studies of neu-
rodegenerative disease.
Standard intensity similarity and segmentation overlap metrics, onthe other hand, do not discriminate performance between the candidate CSRS options.
weights, sample data and training code will be made publicly available after anonymous review
Methods
Software platform : We employ the ANTsX platform [16] version 2.3.5 for anatomical labeling,data augmentation/sampling during model training and to form the tabular data for the statisticalevaluation.
All MRI processing details follow [16].
Tensorflow 2.6.2 is used for deep learningincluding a ResNet implementation and the CSRS architecture.
R version 4.1 is used for statisticalanalysis with packages lmer and ggplot2 .
All MRI processing was done on Amazon Web Servicesparallel cluster with 24 cores and 32GB RAM per process (Intel(R) Xeon(R)
Platinum 8259CL CPU @ 2.50GHz).
Data: Human Connectome Project (HCP): We downloaded 1,113 high-resolution 0.7mm 3 T1weighted images from the HCP on which to train CSRS.
These T1w data were acquired using amagnetization-prepared rapid gradient-echo (MPRAGE) sequence on a customized 3T Siemens
Skyra; see [10] for all details of acquisition.
As such, these images provide both high resolution andhigh quality in comparison to the majority of publicly available T1w MRI.
Critically, they providesuperior resolution for the thin convoluted cortical layer that is critical to the measurement of brainatrophy in frontotemporal disorders.
We transformed these data into numpy blocks with randomlyselected high-resolution 64 3 patches and paired low-resolution 32 3 patches.
For each patch pair, wealso provide a high-resolution binary segmentation and a low-resolution downsampled version ofthat binary segmentation.
Each patch segmentation was gained by 2-class k-means performed onthe patch where the center voxel’s label determines which class (1 or 2) is used as foreground.
Thiscollection of 16,640 patches is then divided randomly into train (n=16,384) and test sets.
Data: Parkinson’s Progression Markers Initiative (PPMI): PPMI is a longitudinal multi-centerclinical study of PD patients and age-matched healthy controls http://www.ppmi-info.org .
PPMI employed(s) over 20 data collection sites with scanners that span the primary manufacturers(Siemens, GE, Phillips), a variety of head coils and also magnet strengths (1.5T, 3T).
This heterogeneity of data collection provides a rich set of T1w images with highly variable image contrast, resolutionand quality.
We manually reviewed and labelled 1,431 raw T1w from PPMI to capture the range ofquality in an ordinal scale.
This resulted in a ground truth dataset with 456 images given grade “A”(superior), 568 given grade “B”, 350 given grade “C” and 57 given grade “F” which represents imagesthat are of little to no use for quantitative studies of brain structure.
We then employed a standard3D ResNet ( antspynet.create_resnet_model_3d with parameters lowest_resolution=32 ,number_of_classification_labels=4 , cardinality=1 , 39,424,004 parameters, 53 3D convolutional layers) to learn to predict this scale automatically and reliably from the input T1w.
We denotethis network as a T1w Quality Rating Resnet (T1wQRResNet).
Details of training T1wQRResNetare in Supplementary Information.
Data: Frontotemporal Lobar Degeneration Neuroimaging Initiative (NIFD) & 4-Repeat
Tauopathy Neuroimaging Initiative (4RTNI) : These inter-related multi-site studies share the goalof improving the quantiﬁcation of frontotemporal spectrum disorders with both imaging and clinicalscores.
Like PPMI and HCP, these studies provide longitudinal T1w images that enable measurementof not only the baseline brain structure differences between controls (individuals without a diseasei.e.
normal aging) and disease groups but also differences in rates of change due to neurodegeneration.
We downloaded and curated 4RTNI and NIFD T1w data and merged these images into a commondatabase.
These images were collected at three different sites using protocols consistent with ADNI3T guidelines [2].
The images overall have a median spacing that is isotropically 1mm with a minorityof subjects with out-of-plane spacing up to 1.2mm.
As such, these data suit the goals of testing PSRfor beneﬁts to the quantiﬁcation of neurodegenerative disease.
After ﬁltering data for very low qualityimages and the presence of longitudinal data collected within 2 years of baseline, we obtained 128baseline/171 followup images for controls, 60/112 for bvFTD, 38/72 for naPPA, 37/71 for svPPA,55/70 for CBS and 75/102 for PSP.
Further cohort details (age, education, sex, etc) are available insupplementary information.
We processed all images consistently and automatically with default
ANTsX pipelines to gain cortical, medial temporal lobe and deep brain structure segmentations forevery subject as described in [16].
By consensus, co-authors selected a priori regions for testingwithin each of four groups CBS/PSP [17], bvFTD, svPPA
and naPPA [18–24].
Details of the regionsand rationale for their selection are available in the Supplementary Information.
See Figure 1 for anoverview of processing, the CSRS method and a visualization of the regions (1.C).
2.1 Concurrent super resolution and segmentation methods
CSRS uses, as a sub-algorithm, a three-dimensional and multi-output version of the neural networkarchitecture deﬁned by the 2D deep back projection network (DBPN) [5].
The DBPN is uniquelyrelevant to medical imaging in that it is perhaps the ﬁrst published SR method that integrates thedownsampling-upsampling error (i.e. residual layers) as a feature map.
This novel architecture mayprevent feature hallucination and constrain the high-resolution image to maintain features that areconsistent with the low-resolution input.
We extend the 2D DBPN to 3D MRI data by, ﬁrst, translating2D convolutions, padding, striding and other relevant parameters to 3D. To generalize the architecturefurther, we allow options for not only convolutional upsampling (transposed convolution) but alsonearest neighbor (or linear) interpolation layers at the user’s choice.
Lastly, we implement ﬂexiblechoices of input channels, the number of residual layers (backprojection) layers and the number ofoutputs.
This 3D DBPN network implementation is available within R and python .
All parameterswere the same as the published work [25] (though in translation to 3D) with the exception of thenumber of back projection layers which has a large impact on the number of parameters.
We reducedthe number of backprojection layers to 5 (16,264,322 parameters) due to the memory limitationscaused by working with large 3D images and limited GPU resources (all GPU computations in thiswork were implemented with Nvidia V100s locally).
Efﬁcient computational strategy is essential for CSRS to be applied to large 3D images (a brainimage may contain 10 million voxels) when CPUs and RAM are limited.
As such, a local patch-workstrategy is necessary.
Sampling, upsampling, mapping and uniﬁcation (SUMU) are the common stepsneeded for not only training but also inference.
“Sampling” decides the form of the input data: fullimages (not used here), image patches (used here in training) or anatomical image regions (used herein inference).
Upscaling determines the core approach to transferring the low-resolution data to ahigher-resolution output.
Mapping compensates for shape or intensity distortion.
Finally, “uniﬁcation”is an ensembling or merging step that brings together several sub-estimates of an SR image into asingle joined (ﬁnal/full) SR image.
We detail each of the 4 components below.
Sampling :
We choose a patch-based model for training as these can easily be applied to input datawith different resolutions and ﬁelds of view.
A second reason for patch-based modeling is that acandidate network does not need to learn the full scope of image variation.
This results in shallowerand faster to train networks that ﬁt more easily onto readily available GPUs.
The choices madeduring sampling step deﬁne the feature basis set.
Because prior super-resolution competitions suggestlarger patches lead to better performance, we choose the largest patches that would permit efﬁcientbatch sizes of 4 (64x64x64).
An additional ad hoc support for this choice is that cortical featuresare relatively well-resolved in sub-1mm training images when voxel cubes of this sized are used.
However, there is no direct evidence that this size of patch domain is optimal for this problem.
Upscaling : is done with the CSRS’s DBPN architecture using nearest neighbor interpolation forthe upsampling layers.
The software interface to CSRS also allows the user to optionally employstandard linear (tri-linear) interpolation.
We use the linear option as a reference in evaluation studiesbelow.
Mapping : may be used to compensate for distortions in the image shape or intensity space.
Becauseeach patch is scaled independently on training data (to have an intensity range of -127.5 to 127.5),the output of the PSR upsampled image intensities must be mapped back to the original quantitativespace.
This is performed by directly comparing the output of the PSR upsampled patch/region to theoriginal data upsampled by nearest neighbor or linear interpolation.
As such, we can accurately retainquantitative intensity data at the original scale/units with minimal distortion and/or stitching artifacts.
Uniﬁcation : this is a general term that, here, refers to the algorithm that is used to derive a single
CSRS image and multi-label segmentation from multiple CSRS sub-images (not necessarily isotropicpatches as in training).
In 2D, multiple input images are typically generated from a single input by“augmentation” e.g. random ﬂipping, translation, etc thus allowing a practitioner to gain multiple“votes” about how the SR image should appear at any given voxel.
Such a step is used in most PSRcompetitions to reduce aliasing or artifacts and may involve averaging, sharpening or more complexmodeling such as joint intensity fusion, multi-channel deep learning or other ensemble methods.
Due to the high memory and computation cost of running CSRS on 3D images, we instead apply
CSRS to either sub-regions of interest or, when a full T1w brain image is desired, each hemisphere.
The uniﬁcation step then maps each local patch intensity range back to the original MRI range andthen joins the sub-regions back together to complete the SR reconstruction.
Augmentation can beemployed beyond this but at substantial increase in computation time (e.g. 10x to see meaningfulgains due to augmentation).
Loss functions for CSRS :
We employ a loss function that seeks to balance reconstruction error(intensity difference, abbreviated here as R), edge preserving denoising (total variation, abbreviatedas TV), perceptual quality (based on VGG or ResNet) and segmentation overlap (Dice, abbreviatedas D).
Each of these terms can be up or down weighted to control the network’s performance wheremean squared error (L2 intensity error) leads to smoother results, L1 (or total variation) providesdenoising and the perceptual loss yields more natural appearing output textures and shapes.
The
Dice loss term seeks to minimize distortions in the shape of segmentation objects on the outputof CSRS.
The Dice loss is only applied to the second output channel of the network which uses asigmoid activation function appropriate for probabilistic/binary data.
We refer to CSRS trained withspeciﬁc combinations of these losses by concatenation of the abbreviations above.
For example,
CSRS.R.TV.D.Res6 refers to a network trained with reconstruction loss, TV regularization, Dice lossand the 6th layer of the T1wQRResNet for perceptual loss.
Recent research demonstrates that deep learning models trained on large-scale object detectionreference datasets (e.g. imagenet ) encode a feature space that may mimic human perception[26].
Such perceptual spaces typically arise from the activations that occur within the layers ofconvolutional networks trained on massive classiﬁcation datasets.
Here, however, we compare astandard VGG based perceptual space ( block2_conv2 ) (mapped to 3D as described before) tothose deﬁned by the T1wQRResNet.
From T1wQRResNet, we choose two different deep layersthat have similar numbers of parameters to the 3D version of the VGG19 block2_conv2 network:res_conv_block_6 (the 2nd convolutional block) and res_conv_block_21 (the 7th convolutionalblock).
This allows us to compare perceptual metrics based on either pseudo-3D VGG19 or ourintrinsically 3D res_conv_block choices.
Quantiﬁcation of medical images requires a high degree of faithfulness to the input data. "
Hallucinated" features are undesirable.
As such, our baseline loss function focuses on reconstruction error and TV for both intensity and segmentation images.
We then add perceptual and Dice losses forfurther comparison.
If we denote I as the estimated super-resolution, I s as the estimated segmentationfrom the sigmoid output channel, J as the real high resolution image, J s as the real high resolutionsegmentation, then the ﬁnal loss function that we optimize is:
[Equation]
where the term (cid:107) · (cid:107) indicates the euclidean norm, TV ( · , · ) indicates the total variation norm (whichprovides denoising), w r,t,f,d (superscripts for intensity or segmentation) indicates a term-speciﬁcscalar weight and f n ( . ) indicates a perceptual feature map.
The weight terms can be tuned forperformance and application area given an objective and quantitative evaluation metric.
We initiallymanually tuned the training of a DBPN model with only the reconstruction metrics ( (cid:107) I − J (cid:107) 2 w Ir +(cid:107) I s − J s (cid:107) 2 w sr with w ir = 5 e − 4 and w sr = 1 ) using adam optimizer and learning rate 5e-5.
Wethen set weights relative to the value of the reconstruction error after convergence such that: the TVloss is roughly 2/3 the reconstruction term (R); the perceptual loss is roughly 3x R; the Dice loss isroughly equivalent to the perceptual loss.
This strategy, based on our task-speciﬁc goals, enables usto compare models consistently and add/subtract terms without extensive weight optimization.
Computation and inference : All models were implemented with tensorflow .
The computation todouble magniﬁcation – for a single T1w – takes (generally on a modern computational platform)between 10 and 40 minutes.
Results are computed region-wise over the set of segmentation labelswhere CSRS is run on each cropped label and its associated intensity.
When multiple regions areused (as is done here), then results are stitched back together while using a linear mapping backto the original intensity space and a arg_max operation to deﬁne the hard segmentation labels atevery voxel in the stitched, joint intensity/probability double magniﬁcation space.
See Figure 2 foran example result of CSRS as applied to the variety of brain regions in this study.
Figure 3 shows azoomed visual comparison of the impact on intensity and the lack of stitching artifacts.
2.2 Quantiﬁcation of CSRS impact on segmentation and intensity in ground truth data
Evaluation of PSR results on simulated downsampled-upsampled data does not constitute real worldconditions.
However, for reference, we include evaluation results based on an independent set oflabeled brain images [27].
For these images, we downsample with nearest neighbor interpolationand upsample with linear interpolation (for the intensity) and a “generic label” interpolation that isdesigned for multi-label images [28] thereby allowing us to report standard metrics of Dice overlap,
PSNR and SSIM to complement our study of brain atrophy detection.
Figure 4 demonstrates exampleresults illustrating this component of our evaluation.
2.3 Quantiﬁcation of effect sizes in frontotemporal disorder atrophy
The frontotemporal disorders produce a profound and debilitating effect on patients with concomitant,symptom-related atrophy.
Measuring this atrophy is critical to detecting the effects, for instance,of disease modifying therapies that may slow atrophy.
Such measurements are challenged by lowresolution and this challenge is compounded by the degeneration process itself.
We use statistical modeling to determine if CSRS can mitigate the known limitations of resolution onatrophy measurement.
We adopt an interpretable mixed effects modeling approach ( lmer )[29] toestimate effect sizes per brain region, per diagnostic category and per resolution/CSRS model.
Thebaseline performance is determined by the effect sizes estimated on the original resolution (OR) data.
We estimate effect sizes following [30, 31].
Better methods, under this design, should more reliablyidentify disease-related atrophy which will be reﬂected in increased effect sizes for a given set of apriori diagnosis-speciﬁc regions.
The model for the region of interest i
( ROI i ) is:
[Equation]
with (1 | ID ) representing a subject-speciﬁc random effect, Age b is the subject’s age at the ﬁrst visit,
BV b is the ﬁrst visit brain volume, DX is the diagnosis for the subject, ∆ T is the change in timesince baseline and the ∆ T ∗ DX represents an interaction between time and diagnosis.
The ROI irepresents the volume for all regions.
However, for cortical regions, we also use the region’s thicknessmeasurement as a second outcome (as this is a standard measurement in morphometry of the humancortex).
We estimate effect sizes for cross-sectional effects via the model’s parameter ﬁt for thediagnosis ( DX ) term; we estimate longitudinal effect sizes via the parameter on the interaction term.
[Table]
Results
Table 1 summarizes overall results where we show original resolution and results from linearupsampling, as baseline, and compare to eight variants of CSRS.
Two of these do not use perceptualmetrics.
The remaining six add or subtract Dice loss and each of our candidate perceptual losses.
Table 1 shows both the aggregate impact of model on effect size estimates in the neurodegenerationdata as well as intensity similarity (reconstruction) and Dice overlap in the ground truth data.
Diceoverlap (a measure that varies between zero and one) improves by a margin of 0.11 to 0.123 (95% CIbootstrapped percentile conﬁdence interval, p < 1 e − 16 .
See Figure 4.
Table 2 focuses on the two perceptual models with the greatest improvement from original resolutionas assessed by pairwise t -test.
It breaks down the effect size results in relation to which type of effectsize is being analyzed (cross-sectional or longitudinal) and by brain region / diagnostic grouping.
Relatedly, Figure 5 shows a Bland-Altman style plot that demonstrates, for the CSRS.R.TV.D.Res6model, the range of effect size changes due to CSRS across all 274 measurement points.
[Table]
Discussion
The PSNR and SSIM improve similarly across all CSRS models and do not substantively differentiateperformance.
Dice overlap is consistently superior than linear upsampling across all models but showslittle difference between models with perhaps a small advantage for the ResNet features.
Greaterstratiﬁcation may be seen when looking at results that relate to quantifying the phenotypic heterogeneity of brain atrophy in frontotemporal spectrum diagnostic groups.
Model CSRS.R.TV.D.Res6stands out under this criteria with Table 2 suggesting that the majority of the improvement arisesfor cortical measurements, particularly longitudinally.
Performance improvements are not, however,perfectly consistent.
Figure 5 shows that CSRS augments effect size in the large majority of regions(some greatly so)
but a few regions are subtly better at OR.
Additional discussion of performanceimplications with respect to individual regions and diagnoses is in supplementary information.
The extension of PSR to 3D raises opportunities as well as challenges.
Parameter exploration isfundamentally limited because training a model on our patch dataset for 1 epoch takes over 12 hours(we trained each model for 2 epochs or until convergence).
Other architectures than DBPN mayperform better with CSRS such as ESRGAN [32] or, potentially, methods with stronger modalityspeciﬁc priors on the convolutional kernels [33].
Speciﬁcally, fast-training, fewer parameter modelsmay ease some of the computational burden and facilitate more parameter exploration.
CSRS performance is fundamentally limited by the quality of its segmentation inputs.
It may bemore beneﬁcial to develop new methods that operate at high resolution (HR) – adding substantialcomputational cost if the goal is to take advantage of HR features – or that take advantage ofintrinsically HR ground truth data.
The primary barrier to such an effort is the current lack of HRground truth labels for neuroimaging and in particular for neurodegenerative disease.
Moreover,most methods embed resolution assumptions in their own processing choices and optimize for thesechoices.
As such, CSRS bridges a performance gap with a practical solution readily available today.
Retooling existing methods and segmentation labels for HR (e.g. 7T MRI) is costly both computationally and in terms of the effort of human experts due to the already high volume of 3D neuroimaging.
We demonstrated that CSRS, in most of its variants, leads to signiﬁcant performance improvementsover our reference of original resolution (1mm 3 ) image processing and ground truth labels.
Because
CSRS operates on existing images and labels, new HR method and segmentation development isnot required.
Thus, CSRS may be used to improve existing ground truth datasets and existingprocessed data, today.
However, comparison to other and/or larger real world datasets is needed tohelp determine the extent to which our results may be deployed to new data without concern.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide
the data, in supplementary material, that is needed to generate the tables shown in the paper.
The raw data is publicly available but we do not have permission to redistribute
these data.
However, we do provide the ability to reproduce key results in the Tables ofthe main manuscript via supplementary information.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See beginning of Methods sectionand CSRS methods section.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] in supplemental information.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] in supplemental information.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] in supplemental information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]