Original: Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis
Revised: Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent self-supervised advances in medical computer vision exploit the globaland local anatomical self-similarity  for pretraining prior to downstream taskssuch as segmentation.
Revised: Recent self-supervised advances in medical computer vision exploit the global andlocal self-similarity of anatomical volumes for pretraining prior to downstreamtasks such as segmentation.

Original: However, current methods assume i.i.d.
Revised: However, current methods assume i.i.d.

Original: image acquisition,which is invalid in  clinical study designs where follow-up longitudinal scans tracksubject-specific temporal changes.
Revised: image acquisition, which is invalid in several clinical studies where follow-up longitudinalscans track subject-specific temporal changes.

Original: Further, existing self-supervised methods formedically-relevant image-to-image architectures exploit only spatial or temporalself-similarity and do so via a loss applied only at a single image-scale, with naivemulti-scale spatiotemporal extensions collapsing to degenerate solutions.
Revised: Further, existing self-supervisedmethods for medically-relevant image-to-image architectures exploit only spatialor temporal self-similarity and do so via a loss applied only at a single image-scale,with naive multi-scale spatiotemporal extensions collapsing to degenerate solutions.

Original: To theseends, this paper makes two contributions: (1) It presents a local and multi-scalespatiotemporal representation learning method for image-to-image architecturestrained on longitudinal images.
Revised: To these ends, this paper contributes two methods: (1) It presents a localand multi-scale spatiotemporal representation learning method for image-to-imagearchitectures trained on longitudinal images.

Original: It exploits the spatiotemporal self-similarity oflearned multi-scale intra-subject image features for pretraining and develops several feature-wise regularizations that avoid degenerate representations; (2) Duringfinetuning, it proposes a surprisingly simple self-supervised segmentation consistency regularization to exploit intra-subject  correlation.
Revised: It exploits the spatiotemporal selfsimilarity of multi-scale intra-subject image features for pretraining and developsseveral feature-wise regularizations that avoid collapsed identity representations;(2) During finetuning, it proposes a surprisingly simple self-supervised consistencyregularization to exploit intra-subject anatomical correlation.

Original: Benchmarked acrossvarious segmentation tasks, the proposed framework outperforms both well-tunedrandomly-initialized baselines and current self-supervised techniques designed forboth i.i.d.
Revised: On finetuning, theproposed framework outperforms both well-tuned randomly-initialized baselinesand current self-supervised techniques designed for both i.i.d.

Original: and longitudinal datasets.
Revised: and longitudinaldatasets in the one-shot segmentation setting.

Original: These improvements are demonstrated acrossboth longitudinal neurodegenerative adult MRI and developing infant brain MRIand yield both higher performance and longitudinal consistency.
Revised: These improvements are demonstrated across both longitudinal neurodegenerative adult MRI and developing infantbrain MRI and yield both higher performance and intra-subject consistency.

Original:  Introduction
Revised:  Introduction

Original: Tracking subject-specific anatomical trends over time is crucial to both clinical diagnostics andlarge-scale biomedical science.
Revised: Tracking subject-specific anatomical trends over time is crucial to both clinical diagnostics andlarge-scale biomedical science.

Original: Such longitudinal imaging is especially relevant to analyzing neurological patterns of growth and degeneration via brain imaging in pediatric and elderly populations,respectively.
Revised: Such longitudinal imaging is especially relevant to analyzing neurological patterns of growth and degeneration via brain imaging in pediatric and elderly populations,respectively.

Original: As tracking individual structural changes requires precise and longitudinally-consistentsegmentation methods with scarce annotated training volumes, we identify two major bottlenecks inexisting self-supervised biomedical image analysis methods which we use to motivate our work.
Revised: As tracking individual structural changes requires precise and longitudinally-consistentsegmentation methods with scarce annotated training volumes, we identify two major bottlenecks inexisting self-supervised biomedical image analysis methods which we use to motivate our work.

Original: Learning with few annotations.
Revised: Learning with few annotations.

Original: While modern imaging studies may scan hundreds to thousandsof individuals, manually outlining volumetric structures of interest across multiple individuals forsupervised segmentation training is prohibitively expensive.
Revised: While modern imaging studies may scan hundreds to thousandsof individuals, manually outlining volumetric structures of interest across multiple individuals forsupervised segmentation training is prohibitively expensive.

Original: Therefore, current work focuses onleveraging large sets of unlabeled images to pretrain image-to-image architectures (e.g., the UNet [48]), which can then be efficiently finetuned in the one or few-shot setting.
Revised: Therefore, current work focuses onleveraging large sets of unlabeled images to pretrain image-to-image architectures (e.g., the UNet [46]), which can then be efficiently finetuned in the one or few-shot setting.

Original: These self-supervisedmethods may handcraft pre-text training objectives [10, 18, 45, 65] or may attempt to pretrain the basenetwork to be equivariant to transformationsin order to preserve semantic meaning as in contrastivelearning in medical [9, 63] and natural image vision [2, 24, 37, 60, 67, 68].
Revised: These self-supervisedmethods may handcraft pre-text training objectives [10, 17, 43, 61] or may attempt to pretrain thebase network to be equivariant to transformations which preserve semantic meaning as in contrastivelearning in medical [9, 59] and natural image vision [2, 23, 35, 57, 63, 64].

Original: However, these methodstypically leverage label supervision in sampling for their losses and impose a self-supervised loss only at a single feature scale (typically the encoder bottleneck or network output
Revised: However, these methodstypically leverage label supervision in sampling for their losses and impose a self-supervised loss only at a single feature scale (typically the encoder bottleneck or network output).

Original: Naive of local unsupervised multi-scale contrastive losses [44] lead to degenerate representations (Fig.
Revised: Naive applicationof local unsupervised multi-scale contrastive losses [42] lead to degenerate representations (Fig.

Original: 1,row A) when applied to both the encoder and decoder of image-to-image architectures.
Revised: 1)when applied to both the encoder and decoder of image-to-image architectures.

Original: Violating i.i.d.
Revised: Violating i.i.d.

Original: assumptions.
Revised: assumptions.

Original: Further, most existing self-supervised frameworks assume i.i.d.data.
Revised: Further, most existing self-supervised frameworks assume i.i.d.

Original: Unfortunately, this assumption does not transfer to longitudinal studies where images of the same subject are highly correlated.
Revised: dataand this assumption does not transfer to longitudinal studies where images of the same subject arehighly correlated.

Original: Emerging longitudinal representation learning methods focus on imposingtemporal-consistency into the encoder bottleneck [15, 43, 66], such that the encoder learns representations which are aware of the order of acquisition [15] and the overall trajectory [43].
Revised: Emerging longitudinal representation learning methods focus on imposing temporalconsistency into the encoder bottleneck [15, 41, 62], such that the encoder learns representationswhich are aware of the order of acquisition [15] and the overall trajectory [41].

Original: These methodsaddress image-level tasks such as disease classification or progression and age prediction.
Revised: These methodsaddress image-level tasks such as disease classification or progression and age prediction.

Original: However,their extension to pixel-level applications with image-to-image architectures remains unclear.
Revised: However,their extension to pixel-level applications with image-to-image architectures remains unclear.

Original: Methods.
Revised: Methods.

Original: Motivated by the above limitations, in this work, we claim that the spatiotemporal dependency of imaging data should be explicitly incorporated in self-supervised frameworks.
Revised: Motivated by the above limitations, in this work, we claim that the spatiotemporaldependency of imaging data should be explicitly incorporated in self-supervised frameworks.

Original: We do soby exploiting the spatial and temporal self-similarity of local multiscale deep features in both encoderand decoder and further learn diverse intermediate representations by developing regularizationsfor self-supervised similarity objectives.
Revised: Wedo so by exploiting the spatial and temporal self-similarity of local multiscale deep features in bothencoder and decoder and do without mode collapse by developing regularizations for self-supervisedsimilarity objectives.

Original: Lastly, when finetuning with limited annotated data, weencourage predictions on unlabeled subject-wise images to be spatiotemporally consistent.
Revised: Further, when finetuning with limited annotated data, we encourage predictionson unlabeled subject-wise images to be spatiotemporally consistent.

Original: Contributions.
Revised: Contributions.

Original: This work makes the following contributions: (1) It presents a longitudinallyconsistent spatiotemporal representation learning framework to learn from image time-series; (2) Toimpose multi-scale local self-supervision while avoiding degenerate solutions, it develops regularization terms on the variance, covariance, orthogonality of local  features within the decoder; (3) Tofurther self-supervise the fine-tuning stage, the proposed method encourages segmentations from adjacent timepoints on unlabeled data to be consistent; (4) Across three large-scale longitudinal one, few,and full-shot segmentation tasks on both elderly and pediatric populations, the developed frameworkyields improved segmentation performance and higher longitudinal segmentation consistency.
Revised: This work makes the following contributions: (1) It presents a longitudinallyconsistent spatiotemporal representation learning framework to learn from image time-series; (2) Toimpose multi-scale local self-supervision without mode collapse, it develops regularization termson the variance, covariance, orthogonality of local decoder features within the decoder; (3) Tofurther self-supervise the fine-tuning stage, the proposed method encourages segmentations frommultiple timepoints on unlabeled data to be consistent; (4) Across three large-scale longitudinal oneshot segmentation tasks on both elderly and pediatric populations, the developed framework yieldsimproved few-shot segmentation performance and higher longitudinal segmentation consistency.

Original:  Related work
Revised:  Related work

Original: Self-supervision.
Revised: Self-supervision.

Original: Self-supervised learning (SSL) methods aim to learn hierarchical visual representations from unannotated data, which can then be transferred to low-annotation regimes.
Revised: Self-supervised learning (SSL) methods aim to learn hierarchical visual representations from unannotated data, which can then be transferred to low-annotation regimes.

Original: Earlywork focused on pretext tasks where a handcrafted loss is used to pretrain networks via orientationprediction [18], context restoration [10, 45], channel prediction [65], among others.
Revised: Earlywork focused on pretext tasks where a handcrafted loss is used to pretrain networks via orientationprediction [17], context restoration [10, 43], channel prediction [61], among others.

Original: Given theirheuristic nature and suboptimal generalization, recent work instead focuses on data-driven SSL losses.
Revised: Given theirheuristic nature and suboptimal generalization, recent work instead focuses on data-driven SSL losses.

Original: Contrastive learning.
Revised: Contrastive learning.

Original: Constrastive learning [11,  21, 22, 23, 49, 55] (CL) typically transforms aninput image and asks the embeddings of the input image and its transformation (the positive pair) to beclose to one another and far apart from embeddings of other images ( negatives ) via a noise contrastiveestimation [20, 42] (NCE) loss.
Revised: Constrastive learning [11, 20, 21, 22, 47, 53] (CL) typically transforms aninput image and asks the embeddings of the input image and its transformation (the positive pair) to beclose to one another and far apart from embeddings of other images ( negatives ) via a noise contrastiveestimation [19, 40] (NCE) loss.

Original: While performant on image-level recognition tasks [12, 31], CLrequires non-trivial modification to extend to segmentation tasks, as described further below.
Revised: While performant on image-level recognition tasks [12, 29], CLrequires non-trivial modification to extend to segmentation tasks, as described further below.

Original: Negative-free representation learning.
Revised: Negative-free representation learning.

Original: In several applications, true negative samples may bedifficult to construct [25].
Revised: In several applications, true negative samples may bedifficult to construct [24].

Original: For example, when learning on intra-domain internal image patches [44],non-local spatial positions may be semantically similar, but NCE objectives push their embeddingsapart.
Revised: For example, when learning on intra-domain internal image patches [42],non-local spatial positions may be semantically similar, but NCE objectives push their embeddingsapart.

Original: This drawback may be mitigated via data-driven SSL methods which only use positive samplesand avoid low-diversity (or collapsed [29]) embeddings solutions  via predictor networks and custombackpropogation [13, 19] and careful regularization [6, 62].
Revised: This drawback may be mitigated via data-driven SSL methods which only use positive samplesand avoid identity solutions ( mode collapse ) via predictor networks and custom backpropogation [13,18] and careful regularization [6, 58].

Original: However, as above, these methodsoperate on global image embeddings and require modification for pixel-level tasks.
Revised: However, as above, these methods operate on global imageembeddings and require modification for pixel-level tasks.

Original: Spatial self-supervision.
Revised: Spatial self-supervision.

Original: Towards downstream  segmentation, recent work [2, 24, 34, 37, 38, 60, 67,68] encourages local single-scale features either within an image or across images  tocluster accordingto their semantic labels by constructing positive pairs using ground-truth labels.
Revised: Towards downstream semantic segmentation, recent work [2, 23, 32, 35,36, 57, 63, 64] encourages local single-scale features either within an image or across images tocluster according to their semantic labels by constructing positive pairs using ground-truth labels.

Original: To incorporate localand multi-scale spatial considerations into unpaired image translation, [44] imposed contrastive losseson randomly-sampled layer-wise encoder features by considering corresponding spatial indices aspositives and all other locations as negatives.
Revised: Toincorporate local and multi-scale spatial considerations into unpaired image translation, [42] imposedcontrastive losses on randomly-sampled layer-wise encoder features by considering correspondingspatial indices as positives and all other locations as negatives.

Original: Our work builds on this by instead only consideringtemporal positives (described below) in the layerwise losses alongside custom regularization whichavoids low-diversity decoder embeddings observed with naive application in Fig.
Revised: Our work builds on this by instead only considering temporal positives (described below) in the layerwise losses alongside custom regularization which avoids the decoder mode collapse observed with naive application in Fig.

Original: Temporal self-supervision.
Revised: Temporal self-supervision.

Original: SSL methods developed for video achieve high performance by exploiting temporally consistent transformation [5] and temporal pretext task [14, 28, 59].
Revised: SSL methods developed for video achieve high performance by ex- ploiting temporally consistent transformation [5] and temporal pretext task [14, 27, 56].

Original: However,longitudinal biomedical image time-series  have sparser sampling (typically 2–5 timepoints/subject)and have greater spatial extents (3D volumes instead of 2D images), which leads to distinct modelingconsiderations.
Revised: However, longitudinal biomedical image time-series typically have sparser sampling (typically 2–5 time- points/subject) and have greater spatial extents (3D volumes instead of 2D images), which leads to distinct modeling considerations.

Original: Emerging biomedical methods [15, 43, 66] enforce smooth trajectories for subjectwise images in the encoder latent space and deploy their methods on image-level downstream taskssuch as disease classification and age regression.
Revised: Emerging biomedical methods [15, 41, 62] enforce smooth trajec- tories for subject-wise images in the encoder latent space and deploy their methods on image-level downstream tasks such as disease classification and age regression.

Original: However, these methods focus on learning a globalembedding, without a clear extension to pixel-level tasks such as segmentation.
Revised: However, these methods focus on learning a global embedding, without a clear extension to pixel-level tasks such as segmentation.

Original: Biomedical image segmentation.
Revised: Biomedical image segmentation.

Original: Major challenges specific to biomedical segmentation include: (1)large 3D volumes; (2) limited sample sizes and annotations; and (3) non-i.i.d.
Revised: Major challenges specific to biomedical segmentation include: (1) large 3D volumes; (2) limited sample sizes and annotations; and (3) non-i.i.d.

Original: longitudinal acquisitionstracking temporal anatomical changes.
Revised: longitudinal acquisitions tracking temporal anatomical changes.

Original: To these ends, conventional approaches use a combinationof intensity-based probabilistic models and registration-driven atlas-based models [1, 26, 39, 58].
Revised: To these ends, conventional approaches use a combination of intensity-based probabilistic models and registration-driven atlas-based models [1, 25, 37, 55].

Original: Inparticular, longitudinal image analysis typically makes use of one or few longitudinal atlases [27, 32,47, 51, 52], which motivates the one-shot segmentation setting benchmarked in this paper.
Revised: In particular, longitudinal image analysis typically makes use of one or few longitudinal atlases [26, 30, 45, 49, 50], which motivates the one-shot segmentation setting benchmarked in this paper.

Original: More recently, deep segmentation networks achieve strong performance [8, 7, 40, 41, 48] givenenough training volumes.
Revised: More recently, deep segmentation networks achieve strong performance [8, 7, 38, 39, 46] given enough training volumes.

Original: In the low-annotation setting, weakly supervised methods develop customloss functions [30, 36], but may have drawbracks analogous to the handcrafted SSL losses describedabove.
Revised: In the low-annotation setting, weakly supervised methods develop custom loss functions [28, 34], but may have drawbracks analogous to the handcrafted SSL losses described above.

Original: Fortunately, recent data-driven self and semi-supervised methods are well-suited to pixel-levelprediction.
Revised: Fortunately, recent data-driven self and semi-supervised methods are well-suited to pixel-level prediction.

Original: For example, to pretrain an encoder for segmentation, [9, 63] develop application-specificpositive and negative sampling for contrastive training, where 2D slices from similar locationsin registered 3D volumes constitute positive pairs.
Revised: For example, to pretrain an encoder for segmentation, [9, 59] develop application-specific positive and negative sampling for contrastive training, where 2D slices from similar locations inregistered 3D volumes constitute positive pairs.

Original: While these methods have been successful intheir applications, they are inherently slice-based methods and are outperformed by well-tunedrandomly-initialized 3D baselines on our datasets (Tab. 1).
Revised: While these methods have been successful in theirapplications, they are inherently slice-based methods and are outperformed by well-tuned randomlyinitialized 3D baselines on our datasets (Tab. 1).

Original: Further, these self-supervised biomedicalsegmentation methods do not explicitly account for non i.i.d acquisitions.
Revised: Further, these self-supervised biomedical segmentation methods do not explicitly account for non i.i.d acquisitions.

Original: Lastly, to our knowledge,existing longitudinal deep learning work developed for biomedical segmentation is currently veryspecific to its target application [17, 35, 56] (for example, in tasks such as MS lesion changedetection [56]) or require supervised pre-training on annotated cross-sectional datasets [61], whereaswe develop a generic self-supervised spatiotemporal representation learning framework for non-i.i.d.data which can be applied to any downstream task in principle.
Revised: Lastly, to our knowledge, existinglongitudinal deep learning work developed for biomedical segmentation is currently very specific to itstarget application [16, 33], whereas we develop a generic self-supervised spatiotemporal representation learning framework for non-i.i.d. data which can be applied to any downstream task in principle.

Original:  Methodology
Revised:  Methodology

Original: Fig.
Revised: Fig.

Original: 2 illustrates the proposed framework.
Revised: 2 gives an overview of the proposed framework.

Original: The base U-Net architecture is pretrained end-to-end ina self-supervised manner with intra-subject spatiotemporal losses.
Revised: The base U-Net architecture is pretrained end-toend in a self-supervised manner with intra-subject temporal and spatial losses.

Original: On convergence, the pretrainednetwork parameters serve as an initialization for training downstream local spatiotemporal pixel-leveltasks (e.g., registration or segmentation).
Revised: On convergence, thepretrained network parameters serve as an initialization for training downstream local spatiotemporalpixel-level tasks (e.g., registration or segmentation).

Revised: This work focuses on one-shot segmentation,where the base network will be finetuned on one annotated subject.

Original: The main similarity loss is applied on multiscale localpatches from different timepoints of the same subject, which attracts features in correspondinglocations in separate timepoints together.
Revised: The main similarity loss is appliedon multiscale local patches from different timepoints of the same subject, which attracts featuresin corresponding locations in separate timepoints together.

Original: The high-dimensional U-Net features corresponding totheboxes at the same locations in Fig.
Revised: The high-dimensional U-Net featurescorresponding to the boxes in Fig.

Original: 2a should maintain high similarity despite varying appearance.
Revised: 2a should maintain high similarity despite varying appearance.

Original: Thefeature regularizers (Fig. 2b,c,d) avoiddegenerate U-Net decoder embeddings in the patch similaritytraining and the output regularizer (Fig. 2e) encourages finetuning consistency on unannotated data.
Revised: The feature regularizers (Fig. 2b,c,d) avoid U-Net decoder mode collapse in the patch similaritytraining and the output regularizer (Fig. 2e) encourages finetuning consistency on unannotated data.

Original: Setup.
Revised: Setup.

Original: The unlabeled dataset is a collection of N subjects, where each subject has at least twolongitudinal image acquisitions available during pretraining.
Revised: The unlabeled dataset is a collection of N subjects, where each subject has at least twolongitudinal image acquisitions available during pretraining.

Original: During every iteration, a pair of images( X ij , X ij +1 ) are randomly sampled from subject i ∈ { 1 , 2 , . . .
Revised: During every iteration, a pair of images( X ij , X ij +1 ) are randomly sampled from subject i ∈ { 1 , 2 , . . .

Original: , N } at distinct timepoints j and j + 1 ,where j ∈ { 1 , 2 , . .
Revised: , N } at distinct timepoints j and j + 1 ,where j ∈ { 1 , 2 , . .

Original: , T i − 1 } . T i indicates the number of registered images from subject i , and
Revised: , T i − 1 } , T i indicates the number of registered images from subject i , and

Original: X ∈ R W × H × D × C is a 3D volume of spatial dimension W × H × D and C channels.
Revised: X ∈ R W × H × D × C is a 3D volume of spatial dimension W × H × D and C channels.

Original: These channelsare typically multi-modality acquisitions (e.g., T1w and T2w MRI from the same subject).
Revised: Thesechannels are typically multi-modality acquisitions (e.g., T1w and T2w MRI from the same subject).

Original: Spatiotemporal patchwise similarity loss.
Revised: Spatiotemporal patchwise similarity loss.

Original: We aim to associate the local embeddings of X ij and
Revised: We aim to associate the local embeddings of X ij and

Original: X ij +1 such that the representations are longitudinally-aware.
Revised: X i j +1 such that the representations are longitudinally-aware.

Original: To this end, a weight-sharing 3D U-Net
Revised: To this end, a weight-sharing 3D U-Net

Original: G takes both images as input and produce a set of multi-scale CNN features { v } L , where eachelement v lij =
Revised: G takes both images as input and produce a set of multi-scale CNN features { v } L , where eachelement v lij =

Original: G l ( x ij ) ∈ R W l × H l × D l × C l indicates the output of the l th layer of interest.
Revised: G l ( x ij ) ∈ R W l × H l × D l × C l indicates the output of the l th layer of interest.

Original: M featurevectors are then randomly sampled from the 3D spatial indices of the feature map, where each featurevector v mlij represents a local patch of the input image. These patch-wise activations from  multiplelayers of G form hierarchical representations of local regions of the input image.
Revised: M featurevectors are then randomly sampled from the 3D spatial indices of the feature map, where each featurevector v mlij represents a local patch of the input image and the patch-wise activations from the multiplelayers of G form hierarchical representations of local regions of the input image.

Original: To maximize the similarity of corresponding local features (e.g., blue boxes in Fig. 2a) without negative samples, we use a projector MLP f and a predictor head p and extend [13] to patchwise operationsuch that matching spatial indices have high agreement.
Revised: To maximize the similarity of corresponding local features (e.g., blue boxes in Fig. 2) without negativesamples, we use a projector MLP f and a predictor head p and extend [13] to patchwise operationsuch that matching spatial indices have high agreement.

Original: The patchwise similarity loss between tworepresentations is defined as follows:
Revised: The patchwise similarity loss between tworepresentations is defined as follows:

Original: L ( v mlij , v mlik )
Revised: L ( v mlij , v mlik )

Original: = 12 D ( p 1 , z 2 ) + 12 D ( p 2 , z 1 )
Revised: = 12 D ( p 1 , z 2 ) + 12 D ( p 2 , z 1 )

Original: where D ( p, z )
Revised: where D ( p, z )

Original: =− p ||
Revised: =− p ||

Original: p || 2 ·
Revised: p || 2 ·

Original: z || z
Revised: z || z

Original: || 2 , z 1 , z 2 = f
Revised: || 2 , z 1 , z 2 = f

Original: ( v mlij ) , f ( v mlik ) and p 1 , p 2 = p ( z 1) , p ( z 2) .
Revised: ( v mlij ) , f ( v mlik ) and p 1 , p 2 = p ( z 1) , p ( z 2) .

Original: The total loss is an average of allsampled patches, across multilayer features: L sim = 1 L (cid:80) l ∈{ 1 , 2 ,...
Revised: The total loss is an average of allsampled patches, across multilayer features: L sim = 1 L (cid:80) l ∈{ 1 , 2 ,...

Original: L } 1 M (cid:80) m ∈{ 1 , 2 ,...
Revised: L } 1 M (cid:80) m ∈{ 1 , 2 ,...

Original: M } L ( v mlij , v mlik ) .
Revised: M } L ( v mlij , v mlik ) .

Original: Architectural challenges in multi-scale representation learning .
Revised: Architectural challenges in multi-scale representation learning .

Original: L sim applied to the hidden layersof a U-Net is found to maximize patchwise similarity of the encoder features (as expected) but leadto low-diversity and semantically-incoherent representations in the decoder layers as observed fromthe similarity maps in Fig.{+(Fig. 
+}
Revised: L sim applied to the hidden layersof a U-Net is found to maximize patchwise similarity of the encoder features (as expected) but leadto mode collapse in the decoder layers as observed from the learned feature similarity maps {+(Fig. 
+}

Original: 1B and empirical observations in
Revised: B).

Original: App E. We speculate that this is partiallyattributable to U-Net skip connections, which lead to a degenerate solution where the encoder learns representations which are good enough to minimize the decoder losses and  the decoder layers do nothave to learn useful representations which transfer.
Revised: We speculate that this is partially due to the skip connections between the encoder and decoder, which lead to a degenerate solution where the encoder learns representations which are good enoughto minimize the decoder losses and therefore the decoder layers do not have to learn meaningfulrepresentations.

Original: To this end, we develop several regularizationstrategies such that the decoder layers obtain diverse and semantically-coherent embeddings.
Revised: To this end, we develop several regularization strategies (described below) such thatthe decoder layers avoid collapse and all layers of the U-Net are adequately pretrained.

Original: Orthogonality.
Revised: Orthogonality.

Original: During model prototyping, we empirically observe that low-diversity embeddingsfirst originate in theU-Net bottleneck (Fig 1B cols 4,5 and App. Fig. 14 rows 2,3) which arethen upsampled hierarchically through the decoder.
Revised: During model prototyping, we empirically observe that decoder mode collapsefirst originates in the bottleneck latent-space of the U-Net.

Original: We therefore encourage decoupled bottleneckfeatures between the encoder and decoder.
Revised: We therefore encourage decoupledlatent features between the encoder and decoder.

Original: Revisiting the U-Net skip-connection, the encoderfeatures from the l -th layer v e ∈ R W l × H l × D l × C l are concatenated with the upsampled decoderfeatures v d ∈ R W l × H l × D l × C d , followed by a convolution to attain the same feature dimension as v e .
Revised: Revisiting the U-Net skip-connection, the encoderfeatures from the l -th layer v e ∈ R W l × H l × D l × C l are concatenated with the upsampled decoderfeatures v d ∈ R W l × H l × D l × C d , followed by a convolution to attain the same feature dimension as v e .

Original: This yields v ˆ d = Conv ( Concat ( v e , v d ))
Revised: This yields v ˆ d = Conv ( Concat ( v e , v d ))

Original: ∈ R W l × H l × D l × C l and by regularizing for orthogonalitybetween v e and v ˆ d in the projector embedding space, we implicitly encourage v d to learn moremeaningful representations instead of converging to degenerate solutions.
Revised: ∈ R W l × H l × D l × C l and by regularizing for orthogonalitybetween v e and v ˆ d in the projector embedding space, we implicitly encourage v d to learn moremeaningful representations instead of converging to degenerate solutions.

Original: Formally, the orthogonalityloss is defined as L O ( z e , z ˆ d ) = 1 M (cid:80) m ∈{ 1 ,...
Revised: Formally, the orthogonalityloss is defined as L O ( z e , z ˆ d ) = 1 M (cid:80) m ∈{ 1 ,...

Original: M }
Revised: M }

Revised: [Equation]

Revised: me ||

Original: [Equation]
Revised: [Equation]

Original:  || 2 where z me and z m ˆ d are projectedrepresentations via f at sampling location m , from the matched encoder/decoder layers, respectively.
Revised: · z m ˆ d || v m ˆ d || 2 where z me and z m ˆ d are projectedrepresentations via f at sampling location m , from the matched encoder/decoder layers, respectively.

Original: Variance and covariance.
Revised: Variance and covariance.

Original: We further encourage spatial variation and channel-wise decorrelation oflocal decoder features to avoid degenerate representations.
Revised: We further encourage spatial variation and channel-wise decorrelation oflocal decoder features to alleviate mode collapse.

Original: We extend the variance and covarianceterms of [6] towards patchwise multi-layer operation and apply them to the decoder layers.
Revised: We do this by extending the variance and covarianceterms of [6] towards patchwise multi-scale operation and apply these terms to the decoder layers.

Original: Astandard deviation loss encourages spatial variation above a threshold of η = 1 and is definedas L S ( z ) =
Revised: A standard deviation loss encourages spatial variation above a threshold of η = 1 and is definedas L S ( z ) =

Original: 1 k (cid:80) l ∈{ L − k,...
Revised: 1 k (cid:80) l ∈{ L − k,...

Original: L } (max(0 , η − S ( z l )) , where S ( z l ) =
Revised: L } (max(0 , η − S ( z l )) , where S ( z l ) =

Original: (cid:112) V ar ( z l )
Revised: (cid:112) V ar ( z l )

Original: + ϵ is the standarddeviation of M randomly-sampled projected features from the l -th layer and ϵ =  − 4 .
Revised: + ϵ is the standarddeviation of M randomly-sampled projected features from the l -th layer, ϵ = 10 − 4 is added fornumerical stability.

Original: A covarianceregularization decorrelates channelwise activations in the decoder to prevent low-diversity embeddingsby minimizing L C ( z )
Revised: A covariance regularization decorrelates channelwise activations in the decoder toprevent collapsed embeddings by minimizing L C ( z )

Original: = 1 k (cid:80) l ∈{ L − k,...
Revised: = 1 k (cid:80) l ∈{ L − k,...

Original: L } 1 n (cid:80) u ̸
Revised: L } 1 n (cid:80) u ̸

Original: = v [ C ( z )] 2 u,v , whereC ( z ) =
Revised: = v [ C ( z )] 2 u,v , where

Original:  1 n − 1 (cid:80) ni ( z i − z ) 
Revised: C ( z ) = 1 n − 1 (cid:80) ni ( z i − z ) ( z i − z

Original: ( z i −z ) T is the covariance matrix of n -D representation z and ( u, v ) indicates its off-diagonal indices.
Revised: T is the covariance matrix of n -D representation z and ( u, v )indicates its off-diagonal indices.

Original: Reconstruction.
Revised: Reconstruction.

Original: To further pretrain the decoder, we investigate reconstruction losses.
Revised: To further pretrain the decoder, we investigate reconstruction losses.

Original: However,as high-resolution information is passed through skip connections, a reconstruction loss for a UNet is near-trivially minimized and does not address degenerate solutions.
Revised: However,as high-resolution information is passed through skip connections, a reconstruction loss for a UNet is near-trivially minimized and does not address collapse.

Original: Therefore, inspiredby denoising autoencoders [57], we encourage network equivariance and invariance to geometric( A g ) and intensity-based transformations ( A i ), respectively, of the input image x , by modifying thereconstruction objective to a denoising loss L rec =
Revised: Therefore, inspired by denoisingautoencoders [54], we encourage network equivariance and invariance to geometric ( A g ) and intensitybased transformations ( A i ), respectively, of the input image x , by modifying the reconstructionobjective to a denoising loss L rec =

Original: ∥ G ( A i ( A g ( x )))
Revised: ∥ G ( A i ( A g ( x )))

Original: − A g ( x ) ∥ 22 .
Revised: − A g ( x ) ∥ 22 .

Original: Finetuning with longitudinal consistency regularization.
Revised: Finetuning with longitudinal consistency regularization.

Original: In a longitudinal segmentation setup, ifthe finetuning data do not cover the overall age range, the network weights may perform well on thefinetuned timepoints, but may perform poorly on out-of-distribution ages even when pretrained onunannotated data across all ages.
Revised: In a longitudinal segmentation setup, ifthe finetuning data do not cover the overall age range, the network weights may perform well on thefinetuned timepoints, but may perform poorly on out-of-distribution ages even when pretrained onunannotated data across all ages.

Original: We therefore develop a self-supervised longitudinal consistencyregularization term applied at the network output during finetuning to increase the intra-subjectagreement.
Revised: We therefore develop an self-supervised longitudinal consistencyregularization term applied at the network output during finetuning, to increase the intra-subjectagreement.

Original: Specific to segmentation, given registered image volumes, we formulate a segmentationconsistency loss as L cs = 1 − Dice ( G ( x ij ) , G ( x ij +1 )) and this is minimized alongside the supervisedsegmentation term below during finetuning.
Revised: Specific to segmentation, given registered image volumes, we formulate a segmentationconsistency loss as L cs = 1 − Dice ( G ( x ij ) , G ( x ij +1 )) and this is minimized alongside the supervisedsegmentation term below during finetuning.

Original: Total objective.
Revised: Total objective.

Original: During pretraining, our overall objective function is a weighted sum of the patchsimilarity loss, reconstruction loss, and three regularizations, and is defined as L PT =
Revised: During pretraining, our overall objective function is a weighted sum of the patchsimilarity loss, reconstruction loss, and three regularizations, and is defined as L PT =

Original: λ L sim +α L rec + µ L S ( z )+ γ L C ( z )+ β L O ( z ) .
Revised: λ L sim +α L rec + µ L S ( z )+ γ L C ( z )+ β L O ( z ) .

Original: During finetuning, we use a combined dice and cross entropyloss on one-shot supervised training pairs as L sup =
Revised: During finetuning, we use a combined dice and cross entropyloss on one-shot supervised training pairs as L sup =

Original: (1 − Dice ( G ( x ) , y ))
Revised: (1 − Dice ( G ( x ) , y ))

Original: +
Revised: +

Original: CE ( G ( x ) , y ) where y isthe groundtruth segmentation label.
Revised: CE ( G ( x ) , y ) where y isthe groundtruth segmentation label.

Original: Lastly, we additionally use the segmentation consistency loss
Revised: Lastly, we additionally use the segmentation consistency loss

Original: L cs for non i.i.d inputs, which forms our final finetuning loss L FT =
Revised: L cs for non i.i.d inputs, which forms our final finetuning loss L FT =

Original: L sup + L cs .
Revised: L sup + L cs .

Original:  Experiments
Revised:  ExperimentsData and segmentation tasks.

Original: Data and segmentation tasks.

Original: We conduct experiments on two de-identified longitudinal neuroimaging datasets, and specifically design three tasks to benchmark different extents of biomedical domaingaps between the finetuning and testing data.The main body of this work focuses on one-shot segmentation using one annotated subject (a common medical image analysis setting).
Revised: We conduct experiments on two de-identified longitudinalneuroimaging datasets, and specifically design three tasks to benchmark different extents ofbiomedical domain gaps between the finetuning and testing data.

Original: Benchmarks onfew-shot and fully-supervised segmentation tasks are provided in Appendix A. For both datasets, weperform a train/validation/test split on a subject-wise basis with 70%, 10% and 20% of the participants.
Revised: For both datasets, we performa train/validation/test split on a subject-wise basis with 70%, 10% and 20% of the participants.

Original: The validation set is used for model and hyperparameter selection and  results are reported on a heldout test set.
Revised: The validation set is used for model and hyperparameter selection, and the results are reported ona held-out test set.

Original: ANTs [3, 4] is used to perform inter-subject affine alignment for all  experiments, followed by intra-subject  deformable registration to obtain accurate spatiotemporal correspondence simand L cs calculation.
Revised: ANTs [3, 4] is used to perform inter-subject affine alignment for all methodsand experiments, followed by intra-subject nonlinear deformable registration to obtain accurate spatiotemporal correspondence for L sim and L cs calculation.

Original: Allimages are skull-stripped, bias-field corrected, and intensity-normalized.
Revised: All are skull corrected, and further details in the

Original: OASIS3 [33] is a publicly-available dataset consisting of 1639brain MRI scans of 992 longitudinallyimaged subjects. Each subject has 1–5 temporal acquisitions over a ∼ 5 -year long observationwindow, resulting in an aging cohort over the span 42 to 95 years which includes cognitivelynormal and mildly impaired individuals alongside subjects with  Alzheimer’s Disease.On OASIS3,we tackle whole-brain segmentation using the FreeSurfer label convention [16].
Revised: OASIS3 [31] is a publicly-available dataset consisting of 1639 MRI scans of 992 longitudinallyimaged subjects (each with 1–5 temporal acquisitions over a ∼ 5 -year long observation window),resulting in an aging cohort over the span 42 to 95 years, including cognitively normal and mildlyimpaired individuals alongside subjects with with early-stage Alzheimer’s Disease.

Original: Cross-sectional
Revised: Cross-sectional

Original: FreeSurfer anatomical segmentation was done as part of the data release.
Revised: FreeSurfer anatomical segmentation was done as part of the data release.

Original: Observing strong temporalinconsistency (see App. C.2), we further perform longitudinal FreeSurfer [47] to improve thetemporal consistency of the reference segmentation.
Revised: Observing strong temporalinconsistency (see appendices), we further perform longitudinal FreeSurfer [45] to improve the temporal consistency of the reference segmentation.

Original: We exclude labels that have less than 100 voxelsin all subjects, which results in 33 labels for segmentation training and evaluation.
Revised: We exclude labels that have less than 100 voxels in allsubjects, which results in 33 labels for segmentation training and evaluation.

Original: Finetuningforone-shot segmentation is performed on a single FreeSurfer-annotated subject with four timepoints.
Revised: Finetuning is performedon a single FreeSurfer-annotated subject with four timepoints with the remainder used for evaluation.

Original: IBIS is an  infant brain imaging study, which longitudinally acquires 1272 structural T1w/T2w MRIfrom 552 infants across both controls and infants at a high-risk for Autism Spectrum Disorder (ASD)over a span of 3 to 36 months of age.
Revised: IBIS is an ongoing infant brain imaging study, which longitudinally acquires 1272 structural T1w/T2w MRI from 552 infants across both controls and infants at a high-risk for Autism Spectrum Disorder(ASD) over a span of 3 to 36 months of age.

Original: We tackle two distinct tasks: subcortical segmentation ( IBISsubcort ) and white/gray matter tissue segmentation ( IBIS-wmgm ).
Revised: We tackle two distinct tasks: subcortical segmentation( IBIS-subcort ) and white/gray matter tissue segmentation ( IBIS-wmgm ).

Original: For IBIS-subcort , a multi-atlasmethod [54] cross-sectionally segments sub-cortical grey matter (relevant to ASD [49]) into 13structures of interest, which are then followed by manual corrections.
Revised: For IBIS-subcort , a multiatlas method [52] cross-sectionally segments sub-cortical grey matter (relevant to ASD [47]) into13 structures of interest, which are then followed by manual corrections.

Original: For one-shot benchmarking,finetuning is performed only on a single longitudinally-labeled subject.
Revised: Similar to OASIS3 above,finetuning is performed only on a single longitudinally-labeled subject.

Original: Further, we use the IBISwmgm setting to simulate a real-world use-case.
Revised: Further, we use the IBISwmgm setting to simulate a real-world use-case.

Original: Briefly, brain MRI segmentation into grey/whitematter is straightforward at 24-36 months of age. However, ∼ 6-month-old brain segmentationremains elusive without manually labeled datasets for supervision due to white matter myelinationleading to isointense appearance at that age [53] (e.g., Fig. 2a and App. D.2).
Revised: Briefly, brain MRI segmentation into grey/whitematter is straightforward at 24-36 months of age, but 6-month-old segmentation remains elusivewithout manually labeled datasets for supervised training due to myelination trends leading toisointense appearance at those ages [51] (examples in Fig. 2a).

Original: We therefore investigatefinetuning all benchmarked methods on a single 36 month old image which can be reliably segmentedwith [46] and then evaluate segmentation deployment with a strong domain shift on 6 month oldisointense images and labels (whose ground truth labels are generated by supervised training [64]).
Revised: We therefore investigate finetuning allbenchmarked methods on a single 36 month old image which can be reliably segmented with [44]and then evaluate segmentation deployment with a strong domain shift on 6 month old isointenseimages and labels (whose ground truth labels are generated by supervised training [60]).

Original: Baselines and Evaluation Strategies.
Revised: Baselines and Evaluation Strategies.

Original: We analyze segmentation performance and longitudinalconsistency against well-tuned randomly initialized 2D/3D U-Nets ( RandInitUnet.2D/3D ) andvarious high-performing self-supervised pretraining methods. These include the pretext-task based 
Revised: We analyze segmentation performance and longitudinalconsistency against well-tuned randomly initialized 2D/3D U-Nets ( RandInitUnet.2D/3D ) and various high-performing self-supervised pretraining methods including the pretext-task based Context

Original: Context Restoration [10], longitudinal representation learning based LNE [43], along with thecontrastive learning based GCL [9] and PCL [63] methods which operate on image slices.
Revised: Restoration [10], longitudinal representation learning based LNE [41], along with contrastivelearning based GCL [9] and PCL [59] methods which operate on image slices.

Original: We alsorepurpose PatchNCE [44] for segmentation to evaluate its generic representation learning capabilities.
Revised: We also repurpose PatchNCE [42] to evaluate its generic representation learning capabilities.

Original: All methods are pretrained and finetuned with both geometric and intensity-based augmentation.
Revised: All methods are bothpretrained and finetuned with both geometric and intensity-based augmentation.

Original: We quantify network performance via commonly used scores such as the Dice coefficient, IoU, andthe 95-th percentile of the Hausdorff distance.
Revised: We quantify network performance via commonly used scores such as the Dice coefficient, IoU, andthe 95-th percentile of the Hausdorff distance.

Original: More importantly, we also quantify the longitudinal agreement between intra-subject non-linearly registered temporal segmentations via scores such as thespatiotemporal consistency of segmentation [35] STCS = 2 | S 1 ∩ S 2 | | S 1 | + | S 2 | where S 1 and S 2 are temporalsegmentation predictions from non-linearly registered input images; and the absolute symmetrized percent change [47] ASPC = 100 | V 2 − V 1 | 0 .
Revised: More importantly, we also quantify the longitudinalagreement between intra-subject non-linearly registered temporal segmentations via scores such as thespatiotemporal consistency of segmentation [33] STCS = 2 | S 1 ∩ S 2 | | S 1 | + | S 2 | where S 1 and S 2 are temporalsegmentation predictions from non-linearly registered input images; and the absolute symmetrized percent change [45] ASPC = 100 | V 2 − V 1 | 0 .

Original: 5( V 1 + V 2 ) , where V 1 and V 2 are the volume of a structure calculatedfrom S 1 and S 2 .
Revised: 5( V 1 + V 2 ) , where V 1 and V 2 are the volume of a structure calculatedfrom S 1 and S 2 .

Original: We also report STCS and ASPC on the groundtruth segmentations as a reference.
Revised: We also report STCS and ASPC on the groundtruth segmentations as a reference.

Original: Implementation details.
Revised: Implementation details.

Original: We train a 3D U-Net [48] as the base image-to-image architecture with fourlevels of up/down sampling and repeated Conv-BN-ReLU blocks (all architectural details provided in App. B).
Revised: We train a 3D U-Net [46] as the base image-to-image architecture withfour levels of up/down sampling and repeated Conv-BN-ReLU blocks, with full architectural detailsprovided in the supplementary material.

Original: The projector head consists of a 3-layer MLP with 2048 nodes per layer.
Revised: The projector head consists of a 3-layer MLP with 2048nodes per layer.

Original: Following [13], weapply batch normalization after each MLP, followed by ReLU activation, and l 2 normalization ofthe final activation.
Revised: Following [13], we apply batch normalization after each MLP, followed by ReLUactivation, and l 2 normalization of the final activation.

Original: The predictor is a 3-layer bottlenecked MLP with widths of 2048-256-2048.
Revised: The predictor is a 3-layer bottlenecked MLPwith widths of 2048-256-2048.

Original: Weapply geometric (left-right flip, random affine warps) and intensity augmentations including randomblur, noise, gamma contrast enhancement.
Revised: We apply geometric (left-right flip, random affine warps) and intensityaugmentations including random blur, noise, gamma contrast enhancement.

Original: Additional MRI-specific augmentations with random biasfield and motion artifacts are also applied and followed by 128 3 random cropping.
Revised: Additional MRI-specificaugmentations with random bias field and motion artifacts are also applied and followed by 128 3random cropping.

Original: We use a batch sizeof 3 crops and an initial learning rate of 2 × 10 − 4 for both pretraining and finetuning.
Revised: For both pretraining and finetuning, we use a batch size of 3 crops, with aninitial learning rate of 2 e − 4 .

Original: All networksare trained with the Adam optimizer ( β 1 = 0 .
Revised: All networks are trained with the Adam optimizer ( β 1 = 0 .

Original: 9 during pretraining and β 1 = 0 .
Revised: 9 duringpretraining and β 1 = 0 .

Original: 5 during finetuning,and β 2 = 0 .  on a single Nvidia RTX8000 GPU (45GB vRAM).
Revised: 5 during finetuning, and β 2 = 0 . 99 ) on a single Nvidia RTX8000 GPU (45GBvRAM).

Original: The networks are pretrainedfor a maximum of 30 , 000 steps and the best model based on validation performance is used forfine-tuning for another 35 , 000 steps, alongside linear learning rate decay.
Revised: The networks are pretrained for a maximum of 30 , 000 steps, and the best model based onvalidation performance is used for fine-tuning for another 35 , 000 steps, alongside linear learning ratedecay.

Original: All experiments are run ona fixed random seed due to limited computational budgets.
Revised: All experiments are run on a fixed random seed due to limited computational budgets.

Original: Based on the ablation analysis in Tab. 2,we empirically choose λ = 1 , α = 10 , γ = 1 e − 3 , β = 100 for all datasets, and use µ = 10 − 2 for OASIS3, µ = 10 − 3 for IBIS.
Revised: Basedon the ablation analysis in Tab. 2, we empirically choose λ = 1 , α = 10 , γ = 1 e − 3 , β = 100 for alldatasets, and use µ = 10 − 2 for OASIS3, µ = 10 − 3 for IBIS.

Original: Further details on the configurations, implementation of our methodand other baselines are provided in Appendix B.
Revised: Further details on the configurations,implementation of our method and other baselines are provided in the appendices.

Original: Segmentation and longitudinal consistency results.
Revised: Segmentation and longitudinal consistency results.

Original: Fig.
Revised: Fig.

Original: 3 qualitatively demonstrates improvedgeneralization using our method on unseen longitudinal data (row 1-3), especially on data displayingrapid intra-subject temporal developments (IBIS- wmgm,subcort ).
Revised: 3 qualitatively demonstrates improvedgeneralization using our method on unseen longitudinal data (row 1-3), especially on data displayingrapid intra-subject temporal developments (IBIS- wmgm,subcort ).

Original: These improvements are consistentwith the quantitative results presented in Fig.
Revised: These improvements are consistentwith the quantitative results presented in Fig.

Original: 4/Tab. 1 which indicate both improved segmentationperformance and longitudinal consistency.
Revised: 4/Tab. 1 which indicate both improved segmentationperformance and longitudinal consistency.

Original: In the strong domain shift setting of IBIS-wmgm , we see anear ten-point increase in median dice over most baselines.
Revised: In the strong domain shift setting of IBIS-wmgm , we see anear ten-point increase in median dice over most baselines.

Original: With moderate shifts in IBIS-subcort , wesee appreciable increases in performance and consistency.
Revised: With moderate shifts in IBIS-subcort ,we see appreciable increases in performance and consistency.

Original: We note that brain segmentation on adultbrain MRI data from OASIS3 is a comparatively easier task as adult neuroimages do not significantlychange appearance between imaging sessions.
Revised: We note that brain segmentationon adult brain MRI data from OASIS3 is a comparatively easier task as adult neuroimages do notsignificantly change appearance between imaging sessions.

Original: Therefore, several baselines are able to match (butnot exceed) the segmentation performance of our method on OASIS3.
Revised: Therefore, several baselines are ableto match (but not exceed) the segmentation performance of our method on OASIS3.

Original: However, all baselines areoutperformed by ours on all datasets in terms of longitudinal-consistency which is essential tonon-i.i.d. statistical analysis.
Revised: However, allbaselines are outperformed by ours on all datasets in terms of longitudinal-consistency which isessential to non-i.i.d. statistical analysis.

Original: In particular, both our pretraining ( Ours w/o L cs ) and finetuning( Ours w/ L cs ) methods show STCS and ASPC improvements over all of the compared settings.
Revised: In particular, both our pretraining ( Ours w/o L cs ) andfinetuning ( Ours w/ L cs ) methods show STCS and ASPC improvements over all of the comparedsettings.

Original: Fig.
Revised: Fig.

Original: 3 (bottom row) shows an example of temporal predictions on two unseen timepoints from IBIS-wmgm. The predictions from Context Restoration [10] match only the input image intensityand lack anatomical and longitudinal consistency, LNE [43] introduces false positive predictions intemporal lobe (within the orange circle), and the proposed method yields a more spatiotemporallyand anatomically consistent segmentation.
Revised: 3 (bottom row) shows an example of temporal predictions on two unseen timepointsfrom IBIS-wmgm, where the predictions from Context Restoration [10] match only the input imageintensity and thus lack anatomical and longitudinal consistency; the predictions from LNE [41]introduce false positive predictions in temporal lobe (within the orange circle) and our method yieldsa more spatiotemporally and anatomically consistent segmentation.

Original: While the main body of this work focuses on one-shotsegmentation, we also observe gains in the few-shot and fully-supervised segmentation settings,with results provided in Appendix A and Suppl.

Original: Tabs. 3 and 4, respectively. Qualitative self-supervised spatiotemporal similarity.
Revised: Qualitative self-supervised spatiotemporal similarity.

Original: In Fig.
Revised: In Fig.

Original: 1, we qualitatively examine thelearned visual representations of the proposed method via intra-subject temporal self-similarity ( C )and compare it to two of its variants which either use contrastive learning with unsupervised negatives(A) or negative-free representation learning (B).
Revised: 1, we qualitatively examine thelearned visual representations of the proposed method via intra-subject temporal self-similarity ( C )and compare it to two of its variants which either use contrastive learning with unsupervised negatives(A) or negative-free representation learning (B).

Original: We calculate the per-layer multiscale feature selfsimilarity between the query and each key from the intra-subject feature maps at a different age (bluebox).
Revised: We calculate the per-layer multiscale feature selfsimilarity between the key and each query from the intra-subject feature maps at a different age (bluebox).

Original: In row A, we see that assuming that all spatial indices not in correspondence constitute negativepairs leads to highly-positionally dependent representations in the decoder which carry low semanticmeaning (e.g., in the adult data, the similarity to localize to the ventricles in the coronal view).
Revised: In row A, we see that assuming that all spatial indices not in correspondence constitute negativepairs leads to highly-positionally dependent representations in the decoder which carry low semanticmeaning (e.g., in the adult data, the similarity to localize to the ventricles in the coronal view).

Original: Bydiscarding all negatives in row B, we observe semantically-incoherent andlow-diversity embeddingsand artifacts in the decoder layers on both datasets.
Revised: Bydiscarding all negatives in row B, we observe mode collapse and artifacts in the decoder feature mapson both datasets.

Original: Finally, with careful regularization, our methodsdiscards all negative pairs and attains semantically and positionally relevant representations.
Revised: Finally, with careful regularization, our methods discards all negative pairs andattains semantically and positionally relevant representations.

Original: Ablations.
Revised: Ablations.

Original: As the proposed method consists of several moving parts, an ablation analysis is conductedover different model configurations, hyperparameters, and loss functions, reported in Tab.
Revised: As the proposed method consists of several moving parts, an ablation analysis is conductedover different model configurations, hyperparameters, and loss functions, reported in Tab.

Original: 2 consistingof average dice coefficients.
Revised: 2 consistingof average dice coefficients.

Original: The combination of all proposed components yields optimal results.
Revised: The combination of all proposed components yields optimal results.

Original: [Table]
Revised: [Table]

Original: Ours w/ L cs 0.

Original: Few-shot and fully-supervised results are provided in Suppl. Tabs. 3 and 4, respectively. Row A starts with a base setting where only four encoder layers from the U-Net are selected for
Revised: Row A starts with a base setting where only four encoder layers from the U-Net are selected for

Original: L sim computation and a small MLP width of 256 is used for the projection and prediction heads.
Revised: L sim computation and a small MLP width of 256 is used for the projection and prediction heads.

Original: Here, IBIS-subcort and OASIS3 results are competitive with randomly initialized U-Net, as expectedgiven the lack of auxiliary losses, data augmentation, and regularizations.
Revised: Here, IBIS-subcort and OASIS3 results are competitive with randomly initialized U-Net, as expectedgiven the lack of auxiliary losses, data augmentation, and regularizations.

Original: However, the IBIS-wmgmexperiment already shows a 2% improvement over random initialization, indicating benefits of usingpatchwise similarity losses for better out-of-distribution generalization even with suboptimal setups.
Revised: However, the IBIS-wmgmexperiment already shows a 2% improvement over random initialization, indicating benefits of usingpatchwise similarity losses for better out-of-distribution generalization even with suboptimal setups.

Original: Row B: With larger projector and predictor networks, we observe improvements on two out of threedatasets, which is consistent with trends observed on natural images [12, 13].
Revised: Row B: With larger projector and predictor networks, we observe improvements on two out of threedatasets, which is consistent with trends observed on natural images [12, 13].

Original: Rows C–F: On adding decoder layers to L sim and introducing L rec alongside data augmentation(without any regularization), we typically observe inconsistent dataset-specific trends which arisefrom unregularized representations (e.g., Fig. 1B).
Revised: Rows C–F: On adding decoder layers to L sim and introducing L rec alongside data augmentation(without any regularization), we typically observe inconsistent dataset-specific trends which arisefrom mode collapsed representations (e.g., Fig. 1 B).

Original: We speculate that a poorly trained decoder (dueto a lack of regularization) may be equivalent to random initialization in this context.
Revised: We speculate that mode collapse (due to alack of regularization) may be equivalent to random initialization.

Original: However, acombination of these components (Row F) leads to an appreciable increase in performance.
Revised: However, a combination of thesecomponents (Row F) leads to an appreciable increase in performance.

Original: Rows G–K: When orthogonal regularization and/or covariance/variance regularization is used, we observe the best performance when they are applied together alongside augmentation and L Rec .
Revised: Rows G–K: When orthogonal regularization and/or covariance/variance regularization is used, we observe the best performance when they are applied together alongside augmentation and L Rec .

Original: In rows
Revised: In rows

Original: J and K, we observe that different hyperparameters are optimal for OASIS3 and IBIS (which alreadyoutperform all baseline methods in Tab. 1), which is intuitive as these are drastically different cohorts.
Revised: J and K, we observe that different hyperparameters are optimal for OASIS3 and IBIS (which alreadyoutperform all baseline methods in Tab. 1), which is intuitive as these are drastically different cohorts.

Original: [Table]
Revised: [Table]

Original: Row L: Finally, the overall proposed model is achieved when L cs is added to the finetuning objective,which yields strong improvements for IBIS-{ wmgm,subcort } and maintains OASIS3 performance.
Revised: Row L: Finally, the overall proposed model is achieved when L cs is added to the finetuning objective,which yields strong improvements for IBIS-{ wmgm,subcort } and maintains OASIS3 performance.

Original:  Discussion
Revised:  Discussion

Original: Limitations and future work.
Revised: Limitations and future work.

Original: The presented work opens up many follow-up questions which willbe tackled in future work: (1) Our proposed losses enable better training and performance, but requirethe tuning of several regularization weights and layer selections which may reveal dataset specificpatterns (e.g., rows J, K in Tab. 2).
Revised: The presented work opens up many follow-up questions which willbe tackled in future work: (1) Our proposed losses enable better training and performance, but requirethe tuning of several regularization weights and layer selections which may reveal dataset specificpatterns (e.g., rows J, K in Tab. 2).

Original: The weights and layers selected here were chosen based on limitedexploratory experiments on the validation sets due to computational budgets and future work willexhaustively search the hyperparameter space for optimal performance.
Revised: The weights and layers selected here were chosen based on limitedexploratory experiments on the validation sets due to computational budgets and future work willexhaustively search the hyperparameter space for optimal performance.

Original: Our pretraining assumesaccurate non-linear intra-subject registration, which may be non-trivial in edge cases like modalitieswith strong distortion and artifacts (e.g., eddy corruption in diffusion MRI).
Revised: Our pretraining assumesaccurate non-linear intra-subject registration, which may be non-trivial in edge cases like modalitieswith strong distortion and artifacts (e.g., eddy corruption in diffusion MRI).

Original: However, when studyingpre and post-operative imaging (e.g., surgical excision of lesions), large topological changes break theassumptions of our model and will require the development of lesion-masked positive patch samplingmethods.
Revised: However, when studyingpre and post-operative imaging (e.g., surgical excision of lesions), large topological changes break theassumptions of our model and will require the development of lesion-masked positive patch samplingmethods.

Original: (3) We use a two-stage pre-training and finetuning approach and it is plausible that theproposed method can be reduced to a single-stage combined framework.
Revised: (3) We use a two-stage pre-training and finetuning approach and it is plausible that theproposed method can be reduced to a single-stage combined framework.

Original: (4) While this paper focusedon the one-shot segmentation application, the pretraining framework is generic to any pixel-level task(e.g.
Revised: (4) While this paper focusedon the one-shot segmentation application, the pretraining framework is generic to any pixel-level task(e.g.

Original: , registration) and its extension to such tasks will be explored.
Revised: , registration) and its extension to such tasks will be explored.

Original: (5) While the proposed methodsyield strong longitudinal segmentation consistency improvements across all datasets, we note thatabsolute segmentation performance gains in an elderly cohort (OASIS3) are modest in comparisonto the rapidly developing infant dataset (IBIS) where higher gains are achieved.
Revised: (5) While the proposed methodsyield strong longitudinal segmentation consistency improvements across all datasets, we note thatabsolute segmentation performance gains in an elderly cohort (OASIS3) are modest in comparisonto the rapidly developing infant dataset (IBIS) where higher gains are achieved.

Original: Future work willfurther investigate these performance differences between populations of differing temporal trends.
Revised: Future work willfurther investigate these performance differences between populations of differing temporal trends.

Original: Potential negative impacts.
Revised: Potential negative impacts.

Original: The proposed method generically applies to medical image time-seriesand we do not anticipate negative impacts beyond those that currently exist for segmentation methods.
Revised: The proposed method generically applies to medical image time-seriesand we do not anticipate negative impacts beyond those that currently exist for segmentation methods.

Original: However, while our tasks have impacts on understanding real-world disease mechanisms, we cannotclaim any further insight into differences between subpopulations, as such analysis requires close collaboration with clinicians, neuroscientists, and biostatisticians, which is beyond the scope of this work.
Revised: However, while our tasks have impacts on understanding real-world disease mechanisms, we cannotclaim any further insight into differences between subpopulations, as such analysis requires close collaboration with clinicians, neuroscientists, and biostatisticians, which is beyond the scope of this work.

Original: Conclusions.
Revised: Conclusions.

Original: This paper addressed several open questions regarding the self-supervised pretrainingand finetuning of image-to-image architectures on longitudinal volumes using objective functionswhich exploit both intra-subject spatial and temporal self-similarity.
Revised: This paper addressed several open questions regarding the self-supervised pretrainingand finetuning of image-to-image architectures on medical volumes using objective functions whichexploit both intra-subject spatial and temporal self-similarity.

Original: It developed a local negativesample-free framework that trains multiple multi-scale hidden layers of image-to-image architecturesthat then enabled improved downstream segmentation performance, all while achieving semanticallymeaningful representations via careful regularization of the decoder activations.
Revised: It developed a local negative samplefree framework that trains multiple multi-scale hidden layers of image-to-image architectures thatthen enabled improved downstream segmentation performance, all while avoiding mode collapse viacareful regularization of the decoder activations.

Original: During finetuning,it similarly developed a simple consistency-regularization objective which encourages longitudinalagreement between predictions on unlabeled data.
Revised: During finetuning, it similarly developed a simpleconsistency-regularization objective which encourages longitudinal agreement between predictionson unannotated data.

Original: When applied to large-scale neurodevelopingand neurodegerative longitudinal images, the proposed framework yielded improved segmentationperformance and temporal consistency, both of which are crucial to statistical analyses of mechanismsof interest such as Alzheimer’s Disease  and Autism Spectrum Disorder.
Revised: When applied to large-scale neurodeveloping and neurodegerative longitudinalimage sets, the proposed framework yielded improved segmentation performance and temporalconsistency, both of which are crucial to statistical analyses of mechanisms of interest such as Alzheimer’s Disease (OASIS3) and Autism Spectrum Disorder (IBIS).

Original: Our  code is attached.
Revised: Our source code is attached.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See abstract and Sec
Revised: [Yes] See abstract and Sec.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Sec. 5.
Revised: [Yes] See Sec. 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Sec. 5.
Revised: [Yes] See Sec. 5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No newtheoretical results claimed.
Revised: No new theoretical results claimed.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: imental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Code: code with instructions is included with the supplementary materials.
Revised: [Yes] Code: Codewith instructions is included with the supplementary materials.

Original: Data: public data links
Revised: Data: OASIS3 is a

Original: and procedures are described in the supplementary material.
Revised: publicly available dataset which can be accessed with grant. However, IBIS is an on-going research study that has not been publicly released as of now.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See ‘Implementation details’ in Sec. 4 and App.
Revised: [Yes] See ‘Implementation details’ in Sec. 4 and supplementary.

Original: B.(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] See Fig.
Revised: [Yes] See Fig.

Original: 4, Tab. 1, Tab.
Revised: 4, Tab. 1, Tab.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See ‘Implementation details’ in Sec.
Revised: [Yes] See ‘Implementation details’ inSec.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See ‘Baselines and
Revised: [Yes] See ‘Baselines and

Original: Evaluation Strategies’ in Sec.
Revised: Evaluation Strategies’ in Sec.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: Our code is included inthe supplementarymaterial
Revised: Our code is included in supplementary

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A] See [33] and [50] for how imaging consent was acquired for
Revised: [N/A] See [31] and [48] for how imaging consent was acquired for

Original: OASIS3 and IBIS, respectively.
Revised: OASIS3 and IBIS, respectively.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See ‘Data and preprocessing’ in Sec. 4.
Revised: [Yes] See ‘Data and preprocessing’ in Sec. 4.

Original: Skull-
Revised: Skull-

Original: stripping deidentifies human faces as only the brain is available.
Revised: stripping deidentifies human faces as only the brain is available.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [Yes] See supplementary materials.
Revised: [Yes] See supplementary materials.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] See supplementary materials.A Additional Results and Experiments [Figure]
Revised: [N/A] See supplementary materials.

Original: Self-supervised similarity maps from anatomically-relevant key points are visualized in Figure 5.

Original: Using only self-supervision, our model learns semantically and positionally-aware representations.

Original: [Figure]

Original: This follows the same nomenclature (A–C) as Fig.

Original: 1 with analysis in Sec.

Original: 4 ( text).

Original: Label-wise longitudinal consistency.

Original: To observe structure-specific performance, we report labelwise longitudinal consistency scores (as measured by STCS [35]) for OASIS3 and IBIS- subcort onthe test set for the best performing models (as measured by Figure 4 of the main text) in Figure 6.

Original: IBIS- wmgm was not included as it only has two anatomical structures.

Original: Few-shot and fully-supervised segmentation.

Original: While we focus on one-shot segmentation in ourmain paper (corresponding to the common real-world setting of single atlas-based segmentation),the proposed framework has benefits beyond one-shot segmentation.

Original: We present results obtainedby using 10% and 100% of all the labeled training sets in Tables 3 and 4, respectively.

Original: As havingmore supervised data may necessitate changing our hyperparameters, we explore reducing the weightof the consistency-regularization from 1 . 0 to 0 .

Original: 1 for both experiments.

Original: In the 10% setting, we findthat Ours w/ L cs with a weight of 0 .

Original: 1 obtains optimal performance and consistency.

Original: In the 100%setting, we find that including L cs actually degrades performance and that optimal results are achievedwith just the proposed pretraining ( Ours w/o L cs ).

Original: We hypothesize that these trends arise fromimperfect deformable registration of the intra-subject images as they were warped considering onlytheir intensity and not semantic structure (see App Section C.1).

Original: Therefore, having the imperfectself-supervision of L cs in the low-annotation regime increases performance.

Original: However, hundredsof annotated samples remove the need for L cs , while maintaining the segmentation performancebenefits of our pretraining framework over existing work.

Original: [Table]

Original: This table quantifies test set performance (Dice, IoU, HD95) and longitudinal consistency (ASPC [47],

Original: [Table]

Original: [Table]

Original: U-Net configuration.

Original: For consistency, we use the same base U-Net architecture for all baselines.

Original: Itsconfiguration is modeled based on one-shot segmentation mean Dice validation results on OASIS3presented in Table 5 and evaluated over several training crop sizes, normalization layers, and channelwidth multipliers.

Original: Importantly, all configurations were trained without any pretraining to obtain abaseline.

Original: We observe that a random crop window of 128 3 is optimal (rows A, B, C) and find that

Original: Instance Normalization (row D) instead of Batch Normalization degrades performance.

Original: Finally, weobserve almost no performance degradation when reducing the model size via the channel widthmultiplier from 24 to 16 (row C vs E) and therefore use 16 for computational efficiency.

Original: [Table]

Original: Training a randomly-initialized U-Net with L cs .

Original: To investigate the standalone benefit of L cswithout any regularized pretraining, we apply L cs to a randomly initialized 3D U-Net in the one-shotsegmentation setting for IBIS- subcort and report its results in Fig.

Original: In the one-shot setting, L cs improves mean dice by 0.027, which is comparable to the proposed model without L cs .

Original: Optimal results are obtained under combination with our regularized pretraining framework.

Original: Using last decoder layer in loss.

Original: We investigate the impact of including the full-resolution feature(layer 21 in Table 9) in the EncDec loss configuration described in Section B.1.

Original: As shown in Figure 8,this notably degrades performance, indicating sensitivity to the exact layers used for L sim calculation.

Original: Additional modeling decisions.

Original: In addition to the quantitative ablations over our modeling choicesincluded in the main paper and appendix, we make a few modeling decisions detailed in Table 6based on qualitative assessments of the representation quality using feature similarity maps.

Original: [Table]

Original: Additional one-shot quantification.

Original: As Table 1 and Figure 4 (main text) report median scores,

Original: Table 7 reports means and standard deviations for further interpretation.

Original: Similar to the medians, ourmethod improves both mean performance and longitudinal consistency on IBIS-{ wmgm, subcort }.

Original: On OASIS3, our Dice and IoU performance improves on all baselines except for LNE (with whom itis strongly competitive) and exceeds all baselines in terms of longitudinal consistency (ASPC, STCS).

Original: [Table]

Original: Additional IBIS- wmgm segmentation results.

Original: For one-shot IBIS- wmgm segmentation, we train ona single 36 month old T1w/T2w MR image and quantitatively evaluate on 6 month old MR images(for additional motivation see Appendix Section D.2).

Original: In Figure 9, we visualize predictions usingadditional baselines and our framework on a held-out test subject.

Original: As the infant brain matures overtime (Appendix Section D.2), the rows are arranged in order of the most to the least amount ofbiological domain-shift w.r.t.

Original: the 36 month-old training image.

Original: Further, as the 12, 24, and 36 monthold held-out images do not have ground truth segmentations available, we are restricted to qualitativeevaluations of segmentation quality.

Original: [Figure]

Original: [Figure]

Original: Figure 10 illustrates an in-depth overview of the proposed pretraining and representation learning framework.

Original: In the proposed loss configuration ( EncDec ), Pre-activation features from layers{ 1 , 3 , 5 , 7 , 9 , 12 , 15 , 18 } in Table 5 are sampled spatially to extract channel-wise vectors which arefed into the projector and predictor MLPs for L sim calculation.

Original: Patch features from layer { 8 , 12 } inthe bottleneck (processed by the projector) are used for the orthogonality regularizer L O and layers{ 12 , 15 , 18 } in the decoder are used for the L S and L C regularizers.

Original: In the ablations in Table 2 (maintext), the Enc loss setting refers to using features from layers { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 } for L sim .

Original: Each iteration of pretraining loads three corresponding crops from two intra-subject images each for

Original: L sim calculation.

Original: Higher batch sizes could have been used for the proposed loss functions but werenot for consistency with one of our baselines ( PatchNCE [44]) which is highly memory-intensive dueto the need for a large number of negative samples alongside 3D computation.

Original: During finetuning, the segmentation loss ( L sup ) is applied to the original unwarped images and theconsistency loss ( L cs ) is applied to the nonlinearly aligned images.

Original: When calculating L sup and L cs ,we perform individual forward/backward passes for each loss due to memory limitations as gradientaccumulation over two forward passes was found to decrease performance on a validation set.

Original: B.2 Architectures

Original: The MLP architectures for the projector and predictor networks used for pretraining and representation learning are given in Table 8.

Original: The U-Net architecture used for both pretraining and finetuning isgiven in Table 9.

Original: During pretraining, an additional convolutional layer is used following layer 23 toreconstruct/denoise the input.

Original: During finetuning, a channel-wise softmax layer is attached followinglayer 23 for segmentation.

Original: [Table]

Original: [Table]

Original: , ReLU bs, w/8, h/8, d/8, 8nc

Original: MaxPool(2), Conv3D(16nc), BN, ReLU bs, w/16, h/16, d/16, 16nc

Original: Conv3D(16nc), BN, ReLU bs, w/16, h/16, d/16, 16nc

Original: Upsample(2), Concatenate withbs, w/8, h/8, d/8, 24nc

Original: Conv3D(16nc), BN, ReLU bs, w/8, h/8, d/8, 8nc

Original: Conv3D(16nc), BN, ReLU bs, w/8, h/8, d/8, 8nc

Original: Upsample(2), Concatenate withbs, w/4, h/4, d/4, 12nc

Original: Conv3D(4nc), BN, ReLU bs, w/4, h/4, d/4, 4nc

Original: Conv3D(4nc), BN, ReLU bs, w/4, h/4, d/4, 4nc

Original: Upsample(2), Concatenate withbs, w/2, h/2, d/2, 6nc

Original: Conv3D(2nc), BN, ReLU bs, w/2, h/2, d/2, 2nc

Original: Conv3D(2nc), BN, ReLU bs, w/2, h/2, d/2, 2nc

Original: Upsample(2), Concatenate withbs, w, h, d, 3nc

Original: Conv3D(nc), BN, ReLU bs, w, h, d, nc

Original: Conv3D(nc), BN, ReLU bs, w, h, d, nc

Original: Conv3D(n) bs, w, h, d, n

Original: B.3 Baseline Reimplementation Details

Original: LNE.

Original: LNE [43] is a self-supervised global representation learning method that models longitudinaleffects in an autoencoder latent space and is repurposed here for longitudinal segmentation.

Original: Duringpretraining, we require the same base U-Net architecture for consistency with other baselines.

Original: Therefore, to extract a 1024D global representation as in the original paper, we add a global poolinglayer on the output of layer 13 in Table 9, followed by a single 1024D fully-connected layer.

Original: Withrespect to their hyperparameters, due to the larger base network size, we reduce the batch size from64 to 16 images due to memory considerations.

Original: We maintain the other hyperparameters of LNEincluding downsampling the images to 64 3 , using N = 5 neighborhoods, and weighing auxiliarylosses as λ dir = 1 and λ rec = 2 (see original work for definitions [43]).

Original: PatchNCE.

Original: PatchNCE [44] proposes a framework for unsupervised multi-scale patchwise contrastivelearning applied to the problem of unpaired image translation and is extended to serve as a baselinein this work as it shares a partially similar motivation.

Original: To adapt it to our generic longitudinal representation learning setting, we impose the same modeling assumption as our method (spatial indicesin correspondence across an image time-series are positive samples and all other spatial indicesare negatives) and modify both its data sampling and loss calculation.

Original: First, we perform forwardpasses on t corresponding intra-subject crops.

Original: At each selected layer, N feature vectors are randomlysampled for the longitudinal InfoNCE [31] loss: L = (cid:80)

Original: i

Original: [Equation]

Original: where for any query feature index i , P ( i ) is the set of indices of all positives (from the same subjectat the same spatial index) and S ( i ) is the set of all query indices (with size ( t × N ) − 1 , where t isthe number of timepoints in the batch and N is the number of patches).

Original: We choose t = 3 timepointsand N = 768 patches in our experiments due to memory limits.

Original: An ablation study of PatchNCEover the layers used for the loss function and the temperature is given in Table 10.

Original: For prototypingefficiency, we perform baseline tuning using a three-layer MLP of size 256 and use validation Diceas the model selection criterion to choose F as our final configuration.

Original: For fair comparison with ourmodel, we increase the MLP width in the PathNCE model to 2048 in all other experiments reported,although this does not significantly alter results.

Original: Context Restoration .

Original: [10] proposes context restoration as a pretext task for self-supervisedrepresentation learning which consists of randomly and repeatedly swapping image patches andtraining the network to restore the original image.

Original: We extend Algorithm 1 in [10] to use 3D inputsand targets and tune it with a varying number of swapping iterations (#swaps) and swapped patchsizes.

Original: Downstream validation dice scores are provided in Table 11 and configuration D is chosen asthe final model.

Original: PCL, GCL, and 2D U-Net.

Original: PCL and GCL [9, 63] are 2D slice-based contrastive learning methodsdesigned for 3D volume segmentation.

Original: We follow the optimal configurations (e.g. temperature,number of partitions, etc.) reported in the original papers.

Original: As these methods use a 2D U-Nettrained on 2D slices, we additionally benchmark against a randomly initialized 2D U-Net to obtaina baseline for 2D methods.

Original: We use a batch size of 128 for all 2D baselines.

Original: Within each batch,we compare two different data sampling strategies: (1) randomly sampling 2D slices across all 3Dvolumes; (2) sampling intra-volume slices within each batch (i.e. treat one of the dimensions from a128 × 128 × 128 crop as the batch dimension for 2D networks).

Original: We train a randomly initialized 2D

Original: U-Net for the one-shot segmentation task, and found that (2) significantly increases the validationmean Dice over (1) on OASIS3 from 0 .

Original: 193) to 0 .

Original: 137) under the same architecture andoptimization up.

Original: We therefore use batch sampling strategy (2) for all 2D model benchmarks (PCL,

Original: GCL, RandInitUnet.2D).

Original: of patchNCE multiscale patchNCE loss

Original: [Table]

Original: A

Original: Enc 0.

Original: 0.834 ( ± 0.060)

Original: B

Original: Enc 0.

Original: [Table]

Original: 834 ( ± 0.062)

Original: C

Original: Enc 0.

Original: 0.823 ( ± 0.067)

Original: D

Original: Enc 0.

Original: 0.830 ( ± 0.064)

Original: E

Original: EncDec 0.

Original: [Table]

Original: [Table]

Original: C Additional data preparation detailsC.1 RegistrationC.1.1 Affine registration

Original: [-To warp all images to a common space, we warp all images affinely to a constructed template [4](with an affine transformation model) using ANTs 1 with the following command:antsMultivariateTemplateConstruction2.sh \-d 3 \-o OUTPUT_FOLDER/T \-i 1 -g 0.2 -j 128 -c 2 -r 1 -n 0 -m MI -l 
-]
Revised: [-To warp all images to a common space, we warp all images affinely to a constructed template [4](with an affine transformation model) using ANTs 1 with the following command:antsMultivariateTemplateConstruction2.sh \-d 3 \-o OUTPUT_FOLDER/T \-i 1 -g 0.2 -j 128 -c 2 -r 1 -n 0 -m MI -l 
-]

Original: \-t Affine INPUT_FOLDER/*t1.nii.gz’

Original: Once affinely aligned on a dataset-wide level, we proceed with longitudinal intra-subject deformablealignment for the calculation of L sim and L cs , as described below.

Original: C.1.2 Nonlinear Deformable Registration

Original: Proper spatial correspondence of positive samples for L sim and label maps for L cs (the similarityand consistency losses, respectively) requires nonlinear/deformable registration of all intra-subjectimages to a common reference.

Original: To this end, we employ the ANTs SyN algorithm from [3] to registerwithin-subject images to a single timepoint in a series of acquisitions with the following command:fix = INPUT_FOLDER/{subj}_{trg_tp}_t1.nii.gzfor src_tp

Original: in tps:moving = INPUT_FOLDER/{subj}_{src_tp}_t1.nii.gzantsRegistration

Original: \--verbose 1 \--dimensionality 3 \--float 1 \--output [OUTPUT_FOLDER/{subj}_{src_tp}_to_{trg_tp}_t1_, \

Original: OUTPUT_FOLDER/{subj}_{src_tp}_to_{trg_tp}_t1_Warped.nii.gz, \

Original: OUTPUT_FOLDER/{subj}_{src_tp}_to_{trg_tp}_t1_ nii gz] \--transform SyN[0.15, 9, 0.2] \--metric CC[{fix}, {moving}, 1, 2, Random, 0.4] \--convergence [250x125x50, 1e-5, 10] \--shrink-factors 4x2x1 \--smoothing-sigmas 2x1x0vox

Original: \--interpolation Linearwhere fix is the intra-subject image from a selected timepoint trg_tp to which all other timepointssrc_tp are registered.

Original: C.2 Additional preprocessing and label generation

Original: With respect to OASIS3, the publicly-available dataset arrives preprocessed with cross-sectional

Original: FreeSurfer processing (intensity and geometric normalization and segmentation).

Original: As cross-sectional

Original: FreeSurfer does not account for longitudinal effects [47], FreeSurfer V6.0 is used for additionallongitudinal processing on top of the publicly-released cross-sectional FreeSurfer segmentations.

Original: Acomparison between longitudinal and cross-sectional FreeSurfer analysis based on training set isgiven in Fig.

Original: 11, where we observe notable improvements to longitudinal segmentation consistency(as quantified by STCS [35]) over all structures.

Original: For IBIS-{ subcort, wmgm }, all T1w/T2w MR images are preprocessed following standard procedures described in [50] of gradient distortion correction, bias-field correction, within-subject andwithin time-point multi-modality registration, and brain extraction.

Original: For IBIS- subcort , ground truthlabel generation follows the segmentation procedures outlined in [50].

Original: For IBIS- wmgm , we use

Original: ) leads to substantial improvements over the publicly-released cross-sectional processing of OASIS3 in longitudinal-consistency metrics such as STCS [35] (higher is better).

Original: See section C.2 for further motivation.

Original: segmentations generated by a fully-supervised network trained on an external dataset, see Appendix

Original: Section D.2 for more details.

Original: Once preprocessed and aligned with methods described in Section C.1, the images are cropped to acommon field-of-view ( 128 × 192 × 160 for IBIS, 160 × 192 × 160 for OASIS3, corresponding tosmaller brain volumes for infants vs. adults) for compatibility with common multi-resolution neuralnetwork architectures.

Original: C.3 Data splitting with repeat acquisitions

Original: We perform a 70-10-20 train, validation, and test split on subject-wise basis from a total of N subjectsand M images ( M > N due to image time-series acquisitions), with the entire procedure illustratedin Figure 12.

Original: This level of the data split hierarchy ( N _ cross subjects and M _ cross images) is usedfor segmentation training and evaluation.

Original: For longitudinal pretraining and longitudinal consistencyevaluation (ASPC, STCS in Table 1 and Figure 4 of the main text), we further filter N _ long subjectswith at least 2 acquisitions per subject to obtain M _ long images.

Original: Finally, IBIS- subcort and IBIS- wmgm are segmentation tasks which share the same unlabeledpretraining data.

Original: Importantly, we note that as IBIS- wmgm is evaluated and tested only on 6-month-old

Original: MR images (representing the strongest domain shift, see Appendix Section D.2 for its motivation),its segmentation validation and test sets contain fewer images than IBIS- subcort .

Original: Table 12 providesthe exact sample sizes for each task.

Original: dataset-specific numbers reported

Original: [Table]

Original: D Miscellaneous Details

Original: D.1 Data availability and IRB information

Original: IBIS.

Original: The IBIS/Autism MRI data is available through NIH NDA 2 .

Original: The Infant Brain Imaging Study(IBIS) Network is a National Institutes of Health–funded Autism Center of Excellence project andconsists of a consortium of eight universities in the United States and Canada.

Original: Parents providedinformed consent and the institutional review board at each site approved the research protocol.

Original: OASIS-3.

Original: OASIS-3 [33] is publicly available through the OASIS webpage 3 .

Original: Ethical approval wasobtained by the relevant ethics committees and informed consent was obtained from all participantsfollowing procedures set by the IRB at the Washington University School of Medicine.

Original: D.2 IBIS-wmgm segmentation task details

Original: Infant white matter maturation.

Original: Neurodevelopment in infants and toddlers is a highly-complexprocess of macro and micro-structural changes (for example, growth and myelination, respectively).

Original: Relevant to the scope of this paper, real-world tissue segmentation (into grey and white matter) ofinfant brains at approximately 6 months of age (post-birth) is highly difficult due to ongoing whitematter myelination giving the brain an “isointense" (flat in intensity) appearance which ambiguatesanatomical edges.

Original: Fig.

Original: 13 visualizes this phenomenon.

Original: Currently, existing datasets for supervised segmentation of isointense infant brains [53] construct theirground truth labels by first nonlinearly registering algorithmic segmentations of an older timepointfrom a given subject image time-series (which is straightforward to segment with existing tools) andthen manually correcting the warped labels with the expertise of neuroradiologists.

Original: In our one-shot segmentation setting, we take a partially analogous approach to segmentation ofisointense infant brains.

Original: We first algorithmically segment [46] a single arbitrarily-selected 36 monthold T1w/T2w MR image and use this reliable segmentation to train all segmentation baselines andthe proposed method.

Original: These methods which have only been trained on a 36 month old brain are thenquantitatively evaluated on isointense 6 month old brain tissue segmentation in this paper.

Original: The targetlabels for the 6 month old evaluation set are generated algorithmically with supervised segmentationnetworks following [64] which have been trained on a separate dataset [53].

Original: E Need for regularization

Original: In this section, we qualitatively and quantitatively illustrate that proper regularization of patch-wisenegative-free multiscale representation learning avoids low-diversity or degenerate decoder representations which hamper downstream segmentation performance.

Original: We primarily compare the completeproposed model ( ‘Ours w/ regularization’ ) against an ablation ( ‘Our ablation w/o regularization’ )that only uses the patch-wise similarity loss and does not use the denoising, variance, covariance, andorthogonality regularization (which corresponds to ablation E from Table 2).

Original: Figure 14 visualizes activations from layers {6,8,10,13,16} of Table 9 (left) and the sorted singularvalues of the spatial covariance matrices of multi-layer feature projections from the correspondinglayer-wise projector output (right) trained with/without regularization.

Original: [-Given a flattened mid-axial feature projection F ∈ R wh × c (where c = 2048 and wh is the vectorized spatial dimensionality), we calculate the spatial covariance as C = 1 c (cid:80) ci =1 ( z i − ¯ z )( z i − ¯ z ) T , where z i ∈ R wh × 1 is a spatial featurevector and ¯ z = 1 c (cid:80) ci =
-]
Revised: [-Given a flattened mid-axial feature projection F ∈ R wh × c (where c = 2048 and wh is the vectorized spatial dimensionality), we calculate the spatial covariance as C = 1 c (cid:80) ci =1 ( z i − ¯ z )( z i − ¯ z ) T , where z i ∈ R wh × 1 is a spatial featurevector and ¯ z = 1 c (cid:80) ci =
-]

Original: z i .

Original: The lower the rank of C , the lower the spatial variability of representations.

Original: previous

Original: 2a