Original: W HEN HIGH - PERFORMING MODELS BEHAVE POORLY IN PRACTICE : PERIODIC SAMPLING CAN HELP
Revised: W HEN HIGH - PERFORMING MODELS BEHAVE POORLY IN PRACTICE : PERIODIC SAMPLING CAN HELP

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training  deep neural networks (DNN) for  cancer detection in medical imaging suffers from the (hopefully) low prevalence of the pathology.
Revised: Training a deep neural network (DNN) for breast cancer detection from medical images suffers from the (hopefully) low prevalence of the pathology.

Original: For a sensible amount of positive cases, images must be collected from numerous places resulting in large heterogeneous datasets.
Revised: For a sensible amount of positive cases, images must be collected from numerous places resulting in large heterogeneous datasets with different acquisition devices, populations, cancer incidences.

Original: Without precaution, this heterogeneity may result in  DNN biased by latent variables a priori independent of the pathology, which can be dramatic in practice.
Revised: Without precaution, this heterogeneity may result in a DNN biased by latent variables a priori independent of the pathology.

Revised: This may be dramatic if this DNN is used inside a software to help radiologists to detect cancers.

Original: This work mitigates this issue by acting on how mini-batches for Stochastic Gradient Descent (SGD) algorithms are constructed.
Revised: This work mitigates this issue by acting on how mini-batches for Stochastic Gradient Descent (SGD) algorithms are constructed.

Original: The dataset is divided into homogeneous subsets sharing some attributes ( e.g. acquisition device, source) called Data Segments (DSs).
Revised: The dataset is divided into homogeneous subsets sharing some attributes ( e.g. acquisition device, source) called Data Segments (DSs).

Original: Batches are built by sampling each DS periodically with a frequency proportional to the rarest label in the DS and by simultaneously preserving an overall balance between  labels within the batch.
Revised: Batches are built by sampling each DS periodically with a frequency proportional to the rarest label in the DS and by simultaneously preserving an overall balance between positive and negative labels within the batch.

Original: Periodicbatch sampling is compared to balanced sampling (equal amount of labels within a batch, independently of DS) and to balanced sampling within
Revised: Periodic sampling is compared to balanced sampling (equal amount of labels within a batch, independently of DS) and to balanced sampling within

Original: DS (equal amount of labels within a batch and eachindividual DS).
Revised: DS (equal amount of labels within a batch and each DS).

Original: We show, on breast cancer prediction from mammography images of various devices and origins, that periodic sampling leads to better generalization than other sampling strategies on similar DS and new ones ( e.g. unknown acquisition device). TODO: harmonize AUC format ! TODO: 3 is missing labels
Revised: We show, on breast cancer prediction from mammography images of various devices and origins, that periodic sampling leads to better generalization than other sampling strategies.

Original: the

Original: todo: 7.

Original: Table 4: It would be easier to capture the results listed here if the baseline results were included in the table instead of reported in the caption

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Breast cancer screening leads to an earlier detection of breast cancer resulting in an improved prognosis and a reduced mortality (Marmot et al., 2013).
Revised: Breast cancer screening leads to an earlier detection of breast cancer resulting in an improved prognosis and a reduced mortality (Marmot et al., 2013).

Original: During a screening exam, Full-Field Digital
Revised: During a screening exam, Full-Field Digital

Original: Mammography (FFDM) images of the breasts of a patient are acquired.
Revised: Mammography (FFDM) images of the breasts of a patient are acquired.

Original: They are then analyzed by 1 or 2 radiologist(s) who look(s) for signs of malignancy.
Revised: They are then analyzed by 1 or 2 radiologist(s) who look(s) for signs of malignancy.

Original: If any sign of malignancy is observed, the patient is recalled to assert the ﬁnal diagnosis with complementary exams.
Revised: If any sign of malignancy is observed, the patient is recalled to assert the ﬁnal diagnosis with complementary exams.

Original: Despite constant technological progress, cancers may still be missed while the recall rate remains high with a limited fraction of recalled patients actually diagnosed with cancer (Rawashdeh et al., 2013).
Revised: Despite constant technological progress, cancers may still be missed while the recall rate remains high with a limited fraction of recalled patients actually diagnosed with cancer (Rawashdeh et al., 2013).

Original: Machine Learning (ML
Revised: Machine Learning (ML

Original: ) based screening systems may help radiologists detecting cancers and improve their sensitivity, speciﬁcity and reading time, (Pacil`e et al., 2020; McKinney et al., 2020;
Revised: ) based screening systems may help radiologists detecting cancers and improve their sensitivity, speciﬁcity and reading time, (Pacil`e et al., 2020; McKinney et al., 2020;

Original: Schaffter et al., 2020; Shen et al., 2019; Kooi et al., 2017).
Revised: Schaffter et al., 2020; Shen et al., 2019; Kooi et al., 2017).

Original: Indeed, Deep Neural Networks (DNNs) can help to solve this problem initially formalized as a binary classiﬁcation problem.
Revised: Indeed, Deep Neural Networks (DNNs) can help to solve this problem initially formalized as a binary classiﬁcation problem.

Original: Recent approaches build upon breasts symmetry (Brhane Hagos et al., 2018; Kooi & Karssemeijer, 2017), the patient history Kooi & Karssemeijer (2017) or the whole set of exam images (Geras et al., 2017) to enhance classiﬁcation performances.
Revised: Recent approaches build upon breasts symmetry (Brhane Hagos et al., 2018; Kooi & Karssemeijer, 2017), the patient history (Kooi & Karssemeijer, 2017) or the whole set of exam images (Geras et al., 2017) to enhance classiﬁcation performances.

Original: Other approaches further aim at localizing the malignant lesion(s) in the breast (Shen et al., 2021;
Revised: Other approaches further aim at localizing the malignant lesion(s) in the breast (Shen et al., 2021;

Original: Pedemonte et al., 2020) or improving the generalization of DNNs trained on FFDM images but used on Digital Breast Tomosynthesis (DBT) images, 3 D images of the breast (Matthews et al., 2021; Singh et al., 2020).
Revised: Pedemonte et al., 2020) or improving the generalization of DNNs trained on FFDM images but used on Digital Breast Tomosynthesis (DBT) images, 3 D images of the breast (Matthews et al., 2021; Singh et al., 2020).

Original: Training a DNN on FFDM images (more generally on medical data) may be hard.
Revised: It is hard to train a DNN on FFDM images that aims to be used in a software delivering nearly real time recommendations for a radiologist investigating screening exams.

Original: First, it requires a large quantity of labeled training data but medical data sources are typically small and partially labeled.
Revised: First, it requires a large quantity of labeled training data but medical data sources are typically small and partially labeled.

Original: To cope with this problem, combining different sources of data appears as a natural and simple solution, a direction investigated through the prism of Domain Adaptation (DA)  (Crammer et al., 2008; Ben-David et al., 2010).
Revised: Transfer Learning (TL) may help to partially cope with this problem (Morid et al., 2021; Karimi et al., 2021) while Domain Adaptation (DA) may help to reduce the potential distribution mismatch between multiple source data distributions and a test data distributions (Crammer et al., 2008; BenDavid et al., 2010; Ganin et al., 2016; Guan & Liu, 2021).

Revised: Yet, TL may deliver little performance gain (Raghu et al., 2019), DA may not be appropriate for the setup considered here due to the potentially large number of training sources and the inﬁnite number of testing sources and the ”nearly real time” prediction constraint may be incompatible with the common ofﬂine setup most DA solutions embrace as they often build upon source and target domains available at the same time (Ben-David et al., 2007; 2010; Ganin et al., 2016).

Original: Yet, as eachsource of data (from a medical institution) exhibits its own characteristics (patients, radiologists experiences, sets of acquisition devices, disease incidences, ...),  theresulting DNN may be biased, the knowledge the DNN has gained from the training data  sources modulating its predictions.
Revised: Second, each medical institution exhibits its own characteristics (patients, radiologists experiences, sets of acquisition devices, disease incidences, ...), thus the DNN trained on data from such sources may still be biased: its predictions being modulated by latent uncontroled variables.

Original: If not addressed, this may result in dramatic consequences once the model is used in production (Barocas & Selbst, 2014).
Revised: If not addressed, this may result in dramatic consequences once the model is used in production (Barocas & Selbst, 2014).

Original: Second, training a DNN on FFDM implies ﬁghting class imbalance (the disease incidence being hopefully small) to avoid technical issues for ML models (He & Garcia, 2009) that could amplify the potential bias issue highlighted above.
Revised: Finally, training a DNN on FFDM implies ﬁghting class imbalance (the disease incidence being hopefully small) to avoid technical issues for ML models (He & Garcia, 2009) that could amplify the potential bias issue highlighted above.

Original: Paradoxically enough, it is hard to train a DNN from multiple sources of data as demonstrated by the following experiment.
Revised: The following experiment exhibits the complexity of training a DNN from only 2 sources of data.

Original: The goal was to train 2 DNNs to classify benign vs malignant FFDM images on 2 sets of data, DS 1 (7258 malignant / 13341 benign images) and DS 2 (160 malignant / 21871 benign) and to evaluate them on a left out set of data DS 3 (1106 malignant / 14192 benign) from another medical institution.
Revised: The goal was to train 2 DNNs to classify benign vs malignant FFDM images on 2 sets of data, DS 1 (7258 malignant / 13341 benign images) and DS 2 (160 malignant / 21871 benign) and to evaluate them on a left out set of data DS 3 (1106 malignant / 14192 benign) from another medical institution.

Original: A train / validation split was used patient-wise and source-wise: 80% (resp.
Revised: A train / validation split was used patient-wise and source-wise: 80% (resp.

Original: 20% ) of patients of a source were used for training (resp. validation).
Revised: 20% ) of patients of a source were used for training (resp. validation).

Original: The proportion of patients with cancers was kept constant in both train / val sets.
Revised: The proportion of patients with cancers was kept constant in both train / val sets.

Original: The ﬁrst DNN, dnn 1 , was trained on DS 1 , the second one, dnn 1 ∪ 2 on DS 1 ∪ DS 2 with a Stochastic Gradient Descent (SGD) like algorithm.
Revised: The ﬁrst DNN, dnn 1 , was trained on DS 1 , the second one, dnn 1 ∪ 2 on DS 1 ∪ DS 2 with a Stochastic Gradient Descent (SGD) like algorithm.

Original: Balancing benign and malignant images within mini-batches was used during training to cope with class imbalance.
Revised: Balancing benign and malignant images within mini-batches was used during training to cope with class imbalance.

Original: The Weighted Logloss per image (WLL / image), log loss with class weights, was monitored on a validation set during the training, see Figure 1 - left.
Revised: The Weighted Logloss per image (WLL / image), log loss with class weights, was monitored on a validation set during the training, see Figure 1 - left.

Original: The model dnn 1 ∪exhibits a low global validation WLL / image on DS 1 ∪ DS 2 , but a higher WLL / image on DScompared to dnn 1 .
Revised: The model dnn 1 ∪ 2 exhibits a low global validation WLL / image on DS 1 ∪ DS 2 , but a higher WLL / image on DS 1 compared to dnn 1 .

Original: The validation WLL / image on DS 2 further indicates that it was over-ﬁtted.
Revised: The validation WLL / image on DS 2 further indicates that it was over-ﬁtted.

Original: Plus, dnn 1 ∪ 2 shows a lower Area Under the ROC Curve (AUC) on DS 3 than dnn 1 , though it was trained on more data, see Figure 1 - center.
Revised: Plus, dnn 1 ∪shows a lower Area Under the ROC Curve (AUC) on DS 3 than dnn 1 , though it was trained on more data, see Figure 1 - center.

Original: Finally, the distributions of models predictions on DSshow that dnn 1 ∪ 2 predictions are shifted towards low values compared to dnn 1 predictions and that even for the malignant images, see Figure 1 - right.
Revised: Finally, the distributions of models predictions on DS 3 show that dnn 1 ∪ 2 predictions are shifted towards low values compared to dnn 1 predictions and that even for the malignant images, see Figure 1 - right.

Original: Hardly, does dnn 1 ∪ 2 beneﬁt from data brought by DS 2 .
Revised: Hardly, does dnn 1 ∪ 2 beneﬁt from data brought by DS 2 .

Original: [Figure]
Revised: [Figure]

Revised: This paper identiﬁes 2 main locks to train efﬁciently a DNN for breast cancer prediction from mammography images: class imbalance and a non homogeneous learning set.

Original: Some DA methods have been proposed to leverage multiple sources of data (Crammer et al., 2008;
Revised: Class imbalance can be addressed via an appropriate sampling strategy (He et al., 2008;

Original: Ben-David et al., 2010) but most of Deep Learning (DL) DA methods aim at using a single labeled source domain to improve the generalization on a target domain (Wang & Deng, 2018).
Revised: Chawla et al., 2002), sample weighting (Byrd & Lipton, 2018) or a learnable sampling (Hu et al., 2019;

Original: If constraining the feature representation learned by a DNN is important to ensure good generalization onto a target domain (Ben-David et al., 2007; 2010; Ganin et al., 2016), using an adapted sampling strategy to build mini-batches of training samples also matters.
Revised: Ren et al., 2018).

Original: Dividing an apparently homogeneous set of training data into low variance clusters of samples and using a non uniform sampling strategy, e.g. stratiﬁed sampling, to  select samples from  clustershas been explored to speed up the convergence of mini-batch optimization algorithms (Csiba & Richt´arik, 2018; Zhao & Zhang, 2014).Furthermore, a dedicated sampling strategy, either ﬁxed or adaptive and learned can help to cope with class imbalance Hu et al.
Revised: The sampling strategy may add further beneﬁts such as stratiﬁed sampling which may help to efﬁciently select samples from low variance clusters to speed up the convergence of mini-batch optimization algorithms (Csiba & Richt´arik, 2018; Zhao & Zhang, 2014).

Original: ; Ren et al. This paper focuses on training a DNN with a mini-batch SGD like optimization algorithm on a training set, divided into a ﬁnite ensemble of independent and homogeneous sets of samples, sharing an explicit and unique set of properties, called Data Segments (DSs).
Revised: This paper focuses on training a DNN with a mini-batch SGD like optimization algorithm on a training set, divided into a ﬁnite ensemble of independent and homogeneous sets of samples, sharing an explicit and unique set of properties, called Data Segments (DSs).

Original: A Periodic Sampling strategy is proposed to build mini-batches of data by sampling each DS periodically with a frequency proportional to its rarest label and by balancing the labels within a batch and a DS to ﬁght class imbalance.
Revised: A Periodic Sampling strategy is proposed to build mini-batches of data by sampling each DS periodically with a frequency propor- tional to its rarest label and by balancing the labels within a batch and a DS to ﬁght class imbalance.

Original: This paper, ﬁrst introduces formally the proposed Periodic Sampling , then provides numerical evidence of its beneﬁts and ﬁnally discusses its main limitations and directions for improvement.
Revised: This paper, ﬁrst introduces formally the proposed Periodic Sampling , then provides numerical evidence of its beneﬁts and ﬁnally discusses its main limitations and directions for improvement.

Original:  M ETHOD
Revised:  M ETHOD

Original: This paper focuses on binary classiﬁcation and introduces a method to train a DNN from several sources of data while reducing the risk of using an identiﬁed bias. The method works for DNN trained with mini-batch optimization algorithms like SGD and aims at building appropriately the mini-batches of data used for training thanks to a dedicated Periodic Sampling strategy.
Revised: This paper focuses on binary classiﬁcation and introduces a method to train a DNN, with a SGD like algorithm, from several segments of data while reducing the risk of using an identiﬁed bias. It aims to build appropriately the mini-batches of data with a dedicated Periodic Sampling strategy.

Original: 2. N OTATION AND PROBLEM STATEMENT
Revised: 2. N OTATION AND PROBLEM STATEMENT

Original: Let X stand for the input space, i.e. the space of FFDM images of a given shape.
Revised: Let X stand for the input space, i.e. the space of FFDM images of a given shape.

Original: Let Y = { 0 , 1 } stand for the space of labels associated to the images, 0 standing for a benign image, 1 for a malignant image.
Revised: Let Y = { 0 , 1 } stand for the space of labels, 0 standing for a benign image, 1 for a malignant image.

Original: x, y ∈ X
Revised: x, y ∈ X

Original: × Y is a learning sample: an image with its label.
Revised: × Y is a learning sample: an image with its label.

Original: Let i, N i ∈ N , then
Revised: Let i, N i ∈ N , then

Original: DS i = { ( x j , y j ) ∈
Revised: DS i = { ( x j , y j ) ∈

Original: X × Y : j = 0 , · · · , N i − 1 } refers to a Data Segment (DS) i.e. a subset of homogeneous learning samples.
Revised: X × Y : j = 0 , · · · , N i − 1 } refers to a Data Segment (DS) i.e. a subset of homogeneous learning samples.

Original: Let M ∈ N be the number of DSs at hand.
Revised: Let M ∈ N be the number of DSs at hand.

Original: Theconsidered learning set is composed of M homogeneous and independent DSs: ∪ M − 1 i =
DS i .
Revised: The learning set is composed of M homogeneous and independent DSs: ∪ M − 1 i =

Original: [-The paper focuses on training a DNN: f : X −→ Y to perform binary classiﬁcation using the labeled samples from ∪ M − 1 i =
-]
Revised: [-The paper focuses on training a DNN: f : X −→ Y to perform binary classiﬁcation using the labeled samples from ∪ M − 1 i =
-]

Original: DS i .
Revised: DS i .

Original: With  (cid:96) standing for the binary cross-entropy, the empirical optimization problem to solve reads:[Equation] [Table] [Equation]
Revised: With a DNN: f : X −→ Y , and with (cid:96) standing for the binary cross-entropy, the empirical optimization problem to solve reads:

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 2. P ERIODIC S AMPLING
Revised: 2. P ERIODIC S AMPLING

Original: This subsection details the proposed Periodic Sampling to efﬁciently construct the mini-batches of training samples to learn from multiple DSs while reducing the risk of learning an unwanted bias.

Original: Motivations First, the DNN should  learn at the same pace on the different DSs to obtain a single model working on all the DSsof interest and to avoid the issue seen in Figure 1. This means that a DS withvery few malignant images should not be over-ﬁtted too quickly.
Revised: General description First, the DNN should work on all the DSs of interest. It should learn at the same pace on the training DSs to avoid the issue seen in Figure 1 and not overﬁt to quickly a DS with few malignant images.

Original: Second, the DNNof interest should not be biased toward a speciﬁc label on any DS of interest.
Revised: Third, the DNN should not be biased toward a speciﬁc label on any DS.

Original: General description To meet these requirements a dedicated Periodic Samplingstrategy is introduced. The strategy is a two step process.
Revised: Periodic Sampling is a two step process introduced to meet these requirements.

Original: First, it builds a sampling pattern that indicates how often samples from thelearning DSs should be included in a mini-batch.
Revised: First, it builds a sampling pattern that indicates how often samples from the DSs should be included in a mini-batch.

Original: Second, it iterates over the sampling pattern and selects samples from the considered DSs while alternating in a balanced way the labels of these samples so that all labels come in equal proportions.
Revised: Second, it iterates over the sampling pattern and selects a sample from the designated DS while alternating in a balanced way the labels to ensure label balance in a DS.

Original: Sampling proportion With the above notation, we deﬁne the number of samples of the least represented label y mini , inside DS i by n mini = (cid:80) N i − 1 j =0 1 y j = y mini .
Revised: Sampling proportion With the above notation, we deﬁne the number of samples of the least represented label y mini , inside DS i by n mini = (cid:80) N i − 1 j =0 1 y j = y mini .

Original: Now, let N min =
Revised: Now, let N min =

Original: (cid:80) M − 1 i =

Revised: (cid:80) M − 1 i =

Original: n mini be the sum of the samples of the least represented label inside each DS.
Revised: n mini be the sum of the samples of the least represented label inside each DS.

Original: The main idea is to feed the
Revised: The main idea is to feed the

Original: DNN with the samples from DS i with a proportion p i = n mini /N min during the training process.
Revised: DNN with the samples from DS i with a proportion p i = n mini /N min during the training process.

Original: Sampling pattern Periodically selecting samples from DS i with a period 1 /p
Revised: Sampling pattern Periodically selecting samples from DS i with a period 1 /p

Original: i allows to feed the
Revised: i allows to feed the

Original: DNN with samples from DS i with a proportion p i .
Revised: DNN with samples from DS i with a proportion p i .

Original: This way, even if p i is dramatically small, samples from DS i would still be selected on a regular basis.
Revised: This way, even if p i is dramatically small, samples from DS i would still be selected on a regular basis.

Original: To do so, a sampling pattern S is built.
Revised: To do so, a sampling pattern S is built.

Original: This is a sequence of DS ids i ∈ { 0 , · · · , M − 1 } of length N min , where the number of occurrences of DS i is given by n mini .
Revised: This is a sequence of DS ids i ∈ { 0 , · · · , M − 1 } of length N min , where the number of occurrences of DS i is given by n mini .

Original: To build the sampling pattern, one  sorts the DS by their numbers n mini , in the ascending order.
Revised: To build it, one ﬁrst sorts the DS by their numbers n mini , in the ascending order.

Original: Then, one iterates over the sequence of i, n mini , to ﬁll in the available slots of a list of length N min and places the indices i at n mini locations, equally spaced while avoiding potential collisions: avoiding to allocate the index i to an already ﬁlled slot.
Revised: Then, one iterates over the sequence of i, n mini , to ﬁll in the available slots of a list of length N min and places the indices i at n mini locations, equally spaced while avoiding potential collisions with already ﬁlled slot.

Original: This is described in Algorithm 1.
Revised: This is described in Algorithm 1.

Revised: Construction of training batches Then, one builds batches of b ∈ N samples by iterating sequentially (and inﬁnitely) over the sampling pattern S to get, for each sample to be included in a batch, the index i of the DS from which it should be drawn.

Original:  Algorithm 
Revised: This way, for every nearly N min /b + 1 seen samples by the DNN, the DNN has been trained on nearly p i × ( N min /b + 1) samples from DS i . To ensure that the labels are presented in equal proportions to the DNN, the mechanism alternates between 0 and 1 labels when selecting a sample from DS i . The batch construction pipeline is summarized in Algorithm 2. An example of the selection of samples according to a sampling pattern with the alternating label mechanism is provided in Figure

Original: Data: N , { DS i : i = 0 , · · · , M − 1 }
Revised: Data: N , { DS i : i = 0 , · · · , M − 1 }

Original: Result:
Revised: Result:

Original: Sampling pattern S 1 step 1 : Compute the numbers of samples with the least represented label for every DS i : (cid:8) n mini :i =0 , · · · , M− 1 (cid:9) 2 step 2 : Compute the length of the sampling pattern L ←(cid:80) M − 1 i =0 n mini 3 step 3 :
Revised: Sampling pattern S 1 step 1 : Compute the numbers of samples with the least represented label for every DS i : (cid:8) n mini : = , M 2 step 2 : Compute the length of the sampling pattern L ←

Original:  Build the 
Revised: M − 1 i =0 n mini 3 step 3 : Build the sampling pattern:

Original: S ← array of size L ﬁlled with -1 values for i ← 0 to M − 1 do pos ← linspace (0 , N min , n min i ) ; for j ← 0 to n mini − 1 do p ← pos [ j ] while S [ p ] (cid:54) = − 1 do p ← p +end
Revised: S ← array of size L ﬁlled with -1 values for i ← 0 to M − 1 do pos ← linspace (0 , N min , n mini ) ; for j ← 0 to n mini − 1 do p ← pos [ j ] while S [ p ] (cid:54) = − 1 do p ← p +end

Original: S [ p ] ← i end end
Revised: S [ p ] ← i end end

Original: Construction of training batches Then, one builds batches of b ∈ N samples by iterating sequentially (and inﬁnitely) over the sampling pattern S to get, for each sample to be included in a batch, the index i of the DS from which it should be drawn.

Original: This way, for every nearly N min /b + 1 seen samples by the DNN, the DNN has been trained on nearly p i × ( N min /b + 1) samples from DS i .

Original: To ensure that the labels are presented in equal proportions to the DNN, the mechanism alternates between 0 and 1 labels when selecting a sample from DS i .

Original: The batch construction pipeline is summarized in Algorithm 2.

Original: An example of the selection of samples according to a sampling pattern with the alternating label mechanism is provided in Figure

Original: [Figure]
Revised: [Figure]

Original: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 Selected Samples DS [Figure] DS
Revised: : b

Original: [Figure] DS [Figure] DS
Revised: S

Original: [Figure] e g m e n t s
Revised: N min

Original: [Figure] [Figure] Practically, some DL libraries ensure that the order of the samples indices returned by a sampling function is maintained inside the mini-batches while others do not. This may not be an issue as long as the proportions of DSs are ensured at a scale of a few (hundreds) learning steps / batches.
Revised: DS = { DS i =

Original: The following experiments aim at training a DNN on FFDM images from multiple DSs to predict the probability of malignancy of these images. Data: b , S , N min , DS = { DS i : i = 0 , · · · , M −1 }
Revised: M −

Original: Result:
Revised: Result:

Original: A collection of batches of size b
Revised: A collection of batches of size b

Original: Y ← { 0 , 1 } M randomly initialized idx ←while training do
Revised: Y ← { 0 , 1 } M randomly initialized idx ←while training do

Original: B ← [ · · · ] empty list for j ← 0 to b − 1 do i ← S [ idx ] y i ← Y [ i ] ( x, y ) ← next samples ( x, y ) ∈ DS i such that
Revised: B ← [ · · · ] empty list for j ← 0 to b − 1 do i ← S [ idx ] y i ← Y [ i ] ( x, y ) ← next samples ( x, y ) ∈ DS i such that

Original: y =
Revised: y =

Original: y
Revised: y

Original: i
Revised: i

Original: B ← B + ( x, y ) idx = idx + 1 (mod N min )
Revised: B ← B + ( x, y ) idx = idx + 1 (mod N min )

Original: Y [ i ] =
Revised: Y [ i ] =

Original: Y [ i ] + 1 (mod 2) end yield B end
Revised: Y [ i ] + 1 (mod 2) end yield B end

Revised: Practically, some Deep Learning (DL) libraries ensure that the order of the samples indices returned by a sampling function is maintained inside the mini-batches while others do not.

Revised: This may not be an issue as long as the proportions of DSs are ensured at a scale of a few (hundreds) learning steps / batches.

Original:   3. E XPERIMENTAL SETUP 3.1. D ATA
Revised:  E XPERIMENTS 3. E XPERIMENTAL SETUP 3.1. D ATA

Original: Image modality and labeling process
Revised: Image modality and labeling process

Original: The considered data are FFDM images e.g. gray-scale images, with pixel values in [0 , 4095] , although other ranges can be used depending on the FFDM device manufacturer.
Revised: The considered data are FFDM images e.g. gray-scale images, with pixel values in [0 , 4095] , although other ranges can be used depending on the FFDM device manufacturer.

Original: Each image is extracted from its native DICOM storing ﬁle Pianykh (2010) and resized to a shape of (1152 , w ) , the width w being adjusted accordingly to keep the same image proportions. The mask of the breast is extracted from the reshaped image using automatic thresholding and morphological operations.
Revised: Every full view (not zoom) FFDM image is considered for training and evaluating the models: Cranio-Caudal (CC), Medio-Lateral Oblique (MLO), ...

Original: Images labels (0 and 1) are assigned as follows.
Revised: Images labels (0 and 1) are assigned as follows.

Original: A positive image (label 1) is an image where a suspicious lesion was localized and for which the malignant status was conﬁrmed by a biopsy shortly after the mammogram was done (within 3 months).
Revised: A positive image (label 1) is an image where a suspicious lesion was localized and for which the malignant status was conﬁrmed by a biopsy shortly after the mammogram was done (within 3 months).

Original: Each malignant lesion is localized as precisely as possible with a bounding box drawn by trained radiologists.
Revised: Each malignant lesion is localized as precisely as possible with a bounding box drawn by trained radiologists.

Original: A positive breast is a breast with at least one positive image.
Revised: A positive breast is a breast with at least one positive image.

Original: When the malignant lesion is not visible inside a speciﬁc view of a malignant breast the image is discarded from the dataset: e.g., a lesion not visible in Cranio-Caudal (CC) view while visible in Medio-Lateral Oblique (MLO) view.
Revised: When the malignant lesion is not visible inside a speciﬁc view of a malignant breast the image is discarded from the dataset e.g. a lesion not visible in CC view while visible in MLO view.

Original: Images anterior and posterior to a positive breast are discarded as well.
Revised: Images anterior and posterior to a positive breast are discarded as well.

Original: On the other hand, a negative image (label 0) is an image of a negative patient, i.e., a patient without a positive breast.
Revised: On the other hand, a negative image (label 0) is an image of a negative patient i.e. a patient without a positive breast.

Original: In addition, a one-year negative follow-up is required to conﬁrm absence of signs of malignancy in the image.
Revised: In addition, a one-year negative follow-up is required to conﬁrm absence of signs of malignancy in the image.

Revised: Additional data information are provided in Appendix.

Original: Data segments The learning set is composed of 2 sources of data, S 1 and S 2 .
Revised: Data segments The learning set is composed of 2 sources of data, S 1 and S 2 .

Original: Each source contains images obtained with acquisition devices of 2 manufacturers M a and M b .
Revised: Each source contains images obtained with acquisition devices of 2 manufacturers M a and M b .

Original: Thus, 4 DSs can naturally be deﬁned: ( S 1 , M a ) , ( S 1 , M b ) , ( S 2 , M a ) , ( S 2 , M b ) .
Revised: Thus, 4 DSs can naturally be deﬁned: ( S 1 , M a ) , ( S 1 , M b ) , ( S 2 , M a ) , ( S 2 , M b ) .

Original: The testing sets are composed of up to 3 sources of data, S 3 , S 4 and S 5 , containing images obtained with acquisition devices frommanufacturers, M a , M b , M c and M d , leading to 5 DSs.
Revised: The testing sets are composed of up to 3 sources of data, S 3 , S 4 and S 5 , containing images obtained with acquisition devices frommanufacturers, M a , M b , M c and M d , leading to 5 DSs.

Original: The numbers of malignant and benign images from all train and test DSs are given in table 1. 
Revised: The numbers of malignant and benign images from all train and test DSs are given in table 1. [Table]

Original: Cross-validation scheme For each experiment, the learning set is divided into training and validation data using a source-wise and patient-wise split: 80% (resp.
Revised: Cross-validation scheme For each experiment, the learning set is divided into training and validation data using a source-wise and patient-wise split: 80% (resp.

Original: 20% ) of the patients of each source are used for training (resp. validation).
Revised: 20% ) of the patients of each source are used for training (resp. validation).

Original: Furthermore, the split is made such that the ratio of malignant / benign patients is similar for the validation and the training sets.
Revised: Furthermore, the split is made such that the ratio of malignant / benign patients is similar for the validation and the training sets.

Revised: Image pre-processing and data augmentation Each image is extracted from its native DICOM storing ﬁle (Pianykh, 2010) and resized to a shape of (1152 , w ) , the width w being adjusted accordingly to keep the same image proportions.

Revised: The mask of the breast is extracted from the reshaped image using automatic thresholding and morphological operations.

Original: Image pre-processing and data augmentation Normalization and preprocessing techniques are used to standardize the FFDM images.
Revised: Normalization and preprocessing techniques are used to standardize the FFDM images.

Original: First, the images are cropped / padded to ﬁt a shape of 1152 ∗ 832 .
Revised: First, the images are cropped / padded to ﬁt a shape of 1152 ∗ 832 .

Original: Then, the pixels inside the breast mask are re-normalized to have values in [0 , 1] .
Revised: Then, the pixels inside the breast mask are re-normalized to have values in [0 , 1] .

Original: The pixels outside mask are set to a constant value such that the average pixel value of the image is always the same.
Revised: The pixels outside mask are set to a constant value such that the average pixel value of the image is always the same.

Original: A standard augmentation pipeline (random rotation, zoom, shearing... ) is used[Table] to augment images used for training while ensuring the malignant lesion remains visible in the augmented image.
Revised: A standard augmentation pipeline (random rotation, zoom, shearing... ) is used to augment images used for training while ensuring the malignant lesion remains visible in the augmented image.

Original: 3.1. M ODEL TRAINING AND EVALUATION
Revised: 3.1. M ODEL TRAINING AND EVALUATION

Original: Metrics As mammography data usually comes with strong class imbalance, the evaluation metrics need to be independent of labels proportions.
Revised: Metrics As mammography data usually comes with strong class imbalance, the evaluation metrics need to be independent of labels proportions.

Original: The Weighted Logloss per image (WLL / image), the standard logloss with samples weights estimated on the evaluation (here validation) set, the Receiver
Revised: The Weighted Logloss per image (WLL / image), the standard logloss with samples weights estimated on the evaluation (here validation) set, the Receiver

Original: Operating Characteristic curve (ROC curve) and the AUC are considered.
Revised: Operating Characteristic curve (ROC curve) and the AUC are considered.

Original: Metrics are computed globally i.e. on the whole (validation or test) set, and also locally i.e. on the DSs taken individually.
Revised: Metrics are computed globally i.e. on the whole (validation or test) set, and also locally i.e. on the DSs taken individually.

Original: To make sure a model exhibits a similar behavior on every DS, one also considers the Operating
Revised: To make sure a model exhibits a similar behavior on every DS, one also considers the Operating

Original: Points (OPs) at threshold in { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } on the ROC curve of this DS.
Revised: Points (OPs) at threshold in { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } on the ROC curve of this DS.

Original: The AUC of a DNN and the difference in AUC between 2 DNNs, ∆ AUC, are evaluated measured using 2000 Efron’s bootstraps with uniform sampling with replacement over the test set (or test data-segment).
Revised: The AUC of a DNN and the difference in AUC between 2 DNNs, ∆ AUC, are evaluated measured using 2000 Efron’s bootstraps with uniform sampling with replacement over the test set (or test data-segment).

Original: Then 95% Conﬁdence Interval (CI), given in brackets, and P-value are provided as well for the ∆ AUC , (Altman & Bland, 2011).
Revised: Then 95% Conﬁdence Interval (CI), given in brackets, and P-value are provided as well for the ∆ AUC , (Altman & Bland, 2011).

Original: A negative CI upper bound of ∆ AUC , with p-value < 0 .
Revised: A negative CI upper bound of ∆ AUC , with p-value < 0 .

Original: 05 is regarded as a signiﬁcant decrease in performance.
Revised: 05 is regarded as a signiﬁcant decrease in performance.

Original: A non negative CI lower bound of ∆ AUC , with p-value < 0 .
Revised: A non negative CI lower bound of ∆ AUC , with p-value < 0 .

Original: 05 is regarded as a signiﬁcant increase in performance.
Revised: 05 is regarded as a signiﬁcant increase in performance.

Original: Neural network The considered DNN, called DreamNet (DN), follows a VGG-like architecture pattern Schaffter et al. DN is composed of 2 parts: a backbone and a classiﬁcation head, the backbone being ﬁrst pre-trained to classify lesions annotated on a subset of images from ( S 1 , M a ) , see supplemental materials.
Revised: Neural network The considered DNN processes every image independently, it makes image wise predictions and assigns for each image a probability of malignancy. It follows a VGG-like architecture pattern (Schaffter et al., 2020) and is composed of 2 parts: a backbone and a classiﬁcation head, the backbone being ﬁrst pre-trained to classify lesions annotated on a subset of images from ( S 1 , M a ) , see Appendix for more details.

Original: The fullDN model is ﬁne-tuned to perform binary classiﬁcation on  Tesla P100, for 150000 steps with Adam optimizer Kingma & Ba (2014) and a learning rate equal to 10 − 4 .
Revised: The full model is ﬁne-tuned to perform binary classiﬁcation on 2 Tesla P100, for 150000 steps with Adam optimizer (Kingma & Ba, 2014) and a learning rate equal to 10 − 4 .

Original: Exponential moving average with decay 0 .
Revised: Exponential moving average with decay 0 .

Original: 99 of all trainable parameters are tracked, their averaged versions are then used for inference on validation and test data.
Revised: 99 of all trainable parameters are tracked, their averaged versions are then used for inference on validation and test data.

Original: A batch size of 24 samples is used ( 12 per GPU).
Revised: A batch size of 24 samples is used ( 12 per GPU).

Original: A validation step is made every 1000 training steps.
Revised: A validation step is made every 1000 training steps.

Original: The checkpoint achieving the best WLL / image on the validation set is kept for inference.
Revised: The checkpoint achieving the best WLL / image on the validation set is kept for inference.

Original: Tensorﬂow 2.3 (Abadi et al., 2015) was used for the experiments.
Revised: Tensorﬂow 2.3 (Abadi et al., 2015) was used for the experiments.

Original: 3.1. B ASELINES AND COMPARED METHODS
Revised: 3.1. B ASELINES AND COMPARED METHODS

Original: Baselines Two baselines are considered and are obtained by training a DN on a single DS: ( S 1 , M a ) for the ﬁrst one, ( S 1 , M b ) for the second.
Revised: Baselines Two baselines are considered and are obtained by training the model on a single DS: ( S 1 , M a ) for baseline (1), ( S 1 , M b ) for baseline (2).

Original: Training is done with standard balanced sampling to cope with class imbalance.
Revised: Training is done with standard balanced sampling to cope with class imbalance.

Original: Baseline 1 (resp. 2) is used as a reference in experiments where models are evaluated on ( S 3 , M a ) (resp. ( S 3 , M b ) ).
Revised: Baseline 1 (resp. 2) is used as a reference in experiments where models are evaluated on ( S 3 , M a ) , ( S 4 , M c ) , ( S 4 , M d ) , ( S 5 , M c ) (resp.

Original: The performances of these models are given in tables captions and are used to compute the differences of AUC with the other methods ( c.f. ∆ AUC).
Revised: ( S 3 , M b ) ) to compute the differences of AUC with the other methods ( c.f. ∆ AUC).

Original: Compared methods Furthermore, 3 other approaches are considered.
Revised: Compared methods Three strategies are benchmarked.

Original: They all aim at training a DN on 2 or more DSs.
Revised: They all aim at training the considered DNN on 2 or more DSs.

Original: The ﬁrst one is called ”balance labels only” and is referred to by ”Bal.
Revised: The ﬁrst one is called ”balance labels only” and is referred to by ”Bal.

Original: labels”.
Revised: labels”.

Original: It aims at sampling equally the labels 0 , 1 independently of the DSs.
Revised: It aims at sampling equally the labels 0 , 1 independently of the DSs.

Original: This boils down to consider that all the training samples come from a unique DS as often assumed in ML.
Revised: This boils down to consider that all the training samples come from a unique DS as often assumed in ML.

Original: The second one is called ”balance every data-segments equally” and is referred to by ”Bal.
Revised: The second one is called ”balance every data-segments equally” and is referred to by ”Bal.

Original: segments”.
Revised: segments”.

Original: It aims at sampling equally the DSs and the labels 0 , 1 .
Revised: It aims at sampling equally the DSs and the labels 0 , 1 .

Original: In other words, it can be seen as a sampling pattern
Revised: In other words, it can be seen as a sampling pattern

Original: S built as if the proportion p i of every DS DS i was given by 1 /M , with M the number of DSs.
Revised: S built as if the proportion p i of every DS DS i was given by 1 /M , with M the number of DSs.

Original: This way, negative and positive samples from DS i are sampled as often as negative and positive samples from DS j , with i (cid:54) =
Revised: This way, negative and positive samples from DS i are sampled as often as negative and positive samples from DS j , with i (cid:54) =

Original: j .
Revised: j .

Original: The last one is the proposed Period Sampling , referred to by ”P. sampling”.
Revised: The last one is the proposed Period Sampling , referred to by ”P. sampling”.

Original: 3. E XPERIMENT 1: TRAINING ON 2 D ATA -S EGMENTS : IMAGES FROM 2 SOURCES BUT THE SAME MANUFACTURER 
Revised: 3. E XPERIMENT 1: TRAINING ON 2 D ATA -S EGMENTS (2 SOURCES , 1 MANUFACTURER )

Original: In this experiment [Table] the training / validation set is divided into 2 DSs: ( S 1 , M a ) and ( S 2 , M a ) . This means that images have been obtained with acquisition devices of the same manufacturer but are originating from 2 different institutions.
Revised: In this experiment, the training / validation set is divided into 2 DSs: ( S 1 , M a ) and ( S 2 , M a ) : images have been obtained with acquisition devices of the same manufacturer but come from 2 different institutions.

Original: Performances on the test set ( S 3 , M a ) are reported in Table 2.[Table]
Revised: Performances on the test set ( S 3 , M a ) are reported in Table 2.

Original: The DNNs trained with ”Bal.
Revised: The DNNs trained with ”Bal.

Original: labels” and ”Bal.
Revised: labels” and ”Bal.

Original: segments” do not beat the baseline on DS ( S 3 , M a ) whereas the DNN trained with ”P. sampling” brings a small gain of performance.
Revised: segments” do not beat the baseline on DS ( S 3 , M a ) whereas the DNN trained with ”P. sampling” brings a small gain of performance.

Original: Such a small gain may be due to the limited amount of malignant images of the DS ( S 2 , M a ) compared to the large number of malignant images in the DS ( S 1 , M a ) 
Revised: Such a small gain may be due to the limited amount of malignant images of the DS ( S 2 , M a ) compared to the large number of malignant images in the DS ( S 1 , M a ) . [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  The global metrics and the local metrics on ( S 1 , M a ) and ( S 2 , M a ) exhibit a signiﬁcantly large discrepancy for the DNN trained with ”Bal.
Revised: [Table] [Figure] The global metrics and the local metrics on ( S 1 , M a ) and ( S 2 , M a ) exhibit a signiﬁcantly large discrepancy for the DNN trained with ”Bal.

Original: labels”, see Figure 3 - blue curves.
Revised: labels”, see Figure 3 - blue curves.

Original: Indeed, this DNN presents a very low global WLL / image and a very (suspiciously) high global AUC / image while it also has much higher WLL / images and much lower AUCs on the DSs ( S 1 , M a ) and ( S 2 , M a ) .
Revised: Indeed, this DNN presents a very low global WLL / image and a very (suspiciously) high global AUC / image while it also has much higher WLL / images and much lower AUCs on the DSs ( S 1 , M a ) and ( S 2 , M a ) .

Original: It is as if the DNN trained with ”Bal labels” learned a wrong task: classify the malignant images from ( S 1 , M a ) vs the benign images of ( S 2 , M a ) .
Revised: It is as if the DNN trained with ”Bal labels” learned a wrong task: classify the malignant images from ( S 1 , M a ) vs the benign images of ( S 2 , M a ) .

Original: Moreover, the WLL / image (resp.
Revised: Moreover, the WLL / image (resp.

Original: the AUC) on ( S 2 , M a ) indicates that the DNN has over-ﬁtted very quickly this DS and does not learn from it after 5000 learning steps.
Revised: the AUC) on ( S 2 , M a ) indicates that the DNN has over-ﬁtted very quickly this DS and does not learn from it after 5000 learning steps.

Original: Everything appears as if this second DS ( S 2 , M a ) is never used in the end.
Revised: Everything appears as if this second DS ( S 2 , M a ) is never used in the end.

Original: The
Revised: The

Original: DNN trained with ”Bal.
Revised: DNN trained with ”Bal.

Original: segments” seems to over-ﬁtting more slowly the DS ( S 2 , M a ) as suggested by the local metrics on this DS, Figure 3 - green curves.
Revised: segments” seems to over-ﬁtting more slowly the DS ( S 2 , M a ) as suggested by the local metrics on this DS, Figure 3 - green curves.

Original: On the contrary, the DNN trained with ”P. sampling” demonstrates much more coherent and reliable global and local metrics, Figure 3 - yellow curves.
Revised: On the contrary, the DNN trained with ”P. sampling” demonstrates much more coherent and reliable global and local metrics, Figure 3 - yellow curves.

Original: The global metrics look like an average of local metrics both for the WLL / image and the
Revised: The global metrics look like an average of local metrics both for the WLL / image and the

Original: AUC / image.
Revised: AUC / image.

Original: Plus, the DNN does not seem to quickly over-ﬁt the segment ( S 2 , M a ) : it seems to learn at the same pace on both DSs.
Revised: Plus, the DNN does not seem to quickly over-ﬁt the segment ( S 2 , M a ) : it seems to learn at the same pace on both DSs.

Original: Last but not least, as a (small) gain of AUC is observed on the test set, this implies that, the DNN trained with ”P. sampling” is able to learn from both DSs and to leverage the second DS ( S 2 , M a ) to improve the test performances.
Revised: Last but not least, as a (small) gain of AUC is observed on the test set, this implies that, the DNN trained with ”P. sampling” is able to learn from both DSs and to leverage the second DS ( S 2 , M a ) to improve the test performances.

Original: The ROC curves obtained on the validation sets of DSs ( S 1 , M a ) and ( S 2 , M a ) are represented for each model as well as the OPs at thresholds 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , see Fig 4.
Revised: The ROC curves obtained on the validation sets of DSs ( S 1 , M a ) and ( S 2 , M a ) are represented for each model as well as the OPs at thresholds 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , see Fig 4.

Original: On the whole validation set, the DNNs trained with ”Bal.
Revised: On the whole validation set, the DNNs trained with ”Bal.

Original: labels” and ”Bal.
Revised: labels” and ”Bal.

Original: segments” seem globally better than the DNN trained with ”P. sampling”.
Revised: segments” seem globally better than the DNN trained with ”P. sampling”.

Original: Indeed, they reach higher global AUC, see black curves in Fig 4.
Revised: Indeed, they reach higher global AUC, see black curves in Fig 4.

Original: Yet, on the DSs taken individually the DNN trained with ”P. sampling” performs better than the others and reduces the discrepancy between OPs of the ROC curves of both DSs.
Revised: Yet, on the DSs taken individually the DNN trained with ”P. sampling” performs better than the others and reduces the discrepancy between OPs of the ROC curves of both DSs.

Original: This likely indicates that the DNN trained with ”P. sampling” works similarly on both DSs.
Revised: This likely indicates that the DNN trained with ”P. sampling” works similarly on both DSs.

Original: On the contrary, the DNN trained with ”Bal.
Revised: On the contrary, the DNN trained with ”Bal.

Original: labels” leads to un-synchronized OPs between the ROC curves of the 2 DSs.
Revised: labels” leads to un-synchronized OPs between the ROC curves of the 2 DSs.

Original: Plus the
Revised: Plus the

Original: OPs of ( S 1 , M a ) look like opposite to the ones of ( S 2 , M a ) suggesting that this DNN may actually classify the malignant images from ( S 1 , M a ) vs the benign images from ( S 2 , M a ) .
Revised: OPs of ( S 1 , M a ) look like opposite to the ones of ( S 2 , M a ) suggesting that this DNN may actually classify the malignant images from ( S 1 , M a ) vs the benign images from ( S 2 , M a ) .

Original: [Figure]
Revised: [Figure]

Revised: ROC curves obtained on the validation sets of DSs ( S 1 , M a ) (resp.

Revised: ( S 2 , M a ) ).

Original: OPs at threshold 0 .
Revised: OPs at threshold 0 .

Original: 8 are added on each ROC curve.
Revised: 8 are added on each ROC curve.

Original: 3. E XPERIMENT 2: TRAINING ON 4 D ATA -S EGMENTS
Revised: 3. E XPERIMENT 2: TRAINING ON 4 D ATA -S EGMENTS

Original: In this experiment, the training / validation set is composed of 4 DSs: ( S 1 , M a ) , ( S 1 , M b ) , ( S 2 , M a ) and ( S 2 , M b ) .
Revised: In this experiment, the training / validation set is composed of 4 DSs: ( S 1 , M a ) , ( S 1 , M b ) , ( S 2 , M a ) and ( S 2 , M b ) .

Original: 2 baselines are considered: a DN trained on ( S 1 , M a ) (resp.
Revised: 2 baselines are considered: a DNN trained on ( S 1 , M a ) (resp.

Original: ( S 1 , M b ) ) which is used when evaluating / comparing the methods on ( S 3 , M a ) (resp.
Revised: ( S 1 , M b ) ) which is used when evaluating / comparing the methods on ( S 3 , M a ) (resp.

Original: ( S 3 , M b ) ).
Revised: ( S 3 , M b ) ).

Original: The models performances on the test sets ( S 3 , M a ) and ( S 3 , M b ) are reported in Table 3.
Revised: The models performances on the test sets ( S 3 , M a ) and ( S 3 , M b ) are reported in Table 3.

Original: [Table]
Revised: [Table]

Original: The DNNs trained with ”Bal.
Revised: The DNNs trained with ”Bal.

Original: labels” and ”Bal.
Revised: labels” and ”Bal.

Original: segments” do not beat the baseline on ( S 3 , M a ) and exhibit a negative ∆ AUC which is statistically signiﬁcant.
Revised: segments” do not beat the baseline on ( S 3 , M a ) and exhibit a negative ∆ AUC which is statistically signiﬁcant.

Original: On the other hand, the DNN trained with ”P. sampling” brings a small gain of performance which is is statistically signiﬁcant.
Revised: On the other hand, the DNN trained with ”P. sampling” brings a small gain of performance which is is statistically signiﬁcant.

Original: On ( S 3 , M b ) , all DNNs beat the baseline, but the one trained with ”P. sampling” gives a larger gain of AUC.On top of AUC gain - loss, training a DNN with ”Bal.
Revised: On ( S 3 , M b ) , all DNNs beat the baseline, but the one trained with ”P. sampling” gives a larger gain of AUC.

Original: labels” leads to a poor synchronization of OPs on the ROC curves obtained on ( S 3 , M a ) and ( S 3 , M b ) , see Figure 5 - left.
Revised: Synchronization of OPs on the ROC curves is again impacted, c.f. Figure 6 in Appendix.

Original: This suggests that the DNN works in 2 different regimes on the 2 test DSs: a non desired property which stresses the manufacturer bias the DNN has integrated.

Original: On the other hand, the DNN trained with ”P. sampling” gives visually better synchronized OPs on the ROC curves of test DSs.

Original: This suggests that this DNN behaves quite similarly on them and that it is not inﬂuenced by the manufacturer to take its decisions.

Original: [Figure]

Original: Last but not least, the DNN trained with the proposed ”P. sampling” seems to better generalize on new manufacturers than the DNNs trained with other compared approaches and seems slightly better than the baseline trained on DS ( S 1 , M a ) although the gain is not signiﬁcant, see Table 4.The baseline reaches an AUC of 85 . 20 (75 . 16 , 92 . 90) on ( S 4 , M c ) , 89 . 01 (85 . 24 , 92 . 29) on ( S 4 , M d ) and 85 . 14 , 89 . 43) on ( S 5 , M c
Revised: Last but not least, the DNN trained with the proposed ”P. sampling” seems to better generalize on new manufacturers than the DNNs trained with other compared approaches and seems slightly better than the baseline trained on DS ( S 1 , M a ) although the gain is not signiﬁcant, see Table 4.

Original: [Table]

Original: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Periodic sampling leads to gains of performance when learning from multiple DSs and harmonized model predictions on different test DSs. Yet, it also has some drawbacks that are now discussed.
Revised: Periodic sampling exhibits some drawbacks that are now discussed.

Original: The main issue is related to the segmentation of the learning set intoDSs.
Revised: The main drawback of the proposed method is related to the segmentation of the learning set into

Original: Although it seems a good idea, little information do we have  on how building DSs.
Revised: DSs as we have little information on how building DSs.

Original: Only accumulated experience and knowledge of the data led us to consider the sources and the manufacturers of acquisition devices as a natural way to split the learning set into DSs.
Revised: Two questions arise: (1) ”when does one have to segment the learning set ?” and (2) ” how ?”.

Original: Nonetheless, by experimenting, we came up with  empirical ways to know whether a set of data should be divided into DSs.
Revised: [Table] By experimenting, we came up with 3 empirical ways to know whether a set of data should be divided into DSs.

Original: First, a poor generalization performance on a left out source of data may be indicating that the learning set should be split into DSs as in experiments 1 and 2.
Revised: First, a poor generalization performance on a left out source of data is likely to indicate that the learning set should be split into DSs as in experiments 1 and 2.

Original: Second, a very surprisingly nice validation metrics is also a good indicator that the learning set should be divided into DSs, see Figure 3.
Revised: Second, a very surprisingly nice validation metrics is also a good indicator that the learning set should be divided into DSs, see Figure 3.

Original: Finally, good knowledge of the learning set and a careful look at positive and negative samples numbers in natural subsets of data may help to divide the learning set intopotential DSs, see Table 1.
Revised: Finally, good knowledge of the learning set and a careful look at positive and negative samples numbers in natural subsets of data may indicate a required division of the learning set into DSs, see Table 1.

Original: What seems to matter is a potential unbalance between negative samples from a subset and positive samples from another one.
Revised: An unbalance between negative samples from a subset and positive samples from another one is likely to indicate a needed segmentation.

Original: Different additional segmentation levelscould be explored to build DSs.
Revised: Through an iterative process, one can deﬁne the DSs relevant to the the learning set and the task to perform: (a) consider N = 1 DS (b) train a model (c) evaluate the models on the potential test DSs of interest, (d) consider partitioning the learning set into N > 1 DSs according to some known characteristics (e.g. source, manufacturer, acquisition device version, population characteristics). Only accumulated knowledge of the data led us to consider the sources and the manufacturers as a segmentation levels to split the learning set into DSs.

Original: A potential segmentation of the learning set according to the ethnicity (which may be already covered by the segmentation by source), the age, the biological sex of patients stand as different levels which may be worth investing to improve fairness.
Revised: A potential segmentation of the learning set according to the ethnicity (which may be already covered by the segmentation by source), the age, the biological sex of patients stand as different levels which may be worth investing to improve fairness.

Original: But dealing withsuch sensible information also raises data acquisition and privacy concerns about whether one (e.g. a machine learner) can have access to such information and how can one safely store them, use them.
Revised: Still, addressing such segmentation levels implies dealing with sensible information raising concerns about data privacy and data security.

Original: This work also raises the question of what a reliable validation metric is. Our experiments show that the global WLL / image or AUC cannot be entirely reliable and blindly trusted.
Revised: The proposed approach still works when there are more than 2 segmentation levels (not shown) but the segmentation into DSs needs to be deﬁned on some categorical criteria.

Original: It seems that only a careful analysis of validation performances on the different DSs can help to point out potential issues.
Revised: Binning a continuous variable could allow one to tackle a continuous source of bias (e.g. age). Further investigation are needed to cover this aspect.

Original: This work builds upon the assumption that 2 DSs DS i and DS j do not have some shared information and are independent.
Revised: This work builds upon the assumption that 2 DSs DS i and DS j do not have some shared information and are independent.

Original: Yet, this may be wrong at some point.
Revised: Yet, this may be wrong at some point.

Original: Indeed, some patients may have data in different DSs when for instance a hospital has used different acquisition devices over time or when a patient goes to another hospital.
Revised: Indeed, some patients may have data in different DSs when for instance a hospital has used different acquisition devices over time or when a patient goes to another hospital.

Original: Adapting the sampling proportions used to generate the sampling pattern could help to cope with this issue.
Revised: Adapting the sampling proportions used to generate the sampling pattern could help to cope with this issue.

Original: Future work should cover this subject.
Revised: Future work should investigate this subject.

Original: The proposed approach relies on a  sampling pattern thatis ﬁxed at the beginning of the learning process and cannot change based on the dynamics of the training processas in (Hu et al., 2019; Ren et al., 2018).
Revised: The proposed approach relies on a ﬁxed sampling pattern that cannot change with the dynamics of the training process (Hu et al., 2019; Ren et al., 2018).

Original: A dynamic and learned sampling could be an advantage and cope with issues when  DSs are not independent.
Revised: A dynamic and learnable sampling could be an advantage and cope with issues when 2 DSs are not independent.

Original: The problem of learning from multiple DSs could be addressed via sample weighting (Shimodaira, 2000). This could result in similaror improved performances provided that one samples all labels and DSs in equal proportions to avoid having batches containing samples of a single labeland / orfrom a single DS.
Revised: Alternatively, the problem addressed in this work could be tackled with sample weighting (Shimodaira, 2000) and may lead to similar performances provided that one samples all labels and DSs in equal proportions to avoid having batches containing samples of a single label or a single DS.

Original: This paper focuses on binary classiﬁcation through the example of breast cancer detection.
Revised: This paper focuses on binary classiﬁcation through the application of breast cancer prediction, a concrete but also restricted scenario.

Original: Yet the formalism is general enough to be used for multi-label classiﬁcation problems
Revised: Yet, the formalism is general enough to be tried with other modalities of data and / or other learning tasks. This shall be covered in follow up studies

Original: D REAM N ET ARCHITECTURE 5. G ENERAL ARCHITECTURE
Revised: A A PPENDIX A. A DDITIONAL INFORMATION ABOUT DATA

Original: DN is a fully convolutional neural network developed during the Digital Mammography DREAM
Revised: This section contains additional information about the training and testing data.

Original: Challenge (Schaffter et al., 2020).
Revised: Table 5 provides information about the countries of origin of the sources and the number of images per status and type of view. Table 6 provides information about the types of the malignant lesions in each DS. Table 7 provides information about the age of patients in the different DSs. A. M ODEL ARCHITECTURE A.2. G ENERAL ARCHITECTURE

Original: This is a classiﬁcation model predicting image wise probability of malignancy on FFDM images.
Revised: The considered DNN is a fully convolutional neural network predicting image wise probability of malignancy on FFDM images.

Original: DN is thus trained on whole FFDM images of shape (1152 , 832) with image-wise labels as explained in the main body of the paper.
Revised: The DNN is thus trained on whole FFDM images of shape (1152 , 832) with image-wise labels as explained in the main body of the paper.

Original: DN exhibits a VGG-like architecture (Simonyan & Zisserman, 2015) and adopts a 2-level architecture as depicted in Figure 6.
Revised: The DNN exhibits a VGG-like architecture (Simonyan & Zisserman, 2015) and adopts a 2-level architecture as depicted in Figure 5.

Original: The ﬁrst part of DN is referred to as the Backbone .
Revised: The ﬁrst part of the DNN is referred to as the Backbone .

Original: The Backbone aims at creating an intermediate representation of the full image which facilitates prediction of the ﬁnal image-wise probability of malignancy.
Revised: The Backbone aims at creating an intermediate representation of the full image which facilitates prediction of the ﬁnal image-wise probability of malignancy.

Original: The second part of the model is the classiﬁer and it is in charge of predicting the ﬁnal image wise probability of malignancy.
Revised: The second part of the model is the classiﬁer and it is in charge of predicting the ﬁnal image wise probability of malignancy.

Original: The Backbone and the classiﬁer are detailed below.
Revised: The Backbone and the classiﬁer are detailed below.

Original: 5. B ACKBONE
Revised: A.2. B ACKBONE

Original: The Backbone is built to take as input a patch of size (224 , 224) and to predict its associated label.
Revised: The Backbone is built to take as input a patch of size (224 , 224) and to predict its associated label.

Original: As 5 labels are possible, the Backbone outputs a vector of size 5 .
Revised: As 5 labels are possible, the Backbone outputs a vector of size 5 .

Original: It is designed in a fully convolutionnal way and thus can take as input, images of arbitrary size to form an output feature map.
Revised: It is designed in a fully convolutionnal way and thus can take as input, images of arbitrary size to form an output feature map.

Original: In the case of images at resolution (1152 , 832) , the Backbone outputs a feature map of size (16 , 11 , 5) .
Revised: In the case of images at resolution (1152 , 832) , the Backbone outputs a feature map of size (16 , 11 , 5) .

Original: This returned feature map plays the role of the intermediate representation mentioned above.
Revised: This returned feature map plays the role of the intermediate representation mentioned above.

Original: This can be seen as a score vector of size 5 for each spatial position in the grid (16 , 11) .
Revised: This can be seen as a score vector of size 5 for each spatial position in the grid (16 , 11) .

Original: The backbone architecture is given in Figure 6.
Revised: The backbone architecture is given in Figure 5.

Original: It builds upon a succession of convolution blocks of the form: (a) batch normalization Ioffe & Szegedy (2015), (b) 2D convolution with (3 , 3) kernels and ReLU activation Nair & Hinton (2010), (c) 2D max pooling with (2 , 2) pool size and stride of (2 , 2) .
Revised: It builds upon a succession of convolution blocks of the form: (a) batch normalization Ioffe & Szegedy (2015), (b) 2D convolution with (3 , 3) kernels and ReLU activation Nair & Hinton (2010), (c) 2D max pooling with (2 , 2) pool size and stride of (2 , 2) .

Original: The numbers of kernels in each convolution block are given in Figure 6.
Revised: The numbers of kernels in each convolution block are given in Figure 5.

Original: Finally, 3 2D convolution layers with resp.
Revised: Finally, 3 2D convolution layers with resp.

Original: 1024 , 512 and 5 kernels of shapes resp.
Revised: 1024 , 512 and 5 kernels of shapes resp.

Original: (2 , 2) , (1 , 1) and (1 , 1) and
Revised: (2 , 2) , (1 , 1) and (1 , 1) and

Original: ReLU activations play the role of the patch classiﬁer to output a vector of size 5 .
Revised: ReLU activations play the role of the patch classiﬁer to output a vector of size 5 .

Original: [ 0 , 1 , 2 , 3 ,] c o n v 1 , c o n v 2 , 1024 512 5 p oo l 2 xc o n v 3 , c o n v 4 , p oo l 2 xc o n v 5 , c o n v 6 , p oo l 2 xc o n v 7 , c o n v 8 , p oo l 2 xc o n v 9 , c o n v 10 , p oo l 2 xc o n v 11 , c o n v 12 , p oo l 2 xp oo l _ f i n a l
Revised: A.2. I MAGE CLASSIFIER

Original: [Figure] [Figure] [Figure]
Revised: To predict the ﬁnal image-wise probability of malignancy, the intermediate representation built by the Backbone is fed into the classiﬁer that processes it with convolutional blocks to obtain the ﬁnal image wise prediction.

Original: [Figure]
Revised: [Table]

Original: 5. D REAM N ET IMAGE CLASSIFIER
Revised: a ) S M a

Original: To predict the ﬁnal image-wise probability of malignancy, the intermediate representation built by the Backbone is fed into the classiﬁer that processes it with convolutional blocks to obtain the ﬁnal image wise prediction.
Revised: France malignant other ( S 2 , M b )

Original: The classiﬁer builds upon a MaxPooling layer with poolsize of (5 , 5) and stride of (5 , 5) .
Revised: S M b France benign CC ( S 2 , M b )

Revised: S

Revised: M b

Revised: France malignant

Revised: CC ( S 2 , M b )

Revised: S

Revised: M b

Revised: France benign

Revised: MLO ( S 2 , M b )

Revised: S

Revised: M b

Revised: France malignant

Revised: MLO ( S 2 , M b )

Revised: S

Revised: M b

Revised: France benign other ( S 2 , M b )

Revised: S

Revised: M b

Revised: France malignant other ( S 3 , M a )

Revised: S

Revised: M a

Revised: U.S.A. benign

Revised: CC ( S 3 , M a )

Revised: S

Revised: M a

Revised: U.S.A. malignant

Revised: CC ( S 3 , M a )

Revised: S

Revised: M a

Revised: U.S.A. benign

Revised: MLO ( S 3 , M a )

Revised: S

Revised: M a

Revised: U.S.A. malignant

Revised: MLO ( S 3 , M a )

Revised: S

Revised: M a

Revised: U.S.A. benign other ( S 3 , M a )

Revised: S

Revised: M a

Revised: U.S.A. malignant other ( S 3 , M b )

Revised: S

Revised: M b

Revised: U.S.A. benign

Revised: CC ( S 3 , M b )

Revised: S

Revised: M b

Revised: U.S.A. malignant

Revised: CC ( S 3 , M b )

Revised: S

Revised: M b

Revised: U.S.A. benign

Revised: MLO ( S 3 , M b )

Revised: S

Revised: M b

Revised: U.S.A. malignant

Revised: MLO ( S 3 , M b )

Revised: S

Revised: M b

Revised: U.S.A. benign other ( S 3 , M b )

Revised: S

Revised: M b

Revised: U.S.A. malignant other ( S 4 , M c )

Revised: S

Revised: M c

Revised: France benign

Revised: CC ( S 4 , M c )

Revised: S

Revised: M c

Revised: France malignant

Revised: CC ( S 4 , M c )

Revised: S

Revised: M c

Revised: France benign

Revised: MLO ( S 4 , M c )

Revised: S

Revised: M c

Revised: France malignant

Revised: MLO ( S 4 , M c )

Revised: S

Revised: M c

Revised: France benign other ( S 4 , M c )

Revised: S

Revised: M c

Revised: France malignant other ( S 4 , M d )

Revised: S

Revised: M d

Revised: France benign

Revised: CC ( S 4 , M d )

Revised: S

Revised: M d

Revised: France malignant

Revised: CC ( S 4 , M d )

Revised: S

Revised: M d

Revised: France benign

Revised: MLO ( S 4 , M d )

Revised: S

Revised: M d

Revised: France malignant

Revised: MLO ( S 4 , M d )

Revised: S

Revised: M d

Revised: France benign other ( S 4 , M d )

Revised: S

Revised: M d

Revised: France malignant other ( S 5 , M c )

Revised: S

Revised: M c

Revised: France benign

Revised: CC ( S 5 , M c )

Revised: S

Revised: M c

Revised: France malignant

Revised: CC ( S 5 , M c )

Revised: S

Revised: M c

Revised: France benign

Revised: MLO ( S 5 , M c )

Revised: S

Revised: M c

Revised: France malignant

Revised: MLO ( S 5 , M c )

Revised: S

Revised: M c

Revised: France benign other ( S 5 , M c )

Revised: S

Revised: M c

Revised: France malignant other

Revised: [Table]

Revised: c )

Revised: S

Revised: M c malignant calciﬁcation ( S 4 , M c )

Revised: S

Revised: M c malignant

Revised: [Table]

Revised: [Table]

Revised: lesion ( S 4 , M

Revised: [Table]

Revised: S

Revised: M c

Revised: [Table]

Revised: [Table]

Revised: 4 , M d )

Revised: S

Revised: M d

Revised: [Table]

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: S

Revised: [Table]

Revised: M d

Revised: [Table]

Revised: lesion

Revised: [Table]

Revised: S

Revised: [Table]

Revised: M c

Revised: [Table]

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: M

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: M

Revised: [Table]

Revised: S

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: S

Revised: [Table]

Revised: , M

Revised: [Table]

Revised: S

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: ( S

Revised: [Table]

Revised: , M

Revised: [Table]

Revised: S

Revised: [Table]

Revised: M

Revised: [Table]

Revised: 54 ±( S

Revised: [Table]

Revised: , M

Revised: [Table]

Revised: S

Revised: M b

Revised: [Table]

Revised: 61 ±( S

Revised: [Table]

Revised: , M

Revised: [Table]

Revised: S

Revised: [Table]

Revised: M a

Revised: [Table]

Revised: [Table]

Revised: ( S 3 , M a )

Revised: S

Revised: M a

Revised: [Table]

Revised: 61 ±( S

Revised: [Table]

Revised: , M

Revised: [Table]

Revised: S

Revised: M b

Revised: [Table]

Revised: [Table]

Revised: ( S

Revised: [Table]

Revised: , M b )

Revised: S

Revised: M b

Revised: [Table]

Revised: 60 ±(

Revised: [Table]

Revised: , M c )

Revised: S

Revised: M c benign 53 ±( S 4 , M c )

Revised: S

Revised: M c malignant 57 ±( S 4 , M d )

Revised: S

Revised: M d benign 55 ±( S 4 , M d )

Revised: S

Revised: M d malignant 59 ±( S 5 , M c )

Revised: S

Revised: M c benign 54 ±( S 5 , M c )

Revised: S

Revised: M c malignant 61 ±

Revised: [ 0 , 1 , 2 , 3 ,] c o n v 1 , c o n v 2 , 1024 512 5 p oo l 2 xc o n v 3 , c o n v 4 , p oo l 2 xc o n v 5 , c o n v 6 , p oo l 2 xc o n v 7 , c o n v 8 , p oo l 2 xc o n v 9 , c o n v 10 , p oo l 2 xc o n v 11 , c o n v 12 , p oo l 2 xp oo l _ f i n a l

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: The classiﬁer builds upon a MaxPooling layer with poolsize of (5 , 5) and stride of (5 , 5) .

Original: Then, it is composed of 2 sub-parts which process the max-pooled feature maps.
Revised: Then, it is composed of 2 sub-parts which process the max-pooled feature maps.

Original: The ﬁrst sub-part builds onsuccessive blocks of convolution of the form: (a) batch normalization layer, (b) ReLU activation (c) 2D convolution with resp.
Revised: The ﬁrst sub-part builds onsuccessive blocks of convolution of the form: (a) batch normalization layer, (b) ReLU activation (c) 2D convolution with resp.

Original: 16 , 8 and 2 kernels of size resp.
Revised: 16 , 8 and 2 kernels of size resp.

Original: (3 , 2) , (1 , 1) and (1 , 1) .
Revised: (3 , 2) , (1 , 1) and (1 , 1) .

Original: This sub-part returns a vector of size 2 .
Revised: This sub-part returns a vector of size 2 .

Original: The second sub-part is composed of a batch normalization layer and a convolution layer with 2 kernels of size (3 , 2) .
Revised: The second sub-part is composed of a batch normalization layer and a convolution layer with 2 kernels of size (3 , 2) .

Original: This sub-part returns a vector of size 2 as well.
Revised: This sub-part returns a vector of size 2 as well.

Original: The outputs of the sub-parts are ﬁnally summed up to form the ﬁnal image wise prediction.
Revised: The outputs of the sub-parts are ﬁnally summed up to form the ﬁnal image wise prediction.

Original: 5. R EMARK ON TRAINING
Revised: A.2. R EMARK ON TRAINING

Original: The 2-level architecture is also reﬂected in the training procedure which is a two-step process.
Revised: The 2-level architecture is also reﬂected in the training procedure which is a two-step process.

Original: First, the Backbone is pre-trained on patches to predict their labels.
Revised: First, the Backbone is pre-trained on patches to predict their labels.

Original: Then, the Backbone weights are used to initialized the full DN weights and DN is then ﬁne-tuned on whole FFDM images to make image wise predictions of malignancy.
Revised: Then, the Backbone weights are used to initialized the full DNN weights and the DNN is then ﬁne-tuned on whole FFDM images to make image wise predictions of malignancy.

Original:  B ACKBONE TRAINING
Revised: A. B ACKBONE TRAINING

Original: The training process of the Backbone is detailed below, the training / ﬁne-tuning process of DN being described in the main body of the paper.
Revised: The training process of the Backbone is detailed below, the training / ﬁne-tuning process of the DNN being described in the main body of the paper.

Original: Practically, the Backbone is fed with patches centered on annotated lesions and is trained to predict which label among 5 possible labels should be assigned to the patch.
Revised: Practically, the Backbone is fed with patches centered on annotated lesions and is trained to predict which label among 5 possible labels should be assigned to the patch.

Original: 6. D ATA
Revised: A.3. D ATA

Original: Lesions have been annotated on some images from the DS ( S 1 , M a ) by trained radiologists.
Revised: Lesions have been annotated on some images from the DS ( S 1 , M a ) by trained radiologists.

Original: Lesions positions and lesions labels have been assigned.
Revised: Lesions positions and lesions labels have been assigned.

Original: Lesions have been categorized into 5 classes: healthy tissue, benign soft tissue, benign calciﬁcation, malignant soft tissue, malignant calciﬁcation.
Revised: Lesions have been categorized into 5 classes: healthy tissue, benign soft tissue, benign calciﬁcation, malignant soft tissue, malignant calciﬁcation.

Original: The numbers of annotated lesions and images are provided in Table 5. [Table]
Revised: The numbers of annotated lesions and images are provided in Table 8.

Original: Cross-validation scheme Similarly as described in the main paper, the learning set is divided into training and validation data using a source-wise and patient-wise split: 80% (resp.
Revised: Cross-validation scheme Similarly as described in the main paper, the learning set is divided into training and validation data using a source-wise and patient-wise split: 80% (resp.

Original: 20% ) of the  patients of each source are used for training (resp. validation).
Revised: 20% ) of the [Table] patients of each source are used for training (resp. validation).

Original: Furthermore, the split is made such that the ratio of malignant / benign patients is similar for the validation and the training sets.
Revised: Furthermore, the split is made such that the ratio of malignant / benign patients is similar for the validation and the training sets.

Original: The same train-validation split as described in the paper is used to train the Backbone.
Revised: The same train-validation split as described in the paper is used to train the Backbone.

Original: This means that the learning samples of a patient used for training (resp.
Revised: This means that the learning samples of a patient used for training (resp.

Original: validation of) the BackBone, are also used for training (resp.
Revised: validation of) the BackBone, are also used for training (resp.

Original: validation of) DN.
Revised: validation of) the DNN.

Original: Image pre-processing and data augmentation Similarly as in the paper, normalization and preprocessing techniques are used to standardize the FFDM images.
Revised: Image pre-processing and data augmentation Similarly as in the paper, normalization and preprocessing techniques are used to standardize the FFDM images.

Original: A patch of size nearly equal to (224 , 224) (see below why) is ﬁrst cropped from a FFDM image.
Revised: A patch of size nearly equal to (224 , 224) (see below why) is ﬁrst cropped from a FFDM image.

Original: The pixels inside the breast mask are re-normalized to have values in [0 , 1] .
Revised: The pixels inside the breast mask are re-normalized to have values in [0 , 1] .

Original: The pixels outside mask are set to a constant value such that the average pixel value of the image is always the same.
Revised: The pixels outside mask are set to a constant value such that the average pixel value of the image is always the same.

Original: A standard augmentation pipeline (random rotation, zoom, shearing... ) is used to augment images used for training and to return an augmented patch of size (224 , 224) .
Revised: A standard augmentation pipeline (random rotation, zoom, shearing... ) is used to augment images used for training and to return an augmented patch of size (224 , 224) .

Original: No augmentation technique is used for validation patches.
Revised: No augmentation technique is used for validation patches.

Original: The augmentation transform is always pre-computed, to know exactly the dimension of the patch to crop.
Revised: The augmentation transform is always pre-computed, to know exactly the dimension of the patch to crop.

Original: This way, every pixel in the augmented patch is an output, through the augmentation transform, of a pixel from the original patch: the transform is surjective.
Revised: This way, every pixel in the augmented patch is an output, through the augmentation transform, of a pixel from the original patch: the transform is surjective.

Original: This is why the cropping step, for training patches, does not return patches with an exact shape of (224 , 224) but with a shape of nearly (224 , 224) .
Revised: This is why the cropping step, for training patches, does not return patches with an exact shape of (224 , 224) but with a shape of nearly (224 , 224) .

Original: 6. B ACKBONE TRAINING
Revised: A.3. T RAINING SETUP

Original: The Backbone is trained to minimize the categorical cross-entropy deﬁned for 5 classes.
Revised: The Backbone is trained to minimize the categorical cross-entropy deﬁned for 5 classes.

Original: The training process is done on 20000 learning steps, a sufﬁciently large number of steps to see convergence.
Revised: The training process is done on 20000 learning steps, a sufﬁciently large number of steps to see convergence.

Original: Samples labels are balanced within a batch to ﬁght potential issues related to class imbalance.
Revised: Samples labels are balanced within a batch to ﬁght potential issues related to class imbalance.

Original: The training relies on Adam optimizer Kingma & Ba (2014) and a learning rate equal to 10 − 3 .
Revised: The training relies on Adam optimizer (Kingma & Ba, 2014) and a learning rate equal to 10 − 3 .

Original: Exponential moving average with decay 0 .
Revised: Exponential moving average with decay 0 .

Original: 99 of all trainable parameters are tracked, their averaged versions are then used for inference on validation data.
Revised: 99 of all trainable parameters are tracked, their averaged versions are then used for inference on validation data.

Original: The training is done on 2 Tesla P100 and a batch size of 64 samples is used ( 32 per GPU).
Revised: The training is done on 2 Tesla P100 and a batch size of 64 samples is used ( 32 per GPU).

Original: A validation step is made every 500 training steps.
Revised: A validation step is made every 500 training steps.

Original: The Weighted Logloss per lesion (WLL / lesion), the standard logloss with samples weights estimated on the validation set was used to monitor the training progress.
Revised: The Weighted Logloss per lesion (WLL / lesion), the standard logloss with samples weights estimated on the validation set was used to monitor the training progress.

Original: The checkpoint achieving the best Weighted Logloss per lesion (WLL / lesion) on the validation set is kept to initialize the DN weights.
Revised: The checkpoint achieving the best Weighted Logloss per lesion (WLL / lesion) on the validation set is kept to initialize the DNN weights.

Original: The models and the experiments were carried out with Tensorﬂow 2.3 (Abadi et al., 2015).
Revised: The models and the experiments were carried out with Tensorﬂow 2.3 (Abadi et al., 2015).

Original: S UBMISSION OF CONFERENCE PAPERS TO ICLR

Original: ICLR requires electronic submissions, processed by https://openreview.net/ .

Original: See ICLR’s website for more instructions.

Original: If your paper is ultimately accepted, the statement \ iclrfinalcopy should be inserted to adjust the format to the camera ready requirements.

Original: The format for the submissions is a variant of the NeurIPS format.

Original: Please read carefully the instructions below, and follow them faithfully.

Original: 7. S TYLE

Original: Papers to be submitted to ICLR 2022 must be prepared according to the instructions presented here.

Original: Authors are required to use the ICLR L A TEX style ﬁles obtainable at the ICLR website.

Original: Please make sure you use the current ﬁles and not previous versions.

Original: Tweaking the style ﬁles may be grounds for rejection.

Original: 7. R ETRIEVAL OF STYLE FILES

Original: The style ﬁles for ICLR and other conference information are available online at: http://www.iclr.cc/

Original: The ﬁle iclr2022_conference.pdf contains these instructions and illustrates the various formatting requirements your ICLR paper must satisfy.

Original: Submissions must be made using L A TEX and the style ﬁles iclr2022_conference.sty and iclr2022_conference.bst (to be used with L A TEX2e).

Original: The ﬁle iclr2022_conference.tex may be used as a “shell” for writing your paper.

Original: All you have to do is replace the author, title, abstract, and text of the paper with your own.

Original: The formatting instructions contained in these style ﬁles are summarized in sections 8, 9, andbelow.

Original: G ENERAL FORMATTING INSTRUCTIONS

Original: The text must be conﬁned within a rectangle 5.5 inches (33 picas) wide and 9 inches (54 picas) long.

Original: The left margin is 1.5 inch (9 picas).

Original: Use 10 point type with a vertical spacing of 11 points.

Original: Times

Original: New Roman is the preferred typeface throughout.

Original: Paragraphs are separated by 1/2 line space, with no indentation.

Original: Paper title is 17 point, in small caps and left-aligned.

Original: All pages should start at 1 inch (6 picas) from the top of the page.

Original: Authors’ names are set in boldface, and each name is placed above its corresponding address.

Original: The lead author’s name is to be listed ﬁrst, and the co-authors’ names are set to follow.

Original: Authors sharing the same address can be on the same line.

Original: Please pay special attention to the instructions in section 10 regarding ﬁgures, tables, acknowledgments, and references.

Original: There will be a strict upper limit of 9 pages for the main text of the initial submission, with unlimited additional pages for citations.

Original: H EADINGS : FIRST LEVEL

Original: First level headings are in small caps, ﬂush left and in point size 12.

Original: One line space before the ﬁrst level heading and 1/2 line space after the ﬁrst level heading.

Original: 9. H EADINGS : SECOND LEVEL

Original: Second level headings are in small caps, ﬂush left and in point size 10.

Original: One line space before the second level heading and 1/2 line space after the second level heading.

Original: 9.1. H EADINGS : THIRD LEVEL

Original: Third level headings are in small caps, ﬂush left and in point size 10.

Original: One line space before the third level heading and 1/2 line space after the third level heading.

Original: C ITATIONS , FIGURES , TABLES , REFERENCES

Original: These instructions apply to everyone, regardless of the formatter being used.

Original: 10. C ITATIONS WITHIN THE TEXT

Original: Citations within the text should be based on the natbib package and include the authors’ last names and year (with the “et al.”

Original: construct for more than two authors).

Original: When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{} (as in “See ? for more information.”).

Original: Otherwise, the citation should be in parenthesis using \citep{} (as in “Deep learning shows promise to make progress towards AI ( ? ).”).

Original: The corresponding references are to be listed in alphabetical order of authors, in the R EFERENCES section.

Original: As to the format of the references themselves, any style is acceptable as long as it is used consistently.

Original: 10. F OOTNOTES

Original: Indicate footnotes with a number 1 in the text.

Original: Place the footnotes at the bottom of the page on which they appear.

Original: Precede the footnote with a horizontal rule of 2 inches (12 picas).

Original: 10. F IGURES

Original: All artwork must be neat, clean, and legible.

Original: Lines should be dark enough for purposes of reproduction; art work should not be hand-drawn.

Original: The ﬁgure number and caption always appear after the ﬁgure.

Original: Place one line space before the ﬁgure caption, and one line space after the ﬁgure.

Original: The ﬁgure caption is lower case (except for ﬁrst word and proper nouns); ﬁgures are numbered consecutively.

Original: Make sure the ﬁgure caption does not get separated from the ﬁgure.

Original: Leave sufﬁcient space to avoid splitting the ﬁgure and ﬁgure caption.

Original: You may use color ﬁgures.

Original: However, it is best for the ﬁgure captions and the paper body to make sense if the paper is printed either in black/white or in color.

Original: [Table]

Original: All tables must be centered, neat, clean and legible.

Original: Do not use hand-drawn tables.

Original: The table number and title always appear before the table.

Original: See Table 6.

Original: Place one line space before the table title, one line space after the table title, and one line space after the table.

Original: The table title must be lower case (except for ﬁrst word and proper nouns); tables are numbered consecutively.

Original: D EFAULT N OTATION

Original: In an attempt to encourage standardized notation, we have included the notation ﬁle from the textbook, Deep Learning ?

Original: available at https://github.com/goodfeli/dlbook_ notation/ .

Original: Use of this style is not required and can be disabled by commenting out math commands.tex .

Original: Numbers and Arrays

Original: a A scalar (integer or real) a A vector A A matrix A

Original: A tensor

Original: I n

Original: Identity matrix with n

Original: rows

Original: and n columns

Original: I

Original: Identity matrix with dimensionality implied by context e i )

Original: Standard basis vector [0 , . . .

Original: . , 0] with a 1 at position i diag a

Original: A square, diagonal matrix with diagonal entries given by

Original: a a

Original: A scalar random variable a A vector-valued random variable

Original: A A matrix-valued random variable

Original: Sets and Graphs

Original: A

Original: A set

Original: R

Original: The set of real numbers { 0 , 1 }

Original: The set containing 0 and{ 0 , 1 , . . . , n }

Original: The set of all integers between 0 and n [ a, b ]

Original: The real interval including a and b ( a, b ]

Original: The real interval excluding a but including b

Original: A \ B

Original: Set subtraction, i.e., the set containing the elements of A that are not in B

Original: G

Original: A graph

Original: Pa G ( x i )

Original: The parents of x i in G

Original: Indexing a

Original: i

Original: Element i of vector a , with indexing starting ata − i

Original: All elements of vector a except for element

Original: i

Original:  A i,j Element i, j of matrix A A i, :
Revised: A. A DDITIONAL FIGURE FOR EXPERIMENT

Original: Row i of matrix

Original: A

Original: A : ,i

Original: Column i of matrix

Original: A

Original: A i,j,k

Original: Element ( i, j, k ) of a 3-D tensor A

Original: A : , : ,i 2-D slice of a 3-D tensor a i

Original: Element i of the random vector

Original: a Calculus dy dx Derivative of y with respect to x ∂y ∂x Partial derivative of y with respect to x ∇ x y
Revised: On top of AUC gain - loss training a DNN with ”Bal.

Original: Gradient of y with respect to x ∇ X y Matrix derivatives of y with respect to X ∇ X y Tensor containing derivatives of y with respect to X ∂f ∂ x Jacobian matrix J ∈ R m × n off : R n → R m ∇ 2 x f ( x ) or H ( f )( x )
Revised: labels” leads to a poor synchronization of

Original: The Hessian matrix of f at input point x (cid:90) f

Original: ( x ) d x

Original: Deﬁnite integral over the entire domain of x (cid:90)

Original: S f ( x ) d x Deﬁnite integral with respect to x over the set S Probability andInformation Theory P ( a ) 
Revised: OPs on the ROC curves obtained on ( S 3 , M a ) and ( S 3 , M b ) , see Figure 6 - left.

Original: A probability distribution over a discrete variable p ( a ) A probability distribution over a continuous variable, or over a variable whose type has not been speciﬁed a ∼ P Random variable a  has distribution P E x ∼ P [ f ( x )] or E f ( x )
Revised: This suggests that the DNN works in 2 different regimes on the 2 test DSs: a non desired property which stresses the manufacturer bias the DNN has integrated.

Original: Expectation of f ( x ) with respect to P ( x )

Original: Var( f ( x ))

Original: Variance of f ( x ) under P ( x )

Original: Cov( f ( x ) , g ( x ))

Original: Covariance of f ( x ) and g ( x ) under P ( x )

Original: H ( x )

Original: Shannon entropy of the random variable x

Original: D KL ( P (cid:107) Q ) Kullback-Leibler divergence of P and Q N ( x ; µ , Σ ) Gaussian distribution over x with mean µ and covariance Σ
Revised: On the other hand, the DNN trained with ”P. sampling” gives visually better synchronized OPs on the ROC curves of test DSs.

Original: Functions f : A → B The function f with domain A and range B f ◦ g Composition of the functions f and g f ( x ; θ ) A function of x parametrized by θ .
Revised: This suggests that this DNN behaves quite similarly on them and that it is not inﬂuenced by the manufacturer to take its decisions.

Original: (Sometimes we write f ( x ) and omit the argument θ to lighten notation) log x

Original: Natural logarithm of x σ ( x )

Original: Logistic sigmoid, 1 + exp( − x ) ζ ( x )

Original: Softplus, log(1 + exp( x )) ||

Original: x || p

Original: L p norm of x || x ||

Original: L 2 norm of x x +

Original: Positive part of x , i.e., max(0 , x ) 1 condition is 1 if the condition is true, 0 otherwise

Original: F INAL INSTRUCTIONS

Original: Do not change any aspects of the formatting parameters in the style ﬁles.

Original: In particular, do not modify the width or length of the rectangle the text should ﬁt into, and do not change font sizes (except perhaps in the R EFERENCES section; see below).

Original: Please note that pages should be numbered.

Original: P REPARING P OST S CRIPT OR PDF FILES

Original: Please prepare PostScript or PDF ﬁles with paper size “US Letter”, and not, for example, “A4”.

Original: The -t letter option on dvips will produce US Letter ﬁles.

Original: Consider directly generating PDF ﬁles using pdflatex (especially if you are a MiKTeX user).

Original: PDF ﬁgures must be substituted for EPS ﬁgures, however.

Original: Otherwise, please generate your PostScript and PDF ﬁles with the following commands:

Original: dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper

Original: [Equation]
Revised: [Figure]

Original: ps ps2pdf mypaper.ps mypaper.pdf

Original: 13. M ARGINS IN L A T E X

Original: Most of the margin problems come from ﬁgures positioned by hand using \special or other commands.

Original: We suggest using the command \includegraphics from the graphicx package.

Original: Always specify the ﬁgure width as a multiple of the line width as in the example below using .eps graphics \usepackage[dvips]{graphicx} ...

Original: \includegraphics[width=0.8\linewidth]{myfile.eps} or \usepackage[pdftex]{graphicx} ...

Original: \includegraphics[width=0.8\linewidth]{myfile.pdf} for .pdf graphics.

Original: See section 4.4 in the graphics bundle documentation ( http://www.ctan.

Original: org/tex-archive/macros/latex/required/graphics/grfguide.ps )

Original: A number of width problems arise when LaTeX cannot properly hyphenate a line.

Original: Please give

Original: LaTeX hyphenation hints using the \- command.

Original: A UTHOR C ONTRIBUTIONS

Original: If you’d like to, you may include a section for author contributions as is done in many journals.

Original: This is optional and at the discretion of the authors.

Original: A CKNOWLEDGMENTS

Original: Use unnumbered third level headings for the acknowledgments.

Original: All acknowledgments, including those to funding agencies, go at the end of the paper.

Original: A A PPENDIX

Original: You may include other additional sections here.