Original: Curriculum Learning with Diversity for Supervised Computer Vision Tasks
Revised: Curriculum Learning with Diversity for Supervised Computer Vision Tasks

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Curriculum learning techniques are a viable solution for improving the accuracy of automatic models, by replacing the traditional random training with an easy to hard strategy.
Revised: Curriculum learning techniques are a viable solution for improving the accuracy of automatic models, by replacing the traditional random training with an easy to hard strategy.

Original: However, the standard curriculum methodology does not automatically provide improved results, but is constrained by multiple elements like the data distribution or the proposed model.
Revised: However, the standard curriculum methodology does not automatically provide improved results, but is constrained by multiple elements like the data distribution or the proposed model.

Original: In this paper, we introduce a novel curriculum sampling strategy which takes into consideration the diversity of the training data together with the difﬁculty of the inputs.
Revised: In this paper, we introduce a novel curriculum sampling strategy which takes into consideration the diversity of the training data together with the difﬁculty of the inputs.

Original: We determine the difﬁculty using a stateof-the-art difﬁculty estimator and we model the diversity during training, giving higher priority to elements from classes visited less.
Revised: We determine the difﬁculty using a stateof-the-art difﬁculty estimator and we model the diversity during training, giving higher priority to elements from classes visited less.

Original: We conduct object detection and instance segmentation experiments on Pascal VOC 2007 and Cityscapes data sets, surpassing both the randomly-trained baseline and the standard curriculum approach.
Revised: We conduct object detection and instance segmentation experiments on Pascal VOC 2007 and Cityscapes data sets, surpassing both the randomly-trained baseline and the standard curriculum approach.

Original: We prove that our strategy is very efﬁcient in unbalanced data sets, leading to faster convergence and more accurate results, where other curriculumbased strategies fail.
Revised: We prove that our strategy is very efﬁcient in unbalanced data sets, leading to faster convergence and more accurate results, where other curriculumbased strategies fail.

Original: 1. Introduction
Revised: 1. Introduction

Original: Although the quality of automatic models highly increased with the development of deep and very deep neural networks, an important and less studied key elementfor the overall performance is the training strategy.
Revised: Although the quality of automatic models highly increased with the development of deep and very deep neural networks, an important and less studied key element is the training strategy.

Original: In this direction, (Bengio et al., 2009) introduced curriculum learning (CL), a set of learning strategies inspired by the way humans teach and learn.
Revised: In this direction, Bengio et al. introduced curriculum learning (CL), a set of learning strategies inspired by the way humans teach and learn.

Original: People learn the easiest concepts at ﬁrst, followed by more and more complex elements.
Revised: People learn the easiest concepts at ﬁrst followed by more and more complex elements.

Original: Similarly, CL proposes feeding the automatic model with easier samples at the beginning of the training, while gradually introducing more difﬁcult data as the training proceeds.
Revised: Similarly, CL proposes feeding the automatic model with easier samples at the beginning of the training, while gradually introducing more difﬁcult data as the training proceeds.

Original: The idea is straightforward, but an important question is how to determine whether a sample is easy or hard.
Revised: The idea is straightforward, but an important question is how to determine whether a sample is easy or hard.

Original: CL requires the existence of a predeﬁned metric which can compute the difﬁculty of the input examples.
Revised: CL requires the existence of a predeﬁned metric which can compute the difﬁculty of the input examples.

Original: In this paper, we use the image difﬁculty estimator from (Ionescu et al., 2016) which is based on the time required by human annotators to identify if a class is present or not in a certain image.
Revised: In this paper, we use the image difﬁculty estimator from (Ionescu et al., 2016) which is based on the time required by human annotators to identify if a class is present or not in a certain image.

Original: The next challenge is building the curriculum schedule, or the rate at which we can augment the training set with more complex information.
Revised: The next challenge is building the curriculum schedule, or the rate at which we can augment the training set with more complex information.

Original: To address this problem, we follow a sampling strategy similar to the one introduced in (Soviany et al., 2020).
Revised: To address this problem, we follow a sampling strategy similar to the one introduced in (Soviany et al., 2020).

Original: Based on the difﬁculty score, we sample according to a probability function, which favors easier samples in the ﬁrst iterations, but converges to give the same weight to all the examples in the later phases of the training.
Revised: Based on the difﬁculty score, we sample according to a probability function which favors easier samples in the ﬁrst iterations, but converges to give the same weight to all the examples in the later phases of the training.

Original: Still, the probability of sampling a harder example in the ﬁrst iterations is not null, and the more difﬁcult samples which are occasionally picked increase the diversity of the data and help training.
Revised: Still, the probability of sampling a harder example in the ﬁrst iterations is not null, and the more difﬁcult samples which are occasionally picked increase the diversity of the data and help training.

Original: The above-mentioned methodology should work well for balanced data sets, but it can fail when the data is unbalanced.
Revised: The above-mentioned methodology should work well for balanced data sets, but it can fail when the data is unbalanced.

Original: (Ionescu et al., 2016) show that some classes may be more difﬁcult than others.
Revised: Ionescu et al. (2016) show that some classes are more difﬁcult than others.

Original: This can make our sampling strategy neglect examples from certain classes and slow down training.
Revised: This can make our sampling strategy neglect examples from harder classes and slow down training.

Original: A more seriousproblem arises when the data is biased towards the easier classes.
Revised: The problem becomes more serious when the data is biased towards the easier classes, making the more difﬁcult categories inaccessible in the ﬁrst epochs.

Original: To solve this problem, we add a new term to our sampling function  which takes into consideration the classes of the elements already sampled, in order to give more importance to images from less-visited classes.
Revised: To solve this problem, we enhance our sampling function with a new term which takes into consideration the classes of the elements already sampled, in order to give more importance to images from less-visited classes.

Original: Since it is a sampling procedure, our CL approach can be applied to any supervised task in machine learning.
Revised: Since it is a sampling procedure, our CL approach can be applied to any supervised task in machine learning.

Original: In this paper, we focus on object detection and instance segmentation, two of the main tasks in computer vision, which require the model to identify the class and the location of objects in images.
Revised: In this paper, we focus on object detection and instance segmentation, two of the main tasks in computer vision, which require the model to identify the class and the location of objects in images.

Original: To test the validity of our approach, we experiment on two data sets: Pascal VOC 2007 (Everingham et al., a) and Cityscapes (Cordts et al., 2016), and compare our curriculum with diversity strategy against the standard random training method, a curriculum sampling (without diversity) procedure and an inverse-curriculum approach, which selects images from hard to easy.
Revised: To test the validity of our approach, we experiment on two data sets: Pascal VOC 2007 (Everingham et al., a) and Cityscapes (Cordts et al., 2016), and compare our curriculum with diversity strategy against the standard random training method, a curriculum sampling (without diversity) procedure and an inverse-curriculum approach, which selects images from hard to easy.

Original: We employ a stateof-the-art Faster R-CNN (Ren et al., 2015) detector with a Resnet-101 (He et al., 2016) backbone for the object detection experiments, and a Mask R-CNN (He et al., 2017) model based on Resnet-50 for instance segmentation.
Revised: We employ a stateof-the-art Faster R-CNN (Ren et al., 2015) detector with a Resnet-101 (He et al., 2016) backbone for the object detection experiments, and a Mask R-CNN (He et al., 2017) model based on Resnet-50 for instance segmentation.

Original: Our main contributions can be resumed as follows:
Revised: Our main contributions can be resumed as follows:

Original: We illustrate the necessity of adding diversity when using CL in unbalanced data sets; 2.
Revised: We illustrate the necessity of adding diversity when using CL in unbalanced data sets

Original: We introduce a novel curricular sampling function, which takes into consideration the class-diversity of the training samples and improves  results when traditional curriculum approaches fail; 3.
Revised: We introduce a novel curriculum sampling function, which takes into consideration the class-diversity of the training samples and improves the results when traditional curriculum approaches fail; 3.

Original: We prove our strategy by experimenting on two computer vision tasks: object detection and instance segmentation, using two data sets of high interest.
Revised: We prove our strategy by experimenting on two computer vision tasks: object detection and instance segmentation, using two data sets of high interest.

Original: We organize the rest of this paper as follows.
Revised: We organize the rest of this paper as follows.

Original: In Section 2, we present the most relevant related works and compare them with our approach.
Revised: In Section 2, we present the most relevant related works and compare them with our approach.

Original: In Section 3, we explain in detail the methodology we follow.
Revised: In Section 3, we explain in detail the methodology we follow.

Original: We present our results in Section 4, and draw our conclusion and discuss possible future work in the last section.
Revised: We present our results in Section 4, and draw our conclusion and discuss possible future work in the last section.

Original: 2. Related Work
Revised: 2. Related Work

Original: Curriculum learning.
Revised: Curriculum learning.

Original: (Bengio et al., 2009) introduced the idea of curriculum learning (CL) to train artiﬁcial intelligence, proving that the standard learning paradigm used in human educational systems could also be applied to automatic models.
Revised: Bengio et al. (2009) introduced the idea of curriculum learning (CL) to train artiﬁcial intelligence, proving that the standard learning paradigm used in human educational systems could also be applied to automatic models.

Original: CL represents a class of easy-to-hard approaches, which have successfully been employed in a wide range of machine learning applications, from natural language processing (Guo et al., 2019; Kocmi & Bojar, 2017; Liu et al., 2018;
Revised: CL represents a class of easy-to-hard approaches, which have successfully been employed in a wide range of machine learning applications, from natural language processing (Guo et al., 2019; Kocmi & Bojar, 2017; Liu et al., 2018;

Original: Platanios et al., 2019; Subramanian et al., 2017), to computer vision (Gong et al., 2016;
Revised: Platanios et al., 2019; Subramanian et al., 2017), to computer vision (Gong et al., 2016;

Original: Gui et al., 2017; Hacohen & Weinshall, 2019; Jiang et al., 2018; Li et al., 2017; Shi & Ferrari, 2016; Weinshall & Cohen, 2018), or audio processing (Amodei, 2016; Ranjan &
Revised: Gui et al., 2017; Hacohen & Weinshall, 2019; Jiang et al., 2018; Li et al., 2017; Shi & Ferrari, 2016; Weinshall & Cohen, 2018), or audio processing (Amodei, 2016; Ranjan &

Original: Hansen, 2017).
Revised: Hansen, 2017).

Original: One of the main limitations of CL is that it assumes the existence of a predeﬁned metric which can rank the samples from easy to hard.
Revised: One of the main limitations of CL is that it assumes the existence of a predeﬁned metric which can rank the samples from easy to hard.

Original: These metrics are usually task-dependent with various solutions being proposed for each.
Revised: These metrics are usually task-dependent with various solutions being proposed for each.

Original: For example, in text processing, the length of the sentence can be used to estimate the difﬁculty of the input (shorter sentences are easier)
Revised: For example, in text processing, the length of the sentence can be used to estimate the difﬁculty of the input (shorter sentences are easier)

Original: (Platanios et al., 2019; Spitkovsky et al., 2009), while the number and the size of objects in a certain sample can provide enough insights about difﬁculty in image processing tasks (images with few large objects are easier)
Revised: (Platanios et al., 2019; Spitkovsky et al., 2009), while the number and the size of objects in a certain sample can provide enough insights about difﬁculty in image processing tasks (images with few large objects are easier)

Original: (Shi & Ferrari, 2016; Soviany & Ionescu, 2018).
Revised: (Shi & Ferrari, 2016; Soviany & Ionescu, 2018).

Original: In our paper, we employ the image difﬁculty estimator of
Revised: In our paper, we employ the image difﬁculty estimator of

Original: Ionescu et al.
Revised: Ionescu et al.

Original: (Ionescu et al., 2016) which was trained considering the time required by human annotators to identify the presence of certain classes in images.
Revised: (2016) which was trained considering the time required by human annotators to identify the presence of certain classes in images.

Revised: A big advantage of this method is that it is not task-dependent, being suitable for a wide range of computer vision challenges.

Revised: Previous results (Soviany et al., 2020), as well as the output on our data set (Figure 1), prove it as a good choice for our experiments.

Original: To alleviate the challenge of ﬁnding a predeﬁned difﬁculty metric, Kumar et al.
Revised: To alleviate the challenge of ﬁnding a predeﬁned difﬁculty metric, Kumar et al.

Original: (Kumar et al., 2010) introduce Selfpaced learning (SPL), a set of approaches in which the model ranks the samples from easy to hard during training, based on its current status.
Revised: (Kumar et al., 2010) introduce Selfpaced learning (SPL), a set of approaches in which the model ranks the samples from easy to hard during training, based on its current progress.

Original: For example, the inputs with the smaller losses at a certain time during training are easier than the samples with higher losses.
Revised: For example, the inputs with the smaller losses at a certain time during training are easier than the samples with higher losses.

Original: Many papers apply SPL successfully (Sangineto et al., 2018; Supancic & Ramanan, 2013; Tang et al., 2012), and some methods combine prior knowledge with live training information, creating self-paced with curriculum techniques (Jiang et al., 2015; Zhang et al., 2019).
Revised: Many papers apply SPL successfully (Sangineto et al., 2018; Supancic & Ramanan, 2013; Tang et al., 2012), and some methods combine prior knowledge with live training information, creating self-paced with curriculum techniques (Jiang et al., 2015; Zhang et al., 2019).

Original: Even so, SPL still has some limitations, requiring a methodology on how to select the samples and on how much to emphasize easier examples.
Revised: Even so, SPL still has some limitations, requiring a methodology on how to select the samples and on how much to emphasize easier examples.

Original: Our approach is on the borderline between CL and SPL, but we consider it to be pure curricular, although we use training information to advantage classes visited less.
Revised: Our approach is on the borderline between CL and SPL, but we consider it to be pure curricular, although we use training information to advantage classes visited less.

Original: The reason behind our statement is that the class of the training samples is a priori information and a similar system could iteratively select examples from every class.
Revised: The reason behind our statement is that the class of the training samples is a priori information and a similar system could iteratively select examples from every class.

Original: Still, we prefer to use the class-diversity as a term in our difﬁculty-based sampling probability function, in order to not massively alter the actual class distribution of our data set.
Revised: Still, we prefer to use the class-diversity as a term in our difﬁculty-based sampling probability function, in order to not massively alter the actual class distribution of our data set.

Original: The easy-to-hard idea behind CL can be implemented in multiple ways.
Revised: The easy-to-hard idea behind CL can be implemented in multiple ways.

Original: One option is to start training on the easiest set of images, while gradually adding more difﬁcult batches (Bengio et al., 2009; Gui et al., 2017; Kocmi &
Revised: One option is to start training on the easiest set of images, while gradually adding more difﬁcult batches (Bengio et al., 2009; Gui et al., 2017; Kocmi &

Original: Bojar, 2017; Shi & Ferrari, 2016; Spitkovsky et al., 2009;
Revised: Bojar, 2017; Shi & Ferrari, 2016; Spitkovsky et al., 2009;

Original: Zhang et al., 2018).
Revised: Zhang et al., 2018).

Original: Although most of the models keep the visited examples in the training set, in (Kocmi & Bojar, 2017) the authors reduce the size of each bin until combining it with the following one, in order to use each example only once during an epoch.
Revised: Although most of the models keep the visited examples in the training set, in (Kocmi & Bojar, 2017) the authors reduce the size of each bin until combining it with the following one, in order to use each example only once during an epoch.

Original: In (Liu et al., 2018; Soviany et al., 2020) the authors propose a sampling strategy according to some probability function, which favors easier examples in the ﬁrst iterations.
Revised: In (Liu et al., 2018; Soviany et al., 2020) the authors propose a sampling strategy according to some probability function, which favors easier examples in the ﬁrst iterations.

Original: As the authors show, the easiness score from (Soviany et al., 2020) could also be added as a new term to the loss function to emphasize the easier examples in the beginning of the training.
Revised: As the authors show, the easiness score from (Soviany et al., 2020) could also be added as a new term to the loss function to emphasize the easier examples in the beginning of the training.

Original: In this paper, we follow their sampling strategy and use a similar probability function to select training examples.
Revised: In this paper, we follow their sampling strategy and use a similar probability function to select training examples.

Original: Despite leading to good results in many related papers, the standard CL procedure is highly inﬂuenced by the task and the data distribution.
Revised: Despite leading to good results in many related papers, the standard CL procedure is highly inﬂuenced by the task and the data distribution.

Original: Simple tasks may not gain much from using curriculum approaches, while employing CL in unbalanced data sets can lead to slower convergence.
Revised: Simple tasks may not gain much from using curriculum approaches, while employing CL in unbalanced data sets can lead to slower convergence.

Original: To address the second problem, (Jiang et al., 2014; Sachan & Xing, 2016) argue that a key element is diversity.
Revised: To address the second problem, Jiang et al. (2014) and Sachan and Xing (2016) argue that a key element is diversity.

Original: (Jiang et al., 2014) introduce a SPL with diversity technique in which they regularize the model using both difﬁculty information and the variety of the samples.
Revised: Jiang et al. (2014) introduce a SPL with diversity technique in which they regularize the model using both difﬁculty information and the variety of the samples.

Original: They suggest using clustering algorithms to split the data in diverse groups.
Revised: They suggest using clustering algorithms to split the data in diverse groups.

Original: (Sachan &
Revised: Sachan and

Original: Xing, 2016) measure diversity using the angle between the hyperplanes the samples induce in the feature space.
Revised: Xing (2016) measure diversity using the angle between the hyperplanes the samples induce in the feature space.

Original: They chose the examples that optimize a convex combination of the curriculum learning objective and the sum of angles between the candidate samples and the examples selected in previous steps.
Revised: They chose the examples that optimize a convex combination of the curriculum learning objective and the sum of angles between the candidate samples and the examples selected in previous steps.

Original: In our model, we deﬁne diversity based on the classes of our data.
Revised: In our model, we deﬁne diversity based on the classes of our data.

Original: We combine our predeﬁned difﬁculty metric with a score which favors images from classes visited less, in order to sample easy and diverse examples at the beginning of the training, then gradually add more complex elements.
Revised: We combine our predeﬁned difﬁculty metric with a score which favors images from classes visited less, in order to sample easy and diverse examples at the beginning of the training, then gradually add more complex elements.

Original: Our idea works well for supervised tasks, but it can be extended to unsupervised learning by replacing the ground-truth labels with a clustering model, like suggested in (Jiang et al., 2014).
Revised: Our idea works well for supervised tasks, but it can be extended to unsupervised learning by replacing the ground-truth labels with a clustering model, like suggested in (Jiang et al., 2014).

Original: Object detection is the task of predicting the location and the class of objects in certain images.
Revised: Object detection is the task of predicting the location and the class of objects in certain images.

Original: As noted in (Soviany & Ionescu, 2018), the state-of-the-art object detectors can be split in two main categories: two-stage and single stage models.
Revised: As noted in (Soviany & Ionescu, 2018), the state-of-the-art object detectors can be split in two main categories: two-stage and single stage models.

Original: The two-stage object detectors (He et al., 2017; Ren et al., 2015) use a Region Proposal Network to generate regions of interest which are then fed to another network for object localization and classiﬁcation.
Revised: The two-stage object detectors (He et al., 2017; Ren et al., 2015) use a Region Proposal Network to generate regions of interest which are then fed to another network for object localization and classiﬁcation.

Original: The single stage approaches (Liu et al., 2016; Redmon et al., 2016) take the whole image as input and solve the problem like a regular regression task.
Revised: The single stage approaches (Liu et al., 2016; Redmon et al., 2016) take the whole image as input and solve the problem like a regular regression task.

Original: These methods are usually faster, but less accurate than the two-stage designs.
Revised: These methods are usually faster, but less accurate than the two-stage designs.

Original: Instance segmentation is similar to object detection, but more complex, requiring to generate a mask instead of a bounding box for the objects in the test image.
Revised: Instance segmentation is similar to object detection, but more complex, requiring to generate a mask instead of a bounding box for the objects in the test image.

Original: Our strategy can be applied on  any detection and segmentation model, but, in order to increase the relevance of our results, we experiment with high quality Faster R-CNN (Ren et al., 2015) andMask
Revised: Our strategy can be applied on top of any detection and segmentation model, but, in order to increase the relevance of our results, we experiment with high quality Faster R-CNN (Ren et al., 2015) and

Original:  R-CNN (He et al., 2017) baselines.
Revised: Mask R-CNN (He et al., 2017) baselines.

Original: 3. Methodology
Revised: 3. Methodology

Original: Training artiﬁcial intelligence using curriculum approaches, from easy to hard, can lead to improved results in a wide range of tasks (Amodei, 2016;
Revised: Training artiﬁcial intelligence using curriculum approaches, from easy to hard, can lead to improved results in a wide range of tasks (Amodei, 2016;

Original: Gong et al., 2016; Gui et al., 2017; Guo et al., 2019; Hacohen & Weinshall, 2019; Jiang et al., 2018; Kocmi & Bojar, 2017;
Revised: Gong et al., 2016; Gui et al., 2017; Guo et al., 2019; Hacohen & Weinshall, 2019; Jiang et al., 2018; Kocmi & Bojar, 2017;

Original: Li et al., 2017; Liu et al., 2018;
Revised: Li et al., 2017; Liu et al., 2018;

Original: Platanios et al., 2019; Ranjan & Hansen, 2017; Shi &
Revised: Platanios et al., 2019; Ranjan & Hansen, 2017; Shi &

Original: Ferrari, 2016; Subramanian et al., 2017; Weinshall & Cohen, 2018).
Revised: Ferrari, 2016; Subramanian et al., 2017; Weinshall & Cohen, 2018).

Original: Still, it is not simple to determine which samples are easy or hard, the available metrics usually being taskdependent.
Revised: Still, it is not simple to determine which samples are easy or hard because many of the available metrics are task-dependent.

Original: Another challenge of CL is ﬁnding the right curricular schedule, i.e. how fast to add more difﬁcult examples to training, as well as introducing the right quantity of harder samples at the right time to positively inﬂuence convergence.
Revised: Another challenge of CL is ﬁnding the right curricular schedule, i.e. how fast to add more difﬁcult examples to training, as well as introducing the right quantity of harder samples at the right time to positively inﬂuence convergence.

Original: In this section, we present our approach for estimating difﬁculty and our curriculum sampling strategy which we enhance by taking into consideration the diversity of the examples.
Revised: In this section, we present our approach for estimating difﬁculty and our curriculum sampling strategy which we enhance by taking into consideration the diversity of the examples.

Original: 3.1. Difﬁculty estimation
Revised: 3.1. Difﬁculty estimation

Original: In (Ionescu et al., 2016), the authors deﬁned image difﬁculty as the required human time for solving a visual search task.
Revised: In (Ionescu et al., 2016), the authors deﬁned image difﬁculty as the required human time for solving a visual search task.

Original: They collected annotations for the Pascal VOC 2012 (Everingham et al.
Revised: They collected annotations for the Pascal VOC 2012 (Everingham et al.

Original: , b) data set, which they normalized and fed as training data for their regression model.
Revised: , b) data set, which they normalized and fed as training data for their regression model.

Original: We follow their strategy as described in the original paper (Ionescu et al., 2016) to determine the difﬁculty scores of the images in our data sets.
Revised: We follow their strategy as described in the original paper (Ionescu et al., 2016) to determine the difﬁculty scores of the images in our data sets.

Original: These scores have values ≈ 3 , with a larger score deﬁning a more difﬁcult sample.
Revised: These scores have values ≈ 3 , with a larger score deﬁning a more difﬁcult sample.

Original: We translate the values between [ − 1 , 1] using Equation 1 to simplify the usage of the score in the next steps.Scale min [Equation]
Revised: We translate the values between [ − 1 , 1] using Equation 1 to simplify the usage of the score in the next steps.

Original: max

Original: [Equation]
Revised: [Equation]

Original: min

Original: [Equation]

Original: [Equation]

Original: 3.2. Curriculum sampling
Revised: 3.2. Curriculum sampling

Original: (Soviany et al., 2020) introduces a curriculum sampling strategy, which favors easier examples in the ﬁrst iterations and converges as the training progresses.
Revised: Soviany et al. (2020) introduce a curriculum sampling strategy, which favors easier examples in the ﬁrst iterations and converges as the training progresses.

Original: It has the advantage of being a continuous method, removing the necessity of a curricular schedule for enhancing the difﬁculty-based batches.
Revised: It has the advantage of being a continuous method, removing the necessity of a curricular schedule for enhancing the difﬁculty-based batches.

Original: Furthermore, the fact that it is a probabilistic sampling method does not constrain the model to only select easy examples in the ﬁrst iterations, as batching does, but adds more diversity in data selection.
Revised: Furthermore, the fact that it is a probabilistic sampling method does not constrain the model to only select easy examples in the ﬁrst iterations, as batching does, but adds more diversity in data selection.

Original: We follow their approach in building our curriculum sampling strategy with only a small change to better emphasize the difﬁculty.
Revised: We follow their approach in building our curriculum sampling strategy with only a small change to better emphasize the difﬁculty.

Original: We use the following function to assign weights to the input images during training:
Revised: We use the following function to assign weights to the input images during training:

Original: [Equation]
Revised: [Equation]

Original: where x i is the training example from the data set X, t is the current iteration, and diff ( x i ) is the difﬁculty score associated with the selected sample.
Revised: where x i is the training example from the data set X, t is the current iteration, and diff ( x i ) is the difﬁculty score associated with the selected sample.

Original: γ is a parameter which sets how fast the function converges to 1 , while k sets how much to emphasize the easier examples.
Revised: γ is a parameter which sets how fast the function converges to 1 , while k sets how much to emphasize the easier examples.

Original: Our function varies from the one proposed in (Soviany et al., 2020) by changing the position of the k parameter.
Revised: Our function varies from the one proposed in (Soviany et al., 2020) by changing the position of the k parameter.

Original: We consider that we can take advantage of the properties of the power function which increases faster for numbers greater than the unit.
Revised: We consider that we can take advantage of the properties of the power function which increases faster for numbers greater than the unit.

Original: Since 1 − s i · e − γ · t ∈ [0 , 2] , and the result is > 1 for easier examples, our function will focus more on the easier samples in the ﬁrst iterations.
Revised: Since 1 − s i · e − γ · t ∈ [0 , 2] , and the result is > 1 for easier examples, our function will focus more on the easier samples in the ﬁrst iterations.

Original: We transform the weights into probabilities and we sample according to them.
Revised: We transform the weights into probabilities and we sample according to them.

Original: 3.3. Curriculum with diversity sampling
Revised: 3.3. Curriculum with diversity sampling

Original: As (Jiang et al., 2014; Sachan & Xing, 2016) note, applying a CL strategy does not guarantee improved quality, the diversity of the selected samples having a great impact on the ﬁnal results.
Revised: As (Jiang et al., 2014; Sachan & Xing, 2016) note, applying a CL strategy does not guarantee improved quality, the diversity of the selected samples having a great impact on the ﬁnal results.

Original: A simple example is the case in which the data set is biased, having fewer samples of certain classes.
Revised: A simple example is the case in which the data set is biased, having fewer samples of certain classes.

Original: Since some classes are more difﬁcult than others (Ionescu et al., 2016), if the data set is not well-balanced, the model will not visit the harder classes until the later stages of the training.
Revised: Since some classes are more difﬁcult than others, as noted in (Ionescu et al., 2016) and illustrated in Figure 2, if the data set is not well-balanced, the model will not visit the harder classes until the later stages of the training.

Original: This is the reason why we enhance our sampling method, by adding a new term, which is based on the diversity of the examples.
Revised: This is the reason why we enhance our sampling method, by adding a new term, which is based on the diversity of the examples.

Original: Our diversity scoring algorithm is simple, taking into consideration the classes of the selected samples.
Revised: Our diversity scoring algorithm is simple, taking into consideration the classes of the selected samples.

Original: During training, we count the number of visited objects from each class ( num objects ( c ) ).
Revised: During training, we count the number of visited objects from each class ( num objects ( c ) ).

Original: We subtract the mean the values to determine how often was each class visited.
Revised: We subtract the mean the values to determine how often was each class visited.

Original: This is formally presented in Equation 3.
Revised: This is formally presented in Equation 3.

Original: We scale and translate the results between [ − 1 , 1] using Equation 1 to get the score of each class, then, for every image, we compute the image-level diversity by averaging the class score for each object in its ground-truth labels  (Equation 4). 
Revised: We scale and translate the results between [ − 1 , 1] using Equation 1 to get the score of each class, then, for every image, we compute the image-level diversity by averaging the class score for each object in its ground-truth labels in order to make our algorithm work in multi-class scenarios (Equation 4). visited ( c i ) = num objects ( c [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Revised: imgV isited

Revised: [Equation]

Revised: objects

Revised: [Equation]

Revised: x

Original:  In our diversity algorithm we want to emphasize the images with objects from classes visited less, i.e. with a small imgV isited value, closer to − 1 .
Revised: [Equation] visited ( [Equation] In our diversity algorithm we want to emphasize the images with objects from classes visited less, i.e. with a small imgV isited value, closer to − 1 .

Original: We compute a scoring function similar to Equation 2, which also takes into consideration how often a class was visited, in order to add diversity:
Revised: We compute a scoring function similar to Equation 2, which also takes into consideration how often a class was visited, in order to add diversity:

Original: [Equation]
Revised: [Equation]

Original: where α controls the impact of each component, the difﬁculty and the diversity, while the rest of the notation follows
Revised: where α controls the impact of each component, the difﬁculty and the diversity, while the rest of the notation follows

Original: Equation 2.
Revised: Equation 2.

Original: We transform the weights into probabilities by dividing them by their sum, and we sample according to them.
Revised: We transform the weights into probabilities by dividing them by their sum, and we sample according to them.

Original: 4. Experiments 4.1. Data sets
Revised: 4. Experiments 4.1. Data sets

Original: In order to test the validity of our method, we experiment on two data sets: Pascal VOC 2007 (Everingham et al., a) and Cityscapes (Cordts et al., 2016).
Revised: In order to test the validity of our method, we experiment on two data sets: Pascal VOC 2007 (Everingham et al., a) and Cityscapes (Cordts et al., 2016).

Original: We conduct detection experiments on 20 classes, training on the 5011 images from the Pascal VOC 2007 trainval split.
Revised: We conduct detection experiments on 20 classes (aeroplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, diningtable, dog, horse, motorbike, person, pottedplant, sheep, sofa, train, tvmonitor), training on the 5011 images from the Pascal VOCtrainval split.

Original: We perform evaluation on the test split which contains 4952 images.
Revised: We perform evaluation on the test split which contains 4952 images.

Original: For our instance segmentation experiments, we use the Cityscapes data set which contains eight labeled object classes.
Revised: For our instance segmentation experiments, we use the Cityscapes data set which contains eight labeled object classes: person, rider, car, truck, bus, train, motorcycle, bicycle.

Original: We train on the training set of 2975 images and we evaluate on the validation split of 500 images.
Revised: We train on the training set of 2975 images and we evaluate on the validation split ofimages.

Original: 4.2. Baselines and conﬁguration
Revised: 4.2. Baselines and conﬁguration

Original: We build our method on top of the Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) implementations available at: https://github.com/facebookresearch/maskrcnnbenchmark.
Revised: We build our method on top of the Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) implementations available at: https://github.com/facebookresearch/maskrcnnbenchmark.

Original: For our detection experiments, we use Faster
Revised: For our detection experiments, we use Faster

Original: R-CNN with Resnet-101 (He et al., 2016) backbone, while for segmentation we employ the Resnet-50 backbone on the Mask R-CNN model.
Revised: R-CNN with Resnet-101 (He et al., 2016) backbone, while for segmentation we employ the Resnet-50 backbone on the Mask R-CNN model.

Original: We use the conﬁgurations available on the web site, with the learning rate adjusted for a training with a batch size of 4.
Revised: We use the conﬁgurations available on the web site, with the learning rate adjusted for a training with a batch size of 4.

Original: In our sampling procedure (Equation 5) we set α = 0 .
Revised: In our sampling procedure (Equation 5) we set α = 0 .

Original: 5 , γ = 6 · 10 − 5 , and k = 5 .
Revised: 5 , γ = 6 · 10 − 5 , and k = 5 .

Original: We do not compare with other models, because the goal of our paper is not surpassing the state of the art, but improving the quality of our baseline model.
Revised: We do not compare with other models, because the goal of our paper is not surpassing the state of the art, but improving the quality of our baseline model.

Original: We also present the results of a hard-to-easy sampling, in order to prove the efﬁciency of the curriculum approach.
Revised: We also present the results of a hard-to-easy sampling, in order to prove the efﬁciency of the curriculum approach.

Original: 4.3. Evaluation metrics
Revised: 4.3. Evaluation metrics

Original: We evaluate our results using the mean Average Precision (AP).
Revised: We evaluate our results using the mean Average Precision (AP).

Original: The AP score is given by the area under the precision-recall curve for the detected objects.
Revised: The AP score is given by the area under the precision-recall curve for the detected objects.

Original: The exact evaluation protocol has some differences for each data set, thus we use the Pascal VOC 

Revised: The exact evaluation protocol has some differences for each data set, thus we use the Pascal VOC

Original: (Everingham et al., a) metric for the detection experiments and the Cityscapes (Cordts et al., 2016) metric for the instance segmentation results.
Revised: (Everingham et al., a) metric for the detection experiments and the Cityscapes (Cordts et al., 2016) metric for the instance segmentation results.

Original: We use the evaluation code available at https://github.com/facebookresearch/maskrcnnbenchmark.
Revised: We use the evaluation code available at https://github.com/facebookresearch/maskrcnnbenchmark.

Original: More details about the evaluation metrics can be found in the original papers (Cordts et al., 2016; Everingham et al., a).
Revised: More details about the evaluation metrics can be found in the original papers (Cordts et al., 2016; Everingham et al., a).

Original: 4.4. Results and discussion
Revised: 4.4. Results and discussion

Original: The class distribution of the objects in Pascal VOCclearly favors class person , with 4690 instances, while classes dinningtable and bus only contain 215 and 229 instances, respectively.
Revised: The class distribution of the objects in Pascal VOCclearly favors class person , with 4690 instances, while classes dinningtable and bus only contain 215 andinstances, respectively (Figure 5).

Original: This would not be a problem if the difﬁculty of the classes was similar, because we can assume the test data set has a matching distribution, but this is not the case.
Revised: This would not be a problem if the difﬁculty of the classes was similar, because we can assume the test data set has a matching distribution, but this is not the case, as shown in Figure 2.

Original: Figure 1 presents how the two sampling methods behave during training on the Pascal VOC 2007 data set.
Revised: Figure 3 presents how the two sampling methods behave during training on the Pascal VOC 2007 data set.

Original: In the ﬁrst 10k iterations, curriculum sampling selects images with almost 20k objects from class person and only 283 instances from class diningtable .
Revised: In the ﬁrst 10k iterations, curriculum sampling selects images with almost 20k objects from class person and only 283 instances from class diningtable .

Original: By adding diversity, we lower the gap between classes, reaching 10k objects of persons and 1000 instances of tables.
Revised: By adding diversity, we lower the gap between classes, reaching 10k objects of persons and 1000 instances of tables.

Original: This behaviour continues as the training progresses, with the differences between classes being smaller when adding diversity.
Revised: This behaviour continues as the training progresses, with the differences between classes being smaller when adding diversity.

Original: It is important to note that we do not want to sample the exact number of objects from each class, but to keep the class distribution of the actual data set, while feeding the model with enough details about every class.
Revised: It is important to note that we do not want to sample the exact number of objects from each class, but to keep the class distribution of the actual data set, while feeding the model with enough details about every class.

Original: Figure 2 shows the difﬁculty of the examples sampled according to our strategies.
Revised: Figure 4 shows the difﬁculty of the examples sampled according to our strategies.

Original: We observe that by adding diversity we do not break our curriculum learning schedule, the examples still being selected from easy to hard.
Revised: We observe that by adding diversity we do not break our curriculum learning schedule, the examples still being selected from easy to hard.

Original: To further prove the efﬁciency of our method, we compute the AP on both object detection and instance segmentation tasks.
Revised: To further prove the efﬁciency of our method [Table] we compute the AP on both object detection and instance segmentation tasks.

Original: The results are presented in Tables 1 and 2.
Revised: The results are presented in Tables 1 and 2.

Original: In our object detection experiments, the  sampling with diversity approach provides an improvement of 0 .
Revised: We repeat our object detection experiments ﬁve times and average the results, in order to ensure their relevance. The sampling with diversity approach provides an improvement of [Table] [Table]

Original: 52% over the standard curriculum method, and of 0 .
Revised: 69% over the standard curriculum method, and of 0 .

Original: 76% over the randomly-trained baseline.
Revised: 79% over the randomly-trained baseline.

Original: Our experiments, with an inverse curriculum approach, from hard to easy, lead to the worst results, showing the utility of CL.
Revised: Our experiments, with an inverse curriculum approach, from hard to easy, lead to the worst results, showing the utility of CL.

Original: Moreover, Figureillustrates the evolution of the AP during training.
Revised: Moreover, Figure 6 illustrates the evolution of the AP during training.

Original: The curriculum with diversity approach has superior results over the baseline from the beginning to the end of the training.
Revised: The curriculum with diversity approach has superior results over the baseline from the beginning to the end of the training.

Original: The standard CL method, on the other hand, starts from lower scores, exactly because it does not visit enough samples from more difﬁcult classes in the early stages of the training.[Table] [Table]
Revised: The standard CL method, on the other hand, starts from lower scores, exactly because it does not visit enough samples from more difﬁcult classes in the early stages of the training.

Original: The instance segmentation results on the Cityscapes data set conﬁrm the conclusion from our previous experiments
Revised: The instance segmentation results on the Cityscapes data set conﬁrm the conclusion from our previous experiments.

Original: [Table] As Table 2 shows [Table] the curriculum with diversity is again the optimal method, surpassing the baseline with 0 .
Revised: As Table 2 shows, the curriculum with diversity is again the optimal method, surpassing the baseline with 0 .

Original: 4% using
Revised: 4% using

Original: AP, 0 .
Revised: AP, 0 .

Original: 71% using AP50% [Table] and 0 .
Revised: 71% using AP50%, and 0 .

Original: 45% using AP75%.
Revised: 45% using AP75%.

Original: It is interesting to point that, although the diverse curriculum approach has a better AP and AP75% than the standard CL method, the former technique surpasses our method with 
Revised: It is interesting to point that, although the diverse curriculum approach has a better AP and AP75% than the standard CL method, the former technique surpasses our method with 0 .

Original: 02% when evaluated using AP50%.
Revised: 02% when evaluated using AP50%.

Original: The inverse curriculum approach has the worst scores again, strengthening our statements on the utility of curriculum learning.
Revised: The inverse curriculum approach has the worst scores again, strengthening our statements on the utility of curriculum learning.

Original: 5. Conclusion and future work
Revised: 5. Conclusion and future work

Original: In this paper, we presented a simple method of optimizing the curriculum learning approaches on unbalanced data sets.
Revised: In this paper, we presented a simple method of optimizing the curriculum learning approaches on unbalanced data sets.

Original: We consider that the diversity of the selected examples is just as important as their difﬁculty, and neglecting this fact may slow down training for more difﬁcult classes.
Revised: We consider that the diversity of the selected examples is just as important as their difﬁculty, and neglecting this fact may slow down training for more difﬁcult classes.

Original: We introduced a novel sampling function, which uses the classes of the visited examples together with a difﬁculty score to ensure the curriculum schedule and the diversity of the selection.
Revised: We introduced a novel sampling function, which uses the classes of the visited examples together with a difﬁculty score to ensure the curriculum schedule and the diversity of the selection.

Original: Our object detection and instance segmentation experiments conducted on two data sets of high interest prove the superiority of our method over the randomly-trained baseline and over the standard CL approach.
Revised: Our object detection and instance segmentation experiments conducted on two data sets of high interest prove the superiority of our method over the randomly-trained baseline and over the standard CL approach.

Original: A beneﬁt of our methodology is that it can be used on top of any deep learning model, for any supervised task.
Revised: A beneﬁt of our methodology is that it can be used on top of any deep learning model, for any supervised task.

Revised: For the future work, we plan on studying more difﬁculty measures to build an extensive view on how the chosen metric affects the performance of our system.

Revised: Furthermore, we aim to create an ablation study on the parameter choice and ﬁnd better ways to detect the right parameter values.

Original: For the future work, we plan on extending the framework to unsupervised tasks, by introducing a novel method of computing the diversity of the examples.
Revised: Another important aspect we are considering is extending the framework to unsupervised tasks, by introducing a novel method of computing the diversity of the examples.