Original: D EEP R EPULSIVE C LUSTERING OF O RDERED D ATA B ASED ON O RDER -I DENTITY D ECOMPOSITION
Revised: D EEP R EPULSIVE C LUSTERING OF O RDERED D ATA B ASED ON O RDER -I DENTITY D ECOMPOSITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose the deep repulsive clustering (DRC) algorithm of ordered data for effective order learning.
Revised: We propose the deep repulsive clustering (DRC) algorithm of ordered data for effective order learning.

Original: First, we develop the order-identity decomposition (ORID) network to divide the information of an object instance into an order-related feature and an identity feature.
Revised: First, we develop the order-identity decomposition (ORID) network to divide the information of an object instance into an order-related feature and an identity feature.

Original: Then, we group object instances into clusters according to their identity features using a repulsive term.
Revised: Then, we group object instances into clusters according to their identity features using a repulsive term.

Original: Moreover, we estimate the rank of a test instance, by comparing it with references within the same cluster.
Revised: Moreover, we estimate the rank of a test instance, by comparing it with references within the same cluster.

Original: Experimental results on facial age estimation, aesthetic score regression and historical color image classiﬁcation show that the proposed algorithm can cluster ordered data effectively and also yield excellent rank estimation performance
Revised: Experimental results on facial age estimation, aesthetic score regression and historical color image classiﬁcation show that the proposed algorithm can cluster ordered data effectively and also yield excellent rank estimation performance

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There are various types of ‘ordered’ data.
Revised: There are various types of ‘ordered’ data.

Original: For instance, in facial age estimation (Ricanek & Tesafaye, 2006), face photos are ranked according to the ages.
Revised: For instance, in facial age estimation (Ricanek & Tesafaye, 2006), face photos are ranked according to the ages.

Original: Also, in a video-sharing platform, videos can be sorted according to the numbers of views or likes.
Revised: Also, in a video-sharing platform, videos can be sorted according to the numbers of views or likes.

Original: In these ordered data, classes, representing ranks or preferences, form an ordered set (Schr ¨ oder, 2003).
Revised: In these ordered data, classes, representing ranks or preferences, form an ordered set (Schr ¨ oder, 2003).

Original: Attempts have been made to estimate the classes of objects, including multi-class classiﬁcation (Pan et al., 2018), ordinal regression (Frank &
Revised: Attempts have been made to estimate the classes of objects, including multi-class classiﬁcation (Pan et al., 2018), ordinal regression (Frank &

Original: Hall, 2001), metric regression (Fu & Huang, 2008).
Revised: Hall, 2001), metric regression (Fu & Huang, 2008).

Original: Recently, a new approach, called order learning (Lim et al., 2020), was proposed to solve this problem.
Revised: Recently, a new approach, called order learning (Lim et al., 2020), was proposed to solve this problem.

Original: Order learning is based on the idea that it is easier to predict ordering relationship between objects than to estimate the absolute classes (or ranks); telling the older one between two people is easier than estimating their exact ages.
Revised: Order learning is based on the idea that it is easier to predict ordering relationship between objects than to estimate the absolute classes (or ranks); telling the older one between two people is easier than estimating their exact ages.

Original: Hence, in order learning, the pairwise ordering relationship is learned from training data.
Revised: Hence, in order learning, the pairwise ordering relationship is learned from training data.

Original: Then, the rank of a test object is estimated by comparing it with reference objects with known ranks.
Revised: Then, the rank of a test object is estimated by comparing it with reference objects with known ranks.

Original: However, some objects cannot be easily compared.
Revised: However, some objects cannot be easily compared.

Original: It is less easy to tell the older one between people of different genders than between those of the same gender.
Revised: It is less easy to tell the older one between people of different genders than between those of the same gender.

Original: Lim et al.
Revised: Lim et al.

Original: (2020) tried to deal with this issue, by dividing an ordered dataset into disjoint chains.
Revised: (2020) tried to deal with this issue, by dividing an ordered dataset into disjoint chains.

Original: But, the chains were not clearly separated, and no meaningful properties were discovered from the chains.
Revised: But, the chains were not clearly separated, and no meaningful properties were discovered from the chains.

Original: In this paper, we propose a reliable clustering algorithm, called deep repulsive clustering (DRC), of ordered data based on order-identity decomposition (ORID).
Revised: In this paper, we propose a reliable clustering algorithm, called deep repulsive clustering (DRC), of ordered data based on order-identity decomposition (ORID).

Original: Figure 1 shows a clustering example of ordered data.
Revised: Figure 1 shows a clustering example of ordered data.

Original: Note that some characteristics of objects, such as genders or races in age estimation, are not related to their ranks, and the ranks of objects sharing such characteristics can be compared more reliably.
Revised: Note that some characteristics of objects, such as genders or races in age estimation, are not related to their ranks, and the ranks of objects sharing such characteristics can be compared more reliably.

Original: To discover such characteristics without any supervision, the proposed ORID network decomposes the information of an object instance into an order-related feature and an identity feature unrelated to the rank.
Revised: To discover such characteristics without any supervision, the proposed ORID network decomposes the information of an object instance into an order-related feature and an identity feature unrelated to the rank.

Original: Then, the proposed DRC clusters object instances using their identity features; in each cluster, the instances share similar identity features.
Revised: Then, the proposed DRC clusters object instances using their identity features; in each cluster, the instances share similar identity features.

Original: Furthermore, given a test instance, we decide its cluster based on the nearest neighbor  rule, and compare it with reference instances within the cluster to estimate its rank.
Revised: Furthermore, given a test instance, we decide its cluster based on the nearest neighbor (NN) rule, and compare it with reference instances within the cluster to estimate its rank.

Original: To this end, we develop a maximum a posteriori (MAP) estimation rule.
Revised: To this end, we develop a maximum a posteriori (MAP) estimation rule.

Original: Experimental results on ordered data for facial age estimation, aesthetic score regression (Kong et al., 2016), and historical color image classiﬁcation (Palermo et al., 2012) demonstrate that the proposed algorithm separates ordered data clearly into meaningful clusters and provides excellent rank estimation performances for unseen test instances.
Revised: Experimental results on ordered data for facial age estimation, aesthetic score regression (Kong et al., 2016), and historical color image classiﬁcation (Palermo et al., 2012) demonstrate that the proposed algorithm separates ordered data clearly into meaningful clusters and provides excellent rank estimation performances for unseen test instances.

Original: The contributions of this paper can be summarized as follows.
Revised: The contributions of this paper can be summarized as follows.

Original: • We ﬁrst propose the notion of identity features of ordered data and develop the ORID network for the order-identity decomposition.
Revised: • We ﬁrst propose the notion of identity features of ordered data and develop the ORID network for the order-identity decomposition.

Original: Cluster Cluster
Revised: Cluster Cluster

Original: Ordered data    

Revised: Ordered data

Original: Clustering
Revised: Clustering

Original: • We develop the DRC algorithm to cluster data on a unit sphere effectively using a repulsive term.
Revised: • We develop the DRC algorithm to cluster data on a unit sphere effectively using a repulsive term.

Original: We also prove the local optimality of the solution.
Revised: We also prove the local optimality of the solution.

Original: • We propose the MAP decision rule for rank estimation.
Revised: • We propose the MAP decision rule for rank estimation.

Original: The proposed algorithm provides the state-of-the-art performances for facial age estimation and aesthetic score regression.
Revised: The proposed algorithm provides the state-of-the-art performances for facial age estimation and aesthetic score regression.

Original:  R ELATED W ORK 2. O RDER L EARNING
Revised:  R ELATED W ORK 2. O RDER L EARNING

Original: The notion of order learning was ﬁrst proposed by Lim et al.
Revised: The notion of order learning was ﬁrst proposed by Lim et al.

Original: It aims to determine the order graph of classes and classify an object into one of the classes.
Revised: It aims to determine the order graph of classes and classify an object into one of the classes.

Original: In practice, it trains a pairwise comparator, which is a ternary classiﬁer, to categorize the relationship between two objects into one of three cases: one object is bigger than, similar to, or smaller than the other.
Revised: In practice, it trains a pairwise comparator, which is a ternary classiﬁer, to categorize the relationship between two objects into one of three cases: one object is bigger than, similar to, or smaller than the other.

Original: Then, it estimates the rank of a test object, by comparing it with reference objects with known ranks.
Revised: Then, it estimates the rank of a test object, by comparing it with reference objects with known ranks.

Original: However, not every pair of objects are comparable.
Revised: However, not every pair of objects can be easily compared.

Original: Although Lim et al.
Revised: Although Lim et al.

Original: (2020) attempted to group objects into clusters, in which objects could be more accurately compared, their clustering results were unreliable.
Revised: (2020) attempted to group objects into clusters, in which objects could be more accurately compared, their clustering results were unreliable.

Original: Pairwise comparison has been used to estimate object ranks, because relative evaluation is easier than absolute evaluation in general.
Revised: Pairwise comparison has been used to estimate object ranks, because relative evaluation is easier than absolute evaluation in general.

Original: Saaty (1977) proposed the scaling method to estimate absolute priorities from relative priorities, which has been applied to various decision processes, including aesthetic score regression (Lee & Kim, 2019).
Revised: Saaty (1977) proposed the scaling method to estimate absolute priorities from relative priorities, which has been applied to various decision processes, including aesthetic score regression (Lee & Kim, 2019).

Original: Also, some learning to rank (LTR) algorithms are based on pairwise comparison (Liu, 2009;  Burges et al., 2005; Tsai et al., 2007).
Revised: Also, some learning to rank (LTR) algorithms are based on pairwise comparison (Liu, 2009; Cohen et al., 1998; Burges et al., 2005; Tsai et al., 2007).

Revised: Order learning attempts to combine (possibly inconsistent) pairwise ordering results to determine the rank of each object.

Original: 2. C LUSTERING
Revised: Thus, it is closely related to the Cohen et al.’s LTR algorithm (1998), which learns a pairwise preference function and obtains a total order of a set to maximize agreements among preference judgments of pairs of elements. Also, order learning is related to rank aggregation (Dwork et al., 2001), in which partially ordered sets are combined into a linearly ordered set to achieve the maximum consensus among those partial sets. Rank aggregation has been studied in various ﬁelds (Br¨uggemann et al., 2004).

Revised: Since optimal aggregation is NP-hard, Dwork et al.

Revised: (2001) proposed an approximate algorithm, called Markov chain ordering.

Revised: There are many other approximate schemes, such as the local Kemenization, Borda count, and scaled footrule aggregation.

Original: Data clustering is a fundamental problem to partition data into disjoint groups, such that elements in the same group are similar to one another but elements from different groups are dissimilar.
Revised: Data clustering is a fundamental problem to partition data into disjoint groups, such that elements in the same group are similar to one another but elements from different groups are dissimilar.

Original: Although various clustering algorithms have been proposed (Hartigan & Wong, 1979; Ester et al., 1996; Kohonen, 1990; Dhillon & Modha, 2001; Reynolds, 2009), conventional algorithms often yield poor performance on high-dimensional data due to the curse of dimensionality and ineffectiveness of similarity metrics.
Revised: Although various clustering algorithms have been proposed (Hartigan & Wong, 1979; Ester et al., 1996; Kohonen, 1990; Dhillon & Modha, 2001; Reynolds, 2009), conventional algorithms often yield poor performance on high-dimensional data due to the curse of dimensionality and ineffectiveness of similarity metrics.

Original: Dimensionality reduction and feature transform methods have been studied to map raw data into a new feature space, in which they are more easily separated.
Revised: Dimensionality reduction and feature transform methods have been studied to map raw data into a new feature space, in which they are more easily separated.

Original: Linear transforms, such as PCA (Wold et al., 1987), and non-linear transformations, including kernel methods (Hofmann et al., 2008) and spectral clustering (Ng et al., 2002), have been proposed.
Revised: Linear transforms, such as PCA (Wold et al., 1987), and non-linear transformations, including kernel methods (Hofmann et al., 2008) and spectral clustering (Ng et al., 2002), have been proposed.

Original: Recently, deep neural networks have been adopted effectively as feature embedding functions (LeCun et al., 2015), and these deep-learning-based feature embedding functions have been combined with classical clustering algorithms.
Revised: Recently, deep neural networks have been adopted effectively as feature embedding functions (LeCun et al., 2015), and these deep-learning-based feature embedding functions have been combined with classical clustering algorithms.

Original: For instance, Caron et al.
Revised: For instance, Caron et al.

Original: (2018) proposed a deep clustering algorithm based on k -means.
Revised: (2018) proposed a deep clustering algorithm based on k -means.

Original: It clusters features from a neural network and then trains the network using the cluster assignments as pseudo-labels.
Revised: It clusters features from a neural network and then trains the network using the cluster assignments as pseudo-labels.

Original: This is done iteratively.
Revised: This is done iteratively.

Original: Also, Yang et al.
Revised: Also, Yang et al.

Original: (2016) jointly learned feature representations and clustered images, based on agglomerative clustering.
Revised: (2016) jointly learned feature representations and clustered images, based on agglomerative clustering.

Original: Chang et al.
Revised: Chang et al.

Original: (2017) recast the image clustering task into a binary classiﬁcation problem to predict whether a pair of images belong to the same cluster or different clusters.
Revised: (2017) recast the image clustering task into a binary classiﬁcation problem to predict whether a pair of images belong to the same cluster or different clusters.

Original: Similarly to these algorithms, we use a neural network to determine a feature space in which clustering is done more effectively.
Revised: Similarly to these algorithms, we use a neural network to determine a feature space in which clustering is done more effectively.

Original: However, we consider the clustering of ordered data, and each cluster should consists of elements, whose ranks can be compared more accurately.
Revised: However, we consider the clustering of ordered data, and each cluster should consists of elements, whose ranks can be compared more accurately.

Revised: There are conventional approaches to use clustering ideas to aid in classiﬁcation or rank estimation.

Revised: For example, Yan et al. (2015) developed a hierarchical classiﬁer, which clusters ﬁne categories into coarse category groups and classiﬁes an object into a ﬁne category within its coarse category group.

Original: P ROPOSED A LGORITHM 3. P ROBLEM D EFINITION
Revised: For extreme multiclass classiﬁcation, Daum´e III et al. (2017) proposed to predict a class label among candidate classes, which are dynamically selected by the recall tree. It is however noted that the leaves of the recall tree do not partition the set of classes. Also, for age estimation, Li et al.

Revised: (2019) proposed a tree-like structure, called bridge-tree, to divide data into overlapping age groups and train a local regressor for each group.

Revised: The set of local regressors can be more accurate than a global regressor to deal with the entire age range.

Revised: Whereas these conventional approaches group data in the label dimension to perform their tasks more effectively, the proposed algorithm cluster data in the dimension orthogonal to the label dimension.

Revised: In other words, we cluster data using identity features, instead of using order features.

Revised:  P ROPOSED A LGORITHM 3. P ROBLEM D EFINITION

Original: An order is a binary relation, often denoted by ≤ , on a set Θ = { θ 1 , θ 2 , . .
Revised: An order is a binary relation, often denoted by ≤ , on a set Θ = { θ 1 , θ 2 , . .

Original: , θ m } (Schr¨oder, 2003).
Revised: , θ m } (Schr¨oder, 2003).

Original: It should satisfy three properties of reﬂexivity ( θ i ≤ θ i for all i ), antisymmetry ( θ i ≤ θ j and θ j ≤ θ i imply θ i
Revised: It should satisfy three properties of reﬂexivity ( θ i ≤ θ i for all i ), antisymmetry ( θ i ≤ θ j and θ j ≤ θ i imply θ i

Original: = θ j ), and transitivity ( θ i ≤ θ j and θ j ≤ θ k imply θ i ≤ θ k ).
Revised: = θ j ), and transitivity ( θ i ≤ θ j and θ j ≤ θ k imply θ i ≤ θ k ).

Original: Then, Θ is called a partially ordered set .
Revised: Then, Θ is called a partially ordered set .

Original: Furthermore, if every pair of elements are comparable ( θ i ≤ θ j or θ j ≤ θ i for all i , j ), Θ is called a chain or linearly ordered set .
Revised: Furthermore, if every pair of elements are comparable ( θ i ≤ θ j or θ j ≤ θ i for all i , j ), Θ is called a chain or linearly ordered set .

Original: An order describes ranks or priorities of classes.
Revised: An order describes ranks or priorities of classes.

Original: For example, in age estimation, θ i may represent the age class of i -year-olds.
Revised: For example, in age estimation, θ i may represent the age class of i -year-olds.

Original: Then, θ 14 ≤ θ 49 represents that 14-year-olds are younger than 49-year-olds.
Revised: Then, θ 14 ≤ θ 49 represents that 14-year-olds are younger than 49-year-olds.

Original: As mentioned previously, it is less easy to tell the older one between people of different genders.
Revised: As mentioned previously, it is less easy to tell the older one between people of different genders.

Original: An algorithm, hence, may compare a subject with reference subjects of the same gender only.
Revised: An algorithm, hence, may compare a subject with reference subjects of the same gender only.

Original: In such a case, the underlying classset Θ is the union of two disjoint chains Θ female = { θ f0 , θ f1 , . . . , θ f p } andΘ male = { θ m0 , θ m1 , . .
Revised: In such a case, each age class θ i represents two subclasses θ female i and θ male i of different types, and the algorithm compares only subjects of the same type.

Original: , θ m p } , where p is the maximum age.
Revised: Lim et al. (2020) assumed that subclasses of different types are incomparable and thus the set of subclasses is the union of k disjoint chains, where k is the number of types. However, in many ranking applications, objects of different types can be compared (although less easily than those of the same type are).

Original: Note that  θ f i and θ m j are assumed to be incomparable with each other for all i , j .
Revised: Thus, instead of assuming incomparability across chains, we assume that there is a total order on Θ = { θ 1 , θ 2 , . . , θ m } , in which each class θ i consists of k types of subclasses, and that object instances of the same type are more easily compared than those of different types.

Original: Suppose that n training instances in X
Revised: Suppose that n training instances in X

Original: = { x 1 , x 2 , . . .
Revised: = { x 1 , x 2 , . . .

Original: , x n } are given.
Revised: , x n } are given.

Original: Also, suppose that there are m ranks and the ground-truth rank of each instance is known.
Revised: Also, suppose that there are m ranks and the ground-truth rank of each instance is known.

Original: In this sense, X contains ordered data.
Revised: In this sense, X contains ordered data.

Original: The problem is twofold.
Revised: The problem is twofold.

Original: The ﬁrst goal is to decompose the class set Θ into k disjoint chains to represent the comparability between instances;
Revised: The ﬁrst goal is to decompose the whole instances X into k disjoint clusters {C j } kj =1 in which instances are more easily compared;

Original: [Equation]
Revised: [Equation]

Original: where Θ i ∩ Θ j = ∅ for i (cid:54) 
Revised: where C i ∩C j = ∅ for i (cid:54) =

Original: = j, each chain is given by Θ l = { θ l 1 , θ l 2 , . . . , θ lm } , and the number k of chains is assumed to be known a priori .
Revised: j .

Original: In other words, we aim to partition the ordered data in X into k clusters (or chains), by grouping them according to their characteristics unrelated to their ranks.
Revised: In other words, we aim to partition the ordered data in X into k clusters, by grouping them according to their characteristics unrelated to their ranks.

Original: These characteristics, which tend to remain the same even when an object experiences rank changes, are referred to as ‘identity’ features in this work.
Revised: These characteristics, which tend to remain the same even when an object experiences rank changes, are referred to as ‘identity’ features in this work.

Original: For example, in age estimation, genders or races can be identity features.
Revised: For example, in age estimation, genders or races can be identity features.

Original: However, we perform the clustering without any supervision for identity features.
Revised: However, we perform the clustering without any supervision for identity features.

Original: Notice that instances within a chain would be compared more easily than those across chains, since they have similar identity features. 
Revised: Notice that instances within a cluster would be compared more easily than those across clusters, since they have similar identity features. The number k of clusters is assumed to be known a priori . Impacts 𝐹 ℎ or𝑥 ℎ id 𝑥 Instance𝑥

Revised: Encoder 𝑥 ≻ 𝑦 or 𝑥 ≈ 𝑦 or 𝑥 ≺ 𝑦

Revised: Real or

Original: The second goal is to assign an unseen test instance into one of the chains and determine its rank by comparing it with reference instances within the chain.
Revised: Fake 𝐹 Instance𝑦 ො𝑦 ො𝑥 Decoder 𝐺 𝐺 𝐹 𝐶 𝐷 Comparator

Revised: Discriminator

Revised: Decoder Encoder ℎ or𝑦 ℎ id𝑦 of k on the clustering performance are discussed in Appendix B.7.The second goal is to assign an unseen test instance into one of the clusters and determine its rank by comparing it with reference instances within the cluster.

Original: To achieve these goals, we propose the ORID network and the DRC algorithm.
Revised: To achieve these goals, we propose the ORID network and the DRC algorithm.

Original: 3. O RDER -I DENTITY D ECOMPOSITION
Revised: 3. O RDER -I DENTITY D ECOMPOSITION

Original: In general, object instances can be compared more easily, as they have more similar identity features irrelevant to order.
Revised: In general, object instances can be compared more easily, as they have more similar identity features irrelevant to order.

Original: Therefore, we decompose the information of each object instance into an order feature and an identity feature.
Revised: Therefore, we decompose the information of each object instance into an order feature and an identity feature.

Original: To this end, we propose the ORID network in Figure 2, composed of three parts: autoencoder, discriminator, and comparator.
Revised: To this end, we propose the ORID network in Figure 2, composed of three parts: autoencoder, discriminator, and comparator.

Original: 1) Autoencoder:
Revised: 1) Autoencoder:

Original: Similarly to deep clustering algorithms in (Yang et al., 2017; Dizaji et al., 2017;
Revised: Similarly to deep clustering algorithms in (Yang et al., 2017; Dizaji et al., 2017;

Original: Chen et al., 2017; Ji et al., 2017), we use the autoencoder G ◦ F ( · ) , based on a neural network, to extract feature vectors.
Revised: Chen et al., 2017; Ji et al., 2017), we use the autoencoder G ◦ F ( · ) , based on a neural network, to extract feature vectors.

Original: The encoder h x = F ( x ) maps an input vector x to a feature vector h x , while the decoder ˆ x = G ( h x ) reconstructs ˆ x from h x .
Revised: The encoder h x = F ( x ) maps an input vector x to a feature vector h x , while the decoder ˆ x = G ( h x ) reconstructs ˆ x from h x .

Original: By minimizing the reconstruction loss (cid:107) x − ˆ x (cid:107) 1 , F is trained to represent x compactly with as little loss of information as possible.
Revised: By minimizing the reconstruction loss (cid:107) x − ˆ x (cid:107) 1 , F is trained to represent x compactly with as little loss of information as possible.

Original: We decompose the overall feature h x ∈ R d or + d id into the order feature h x or and the identity feature h x id , given by h x
Revised: We decompose the overall feature h x ∈ R d or + d id into the order feature h x or and the identity feature h x id , given by [Equation]

Original: [Equation]
Revised: [Equation]

Original: where d or and d id are the dimensions of h x or and h x id .
Revised: where d or and d id are the dimensions of h x or and h x id .

Original: However, without additional control, the output h x of the neural network F would be highly entangled (Higgins et al., 2018).
Revised: However, without additional control, the output h x of the neural network F would be highly entangled (Higgins et al., 2018).

Original: To put together order-related information into h x or , we employ the comparator.
Revised: To put together order-related information into h x or , we employ the comparator.

Original: 2) Comparator: Using the order features h x or and h y or of a pair of instances x and y , we train the comparator, which classiﬁes their ordering relationship into one of three categories ‘bigger,’ ‘similar,’ and ‘smaller’:
Revised: 2) Comparator: Using the order features h x or and h y or of a pair of instances x and y , we train the comparator, which classiﬁes their ordering relationship into one of three categories ‘bigger,’ ‘similar,’ and ‘smaller’:

Original: [Equation]
Revised: [Equation]

Original: 𝐹 ℎ or𝑥 ℎ id 𝑥 Instance𝑥

Original: Encoder 𝑥 ≻ 𝑦 or 𝑥 ≈ 𝑦 or 𝑥 ≺ 𝑦

Original: Real or

Original: Fake 𝐹

Original: Instance𝑦 ො𝑦 ො𝑥

Original: Decoder 𝐺 𝐺 𝐹 𝐶 𝐷 Comparator Discriminator Decoder Encoder ℎ or𝑦 ℎ id𝑦 where θ ( · ) denotes the class of an instance.
Revised: where θ ( · ) denotes the class of an instance.

Original: As in (Lim et al., 2020), ‘ (cid:31) , ≈ , ≺ ’ represent the ordering relationship between instances, while ‘ >, = , < ’ do the mathematical order between classes.
Revised: As in (Lim et al., 2020), ‘ (cid:31) , ≈ , ≺ ’ represent the ordering relationship between instances, while ‘ >, = , < ’ do the mathematical order between classes.

Original: The comparator outputs the softmax probability p xy =
Revised: The comparator outputs the softmax probability p xy =

Original: ( p xy (cid:31) , p xy ≈ , p xy ≺ ) .
Revised: ( p xy (cid:31) , p xy ≈ , p xy ≺ ) .

Original: It is trained to minimize the cross-entropy between p xy and the ground-truth one-hot vector q xy =
Revised: It is trained to minimize the cross-entropy between p xy and the ground-truth one-hot vector q xy =

Original: ( q xy (cid:31) , q xy ≈ , q xy ≺ ) .
Revised: ( q xy (cid:31) , q xy ≈ , q xy ≺ ) .

Original: Because it is trained jointly with the autoencoder, the information deciding the ordering relationship tends to be encoded into the order features h x or and h y or .
Revised: Because it is trained jointly with the autoencoder, the information deciding the ordering relationship tends to be encoded into the order features h x or and h y or .

Original: On the other hand, the remaining information necessary for the reconstruction of ˆ x and ˆ y are encoded into the identity features h x id and h y id .
Revised: On the other hand, the remaining information necessary for the reconstruction of ˆ x and ˆ y are encoded into the identity features h x id and h y id .

Original: 3) Discriminator: We adopt the discriminator D that tells real images from synthesized images, generated by the decoder G .
Revised: 3) Discriminator: We adopt the discriminator D that tells real images from synthesized images, generated by the decoder G .

Original: Using the GAN loss (Goodfellow et al., 2014), the discriminator helps the decoder to reconstruct more realistic output ˆ x and ˆ y .
Revised: Using the GAN loss (Goodfellow et al., 2014), the discriminator helps the decoder to reconstruct more realistic output ˆ x and ˆ y .

Original: Appendix A provides detailed network structures of these components in ORID.
Revised: Appendix A provides detailed network structures of these components in ORID.

Original: 3. D EEP R EPULSIVE C LUSTERING
Revised: 3. D EEP R EPULSIVE C LUSTERING

Original: After obtaining the identity features h x 1 id , h x 2 id , . . .
Revised: After obtaining the identity features h x 1 id , h x 2 id , . . .

Original: , h x n id of all instances x i ∈ X , we partition them into k clusters, corresponding to k disjoint chains.
Revised: , h x n id of all instances x i ∈ X , we partition them into k clusters.

Original: Each chain contains instances  comparable to one another.
Revised: Each cluster contains instances that are more easily comparable to one another.

Original: The identity features are normalized in Eq.
Revised: The identity features are normalized in Eq.

Original: (3) and lie on the unit sphere in R d id .
Revised: (3) and lie on the unit sphere in R d id .

Revised: In other words, we cluster data points on the unit sphere.

Original: Thus, the cosine similarity is a natural afﬁnity metric.
Revised: Thus, the cosine similarity is a natural afﬁnity metric.

Original: Let C j , 1 ≤ j ≤ k , denote the k clusters.
Revised: Let C j , 1 ≤ j ≤ k , denote the k clusters.

Original: Also, let c j , constrained to be on the unit sphere, denote the ‘centroid’ or the representative vector for the instances in cluster C j .
Revised: Also, let c j , constrained to be on the unit sphere, denote the ‘centroid’ or the representative vector for the instances in cluster C j .

Original: We deﬁne the quality of cluster C j as
Revised: We deﬁne the quality of cluster C j as

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term is the similarity of an instance in C j to the centroid c j , the second term with the negative sign quantiﬁes the average dissimilarity of the instance from the other centroids, and α is a nonnegative weight.
Revised: where the ﬁrst term is the similarity of an instance in C j to the centroid c j , the second term with the negative sign quantiﬁes the average dissimilarity of the instance from the other centroids, and α is a nonnegative weight.

Original: For a high quality cluster, instances should be concentrated around the centroid and be far from the other clusters.
Revised: For a high quality cluster, instances should be concentrated around the centroid and be far from the other clusters.

Original: The second term is referred to as the repulsive term, as its objective is similar to the repulsive rule in (Lee et al., 2015).
Revised: The second term is referred to as the repulsive term, as its objective is similar to the repulsive rule in (Lee et al., 2015).

Revised: Although conventional methods also try to increase inter-cluster dissimilarity (Ward Jr, 1963; Lee et al., 2015), to the best of our knowledge,

Revised: DRC is the ﬁrst attempt to use an explicit repulsive term in deep clustering, which jointly optimizes clustering and feature embedding.

Original: Next, we measure the overall quality of the clustering by
Revised: Next, we measure the overall quality of the clustering by

Original: [Equation]
Revised: [Equation]

Original:  We aim to ﬁnd the optimum clusters to maximize this objective function J , yet ﬁnding the global optimum is NP-complete (Kleinberg et al., 1998; Garey et al., 1982).
Revised: [Equation] We aim to ﬁnd the optimum clusters to maximize this objective function J , yet ﬁnding the global optimum is NP-complete (Kleinberg et al., 1998; Garey et al., 1982).

Original: Hence, we propose an iterative algorithm, called DRC, to ﬁnd a local optimum, as in the k -means algorithm (Gersho & Gray, 1991).
Revised: Hence, we propose an iterative algorithm, called DRC, to ﬁnd a local optimum, as in the k -means algorithm (Gersho & Gray, 1991).

Original: Centroid rule: After ﬁxing the clusters {C j } kj =1 , we update the centroids { c j } kj =1 to maximize J in Eq.
Revised: Centroid rule: After ﬁxing the clusters {C j } kj =1 , we update the centroids { c j } kj =1 to maximize J in Eq.

Original: Because the centroids should lie on the unit sphere, we solve the constrained optimization problem:
Revised: Because the centroids should lie on the unit sphere, we solve the constrained optimization problem:

Original: maximize J ( { c j } kj =1 ) subject to c tj c j = 1 for

Original: all j

Original: [Equation]
Revised: [Equation]

Original: Using Lagrangian multipliers (Bertsekas, 1996), the optimal centroids are obtained as
Revised: Using Lagrangian multipliers (Bertsekas, 1996), the optimal centroids are obtained as

Original: [Equation]
Revised: [Equation]

Original: Nearest neighbor NN) rule: On the other hand, after ﬁxing the centroids, we update the membership of each instance to maximize J in Eq.
Revised: NN rule: On the other hand, after ﬁxing the centroids, we update the membership of each instance to maximize J in Eq.

Original: The optimal cluster C j is given by
Revised: The optimal cluster C j is given by

Original: [Equation]
Revised: [Equation]

Original: In other words, an instance should be assigned to C j if its nearest centroid is c j .
Revised: In other words, an instance should be assigned to C j if its nearest centroid is c j .

Original: Output: Chains {C j } kj =1 , centroids { c j } kj =1 , ORID network

Original: We apply the centroid rule and the NN rule iteratively until convergence.
Revised: We apply the centroid rule and the NN rule iteratively until convergence.

Original: Since both rules monotonically increase the same objective function J , DRC is guaranteed to converge to a local maximum.
Revised: Because both rules monotonically increase the same objective function J and the inequality J ≤ n + αk − 1 n always holds, J is guaranteed to converge to a local maximum.

Revised: Readers interested in the convergence are referred to (Sabin & Gray, 1986; Pollard, 1982).

Original: Without the repulsive term in Eq.
Revised: Without the repulsive term in Eq.

Original: (6) ( i.e. when α = 0 ), centroid c j in Eq.
Revised: (6) ( i.e. when α = 0 ), centroid c j in Eq.

Original: (8) is updated by
Revised: (8) is updated by

Original: [Equation]
Revised: [Equation]

Original: (10) as done in the spherical k -means (Dhillon & Modha, 2001).
Revised: as done in the spherical k -means (Dhillon & Modha, 2001).

Original: In contrast, with a positive α , the objective function J is reduced when the centroids are far from one another.
Revised: In contrast, with a positive α , the objective function J is reduced when the centroids are far from one another.

Original: Ideally, in equilibrium, the centroid of a cluster should be the opposite of the centroid of all the other clusters
Revised: Ideally, in equilibrium, the centroid of a cluster should be the opposite of the centroid of all the other clusters;

Original: [Equation]
Revised: [Equation]

Original: (cid:18) (cid:80) x ∈C j h x id (cid:107) (cid:80) x ∈C j h x id (cid:107) (cid:19) t (cid:18) (cid:80) x

Original: ∈X\C j h x id

Original: [Equation]

Original: (cid:80) x

Original: ∈X\C j h x id

Original: [Equation]

Original: for all j = 1 , 2 , . . .

Original: , k. [Equation] Note that the ORID network and thus the encoded feature space are trained jointly with the repulsive clustering.
Revised: Note that the ORID network and thus the encoded feature space are trained jointly with the repulsive clustering.

Original: As the training goes on, the centroids repel one another, and the clusters are separated more clearly due to the repulsive term.
Revised: As the training goes on, the centroids repel one another, and the clusters are separated more clearly due to the repulsive term.

Original: We jointly optimize the clusters(or chains) and the ORID network parameters, as described in Algorithm 1 .
Revised: We jointly optimize the clusters and the ORID network parameters, as described in Algorithm 1 .

Original: First, we train the ORID network for warm-up epochs, assuming that every pair of instances x and y are comparable.
Revised: First, we train the ORID network for warm-up epochs, by employing every pair of instances x and y as input.

Original: Then, using the identity features, we partition the input data into k chains by employing k -means.
Revised: Then, using the identity features, we partition the input data into k clusters using k -means.

Original: Second, we repeat the ﬁne-tuning of the ORID network and the repulsive clustering alternately.
Revised: Second, we repeat the ﬁne-tuning of the ORID network and the repulsive clustering alternately.

Original: In the ﬁne-tuning, a pair of x and y are constrained to be from the same chain, and the following loss function is employed.
Revised: In the ﬁne-tuning, a pair of x and y are constrained to be from the same cluster, and the following loss function is employed.

Original: [Equation]
Revised: [Equation]

Original: Appendix B describes this loss function in detail, proves the optimality of the centroid and NN rules in Eq.
Revised: Appendix B describes this loss function in detail, proves the optimality of the centroid and NN rules in Eq.

Original: (8) and (9), and analyzes the impacts of the repulsive term in Eq.
Revised: (8) and (9), and analyzes the impacts of the repulsive term Output: Clusters {C j } kj

Revised: 1 , centroids { c j } kj network

Original: 3. R ANK E STIMATION
Revised: 3. R ANK E STIMATION

Original: Using the output of the DRC-ORID algorithm, we can estimate the rank of an unseen test instance x .
Revised: Using the output of the DRC-ORID algorithm, we can estimate the rank of an unseen test instance x .

Original: First, we extract its identity feature
Revised: First, we extract its identity feature

Original: h x id using the ORID encoder.
Revised: h x id using the ORID encoder.

Original: By comparing h x id with the centroids { c j } kj =1 based on the NN rule, we ﬁnd the most similar centroid c l .
Revised: By comparing h x id with the centroids { c j } kj =1 based on the NN rule, we ﬁnd the most similar centroid c l .

Original: Then, x is declared to belong to chain C l, whose class set is Θ l = { θ l 1 , θ l 2 , . . , θ lm } .
Revised: Then, x is declared to belong to cluster C l .

Original: Without loss of generality, let us assume that the classes (or ranks) are the ﬁrst m natural numbers, Θl = { 1 , 2 , . . .
Revised: Without loss of generality, let us assume that the classes (or ranks) are the ﬁrst m natural numbers, Θ = { 1 , 2 , . . .

Original: m } .
Revised: m } .

Original: Then, for each i ∈ Θl , we select a reference instance y i with rank i from chain C l , so that it is the most similar to x .
Revised: Then, for each i ∈ Θ , we select a reference instance y i with rank i from cluster C l , so that it is the most similar to x .

Original: Speciﬁcally, 
Revised: Speciﬁcally, y i

Original: [Equation]
Revised: [Equation]

Original: We estimate the rank θ ( x ) of the test instance x , by comparing it with the chosen references y i , 1 ≤ i ≤ m .
Revised: We estimate the rank θ ( x ) of the test instance x , by comparing it with the chosen references y i , 1 ≤ i ≤ m .

Original: For the rank estimation, Lim et al.
Revised: For the rank estimation, Lim et al.

Original: (2020) developed the maximum consistency rule, C h a i n C h a i n C h a i n
Revised: (2020) developed the maximum consistency rule, which however does not exploit the probability information, generated by the comparator.

Original: C h a i n

Original: C h a i n which however does not exploit the probability information, generated by the comparator.

Original: In this paper, we propose the maximum a posteriori (MAP) estimation algorithm as follows. By comparing x with y i , the comparator yields the probability vector p xy i =
Revised: In this paper, we use the maximum a posteriori (MAP) estimation rule, which is described in detail in

Original: ( p xy i (cid:31) , p xy i ≈ , p xy i ≺ ) for the three cases in Eq.

Original: Thus, given y i , the probability of θ ( x ) = r can be written as

Original: [Equation] Suppose that x (cid:31) y i . Then, θ ( x ) − θ ( y i ) = r − i > τ from Eq.
Revised: Appendix B.10.

Original: Also, the maximum possible rank is m .

Original: We hence assume that θ ( x ) has the uniform distribution between i + τ + 1 and m .

Original: In other words, P θ ( x ) ( r | x (cid:31) y i ) ∼ U ( i + τ + 1 , m ) , where U denotes a discrete uniform distribution.

Original: Similarly, we have P θ ( x ) ( r | x ≈ y i ) ∼ U ( i − τ, i + τ ) and P θ ( x ) ( r | x ≺ y i ) ∼ U (1 , i − τ − 1) .

Original: Then, we approximate the a posteriori probability P θ ( x ) ( r | y 1 , . . .

Original: y m ) by averaging those single-reference inferences in Eq.

Original: [Equation]

Original: Finally, we obtain the MAP estimate of the rank of x , which is given by

Original: [Equation]

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: This section provides various experimental results.
Revised: This section provides various experimental results.

Original: Due to space limitation, implementation details and more results are available in Appendices C, D, and E.
Revised: Due to space limitation, implementation details and more results are available in Appendices C, D, and E.

Original: 4. F ACIAL A GE E STIMATION
Revised: 4. F ACIAL A GE E STIMATION

Original: Datasets: We use two datasets.
Revised: Datasets: We use two datasets.

Original: First, MORPH II (Ricanek & Tesafaye, 2006) is a collection of about 55,000 facial images in the age range [16 , 77] .
Revised: First, MORPH II (Ricanek & Tesafaye, 2006) is a collection of about 55,000 facial images in the age range [16 , 77] .

Original: It provides gender (female, male) and race (African
Revised: It provides gender (female, male) and race (African

Original: American, Asian, Caucasian, Hispanic) labels as well.
Revised: American, Asian, Caucasian, Hispanic) labels as well.

Original: We employ the four evaluation settings A, B,
Revised: We employ the four evaluation settings A, B,

Original: C, and D in Appendix C.2.
Revised: C, and D in Appendix C.2.

Original: Second, the balanced dataset (Lim et al., 2020) is sampled from the three datasets of MORPH II, AFAD (Niu et al., 2016), and UTK (Zhang et al., 2017) to overcome bias to speciﬁc ethnic groups or genders.
Revised: Second, the balanced dataset (Lim et al., 2020) is sampled from the three datasets of MORPH II, AFAD (Niu et al., 2016), and UTK (Zhang et al., 2017) to overcome bias to speciﬁc ethnic groups or genders.

Original: It contains about 6,000 images for each combination of gender in { female, male } and ethnic group in { African, Asian, European } .
Revised: It contains about 6,000 images for each combination of gender in { female, male } and ethnic group in { African, Asian, European } .

Original: Clustering: Figure 3 shows clustering results on MORPH II (setting A), when the number of chains is k = 2 .
Revised: Clustering: Figure 3 shows clustering results on MORPH II (setting A), when the number of clusters is k = 2 .

Original: Setting A contains faces of Caucasian descent only.
Revised: Setting A contains faces of Caucasian descent only.

Original: Thus, the proposed DRC-ORID divides those faces into two chains according to genders in general, although the annotated gender information is not used.
Revised: Thus, the proposed DRC-ORID divides those faces into two clusters according to genders in general, although the annotated gender information is not used.

Original: Most males are assigned to chain 1, while a majority of females to chain 2.
Revised: Most males are assigned to cluster 1, while a majority of females to cluster 2.

Original: On the other hand, setting B consists of Africans and Caucasians.
Revised: On the other hand, setting B consists of Africans and Caucasians.

Original: Thus, those images are clustered according to the races, as shown in Appendix C.3.
Revised: Thus, those images are clustered according to the races, as shown in Appendix C.3.

Original: Figure 4 is the results on the balanced dataset at k = 3 , which is composed of MORPH II, AFAD, and UTK images.
Revised: Figure 4 is the results on the balanced dataset at k = 3 , which is composed of MORPH II, AFAD, and UTK images.

Original: Due to different characteristics of these sources, images are clearly divided according to their sources.
Revised: Due to different characteristics of these sources, images are clearly divided according to their sources.

Original: At k = 2 , MORPH II images are separated from the others.
Revised: At k = 2 , C l u s t e r C l u s t e r C l u s t e r

Original: This is because, unlike the MORPH II images, the boundaries of most
Revised: C l u s t e r C l u s t e r [Table] MORPH II images are separated from the others.

Original:  AFAD and UTK images are zeroed for alignment using SeetaFaceEngine (Zhang et al., 2014).[Figure] [Figure] [Table]
Revised: This is because, unlike the MORPH II images, the boundaries of most AFAD and UTK images are zeroed for alignment using SeetaFaceEngine (Zhang et al., 2014).

Original: Lim et al.
Revised: Lim et al.

Original: (2020) also tried the clustering of the balanced dataset.
Revised: (2020) also tried the clustering of the balanced dataset.

Original: Figure 5 visualizes the feature space using t-SNE (Maaten & Hinton, 2008).
Revised: Figure 5 visualizes the feature space using t-SNE (Maaten & Hinton, 2008).

Original: Although their method aligns the features according to ages, their chains are not separated, overlapping one another.
Revised: Although their method aligns the features according to ages, their clusters are not separated, overlapping one another.

Original: In contrast, the proposed DRC-ORID separates the three chains clearly, as well as sorts features according to the ages within each chain.
Revised: In contrast, the proposed DRC-ORID separates the three clusters clearly, as well as sorts features according to the ages within each cluster.

Original: More t-SNE plots for analyzing the impacts of the repulsive term are available in Appendix B.5.
Revised: More t-SNE plots for analyzing the impacts of the repulsive term are available in Appendix B.5.

Original: Age transformation: We assess the decomposition performance of ORID.
Revised: Age transformation: We assess the decomposition performance of ORID.

Original: Although ORID is not designed for age transformation (Or-El et al., 2020), it decomposes an image x into the order and identity features, h x or and h x id .
Revised: Although ORID is not designed for age transformation (Or-El et al., 2020), it decomposes an image x into the order and identity features, h x or and h x id .

Original: Thus, the age can be transformed in two steps.
Revised: Thus, the age can be transformed in two steps.

Original: First, we replace h x or of x with h y or of a reference image y at a target age.
Revised: First, we replace h x or of x with h y or of a reference image y at a target age.

Original: Second, we decode the resulting feature (concatenation of h y or and h x id ) to obtain the transformed image.
Revised: Second, we decode the resulting feature (concatenation of h y or and h x id ) to obtain the transformed image.

Original: Figure 6 shows some results on
Revised: Figure 6 shows some results on

Original: MORPH II images.
Revised: MORPH II images.

Original: Order-related properties, such as skin textures and hair colors, are modiﬁed plausibly, but identity information is preserved.
Revised: Order-related properties, such as skin textures and hair colors, are modiﬁed plausibly, but identity information is preserved.

Original: This indicates the reliability of ORID.
Revised: This indicates the reliability of ORID.

Revised: Reconstruction: Figure 7 shows reconstructed faces using whole feature ( h x or ⊕ h x id ), order feature only ( h x or ⊕ 0 ), and identity feature only ( 0 ⊕

Revised: h x id ).

Revised: Without the order feature, each decoded face is degraded but the person can be identiﬁed.

Revised: In contrast, without the identity feature, the reconstruction is not related to the person except that it seems to be an average face of people at the same age as the person.

Revised: These results conﬁrm that order and identity features are complementary.

Original: Age estimation:
Revised: Age estimation:

Original: Table 1 compares the proposed algorithm with conventional age estimators on the four evaluation settings of MORPH II.
Revised: Table 1 compares the proposed algorithm with conventional age estimators on the four evaluation settings of MORPH II.

Original: These conventional algorithms take 224 × 224 or bigger images as input, while ORID takes 64 × 64 images.
Revised: These conventional algorithms take 224 × 224 or bigger images as input, while ORID takes 64 × 64 images.

Original: Moreover, most of them adopt VGG16 (Simonyan & Zisserman, 2015) as their backbones, which is more complicated than the ORID encoder.
Revised: Moreover, most of them adopt VGG16 (Simonyan & Zisserman, 2015) as their backbones, which is more complicated than the ORID encoder.

Original: Thus, for comparison, after ﬁxing chains using DRC-ORID, we train another pairwise comparator based on VGG16, whose architecture is the same as Lim et al. 
Revised: Thus, for comparison, after ﬁxing clusters using DRC-ORID, we train another pairwise comparator based on VGG16, whose architecture is the same as Lim et al. We measure the age estimation [Figure]

Original: We measure the age estimation performance by the mean absolute error (MAE) and the cumulative score (CS).
Revised: [Figure] [Figure] performance by the mean absolute error (MAE) and the cumulative score (CS).

Original: MAE is the average absolute error between estimated and ground-truth ages, and CS computes the percentage of test samples whose absolute errors are less than or equal to a tolerance level of 5 .
Revised: MAE is the average absolute error between estimated and ground-truth ages, and CS computes the percentage of test samples whose absolute errors are less than or equal to a tolerance level of 5 .

Original: Mainly due to the smaller input size of 64 × 64 , the vanilla version yields poorer performances than the conventional algorithms.
Revised: Mainly due to the smaller input size of 64 × 64 , the vanilla version yields poorer performances than the conventional algorithms.

Original: The VGG version, however, outperforms them signiﬁcantly.
Revised: The VGG version, however, outperforms them signiﬁcantly.

Original: First, in the proposed-VGG-1CH, all classes form a single chain, as in the OL algorithm.
Revised: First, in the proposed-VGG ( k = 1 ), all instances can be compared, as in the OL algorithm.

Original: In other words, the clustering is not performed.
Revised: In other words, the clustering is not performed.

Original: Thus, the pairwise comparators of OL and the proposed-VGG-1CH are trained in the same way, but their rank estimation rules are different.
Revised: Thus, the pairwise comparators of OL and the proposedVGG ( k = 1 ) are trained in the same way, but their rank estimation rules are different.

Original: Whereas OL uses the maximum consistency rule, the proposed algorithm performs the MAP estimation.
Revised: Whereas OL uses the maximum consistency rule, the proposed algorithm performs the MAP estimation.

Original: The score gaps between them conﬁrm that the MAP estimation is more accurate.
Revised: The score gaps between them conﬁrm that the MAP estimation is more accurate.

Original: Moreover, by clustering facial[Table] [Table] images into two chains, the proposed-VGG-2CH improves the results meaningfully.
Revised: Moreover, by clustering facial images into two groups, the proposed-VGG ( k = 2 ) improves the results meaningfully.

Original: The proposedVGG-2CH provides the state-of-the-art results, except for the MAE test in setting D.
Revised: The proposed-VGG ( k = 2 ) provides the state-of-the-art results, except for the MAE test in setting D.

Original: 4. A ESTHETIC S CORE R EGRESSION
Revised: A ESTHETIC S CORE R EGRESSION

Original: The aesthetics and attribute database (AADB) is composed of 10,000 photographs of various themes such as scenery and close-up (Kong et al., 2016).
Revised: The aesthetics and attribute database (AADB) is composed of 10,000 photographs of various themes such as scenery and close-up (Kong et al., 2016).

Original: Each image is annotated with an aesthetic score in [0 , 1] .
Revised: Each image is annotated with an aesthetic score in [0 , 1] .

Original: We quantize the continuous score with a step size of 0.01 to make 101 score classes.
Revised: We quantize the continuous score with a step size of 0.01 to make 101 score classes.

Original: Compared to facial images, AADB contains more diverse data.
Revised: Compared to facial images, AADB contains more diverse data.

Original: It is hence more challenging to cluster AADB images into chains.
Revised: It is hence more challenging to cluster AADB images.

Original: Figure 7 shows example images in each chain at k = 8 .
Revised: Figure 8 shows example images in each cluster at k = 8 .

Original: Images in the same chain have similar colors, similar contents, or similar composition.
Revised: Images in the same cluster have similar colors, similar contents, or similar composition.

Original: This means that ORID extracts identity features effectively, corresponding to contents or styles that are not directly related to aesthetic scores.
Revised: This means that ORID extracts identity features effectively, corresponding to contents or styles that are not directly related to aesthetic scores.

Original: Using those identity features, DRC discovers meaningful clusters.
Revised: Using those identity features, DRC discovers meaningful clusters.

Original: Tablecompares regression results.
Revised: Tablecompares regression results.

Original: Even with a single chain, the proposed algorithm outperforms the Reg-Net and ASM algorithms.
Revised: Even without clustering process, the proposed algorithm outperforms the Reg-Net and ASM algorithms.

Original: Moreover, by using the eight unsupervised chains in Figure 7, the proposed algorithm further reduces the MAE to yield the state-of-the-art result.[Table] [Table]
Revised: Moreover, by using the eight unsupervised clusters in Figure 8, the proposed algorithm further reduces the MAE to yield the state-of-the-art result.

Original: 4. H ISTORICAL C OLOR I MAGE C LASSIFICATION
Revised: 4. H ISTORICAL C OLOR I MAGE C LASSIFICATION

Original: HCI (Palermo et al., 2012) is a dataset for determining the decade when a photograph was taken.
Revised: HCI (Palermo et al., 2012) is a dataset for determining the decade when a photograph was taken.

Original: It contains images from ﬁve decades from 1930s to 1970s.
Revised: It contains images from ﬁve decades from 1930s to 1970s.

Original: Each decade category has 265 images: 210, 5, and 50 are used for training, validation and testing.
Revised: Each decade category has 265 images: 210, 5, and 50 are used for training, validation and testing.

Original: Figure 8 shows the clustering results at k = 4 .
Revised: Figure 9 shows the clustering results at k = 4 .

Original: We observe similarity of contents in each chain.
Revised: We observe similarity of contents in each cluster.

Original: Table 3 compares the quinary classiﬁcation results.
Revised: Table 3 compares the quinary classiﬁcation results.

Original: Frank & Hall (2001), Cardoso & da Costa (2007), Palermo et al.
Revised: Frank & Hall (2001), Cardoso & da Costa (2007), Palermo et al.

Original: (2012), and RED-SVM use traditional features, while the others deep features.
Revised: (2012), and RED-SVM use traditional features, while the others deep features.

Original: The performance gaps between these two approaches are not huge, since 1,050 images are insufﬁcient for training deep networks.
Revised: The performance gaps between these two approaches are not huge, since 1,050 images are insufﬁcient for training deep networks.

Original: However, the proposed algorithm yields competitive results in terms of both accuracy and MAE. 
Revised: However, the proposed algorithm yields competitive results in terms of both accuracy and MAE. [Table] [Table] [Table]

Original:  C ONCLUSIONS
Revised:  I MPACTS OF A PPLICATIONS

Revised: Potential uses: The proposed algorithm can be applied to various ranking problems. In this paper, we demonstrated three vision applications: facial age estimation, aesthetic score regression and historical image classiﬁcation. In particular, the proposed age estimator has various potential uses.

Revised: For example, it can block or recommend media contents to people according to their age groups, can check whether a customer is older than the legal smoking or drinking age, and can facilitate age-based customization of advertisements.

Revised: However, it has harmful impacts, as well as positive ones.

Revised: In the aforementioned examples, age estimation errors lead to undesirable results, such as recommending unsuitable contents.

Revised: Moreover, since it deals with facial image data, it can be used in unwanted surveillance and infringe privacy.

Revised: Hence, ethical considerations should be made before the use of the proposed algorithm.

Revised: Ethical considerations: Recently, ethical concerns about the fairness and safety of automated systems have been raised as critical issues (Corbett-Davies et al., 2017;

Revised: Hardt et al., 2016; Raji &

Revised: Buolamwini, 2019; Buolamwini & Gebru, 2018).

Revised: Especially, due to the intrinsic imbalance of popular facial datasets (Ricanek & Tesafaye, 2006; Zhang et al., 2017;

Revised: Niu et al., 2016), most deep learning methods on facial analysis (Pan et al., 2018; Li et al., 2019; Wen et al., 2020; Or-El et al., 2020) have unwanted gender or racial bias and asymmetric accuracy across demographic subgroups.

Revised: The proposed algorithm is not free from this bias either when trained on such datasets.

Revised: Hence, for any practical usage, it is urged to resolve this unfair performance gap by balancing training data or adopting other techniques.

Revised: Also, even though the proposed algorithm groups data in an unsupervised manner, data are clustered according to genders or races on MORPH II (Ricanek & Tesafaye, 2006).

Revised: These experimental results should never be misinterpreted in such a way as to encourage any racial or gender discrimination or to induce any kind of prejudice and injustice.

Original: The DRC algorithm of ordered data based on ORID was proposed in this work.
Revised: The DRC algorithm of ordered data based on ORID was proposed in this work.

Original: First, the ORID network decomposes the information of an object into the order and identity features.
Revised: First, the ORID network decomposes the information of an object into the order and identity features.

Original: Then, DRC groups objects into clusters using their identity features in a repulsive manner.
Revised: Then, DRC groups objects into clusters using their identity features in a repulsive manner.

Original: Also, we can estimate the rank of an unseen test by comparing it with references within the corresponding cluster based on the MAP decision.
Revised: Also, we can estimate the rank of an unseen test by comparing it with references within the corresponding cluster based on the MAP decision.

Original: Extensive experimental results on various ordered data demonstrated that the proposed algorithm provides excellent clustering and rank estimation performances.
Revised: Extensive experimental results on various ordered data demonstrated that the proposed algorithm provides excellent clustering and rank estimation performances.

Original: A N ETWORK S TRUCTURE OF ORID
Revised: A N ETWORK S TRUCTURE OF ORID

Original: As described in Section 3.2, the ORID network consists of the encoder F , the decoder G , the comparator C
Revised: As described in Section 3.2, the ORID network consists of the encoder F , the decoder G , the comparator C

Original: [Table]
Revised: [Table]

Original: and the discriminator D .
Revised: and the discriminator D .

Original: The network structures of these components are detailed in
Revised: The network structures of these components are detailed in

Original: Tables 4 ∼ 7, where ‘ k h × k w - s - c Conv’ and ‘ k h × k w - s - c Deconv’ denote the 2D convolution and 2D deconvolution with kernel size k h × k w , stride s , and c output channels, respectively.
Revised: Tables 4 ∼ 7, where ‘ k h × k w - s - c Conv’ and ‘ k h × k w - s - c Deconv’ denote the 2D convolution and 2D deconvolution with kernel size k h × k w , stride s , and c output channels, respectively.

Original: ‘BN’ means batch normalization (Ioffe & Szegedy, 2015)
Revised: ‘BN’ means batch normalization (Ioffe & Szegedy, 2015)

Original: [Table]
Revised: [Table]

Original: and ‘ c Dense’ is a dense layer with c output channels.
Revised: and ‘ c Dense’ is a dense layer with c output channels.

Original: Note that the encoder takes a 64 × 64 RGB image as input
Revised: Note that the encoder takes a 64 × 64 RGB image as input

Original: [Table]
Revised: [Table]

Original: and the identity feature of the encoder output is l 2 -normalized in Eq.
Revised: and the identity feature of the encoder output is l 2 -normalized in Eq.

Revised: Also, we set d or = 128 and d id = 896 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B D EEP R EPULSIVE C LUSTERING – D ETAILS B. O PTIMALITY OF C ENTROID R ULE
Revised: B A LGORITHMS – D ETAILS B. O PTIMALITY OF C ENTROID R ULE

Original: To solve the constrained optimization problem in Eq.
Revised: To solve the constrained optimization problem in Eq.

Original: [Equation]
Revised: [Equation]

Original: we construct the Lagrangian function
Revised: we construct the Lagrangian function

Original: [Equation]
Revised: [Equation]

Original: ) where λ j , 1 ≤ j ≤ k , are Lagrangian multipliers (Bertsekas, 1996).
Revised: 14) where λ j , 1 ≤ j ≤ k , are Lagrangian multipliers (Bertsekas, 1996).

Original: By differentiating L with respect to c j and setting it to zero
Revised: By differentiating L with respect to c j and setting it to zero

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] 
Revised: [Equation] [Equation]

Original: 20) for j =
Revised: 17) for j =

Original: [Equation] 
Revised: [Equation] [Equation]

Original: k .
Revised: k .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: the optimal centroid c j is given by
Revised: the optimal centroid c j is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Because of the normalization constraint c jt c j =
Revised: Because of the normalization constraint c jt c j =

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: which leads to the centroid rule in Eq.
Revised: which leads to the centroid rule in Eq.

Original: B. O PTIMALITY OF NN R ULE
Revised: B. O PTIMALITY OF NN R ULE

Original: Let us consider two cases.
Revised: Let us consider two cases.

Original: First, instance x is declared to belong to cluster C j .
Revised: First, instance x is declared to belong to cluster C j .

Original: It then contributes to the objective function J in Eq.
Revised: It then contributes to the objective function J in Eq.

Original: (6) by
Revised: (6) by

Original: [Equation]
Revised: [Equation]

Original: Second, x is declared to belong to another cluster C j (cid:48) .
Revised: Second, x is declared to belong to another cluster C j (cid:48) .

Original: Then, its contribution is
Revised: Then, its contribution is

Original: [Equation]
Revised: [Equation]

Original: By comparing the two contributions, we have
Revised: By comparing the two contributions, we have

Original: [Equation]
Revised: [Equation]

Original: This means that β j ≥ β j (cid:48) when ( h x id ) t c j ≥ ( h x id )
Revised: This means that β j ≥ β j (cid:48) when ( h x id ) t c j ≥ ( h x id )

Original: t c j (cid:48) .
Revised: t c j (cid:48) .

Original: Therefore, x should be assigned to the optimal cluster C j ∗ such that the cosine similarity ( h x id ) t c j ∗ is maximized.
Revised: Therefore, x should be assigned to the optimal cluster C j ∗ such that the cosine similarity ( h x id ) t c j ∗ is maximized.

Original: Equivalently, we have the NN rule in Eq.
Revised: Equivalently, we have the NN rule in Eq.

Original: B. R EGULARIZATION C ONSTRAINT 
Revised: B. R EGULARIZATION C ONSTRAINT IN DRC

Original: To prevent empty clusters and balance the partitioning, we enforce a regularization constraint so that every cluster contains at least a predeﬁned number of instances.
Revised: To prevent empty clusters and balance the partitioning, we enforce a regularization constraint so that every cluster contains at least a predeﬁned number of instances.

Original: More speciﬁcally, when applying the NN rule, we enforce that at least 12 k of instances are assigned to each cluster C j .
Revised: More speciﬁcally, when applying the NN rule, we enforce that at least 12 k of instances are assigned to each cluster C j .

Original: The instances are selected in the decreasing order of cosine similarity ( h x id )
Revised: The instances are selected in the decreasing order of cosine similarity ( h x id )

Original: t c j .
Revised: t c j .

Original: B. L OSS F UNCTIONS
Revised: B. L OSS F UNCTIONS

Original: In the DRC-ORID algorithm, we use the loss function 
Revised: In the DRC-ORID algorithm, we use the loss function [Equation] com

Original: [Equation]
Revised: [Equation]

Original: where the reconstruction, clustering, comparator, and GAN losses are given by
Revised: where the reconstruction, clustering, comparator, and GAN losses are given by

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: Here, N is the number of image pairs in a minibatch.
Revised: Here, N is the number of image pairs in a minibatch.

Original: The weighting parameters are set to λ rec = 5 , λ clu = 0 .
Revised: The weighting parameters are set to λ rec = 5 , λ clu = 0 .

Original: 1 , λ com = 1 , and λ gan = 1 .
Revised: 1 , λ com = 1 , and λ gan = 1 .

Original: B. I MPACTS OF R EPULSIVE T ERM 
Revised: B. I MPACTS OF R EPULSIVE T ERM ON C LUSTERING

Original: To analyze the impacts of the repulsive term in Eq.
Revised: To analyze the impacts of the repulsive term in Eq.

Original: (6), we ﬁrst compare clustering qualities with α = 0 and α = 0 .
Revised: (6), we ﬁrst compare clustering qualities with α = 0 and α = 0 .

Original: At α = 0 , the repulsive term is excluded from the objective function J and the centroid rule is reduced to Eq.
Revised: At α = 0 , the repulsive term is excluded from the objective function J and the centroid rule is reduced to Eq.

Original: (10) in the spherical k -means (Dhillon & Modha, 2001).
Revised: (10) in the spherical k -means (Dhillon & Modha, 2001).

Original: However, different from the spherical k -means, even at α = 0 , the clustering is jointly performed with the training of the ORID network.
Revised: However, different from the spherical k -means, even at α = 0 , the clustering is jointly performed with the training of the ORID network.

Original: We adopt two metrics to measure the quality of clustering: normalized mutual information (NMI) (Strehl & Ghosh, 2002) and centroid afﬁnity (CA).
Revised: We adopt two metrics to measure the quality of clustering: normalized mutual information (NMI) (Strehl & Ghosh, 2002) and centroid afﬁnity (CA).

Original: NMI measures the information shared between two different partitioning of the same data
Revised: NMI measures the information shared between two different partitioning of the same data

Original: A = ∪ Ui =

Revised: A = ∪ Ui =

Original: A i and B = ∪ Vj =
Revised: A i and B = ∪ Vj =

Original: 1 B j ,
Revised: 1 B j ,

Original: [Equation]
Revised: [Equation]

Original:  where U and V are the numbers of clusters in A and B , respectively, N is the total number of samples, and | · | denotes the cardinality.
Revised: 29) where U and V are the numbers of clusters in A and B , respectively, N is the total number of samples, and | · | denotes the cardinality.

Original: Also, we deﬁne the centroid afﬁnity (CA) as
Revised: Also, we deﬁne the centroid afﬁnity (CA) as

Original: [Equation]
Revised: [Equation]

Original: For high-quality clustering, the centroids should be far from one another and thus should yield a low
Revised: For high-quality clustering, the centroids should be far from one another and thus should yield a low

Original: CA score.
Revised: CA score.

Original: Figure 9 plots how NMI and CA vary as the iteration goes on.
Revised: Figure 10 plots how NMI and CA vary as the iteration goes on.

Original: In this test, MORPH II (setting B) is used and the number of clusters k is set to 2.
Revised: In this test, MORPH II (setting B) is used and the number of clusters k is set to 2.

Original: Since setting B consists of Africans and Caucasians, we use the race groups as the ground-truth partitioning for the NMI measurement.
Revised: Since setting B consists of Africans and Caucasians, we use the race groups as the ground-truth partitioning for the NMI measurement.

Original: At early iterations, the NMI score of DRC-ORID with α = 0 .
Revised: At early iterations, the NMI score of DRC-ORID with α = 0 .

Original: 1 is slightly better than that with α = 0 .
Revised: 1 is slightly better than that with α = 0 .

Original: However, as the iterative training and clustering go on, the score gap gets larger.
Revised: However, as the iterative training and clustering go on, the score gap gets larger.

Original: After the convergence, DRC-ORID with α = 0 .
Revised: After the convergence, DRC-ORID with α = 0 .

Original: 1 outperforms the option α = 0 by a signiﬁcant NMI gap of 0.13.
Revised: 1 outperforms the option α = 0 by a signiﬁcant NMI gap of 0.13.

Original: Also, CA of the option α = 0 .
Revised: Also, CA of the option α = 0 .

Original: 1 gradually decreases, whereas that of α = 0 does not.
Revised: 1 gradually decreases, whereas that of α = 0 does not.

Original: At α = 0 . 1 , the repulsive term makes the centroids repel each other.
Revised: At α = 0 . 1 , the repulsive term makes the centroids repel each other.

Original: As a result, CA, which is the cosine similarity between the two centroids, becomes almost − 1 , which means the equilibrium state in Eq.
Revised: As a result, CA, which is the cosine similarity between the two centroids, becomes almost − 1 , which means the equilibrium state in Eq.

Original: (11) is almost achieved
Revised: (11) is almost achieved

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Iteration
Revised: Iteration

Original: Iteration
Revised: [Figure]

Original: Chain 1 Chain
Revised: Cluster Cluster

Original: [Figure]
Revised: [Figure]

Original: We also visualize the feature spaces of the two options, α = 0 and α = 0 .
Revised: We also visualize the feature spaces of the two options, α = 0 and α = 0 .

Original: 1 , using t-SNE in
Revised: 1 , using t-SNE in

Original: Figure 10.
Revised: Figure 11.

Original: It is observed that two clusters are more clearly separated by DRC-ORID with α = 0 .
Revised: It is observed that two clusters are more clearly separated by DRC-ORID with α = 0 .

Original: Figure 11 shows the t-SNE results after the convergence with age labels.
Revised: Figure 12 shows the t-SNE results after the convergence with age labels.

Original: Figure 12 compares the NMI curves at different α ’s.
Revised: Figure 13 compares the NMI curves at different α ’s.

Original: The choice of α affects the quality of clustering, as α controls the intensity of the repulsive force between centroids.
Revised: The choice of α affects the quality of clustering, as α controls the intensity of the repulsive force between centroids.

Original: When α is too large, the centroids move too quickly, making the training of the ORID network difﬁcult.
Revised: When α is too large, the centroids move too quickly, making the training of the ORID network difﬁcult.

Original: On the other hand, when α is too small, the repulsive term does not affect the clustering meaningfully.
Revised: On the other hand, when α is too small, the repulsive term does not affect the clustering meaningfully.

Original: Hence, α should be selected to strike a balance between training reliability and effective repulsion.
Revised: Hence, α should be selected to strike a balance between training reliability and effective repulsion.

Original: It was found experimentally that clustering is performed well around α = 0 .
Revised: It was found experimentally that clustering is performed well around α = 0 .

Original: [Figure]
Revised: [Figure]

Original: Finally, it is worth pointing out that, if the identity features were not normalized as in Eq.
Revised: Finally, it is worth pointing out that, if the identity features were not normalized as in Eq.

Original: (3) and the repulsive clustering were performed in an unbounded space, the distances between centroids would get larger and larger as the iteration goes on.
Revised: (3) and the repulsive clustering were performed in an unbounded space, the distances between centroids would get larger and larger as the iteration goes on.

Original: Thus, convergence would not be achieved.
Revised: Thus, convergence would not be achieved.

Original: This is why we performDRC on the bounded unit sphere.
Revised: This is why we perform on the bounded unit sphere.

Original: C F ACIAL A GE E STIMATION– M ORE E XPERIMENTS AND D ETAILS C. I MPLEMENTATION D ETAILS
Revised: B. I MPACTS OF R EPULSIVE T ERM ON R ANK E STIMATION

Revised: Table 8 compares the rank estimation results when the clustering is performed with and without the repulsive term. In this experiment, we use MORPH II (setting A) and set k = 2 . Without the repulsive term, lower-quality clusters make the training of the comparator more difﬁcult.

Revised: As a result, the age estimation performance degrades signiﬁcantly in terms of both MAE and CS.

Revised: In other words, the quality of clustering affects the rank estimation performance greatly, and the proposed

Revised: DRC algorithm provides high quality clusters suitable for the rank estimation.

Revised: [Table]

Revised: B. I MPACTS OF THE N UMBER k OF C LUSTERS ON R ANK E STIMATION

Revised: Tables 9 and 10 compare the rank estimation results according to the number k of clusters on the

Revised: MORPH II (setting A) and AADB datasets, respectively.

Revised: On MORPH II, the age estimation performance decreases as k increases.

Revised: Since the training set in setting A consists of only 4,394 images, each cluster at a large k contains too few instances.

Revised: Thus, the comparator is trained inefﬁciently with fewer training pairs, degrading the performance.

Revised: In contrast, AADB contains a large number of diverse images.

Revised: Due to the diversity, a relatively large k should be used to group images into meaningful clusters.

Revised: Also, even at a large k , each cluster contains a sufﬁcient number of data.

Revised: Thus, as compared to MORPH II, results on AADB are less sensitive to k .

Revised: In addition, we provide age estimation results on the balanced dataset in Table 14, in which k has marginal impacts on the rank estimation performance.

Revised: As mentioned previously, the quality of clustering signiﬁcantly affects the rank estimation performance.

Revised: Also, similarly to other algorithms based on k -means, the clustering quality of DRC is affected by k .

Revised: Hence, for the proposed algorithm to be used on a new ordered dataset, k should be determined effectively to obtain good clustering and rank estimation results.

Revised: Readers interested in the selection of k are referred to Pham et al.

Revised: [Table]

Revised: [Table]

Revised: C LUSTERING U SING O THER F EATURES

Revised: Instead of clustering identity features h x 1 id , h x 2 id , . . .

Revised: , h x n id , we test clustering order features h x 1 or , h x 2 or , . . .

Revised: , h x n or or whole features h x 1 , h x 2 , . . .

Revised: , h x n .

Revised: In this test, MORPH II (setting A) is used and k = 2 .

Revised: Figure 14 compares the clustering results.

Revised: When using order features or whole features, instances are divided by their ages.

Revised: We see that instances younger than 30 mostly belong to clusterand the others to cluster 2.

Revised: Table 11 compares the performances of the age estimators trained using these clustering results.

Revised: The best performance is achieved when the clustering is done on identity features.

Revised: [Table]

Revised: [Table]

Revised: R ELIABILITY OF F EATURE D ECOMPOSITION

Revised: Performing the comparison using order features only does not theoretically guarantee that orderrelated information is fully excluded from identity features.

Revised: However, we observed empirically that the decomposition is sufﬁciently reliable if the dimension of an identity feature is selected properly.

Revised: If the dimension is too small, the encoder may lose a signiﬁcant portion of order-irrelevant information.

Revised: On the contrary, if the dimension is too large, the encoder may encode order information redundantly.

Revised: In our experiments, we use 128 and 896 dimensional vectors for order and identity features ( d or = 128 and d id = 896 ), and obtain satisfactory decomposition results.

Revised: To show that order-related information is excluded from identity features, we compare the accuracies of the comparator ( i.e. ternary classiﬁer), when identity features are used instead of order features.

Revised: Speciﬁcally, we ﬁrst extract order features and identity features from all instances in MORPH II using the pretrained ORID network.

Revised: Then, we train two comparators that predict the ordering relationship between two instances x and y : one takes the order features h x or and h y or as input and the other takes the identity features h x id and h y id .

Revised: Table 12 lists the comparator accuracies.

Revised: We see that the comparator fails to predict ordering relationships from identity features.

Revised: Also, Figure 15 is t-SNE visualization of the identity feature spaces with age or cluster labels, which conﬁrms that order-related information is reliably excluded from identity features.

Revised: [Table]

Revised: [Figure]

Revised: B. MAP E STIMATION

Revised: Let us describe the MAP estimation rule for rank estimation in Section 3.4.

Revised: Given a test instance x , we select references y i by Eq.

Revised: Then, by comparing x with y i , the comparator yields the probability vector p

Revised: xy i

Revised: =

Revised: ( p xy i (cid:31) , p xy i ≈ , p xy i ≺ ) for the three cases in Eq.

Revised: Thus, given y i , the probability of θ ( x ) = r can be written as

Revised: [Equation]

Revised: Suppose that x (cid:31) y i .

Revised: Then, θ ( x ) − θ ( y i ) =

Revised: r − i > τ from Eq.

Revised: Also, the maximum possible rank is m .

Revised: We hence assume that θ ( x ) has the uniform distribution between i + τ + 1 and m .

Revised: In other words, P θ ( x ) ( r | x (cid:31) y i ) ∼ U ( i + τ + 1 , m ) , where U denotes a discrete uniform distribution.

Revised: Similarly, we have P θ ( x ) ( r | x ≈ y i ) ∼ U ( i − τ, i + τ ) and P θ ( x ) ( r | x ≺ y i ) ∼ U (1 , i − τ − 1) .

Revised: Then, we approximate the a posteriori probability P θ ( x ) ( r | y 1 , . . .

Revised: y m ) by averaging those single-reference inferences in Eq.

Revised: [Equation]

Revised: Finally, we obtain the MAP estimate of the rank of x , which is given by

Revised: [Equation]

Revised: C F ACIAL A GE E STIMATION – M ORE E XPERIMENTS AND D ETAILS C. I MPLEMENTATION D ETAILS

Original: We initialize the parameters of the ORID network for facial age estimation using the Glorot normal method (Glorot & Bengio, 2010).
Revised: We initialize the parameters of the ORID network for facial age estimation using the Glorot normal method (Glorot & Bengio, 2010).

Original: We use the Adam optimizer with a learning rate of 10 − 4 and decrease the rate by a factor of 0.5 every 50,000 steps.
Revised: We use the Adam optimizer with a learning rate of 10 − 4 and decrease the rate by a factor of 0.5 every 50,000 steps.

Original: For data augmentation, we do random horizontal ﬂips only.
Revised: For data augmentation, we do random horizontal ﬂips only.

Original: This is because other augmentation schemes, such as brightness or contrast modiﬁcation, may deform identity information such as skin colors. 
Revised: This is because other augmentation schemes, such as brightness or contrast modiﬁcation, may deform identity information such as skin colors. Also, d or and d id are set to be 128 and 896, respectively.

Original: In Eq.
Revised: In Eq.

Original: (6), we set α to 0.1 and decrease it to 0.05 after 200 epochs.
Revised: (6), we set α to 0.1 and decrease it to 0.05 after 200 epochs.

Original: C. E VALUATION S ETTINGS
Revised: C. E VALUATION S ETTINGS

Original: For evaluation on the MORPH II dataset, we adopt four widely used testing protocols.
Revised: For evaluation on the MORPH II dataset, we adopt four widely used testing protocols.

Original: • Setting A – 5,492 images of the Caucasian race are selected and then randomly divided into two non-overlapping parts: 80% for training and 20% for test.
Revised: • Setting A – 5,492 images of the Caucasian race are selected and then randomly divided into two non-overlapping parts: 80% for training and 20% for test.

Original: • Setting B – 21,000 images of Africans and Caucasians are selected to satisfy two constraints: the ratio between Africans and Caucasians should be 1 : 1 , and that between females and males 1 : 3 .
Revised: • Setting B – 21,000 images of Africans and Caucasians are selected to satisfy two constraints: the ratio between Africans and Caucasians should be 1 : 1 , and that between females and males 1 : 3 .

Original: They are split into three disjoint subsets S1, S2, and S3.
Revised: They are split into three disjoint subsets S1, S2, and S3.

Original: The training and testing are repeated twice: 1) training on S1, testing on S2 + S3, and 2) training on S2, testing on S1 + S3.
Revised: The training and testing are repeated twice: 1) training on S1, testing on S2 + S3, and 2) training on S2, testing on S1 + S3.

Original: The average performance of the two experiments is reported.
Revised: The average performance of the two experiments is reported.

Original: • Setting C – This setting is the 5-fold cross-validation on the entire dataset.
Revised: • Setting C – This setting is the 5-fold cross-validation on the entire dataset.

Original: Images are randomly split into ﬁve folds, but the same person’s images should belong to only one fold.
Revised: Images are randomly split into ﬁve folds, but the same person’s images should belong to only one fold.

Original: The average performance of the ﬁve experiments is reported.
Revised: The average performance of the ﬁve experiments is reported.

Original: • Setting D
Revised: • Setting D

Original: This is called the 80-20 protocol.
Revised: This is called the 80-20 protocol.

Original: Without any constraint, the entire dataset is randomly divided into the training and test sets with ratio 8 : 2 .
Revised: Without any constraint, the entire dataset is randomly divided into the training and test sets with ratio 8 : 2 .

Original: Thus, setting D is similar to one experiment in setting C, but the same person’s images may belong to both training and test sets.
Revised: Thus, setting D is similar to one experiment in setting C, but the same person’s images may belong to both training and test sets.

Original: C. 
Revised: C. C LUSTERING

Original: C LUSTERING

Original: We provide more clustering results on MORPH II.
Revised: We provide more clustering results on MORPH II.

Original: Figure 13 is the clustering results on setting B at k = 2 .
Revised: Figure 16 is the clustering results on setting B at k = 2 .

Original: Since setting B consists of Africans and Caucasians, the images are clustered according to the races.
Revised: Since setting B consists of Africans and Caucasians, the images are clustered according to the races.

Original: Also, Table 8 summarizes the clustering results for settings A, B, and C at k = 2 .
Revised: Also, Table 13 summarizes the clustering results for settings A, B, and C at k = 2 .

Original: The clustering result on setting D is omitted, since it is almost identical with that on setting C. In all settings, the proposed DRC-ORID divides facial images into two chains with meaningful criteria, which are gender for setting A and race for settings B, C, and D.
Revised: The clustering result on setting D is omitted, since it is almost identical with that on setting C. In all settings, the proposed DRC-ORID divides facial images into two clusters with meaningful criteria, which are gender for setting A and race for settings B, C, and D.

Original: C h a i n C h a i n
Revised: C l u s t e r C l u s t e r

Original: [Table]
Revised: [Table]

Original: C. A GE E STIMATION
Revised: C. A GE E STIMATION

Original: We implement a VGG-based pairwise comparator and follow the settings of Lim et al.
Revised: We implement a VGG-based pairwise comparator and follow the settings of Lim et al.

Original: Speciﬁcally, instead of Eq.
Revised: Speciﬁcally, instead of Eq.

Original: (4), we use the ternary categorization based on the geometric ratio and set τ = 0 .
Revised: (4), we use the ternary categorization based on the geometric ratio and set τ = 0 .

Original: We initialize its feature extractor using VGG16 pre-trained on the ILSVRC dataset (Deng et al., 2009) and its fully connected layers using the Glorot normal method.
Revised: We initialize its feature extractor using VGG16 pre-trained on the ILSVRC dataset (Deng et al., 2009) and its fully connected layers using the Glorot normal method.

Original: We employ the Adam optimizer with a minibatch size of 32.
Revised: We employ the Adam optimizer with a minibatch size of 32.

Original: We start with a learning rate of 10 − 4 and shrink it by a factor of 0.5 after every 80,000 steps.
Revised: We start with a learning rate of 10 − 4 and shrink it by a factor of 0.5 after every 80,000 steps.

Original: [Table]
Revised: [Table]

Original: Table 9 lists age estimation results on the balanced dataset according to the number  of chains.
Revised: Table 14 lists age estimation results on the balanced dataset according to the number k of clusters.

Original: OLsupervised trains the comparator using supervised chains separated according to gender or ethnic group annotations.
Revised: OL-supervised trains the comparator using supervised clusters separated according to gender or ethnic group annotations.

Original: Speciﬁcally, the supervised 2CH, 3CH, and 6CH are divided according to genders, ethnic groups, and both genders and ethnic groups, respectively.
Revised: Speciﬁcally, the supervised clusters at k = 2 , 3 , and 6 are divided according to genders, ethnic groups, and both genders and ethnic groups, respectively.

Original: On the other hand, OLunsupervised and the proposed algorithm determine their chains in unsupervised manners.
Revised: On the other hand, OL-unsupervised and the proposed algorithm determine their clusters in unsupervised manners.

Original: We see that the proposed algorithm performs better than the conventional algorithms in all tests.
Revised: We see that the proposed algorithm performs better than the conventional algorithms in all tests.

Original: By employing multiple chains, the proposed algorithm improves MAE by 0.12 and CS by 0.73% on average.
Revised: By employing multiple clusters, the proposed algorithm improves MAE by 0.12 and CS by 0.73% on average.

Original: In contrast, OL-unsupervised improves MAE by 0.04 and CS by 0.07% only.
Revised: In contrast, OL-unsupervised improves MAE by 0.04 and CS by 0.07% only.

Original: This indicates that, by employing identity features, the proposed DRC-ORID algorithm groups instances into meaningful chains, in which instance ranks can be compared more accurately.
Revised: This indicates that, by employing identity features, the proposed DRC-ORID algorithm groups instances into meaningful clusters, in which instance ranks can be compared more accurately.

Original: C. A GE T RANSFORMATION
Revised: C. A GE T RANSFORMATION

Original: More age transformation results are in Figure 14.
Revised: More age transformation results are in Figure 17.

Original: Note that, in Figure 6, given an image x , we select the reference y at a target age, whose identity feature is the most similar to that of x , as in Eq.
Revised: Note that, in Figure 6, given an image x , we select the reference y at a target age, whose identity feature is the most similar to that of x , as in Eq.

Original: Hence, the image x and the reference y have similar appearance.
Revised: Hence, the image x and the reference y have similar appearance.

Original: On the other hand, Figureshows transformed images using randomly selected references.
Revised: On the other hand, Figureshows transformed images using randomly selected references.

Original: The ﬁrst two cases transform the same image x with different references, but the transformed images are similar.
Revised: The ﬁrst two cases transform the same image x with different references, but the transformed images are similar.

Original: Also, even when the gender and/or race of y are different from those of x , the identity information of x is preserved well in the transformed image.
Revised: Also, even when the gender and/or race of y are different from those of x , the identity information of x is preserved well in the transformed image.

Original: This conﬁrms the reliability of ORID.
Revised: This conﬁrms the reliability of ORID.

Original: [Figure]
Revised: [Figure]

Original: C. R EFERENCE I MAGES
Revised: C. R EFERENCE I MAGES

Original: Figure 15 shows examples of reference images, which are used for the rank estimation on MORPH II (setting D) at k = 2 .
Revised: Figure 18 shows examples of reference images, which are used for the rank estimation on MORPH II (setting D) at k = 2 .

Original: Given a test image x , reference image y i of age class i is selected via Eq.
Revised: Given a test image x , reference image y i of age class i is selected via Eq.

Original: (13) from the training set.
Revised: (13) from the training set.

Original: In the default mode, for each age i , a single reference image is selected.
Revised: In the default mode, for each age i , a single reference image is selected.

Original: However, the top r most similar references can be selected and used for the estimation.
Revised: However, the top r most similar references can be selected and used for the estimation.

Original: We use a single reference because multiple references improve the estimation performance only negligibly.
Revised: We use a single reference because multiple references improve the estimation performance only negligibly.

Original: In Figure 15, the top three reference images are shown for each age from 16 to 53.
Revised: In Figure 18, the top three reference images are shown for each age from 16 to 53.

Original: In setting D, the two chains are divided by Africans and the others in general.
Revised: In setting D, the two clusters are divided by Africans and the others in general.

Original: However, we see that test and reference images tend to have the same gender, as well as the same race.
Revised: However, we see that test and reference images tend to have the same gender, as well as the same race.

Original: Furthermore, they have similar appearance, even when they have a big age difference.
Revised: Furthermore, they have similar appearance, even when they have a big age difference.

Original:  T e s ti m a g e
Revised: 30 T e s t

Original: R e f e r e n cei m a g e s                                                           T e s ti m a g e
Revised: R e f e r e n ce s                                                          26 T e s t

Original: R e f e r e n ce 
Revised: R e f e r e n ce s

Original: i m a g e s T e s ti m a g e
Revised: T e s t

Original: R e f e r e n ce i m a g e s [-
-]
Revised: R e f e r e n ce s                                                        27 T e s [-
-]t

Original: T e s t i m a g e R e f e r e n ce i m a g e s [-
-]
Revised: R e f e r e n ce s                    1                 39 T e s [-
-]t

Original: T e s t i m a g e R e f e r e n cei m a g e s                     T e s ti m a g e
Revised: R e f e r e n ce s                    1                 41 T e s t

Original: R e f e r e n ce 
Revised: R e f e r e n ce s

Original: i m a g e s D A ESTHETIC S CORE R EGRESSION – M ORE E XPERIMENTS AND D ETAILS D. I MPLEMENTATION D ETAILS
Revised: D A ESTHETIC S CORE R EGRESSION – M ORE E XPERIMENTS AND D ETAILS D. I MPLEMENTATION D ETAILS

Original: For aesthetic score regression, we implement a pairwise comparator based on EfﬁcientNetB4 (Tan & Le, 2019).
Revised: For aesthetic score regression, we implement a pairwise comparator based on EfﬁcientNetB4 (Tan & Le, 2019).

Original: The pairwise comparator has the same architecture as that for facial age estimation, except for the backbone network.
Revised: The pairwise comparator has the same architecture as that for facial age estimation, except for the backbone network.

Original: To initialize the backbone, we adopt the parameters pre-trained on the ILSVRC2012 dataset.
Revised: To initialize the backbone, we adopt the parameters pre-trained on the ILSVRC2012 dataset.

Original: We initialize the other layers using the Glorot normal method.
Revised: We initialize the other layers using the Glorot normal method.

Original: We update the network parameters using the Adam optimizer with a minibatch size of 16.
Revised: We update the network parameters using the Adam optimizer with a minibatch size of 16.

Original: We start with a learning rate of 10 − 4 and shrink it by a factor of 0.8 every 8000 steps.
Revised: We start with a learning rate of 10 − 4 and shrink it by a factor of 0.8 every 8000 steps.

Original: Training images are augmented by random horizontal ﬂipping.
Revised: Training images are augmented by random horizontal ﬂipping.

Original: We set τ = 0 .
Revised: We set τ = 0 .

Original: 15 for the ternary categorization in Eq.
Revised: 15 for the ternary categorization in Eq.

Original: D. C LUSTERING
Revised: D. C LUSTERING

Original: Notice that the AADB dataset contains images of diverse contents and styles.
Revised: Notice that the AADB dataset contains images of diverse contents and styles.

Original: Hence, when clustering with a small k , it is hard to observe the characteristics shared by images within each cluster, whereas k = 2 or 3 is sufﬁcient for facial age data.
Revised: Hence, when clustering with a small k , it is hard to observe the characteristics shared by images within each cluster, whereas k = 2 or 3 is sufﬁcient for facial age data.

Original: We empirically found that at least eight clusters are required ( k = 8 ) to partition the AADB dataset by meaningful criteria.
Revised: We empirically found that at least eight clusters are required ( k = 8 ) to partition the AADB dataset by meaningful criteria.

Original: Figure 16 provides more examples of clustering results at k = 8 .
Revised: Figure 19 provides more examples of clustering results at k = 8 .

Original: Figure 17 visualizes the feature space of AADB.
Revised: Figure 20 visualizes the feature space of AADB.

Original: Aesthetic scores are sorted along one direction, while chains are separated in the other orthogonal direction.
Revised: Aesthetic scores are sorted along one direction, while clusters are separated in the other orthogonal direction.

Original: In other words, the scores look like latitudes, while the chains appear to be separated by meridians (or lines of longitude).
Revised: In other words, the scores look like latitudes, while the clusters appear to be separated by meridians (or lines of longitude).

Original: As a point on the earth surface can be located by its latitude and longitude, an image is represented by its aesthetic score (order feature) and chain (identity feature).
Revised: As a point on the earth surface can be located by its latitude and longitude, an image is represented by its aesthetic score (order feature) and cluster (identity feature).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Chain Chain
Revised: Cluster Cluster

Original: Chain Chain
Revised: Cluster Cluster

Original: Chain Chain
Revised: Cluster Cluster

Original: Chain
Revised: Cluster

Original: Chain[0.1, 0.2) [0.2, 0.3) [0.3, 0.4) [0.4, 0.5) [0.5, 0.6) [0.6, 0.7) [0.7, 0.8) [0.8, 0.9) [0.9, 1.0] 
Revised: Cluster[0.1, 0.2) [0.2, 0.3) [0.3, 0.4) [0.4, 0.5) [0.5, 0.6) [0.6, 0.7) [0.7, 0.8) [0.8, 0.9) [0.9, 1.0] [Figure]

Original: Chain
Revised: Cluster

Original: Chain
Revised: [Figure]

Original: Chain Chain Chain
Revised: Cluster Cluster

Original: Chain Chain
Revised: Cluster Cluster

Original: D. R EFERENCE I MAGES
Revised: D. R EFERENCE I MAGES

Original: Figure 18 shows examples of reference images, which are used for the aesthetic score regression.
Revised: Figure 21 shows examples of reference images, which are used for the aesthetic score regression.

Original: Given a test image x , the reference image y i of aesthetic class i is selected by Eq.
Revised: Given a test image x , the reference image y i of aesthetic class i is selected by Eq.

Original: For the aesthetic score regression, we use a single reference image for each aesthetic class, as done in the facial age estimation.
Revised: For the aesthetic score regression, we use a single reference image for each aesthetic class, as done in the facial age estimation.

Original: Thus, 101 reference images are used in total.
Revised: Thus, 101 reference images are used in total.

Original: References Test 0.
Revised: References Test 0.

Original: References Test 0. References Test 0.
Revised: References Test 0. References Test 0.

Original: References Test 0.
Revised: References Test 0.

Original: [Figure]
Revised: [Figure]

Original: E HCI C LASSIFICATION – M ORE E XPERIMENTS AND D ETAILS E. I MPLEMENTATION D ETAILS
Revised: E HCI C LASSIFICATION – M ORE E XPERIMENTS AND D ETAILS E. I MPLEMENTATION D ETAILS

Original: For DRC-ORID for HCI classiﬁcation, we set all hyper-parameters in the same way as we do in
Revised: For DRC-ORID for HCI classiﬁcation, we set all hyper-parameters in the same way as we do in

Original: Appendix C.1.
Revised: Appendix C.1.

Original: We set τ = 1 for the ternary categorization of ordering relationship in Eq.
Revised: We set τ = 1 for the ternary categorization of ordering relationship in Eq.

Original: Note that there are ﬁve decade classes from 1 to 5.
Revised: Note that there are ﬁve decade classes from 1 to 5.

Original: E. C LUSTERING
Revised: E. C LUSTERING

Original: Figure 19 shows some sample images in the HCI dataset, which are ordered according to the decade classes.
Revised: Figure 22 shows some sample images in the HCI dataset, which are ordered according to the decade classes.

Original: Figure 20 shows more example HCI images grouped into four chains ( k = 4 ).
Revised: Figure 23 shows more example HCI images grouped into four clusters ( k = 4 ).

Original: 1930s 1940s 1950s 1960s 1970s
Revised: 1930s 1940s 1950s 1960s 1970s

Original: [Figure]
Revised: [Figure]

Original: E. R EFERENCE I MAGES
Revised: E. R EFERENCE I MAGES

Original: Figure 21 shows the ﬁve reference images for each of six test image examples.
Revised: Figure 24 shows the ﬁve reference images for each of six test image examples.

Original: Note that, given a test image, the reference images of similar contents, tones, or composition are selected from the ﬁve decade classes.
Revised: Note that, given a test image, the reference images of similar contents, tones, or composition are selected from the ﬁve decade classes.

Original: Test
Revised: Test

Original: References 1930s 1930s 1940s 1950s 1960s 1970s
Revised: References 1930s 1930s 1940s 1950s 1960s 1970s

Original: Test
Revised: Test

Original: References 1940s 1930s 1940s 1950s 1960s 1970s
Revised: References 1940s 1930s 1940s 1950s 1960s 1970s

Original: Test
Revised: Test

Original: References 1950s 1930s 1940s 1950s 1960s 1970s
Revised: References 1950s 1930s 1940s 1950s 1960s 1970s

Original: Test
Revised: Test

Original: References 1960s 1930s 1940s 1950s 1960s 1970s
Revised: References 1960s 1930s 1940s 1950s 1960s 1970s

Original: Test
Revised: Test

Original: References 1970s 1930s 1940s 1950s 1960s 1970s
Revised: References 1970s 1930s 1940s 1950s 1960s 1970s

Original: Test
Revised: Test

Original: References 1960s 1930s 1940s 1950s 1960s 1970s
Revised: References 1960s 1930s 1940s 1950s 1960s 1970s