Original: S EMI - RELAXED G ROMOV -W ASSERSTEIN DIVERGENCE WITH APPLICATIONS ON GRAPHS
Revised: S EMI - RELAXED G ROMOV -W ASSERSTEIN DIVERGENCE WITH APPLICATIONS ON GRAPHS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Comparing structured objects such as graphs is a fundamental operation involved in many learning tasks.
Revised: Comparing structured objects such as graphs is a fundamental operation involved in many learning tasks.

Original: To this end, the Gromov-Wasserstein (GW) distance, based on Optimal Transport (OT), has proven to be successful in handling the speciﬁc nature of the associated objects.
Revised: To this end, the Gromov-Wasserstein (GW) distance, based on Optimal Transport (OT), has proven to be successful in handling the speciﬁc nature of the associated objects.

Original: More speciﬁcally, through the nodes connectivity relations, GW operates on graphs, seen as probability measures over speciﬁc spaces.
Revised: More speciﬁcally, through the nodes connectivity relations, GW operates on graphs, seen as probability measures over speciﬁc spaces.

Original: At the core of OT is the idea of conservation of mass, which imposes a coupling between all the nodes from the two considered graphs.
Revised: At the core of OT is the idea of conservation of mass, which imposes a coupling between all the nodes from the two considered graphs.

Original: We argue in this paper that this property can be detrimental for tasks such as graph dictionary or partition learning, and we relax it by proposing a new semi-relaxed
Revised: We argue in this paper that this property can be detrimental for tasks such as graph dictionary or partition learning, and we relax it by proposing a new semi-relaxed

Original: Gromov-Wasserstein divergence.
Revised: Gromov-Wasserstein divergence.

Original: Aside from immediate computational beneﬁts, we discuss its properties, and show that it can lead to an efﬁcient graph dictionary learning algorithm.
Revised: Aside from immediate computational beneﬁts, we discuss its properties, and show that it can lead to an efﬁcient graph dictionary learning algorithm.

Original: We empirically demonstrate its relevance for complex tasks on graphs such as partitioning, clustering and completion.
Revised: We empirically demonstrate its relevance for complex tasks on graphs such as partitioning, clustering and completion.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One of the main challenges in machine learning (ML) is to design efﬁcient algorithms that are able to learn from structured data (Battaglia et al., 2018).
Revised: One of the main challenges in machine learning (ML) is to design efﬁcient algorithms that are able to learn from structured data (Battaglia et al., 2018).

Original: Learning from datasets containing such nonvectorial objects is a difﬁcult task that involves many areas of data analysis such as signal processing (Shuman et al., 2013), Bayesian and kernel methods on graphs (Ng et al., 2018; Kriege et al., 2020) or more recently geometric deep learning (Bronstein et al., 2017; 2021) and graph neural networks (Wu et al., 2020).
Revised: Learning from datasets containing such nonvectorial objects is a difﬁcult task that involves many areas of data analysis such as signal processing (Shuman et al., 2013), Bayesian and kernel methods on graphs (Ng et al., 2018; Kriege et al., 2020) or more recently geometric deep learning (Bronstein et al., 2017; 2021) and graph neural networks (Wu et al., 2020).

Original: In terms of applications, building algorithms that go beyond Euclidean data has led to many progresses, e.g. in image analysis (Harchaoui & Bach, 2007), brain connectivity (Ktena et al., 2017), social networks analysis (Yanardag & Vishwanathan, 2015) orfor protein structure prediction (Jumper et al., 2021).
Revised: In terms of applications, building algorithms that go beyond Euclidean data has led to many progresses, e.g. in image analysis (Harchaoui & Bach, 2007), brain connectivity (Ktena et al., 2017), social networks analysis (Yanardag & Vishwanathan, 2015) or protein structure prediction (Jumper et al., 2021).

Original: Learning from graph data is ubiquitous in a number of ML tasks.
Revised: Learning from graph data is ubiquitous in a number of ML tasks.

Original: A ﬁrst one is to learn graph representations that can encode the graph structure (a.k.a.
Revised: A ﬁrst one is to learn graph representations that can encode the graph structure (a.k.a.

Original: graph representation learning ).
Revised: graph representation learning ).

Original: In this domain, advances on graph neural networks provided end-to-end embeddings showing state-of-the-art results given sufﬁciently  labeled data (Ying et al., 2018; Morris et al., 2019; Gao & Ji, 2019; Wu et al., 2020).
Revised: In this domain, advances on graph neural networks led to state-of-the-art end-to-end embeddings, although requiring a sufﬁciently large amount of labeled data (Ying et al., 2018; Morris et al., 2019; Gao & Ji, 2019; Wu et al., 2020).

Original: Another problem is to ﬁnd a meaningful notion of similarity/distance between graphs.
Revised: Another task is to ﬁnd a meaningful notion of similarity/distance between graphs.

Original: A way to address this problem is by leveraging geometric or signal properties though the use of graph kernels (Kriege et al., 2020) or other embeddings that deal with graph isomorphisms (Zambon et al., 2020).
Revised: A way to address this problem is to leverage geometric or signal properties through the use of graph kernels (Kriege et al., 2020) or other embeddings accounting for graph isomorphisms (Zambon et al., 2020).

Original: Finally, it is often interesting to establish meaningful structural correspondences between the nodes ofthe different graphs, also known as graph matching (Zhou & De la Torre, 2012;Maron &
Revised: Finally, it is often of interest either to establish meaningful structural correspondences between the nodes of different graphs, also known as graph matching (Zhou & De la Torre, 2012;

Original:  Lipman, 2018; Bernard et al., 2018; Yan et al., 2016) or to ﬁnd a representative partition of the nodes of a graph,  astackled by graph partitioningor graph clustering methods (Chen et al., 2014; Nazi et al., 2019; Kawamoto et al., 2018; Bianchi et al., 2020).
Revised: Maron & Lipman, 2018; Bernard et al., 2018; Yan et al., 2016) or to ﬁnd a representative partition of the nodes of a graph, which we refer to as graph partitioning (Chen et al., 2014; Nazi et al., 2019; Kawamoto et al., 2018; Bianchi et al., 2020).

Original: Optimal Transport for structured data.
Revised: Optimal Transport for structured data.

Original: A novel approach to graph modeling, based on the theory of Optimal Transport (OT, Peyr´e & Cuturi, 2019),  has recently emerged from a series of works.
Revised: Based on the theory of Optimal Transport (OT, Peyr´e & Cuturi, 2019), a novel approach to graph modeling has recently emerged from a series of works.

Original: Informally, the goal of OT is to match two probability distributions under the constraint of mass conservation and in order to minimize a given matching cost.
Revised: Informally, the goal of OT is to match two probability distributions under the constraint of mass conservation and in order to minimize a given matching cost.

Original: OT originally tackles the problem of comparing probability distributions whose supports lie on the same metric space, by means of the so-called Wasserstein distance.
Revised: OT originally tackles the problem of comparing probability distributions whose supports lie on the same metric space, by means of the so-called Wasserstein distance.

Original: Extensions to graph data analysis were introduced by either embedding the graphs in a space endowed with Wasserstein geometry (Nikolentzos et al., 2017;
Revised: Extensions to graph data analysis were introduced by either embedding the graphs in a space endowed with Wasserstein geometry (Nikolentzos et al., 2017;

Original: [Figure]
Revised: [Figure]

Original: Togninalli et al., 2019; Petric Maretic et al., 2019) or relying on the Gromov-Wasserstein ( GW ) distance (M´emoli, 2011; Sturm, 2012).
Revised: Togninalli et al., 2019; Petric Maretic et al., 2019) or relying on the Gromov-Wasserstein ( GW ) distance (M´emoli, 2011; Sturm, 2012).

Original: The latter approach is a variant of the classical OT in which one aims at comparing probability distributions whose supports lie on different metric spaces by ﬁnding a matching of these distributions that is as close as possible to an isometry.
Revised: The latter approach is a variant of the classical OT in which one aims at comparing probability distributions whose supports lie on different metric spaces by ﬁnding a matching of these distributions being as close as possible to an isometry.

Original: By expressing graphs as probability measures over spaces speciﬁc to their topology, where the structure of a graph is represented through a pairwise distance/similarity matrix between its nodes, the GW distance computes both a soft assignment matrix between the nodes of the two graphs and a notion of distance between them (see the left part of Figure 1).
Revised: By expressing graphs as probability measures over spaces speciﬁc to their topology, where the structure of a graph is represented through a pairwise distance/similarity matrix between its nodes, the GW distance computes both a soft assignment matrix between the nodes of the two graphs and a notion of distance between them (see the left part of Figure 1).

Original: These properties have proven to be useful for a wide range of tasks such as graphs matching and partitioning (Xu et al., 2019a; Chowdhury & Needham, 2021; Vayer et al., 2019), estimation of nonparametric graph models ( graphons , Diaconis & Janson, 2007;
Revised: These properties have proven to be useful for a wide range of tasks such as graph matching and partitioning (Xu et al., 2019a; Chowdhury & Needham, 2021; Vayer et al., 2019), estimation of nonparametric graph models ( graphons , Diaconis & Janson, 2007;

Original: Xu et al., 2021) or for graphs dictionary learning (Vincent-Cuaz et al., 2021; Xu, 2020a).
Revised: Xu et al., 2021) or for graph dictionary learning (Vincent-Cuaz et al., 2021; Xu, 2020).

Original: GW has also been studied for directed graphs in Chowdhury & M´emoli (2019) andhas been extended to labeled graphs thanks to the Fused Gromov-Wasserstein (FGW) distance in (Vayer et al., 2020).
Revised: GW was also extended to directed graphs in Chowdhury & M´emoli (2019) and to labeled graphs via the Fused Gromov-Wasserstein (FGW) distance in Vayer et al.

Original: Despite those recent successes, applications of GW to graph modeling still have several limitations.
Revised: Despite those recent successes, applications of GW to graph modeling still have several limitations.

Original: First, ﬁnding the GW distance remains challenging, as it boils down to solving a difﬁcult nonconvex quadratic program (Peyr ´ e et al., 2016; Solomon et al., 2016; Vayer et al., 2019), which, in practice, limits the size of the graphs that can be processed.
Revised: First, ﬁnding the GW distance remains challenging, as it boils down to solving a difﬁcult nonconvex quadratic program (Peyr ´ e et al., 2016; Solomon et al., 2016; Vayer et al., 2019), which, in practice, limits the size of the graphs that can be processed.

Original: A second limit naturally emerges when a probability mass function is introduced over the set of the graph nodes.
Revised: A second limit naturally emerges when a probability mass function is introduced over the set of the graph nodes.

Original: The mass associated with a node refers to its relative importance and, without prior knowledge, each node is either assumed to share the same probability mass or to have one proportional to its degree.
Revised: The mass associated with a node refers to its relative importance and, without prior knowledge, each node is either assumed to share the same probability mass or to have one proportional to its degree.

Original: In this paper, we somehow argue that these choices can be suboptimal in several cases, and should be relaxed, with the additional beneﬁt of lowering the computational complexity.
Revised: In this paper, we somehow argue that these choices can be suboptimal in several cases, and should be relaxed, with the additional beneﬁt of lowering the computational complexity.

Original: As an illustration, consider Figure 1, where on the left image the GW matching is given between two graphs, with respectively two and three clusters, associated to uniform weights on the nodes.
Revised: As an illustration, consider Figure 1: on the left image, the GW matching is given between two graphs, with respectively two and three clusters, associated with uniform weights on the nodes.

Original: By relaxing the weight constraints over the second (middle image) or ﬁrst graph (right image) we obtain different matchings, that can better preserve the structure of the source graph by reweighing the target nodes and thus selecting a meaningful sub-graph.
Revised: By relaxing the weight constraints over the second (middle image) or ﬁrst graph (right image) we obtain different matchings, that can better preserve the structure of the source graph by reweighing the target nodes and thus selecting a meaningful subgraph.

Original: Contributions.
Revised: Contributions.

Original: We introduce a new  transport based divergence between graphs derived from the GW distance, that we call  the semi-relaxed Gromov-Wasserstein ( srGW ) divergence.
Revised: We introduce a new optimal transport based divergence between graphs derived from the GW distance. We call it the semi-relaxed Gromov-Wasserstein ( srGW ) divergence.

Original: After presenting its properties and motivating its use for ML applications, we propose an efﬁcient solver for the corresponding optimization problem which better ﬁts to modern parallel programming than exact solvers for GW.
Revised: After discussing its properties and motivating its use in ML applications, we propose an efﬁcient solver for the corresponding optimization problem. Our solver better ﬁts to modern parallel programming than exact solvers for GW do.

Original: We empirically demonstrate the relevance of our divergence for graph partitioning tasks, Dictionary Learning (DL), clustering of graphs and graph completion datasets.
Revised: We empirically demonstrate the relevance of our divergence for graph partitioning, Dictionary Learning (DL), clustering of graphs and graph completion tasks.

Original: With srGW , we recover SOTA performances at a signiﬁcantly lower computational cost compared to methods based on  GW .
Revised: With srGW , we recover SOTA performances at a signiﬁcantly lower computational cost compared to methods based on pure GW .

Original:  M ODELING GRAPHS WITH THE G ROMOV -W ASSERSTEIN DIVERGENCE
Revised:  M ODELING GRAPHS WITH THE G ROMOV -W ASSERSTEIN DIVERGENCE

Original: In this section we introduce more formally the GW distance and discuss two of its applications on graphs, namely graph partitioning and unsupervised graph representation learning.
Revised: In this section we introduce more formally the GW distance and discuss two of its applications on graphs, namely graph partitioning and unsupervised graph representation learning.

Original: In the following the probability simplex with N-bins is denoted as Σ N := { h ∈ R N + | (cid:80) i h i = 1 } .
Revised: In the following the probability simplex with N-bins is denoted as Σ N := { h ∈ R N + | (cid:80) i h i = 1 } .

Original: GW as graphs similarity.
Revised: GW as graphs similarity.

Original: In the OT context, a graph G with n nodes can be modeled as a couple ( C , h ) where C ∈ R n × n is a matrix encoding the connectivity between nodes and h ∈ Σ n is a histogram, referred here as distribution, modeling the relative importance of the nodes within the graph.
Revised: In the OT context, a graph G with n nodes can be modeled as a couple ( C , h ) where C ∈ R n × n is a matrix encoding the connectivity between nodes and h ∈ Σ n is a histogram, referred here as distribution, modeling the relative importance of the nodes within the graph.

Original: The matrix C can be arbitrarily chosen as the graph adjacency matrix, or any other matrix describing the relationships betweenthe nodes in the topology of the graph ( e.g.  shortest-path,
Revised: The matrix C can be arbitrarily chosen as the graph adjacency matrix, or any other matrix describing the relationships between nodes in the topology of the graph ( e.g. adjacency, shortest-path,

Original: Laplacian).
Revised: Laplacian).

Original: The distribution is often considered as uniform ( h = 1 n 1 n ) but can also convey prior knowledge e.g. using the normalized degree distribution (Xu et al., 2019a).
Revised: The distribution is often considered as uniform ( h = 1 n 1 n ) but can also convey prior knowledge e.g. using the normalized degree distribution (Xu et al., 2019a).

Original: Consider now two graphs
Revised: Consider now two graphs

Original: G =
Revised: G =

Original: ( C , h ) and G =
Revised: ( C , h ) and G =

Original: ( C , h ) , respectively with n and m nodes, potentially different ( n (cid:54) = m ).
Revised: ( C , h ) , respectively with n and m nodes, potentially different ( n (cid:54) = m ).

Original: The
Revised: The

Original: GW distance between G and G is deﬁned as the result of the following optimization problem:
Revised: GW distance between G and G is deﬁned as the result of the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: The optimal coupling T (cid:63) acts as a probabilistic matching of nodes which tends to associate pairs of nodes that have similar pairwise relations in C and C respectively, while preserving masses h and h through its marginals constraints.
Revised: The optimal coupling T (cid:63) acts as a probabilistic matching of nodes which tends to associate pairs of nodes that have similar pairwise relations in C and C respectively, while preserving masses h and h through its marginals constraints.

Original: GW deﬁnes a distance on the space of metric measure spaces  invariant to measure preserving isometriessuch as nodes permutation (M´emoli, 2011; Sturm, 2012).
Revised: GW deﬁnes a distance on the space of metric measure spaces (mm-spaces) invariant to measure preserving isometries (M´emoli, 2011; Sturm, 2012).

Revised: In the case of graphs, which corresponds to discrete mm-spaces, such invariant is a permutation of the nodes that preserves the structures and the weights of the graphs (Vayer et al., 2020).

Original: First applications of GW to ML on graphics problems (Peyr ´ e et al., 2016; Solomon et al., 2016) motivated further connections with graph partitioning.
Revised: First applications of GW to ML on graphics problems (Peyr´e et al., 2016; Solomon et al., 2016) motivated further connections with graph partitioning.

Original: In the GW sense, this paradigm is illustrated by ﬁnding an ideally partitioned graph G =
Revised: In the GW sense, this paradigm is illustrated by ﬁnding an ideally partitioned graph G =

Original: ( D , h ) of m clustrers whose structure is a diagonal matrix D  , representing the cluster’s connections, and its distribution h estimates the proportion of the nodes in each cluster (Xu et al., 2019a).
Revised: ( D , h ) of m clustrers whose structure is a diagonal matrix D ∈ R m × m , representing the cluster’s connections, and its distribution h estimates the proportion of the nodes in each cluster (Xu et al., 2019a).

Original: The OT plan between a graph G =
Revised: The OT plan between a graph G =

Original: ( C , h ) represented by its adjacency matrix C to this ideal graph can be used to recover G ’s clusters, if their number and proportions match the number and the weights of G ’s nodes.
Revised: ( C , h ) represented by its adjacency matrix C to this ideal graph can be used to recover G ’s clusters, if their number and proportions match the number and the weights of G ’s nodes.

Original: Xu et al.
Revised: Xu et al.

Original: (2019a) suggested to empirically reﬁne both graph distribution by choosing h based on power-law transformations of the degree distribution of G and to deduce h from h by linear interpolation.
Revised: (2019a) suggested to empirically reﬁne both graph distributions, by choosing h based on power-law transformations of the degree distribution of G and to deduce h from h by linear interpolation.

Original: Chowdhury &
Revised: Chowdhury &

Original: Needham (2020) proposed to use heat kernels from the Laplacian of G instead of its adjacency binary representation, and proved that the resulting GW partitioning is closely related to the well-known spectral clustering (Fiedler, 1973).
Revised: Needham (2021) proposed to use heat kernels from the Laplacian of G instead of its adjacency binary representation, and proved that the resulting GW partitioning is closely related to the well-known spectral clustering (Fiedler, 1973).

Original: The GW distance has been extended to graphs with node attributes (typically R d vectors) thanks to the Fused Gromov-Wasserstein distance (FGW)
Revised: The GW distance has been extended to graphs with node attributes (typically R d vectors) thanks to the Fused Gromov-Wasserstein distance (FGW)

Original: (Vayer et al., 2019).
Revised: (Vayer et al., 2019).

Original: In this context, an attributed graph G with n nodes is a tuple G =
Revised: In this context, an attributed graph G with n nodes is a tuple G =

Original: ( C , F , h ) where F ∈ R n × d is its matrix of nodes features.
Revised: ( C , F , h ) where F ∈ R n × d is its matrix of node features.

Original: FGW between two attributed graphs aims at ﬁnding an OT by minimizing a weighted sum of a GW cost on structures and a Wasserstein cost on features (balanced by a parameter α ∈ [0 , 1] ).
Revised: FGW between two attributed graphs aims at ﬁnding an OT by minimizing a weighted sum of a GW cost on structures and a Wasserstein cost on features (balanced by a parameter α ∈ [0 , 1] ).

Original: Most of the applications of GW can be naturally extended with FGW to attributed graphs.
Revised: Most of the applications of GW can be naturally extended with FGW to attributed graphs.

Original: GW for Unsupervised Graph Representation Learning.
Revised: GW for Unsupervised Graph Representation Learning.

Original: More recently, GW has been used as a data ﬁtting term for unsupervised graph representation learning by means of Dictionary Learning (DL) (Mairal et al., 2009; Schmitz et al., 2018).
Revised: More recently, GW has been used as a data ﬁtting term for unsupervised graph representation learning by means of Dictionary Learning (DL) (Mairal et al., 2009; Schmitz et al., 2018).

Original: While DL methods mainly focus on vectorial data (Ng et al., 2002; Cand ` es et al., 2011; Bobadilla et al., 2013), DL applied to graphs datasets consists in factorizing them as composition of graph primitives (or atoms) encoded as { ( C k , h k )
Revised: While DL methods mainly focus on vectorial data (Ng et al., 2002; Cand`es et al., 2011; Bobadilla et al., 2013), DL applied to graphs datasets consists in factorizing them as composition of graph primitives (or atoms) encoded as { ( C k , h k )

Original: } k ∈ [[ K ]] .
Revised: } k ∈ [[ K ]] .

Original: The ﬁrst approach proposed by (Xu, 2020a) consists in a non-linear DL based on entropic GW barycenters.
Revised: The ﬁrst approach proposed by (Xu, 2020) consists in a non-linear DL based on entropic GW barycenters.

Original: On the other hand, Vincent-Cuaz et al.
Revised: On the other hand, Vincent-Cuaz et al.

Original: (2021) proposed a linear DL method by modeling graphs as a linear combination of graph atoms thus reducing the computational cost.
Revised: (2021) proposed a linear DL method by modeling graphs as a linear combination of graph atoms thus reducing the computational cost.

Original: In both cases, the embedding problem that consists in the projection of any graph on the learned graph subspace requires solving a computationally intensive optimization problem.
Revised: In both cases, the embedding problem that consists in the projection of any graph on the learned graph subspace requires solving a computationally intensive optimization problem.

Original:  S EMI - RELAXED G ROMOV -W ASSERSTEIN DIVERGENCE 3. D EFINITION AND PROPERTIES
Revised:  S EMI - RELAXED G ROMOV -W ASSERSTEIN DIVERGENCE 3. D EFINITION AND PROPERTIES

Original: Given two observed graphs G =
Revised: Given two observed graphs G =

Original: ( C , h ) and G =
Revised: ( C , h ) and G =

Original: ( C , h ) of n and m nodes, we propose to ﬁnd a correspondence between them while relaxing the weights h on the second graph.
Revised: ( C , h ) of n and m nodes, we propose to ﬁnd a correspondence between them while relaxing the weights h on the second graph.

Original: To this end we introduce the semi-relaxed Gromov-Wasserstein divergence as :
Revised: To this end we introduce the semi-relaxed Gromov-Wasserstein divergence as :

Original: [Equation]
Revised: [Equation]

Original: This means that we search for a reweighing of the nodes of G leading to a graph with structure C with minimal GW distance from G .
Revised: This means that we search for a reweighing of the nodes of G leading to a graph with structure C with minimal GW distance from G .

Original: While the optimization problem (2) above might seems complex to solve, it is actually equivalent to a GW problem where the mass constraints on the second marginal of T are relaxed, reducing the problem to:[Equation] T
Revised: While the optimization problem (2) above might seem complex to solve, it is actually equivalent to a GW problem where the mass constraints on the second marginal of T are relaxed, reducing the problem to:

Original: [Equation]
Revised: [Equation]

Original: From an optimal coupling T (cid:63) of problem (3), the optimal weights h (cid:63) expressed in problem (2) can be recovered by computing T (cid:63) ’s second marginal, i.e h (cid:63)
Revised: From an optimal coupling T (cid:63) of problem (3), the optimal weights h (cid:63) expressed in problem (2) can be recovered by computing T (cid:63) ’s second marginal, i.e h (cid:63)

Original: =
Revised: =

Original: T (cid:63) (cid:62) 1 n .
Revised: T (cid:63) (cid:62) 1 n .

Original: We refer the reader interested in the proof of this equivalence to the annex. This reformulation with relaxed marginal has been investigated in the context of the Wasserstein distance (Rabin et al., 2014; Flamary et al., 2016)  but  never investigated for the GW distance.
Revised: This reformulation with relaxed marginal has been investigated in the context of the Wasserstein distance (Rabin et al., 2014; Flamary et al., 2016) and for relaxations of the GW problem in (Schmitzer & Schn¨orr, 2013) but was never investigated for the GW distance itself.

Original: To the best of our knowledge, the most similar related work is the Unbalanced GW S´ejourn´e et al.
Revised: To the best of our knowledge, the most similar related work is the Unbalanced GW S´ejourn´e et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: Chapel et al.
Revised: Chapel et al.

Original: (2019) where one could recover srGW with different weighting over the marginal relaxations ( ∞ on the ﬁrst marginal andon the second)
Revised: (2019) where one could recover srGW with different weighting over the marginal relaxations ( ∞ on the ﬁrst marginal andon the second)

Original: but this speciﬁc case was not discussed nor studied in these works.
Revised: but this speciﬁc case was not discussed nor studied in these works.

Original: A ﬁrst interesting property of srGW is that since h is optimized in the the simplex Σ m , its optimal value h (cid:63) can be sparse.
Revised: A ﬁrst interesting property of srGW is that since h is optimized in the the simplex Σ m , its optimal value h (cid:63) can be sparse.

Original: As a consequence, parts of the graph G can be omitted in the comparison, similarly to a partial matching.
Revised: As a consequence, parts of the graph G can be omitted in the comparison, similarly to a partial matching.

Original: This behavior is illustrated in the Figure 1, where two graphs with uniform distributions and structures C and C forming respectively 2 and 3 clusters are matched.
Revised: This behavior is illustrated in the Figure 1, where two graphs with uniform distributions and structures C and C forming respectively 2 and 3 clusters are matched.

Original: The
Revised: The

Original: GW matching (left) between both graphs forces nodes of different clusters from C to be transported on one of the three clusters of C , leading to a high GW cost where clusters are not preserved.
Revised: GW matching (left) between both graphs forces nodes of different clusters from C to be transported on one of the three clusters of C , leading to a high GW cost where clusters are not preserved.

Original: Whereas srGW can ﬁnd a reasonable approximation of the structure of the left graph either though transporting on only two clusters (middle) or ﬁnding a structure with 3 clusters in a subgraph of the target graph with two clusters (right).
Revised: Whereas srGW can ﬁnd a reasonable approximation of the structure of the left graph either though transporting on only two clusters (middle) or ﬁnding a structure with 3 clusters in a subgraph of the target graph with two clusters (right).

Original: A second natural observation resulting from the dependence of srGW 2 of only one input distributions is its asymmetry, i.e. srGW 2 ( C , h , C ) (cid:54) =
Revised: A second natural observation resulting from the dependence of srGW 2 of only one input distributions is its asymmetry, i.e. srGW 2 ( C , h , C ) (cid:54) =

Original: srGW 2 ( C , h , C ) .
Revised: srGW 2 ( C , h , C ) .

Original: Interestingly, srGW 2 shares similar properties than GW as described in the next proposition:
Revised: Interestingly, srGW 2 shares similar properties than GW as described in the next proposition:

Original: Proposition 1 Let C ∈ R n × n and C ∈ R m × m  and h ∈ Σ n with supp( h ) =
Revised: Proposition 1 Let C ∈ R n × n and C ∈ R m × m be distance matrices and h ∈ Σ n with supp( h ) =

Original: [[ n ]] .
Revised: [[ n ]] .

Original: Then srGW 2 ( C , h , C ) = 

Revised: Then srGW 2 ( C , h , C ) =

Original: iff there exists h ∈ Σ m such that GW 2 ( C , h , C , h ) = 0 .
Revised: iff there exists h ∈ Σ m with card(supp( h ))

Original: [-As a consequence, if C , C are distances srGW 2 ( h , C , C ) = 
 iff there exists T ∈ R n × m + with n ≤ m , with ﬁrst marginal h and whose second marginal h ∈ Σ m is such that (cid:12)(cid:12) supp( h ) (cid:12)(cid:12)-] =
Revised: [-As a consequence, if C , C are distances srGW 2 ( h , C , C ) = 
 iff there exists T ∈ R n × m + with n ≤ m , with ﬁrst marginal h and whose second marginal h ∈ Σ m is such that (cid:12)(cid:12) supp( h ) (cid:12)(cid:12)-] =

Original: n andthere is a bijection σ : supp( h ) →
Revised: n and a bijection σ : supp( h ) →

Original: [[ n ]] which satisﬁes:
Revised: [[ n ]] such that:

Original: [Equation]
Revised: [Equation]

Original: And:
Revised: And:

Original: [Equation]
Revised: [Equation]

Original: In other words srGW 2 ( C , h , C ) vanishes iff there exists a reweighting h (cid:63) ∈ Σ m of the nodes of the second graph which cancels the GW distance.
Revised: In other words srGW 2 ( C , h , C ) vanishes iff there exists a reweighing h (cid:63) ∈ Σ m of the nodes of the second graph which cancels the GW distance.

Original: When it is the case, the induced graphs ( C , h ) and ( C , h (cid:63) ) arethus isomorphic (M´emoli, 2011; Sturm, 2012).
Revised: When it is the case, the induced graphs ( C , h ) and ( C , h (cid:63) ) are isomorphic (M´emoli, 2011; Sturm, 2012).

Revised: We refer the reader interested in the proofs of the equivalence and the Proposition 1 to the annex (section 7.2).

Original: 3. O PTIMIZATION AND ALGORITHMS
Revised: 3. O PTIMIZATION AND ALGORITHMS

Original: In this section we discuss the computational aspects of the srGW divergence and propose an algorithm to solve the related optimization problem (3).
Revised: In this section we discuss the computational aspects of the srGW divergence and propose an algorithm to solve the related optimization problem (

Original: We also discuss variations resulting from entropic or/and sparse regularization of the initial quadratic problem.
Revised: We also discuss variations resulting from entropic or/and sparse regularization of the initial quadratic problem

Original: Solving for the semi-relaxed GW.
Revised: Solving for the semi-relaxed GW.

Original: The optimization problem related to the calculation of srGW 22 ( C , h , C ) is a non-convex quadratic program similar to the one of GW with the important difference that the linear constraints are independent.
Revised: The optimization problem related to the calculation of srGW 22 ( C , h , C ) is a non-convex quadratic program similar to the one of GW with the important difference that the linear constraints are independent.

Original: Consequently, we propose to solve (3) with a Conditional Gradient (CG) algorithm
Revised: Consequently, we propose to solve (3) with a Conditional Gradient (CG) algorithm

Original: (Jaggi, 2013) that can beneﬁt from those independent constraints and is known to converge to local stationary point on non-convex problems (Lacoste-Julien, 2016).
Revised: (Jaggi, 2013) that can beneﬁt from those independent constraints and is known to converge to local stationary point on non-convex problems (Lacoste-Julien, 2016).

Original: This algorithm, provided in Alg. 1, consists in solving at each iteration  a linearization (cid:104) X , G (cid:105) of the problem (3) where G is the gradient of the objective in (3).
Revised: This algorithm, provided in Alg. 1, consists in solving at each iteration ( t ) a linearization (cid:104) X , G (cid:105) of the problem (3) where G is the gradient of the objective in (3).

Original: 1: repeat 2:
Revised: 1: repeat 2:

Original: G ( t ) ← Compute gradient w.r.t T of (2).
Revised: G ( t ) ← Compute gradient w.r.t T of (2).

Original: Get direction X ( t ) problem: Solve independent subproblems on rows of G ( t ) .{+← min X 1 m = h X ≥ 
+}
Revised: X ( t ) {+← min X 1 m = h X ≥ 
+}

Original: T ( t [Table]
Revised: (cid:104) X , G ( t ) (cid:105) 4:

Original: 1) ← (1 − γ (cid:63) ) T ( t ) + γ (cid:63) X ( t ) with γ (cid:63) ∈ [0 , 1] from exact-line search.
Revised: T ( t +1) ← (1 − γ (cid:63) ) T ( t ) + γ (cid:63) X ( t ) with γ (cid:63) ∈ [0 , 1] from exact-line search.

Original: 5: until convergence.
Revised: 5: until convergence.

Original: The solution of the linearized problem provides a descent direction X (cid:63) − T , and a linesearch whose optimal step can be found in closed form to update the current solution T (Vayer et al., 2019).
Revised: The solution of the linearized problem provides a descent direction X (cid:63) − T , and a linesearch whose optimal step can be found in closed form to update the current solution T (Vayer et al., 2019).

Original: While both srGW and GW CG require at each iteration the computation of a gradient with complexity O ( n 2 m + m 2 n ) , the main source of efﬁciency of our algorithm comes from the computation of the descent directions.
Revised: While both srGW and GW CG require at each iteration the computation of a gradient with complexity O ( n 2 m + m 2 n ) , the main source of efﬁciency of our algorithm comes from the computation of the descent directions.

Original: In the
Revised: In the

Original: GW case, one needs to solve an exact linear OT problem, while in our case, one just needs to solve several independent linear problems under a simplex constraint.
Revised: GW case, one needs to solve an exact linear OT problem, while in our case, one just needs to solve several independent linear problems under a simplex constraint.

Original: This simply amounts to ﬁnding the minimum on the lines of G , within O ( mn ) operations, with potential parallelization with GPUs.
Revised: This simply amounts to ﬁnding the minimum on the rows of G as discussed in (Flamary et al., 2016, Equation (8)), within O ( mn ) operations, with potential parallelization with GPUs.

Original: Performance gains are illustrated in the experimental section.
Revised: Performance gains are illustrated in the experimental section.

Original: Entropic regularization.
Revised: Entropic regularization.

Original: Recent OT applications have shown the interest of adding an entropic regularization to the exact problem (3) , e.g. (Cuturi, 2013; Peyr´e et al., 2016).
Revised: Recent OT applications have shown the interest of adding an entropic regularization to the exact problem (3) , e.g. (Cuturi, 2013; Peyr´e et al., 2016).

Original: Entropic regularization makes the optimization problem smooth and more robust while densifying the optimal transport plan.
Revised: Entropic regularization makes the optimization problem smooth and more robust while densifying the optimal transport plan.

Original: Similar to (Peyr ´ e et al., 2016; Xu et al., 2019b; Xie et al., 2020), we can use a mirror-descent scheme w.r.t.
Revised: Similar to (Peyr ´ e et al., 2016; Xu et al., 2019b; Xie et al., 2020), we can use a mirror-descent scheme w.r.t.

Original: the Kullback-Leibler divergence (KL) to solve entropic regularized srGW.
Revised: the Kullback-Leibler divergence (KL) to solve entropic regularized srGW.

Original: The problem boils down to ﬁnd, at iteration t the couplingT ( t +1) ← arg min T 1 m = h ; T ≥
Revised: The problem boils down to ﬁnd, at iteration t the coupling

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 , G ( t ) denotes the gradient of the GW loss at iteration t and KL( T | T ( t ) ) is the KL divergence.
Revised: where (cid:15) > 0 , G ( t ) denotes the gradient of the GW loss at iteration t and KL( T | T ( t ) ) is the KL divergence.

Original: These updates can be solved using the following closed-form Bregman Projections :
Revised: These updates can be solved using the following closed-form Bregman Projections :

Original: [Equation]
Revised: [Equation]

Original: where K ( t ) = exp (cid:0) G ( t ) − (cid:15) log( T ( t ) )
Revised: where K ( t ) = exp (cid:0) G ( t ) − (cid:15) log( T ( t ) )

Original: (cid:1) ( exp and log are applied componentwise).
Revised: (cid:1) ( exp and log are applied componentwise).

Original: Unlike existing solvers for GW based on entropic regularization (Peyr ´ e et al., 2016; Solomon et al., 2016) that rely on a Sinkhorn’s matrix scaling algorithm on K ( t ) at each iteration , our problem requires only one (left) scaling of K ( t ) per iteration.
Revised: Unlike existing solvers for GW based on entropic regularization (Peyr ´ e et al., 2016; Solomon et al., 2016) that rely on a Sinkhorn’s matrix scaling algorithm on K ( t ) at each iteration , our problem requires only one (left) scaling of K ( t ) per iteration.

Original: We denote by srGW e ( C , h , C ) the result of this procedure.
Revised: We denote by srGW e ( C , h , C ) the result of this procedure.

Original: Sparsity promoting regularization.
Revised: Sparsity promoting regularization.

Original: As illustrated in Figure 1, srGW naturally leads to sparse solutions in ¯ h .
Revised: As illustrated in Figure 1, srGW naturally leads to sparse solutions in ¯ h .

Original: To compress even more the localization over a few nodes of C , we can promote the sparsity of h through a penalization Ω( T ) =
Revised: To compress even more the localization over a few nodes of C , we can promote the sparsity of h through a penalization Ω( T ) =

Original: (cid:80) j | h
Revised: (cid:80) j | h

Original: j | 1 / 2 which deﬁnes a concave function in the positive orthant R + .
Revised: j | 1 / 2 which deﬁnes a concave function in the positive orthant R + .

Original: We adapt the Majorisation-Minimisation (MM) of Courty et al.
Revised: We adapt the Majorisation-Minimisation (MM) of Courty et al.

Original: (2014) that was introduced to solve classical OT with a similar regularizer.
Revised: (2014) that was introduced to solve classical OT with a similar regularizer.

Original: The resulting algorithm, which relies on a local linearization of Ω( T ( t ) ) , consists in iteratively solving the srGW or srGW e problems, regularized at iteration t + 1 by a linear OT cost of components R ( t ) i,j =
Revised: The resulting algorithm, which relies on a local linearization of Ω( T ( t ) ) , consists in iteratively solving the srGW or srGW e problems, regularized at iteration t + 1 by a linear OT cost of components R ( t ) i,j =

Original: λ
Revised: λ

Original: g 2 ( h ( t ) j ) − 1 / 2 (see annex).
Revised: g 2 ( h ( t ) j ) − 1 / 2 .

Revised: Further detailed explanations on these algorithms can be found in section 7.3 of the annex.

Original:  L EARNING THE TARGET STRUCTURE
Revised:  L EARNING THE TARGET STRUCTURE

Original: A dataset of K graphs D = { ( C k , h k ) } k ∈ [[ K ]] is now considered, with heterogeneous structures and a variable number of nodes, denoted by ( n k ) k ∈ [[ K ]] .
Revised: A dataset of K graphs D = { ( C k , h k ) } k ∈ [[ K ]] is now considered, with heterogeneous structures and a variable number of nodes, denoted by ( n k ) k ∈ [[ K ]] .

Original: We introduce in the following a novel graph dictionary learning (DL) that has the peculiarity  to learn a unique dictionary element.
Revised: In the following, we introduce a novel graph dictionary learning (DL) whose peculiarity is to learn a unique dictionary element.

Original: Then we discuss how this dictionary can be used to perform graph completion, i.e. reconstruct the full structure of a graph from an observed sub-graph.
Revised: Then we discuss how this dictionary can be used to perform graph completion, i.e. reconstruct the full structure of a graph from an observed subgraph.

Original: 4. A NEW GRAPH DICTIONARY LEARNING
Revised: 4. A NEW GRAPH DICTIONARY LEARNING

Original: Semi-relaxed Gromov-Wasserstein embedding.
Revised: Semi-relaxed Gromov-Wasserstein embedding.

Original: We ﬁrst discuss how an observed graph can be represented in a dictionary with a unique element (or atom ) C ∈ R m × m , assumed to be known or designed through expert knowledge.
Revised: We ﬁrst discuss how an observed graph can be represented in a dictionary with a unique element (or atom ) C ∈ R m × m , assumed to be known or designed through expert knowledge.

Original: First, one computes srGW 22 ( C k , h k , C ) using the algorithmic solutions and regularization strategies detailed in Section 3.1.
Revised: First, one computes srGW 22 ( C k , h k , C ) using the algorithmic solutions and regularization strategies detailed in Section 3.1.

Original: From the resulting optimal coupling
Revised: From the resulting optimal coupling

Original: T (cid:63)k , the optimal weights for the target graph C are recovered with h (cid:63)k = T (cid:63) (cid:62) k 1 n k .
Revised: T (cid:63) k , the optimal weights for the target graph C are recovered with h (cid:63) k = T (cid:63) (cid:62) k 1 n k .

Original: The graph ( C , h 
Revised: The graph ( C , h (cid:63) k ) [Equation]

Original: (cid:63)k ) can be seen as a projection of ( C k , h k ) in the GW sense and the distribution on the nodes h (cid:63) k is an embedding of the graph ( C k , h k ) .
Revised: [Equation] can be seen as a projection of ( C k , h k ) in the GW sense and the distribution on the nodes h (cid:63)k is an embedding of the graph ( C k , h k ) .

Original: Representing a graph as a vector of weights h
Revised: Representing a graph as a vector of weights h

Original: (cid:63)k on a graph C is a new and elegant way to deﬁne a graph subspace that is orthogonal to other DL methods that either rely on GW barycenters Xu (2020a) or linear models Vincent-Cuaz et al.
Revised: (cid:63)k on a graph C is a new and elegant way to deﬁne a graph subspace that is orthogonal to other DL methods that either rely on GW barycenters (Xu, 2020) or linear representations (Vincent-Cuaz et al., 2021).

Original: One particularly interesting aspect of this modeling is that when h (cid:63)k  sparse, only a subpart or subgraph of C, corresponding to the nodes with non-zero weights in h (cid:63)k, is used in the model.
Revised: One particularly interesting aspect of this modeling is that when h (cid:63)k is sparse, only the subpart (or subgraph) of C corresponding to the nodes with non-zero weights in h (cid:63)k is used.

Original: srGW Dictionary Learning and online algorithm.
Revised: srGW Dictionary Learning and online algorithm.

Original: Given a dataset of graphs D = { [Equation] C k , h k ) } k ∈ [ K ] , we propose to learn the graph dictionary C ∈ R m × m from the observed data, by optimizing:
Revised: Given a dataset of graphs D = { ( C k , h k ) } k ∈ [ K ] , we propose to learn the graph dictionary C ∈ R m × m from the observed data, by optimizing:

Original: [Equation]
Revised: [Equation]

Original: This problem is denoted as srGW Dictionary Learning .
Revised: This problem is denoted as srGW Dictionary Learning .

Original: It can be seen as a srGW barycenter problem (Peyr ´ e et al., 2016) where we search for a graph structure C for which there exists node weights ( h (cid:63)k ) k ∈
Revised: It can be seen as a srGW barycenter problem (Peyr ´ e et al., 2016) where we look for a graph structure C for which there exists node weights ( h (cid:63)k ) k ∈

Original: [[ K ]] leading to a minimal GW error.
Revised: [[ K ]] leading to a minimal GW error.

Original: Interestingly this DL model requires only to solve the srGW problem to compute the embedding h (cid:63)k since it can be recovered from the solution
Revised: Interestingly this DL model requires only to solve the srGW problem to compute the embedding h (cid:63)k since it can be recovered from the solution

Original: T (cid:63)k of the problem.
Revised: T (cid:63)k of the problem.

Original: We solve this non-convex optimization problem above with an online algorithm similar to the one ﬁrst proposed in (Mairal et al., 2009) for vectorial data and adapted by Vincent-Cuaz et al.
Revised: We solve this non-convex optimization problem above with an online algorithm similar to the one ﬁrst proposed in Mairal et al. (2009) for vectorial data and adapted by Vincent-Cuaz et al.

Original: (2021) for graphs data.
Revised: (2021) for graph data.

Original: The core of the stochastic algorithm is depicted in Alg. 2.
Revised: The core of the stochastic algorithm is depicted in Alg. 2.

Original: The main idea is to use batches of graphs to independently solve the embedding problems (via Alg.1), then compute estimates of the gradient (cid:101) ∇ C with respect to C on each batch B .
Revised: The main idea is to use batches of graphs to independently solve the embedding problems (via Alg.1), then compute estimates of the gradient (cid:101) ∇ C with respect to C on each batch B .

Original: Finally a projected gradient on the set S of symmetric non-negative matrices is performed to update C .
Revised: Finally a projected gradient on the set S of symmetric non-negative matrices is performed to update C .

Original: In practice we use Adam optimizer (Kingma & Ba, 2015) in all our experiments.
Revised: In practice we use Adam optimizer (Kingma & Ba, 2015) in all our experiments.

Original: The complexity of this stochastic algorithm is mostly bounded by computing the gradients, which can be done in O ( n 2 k m + m 2 n k ) (see Section 3.2).
Revised: The complexity of this stochastic algorithm is mostly bounded by computing the gradients, which can be done in O ( n 2 k m + m 2 n k ) (see Section 3.2).

Original: Hence, in a factorization context i.e. max k ( n k ) >> m , the overall learning procedure has a quadratic complexity w.r.t.
Revised: Hence, in a factorization context i.e. max k ( n k ) >> m , the overall learning procedure has a quadratic complexity w.r.t.

Original: the maximum graphs size.
Revised: the maximum graphs size.

Original: Since the embedding h (cid:63)k is a by-product of computing the different srGW, we do not need an iterative solver to estimate it.
Revised: Since the embedding h (cid:63)k is a by-product of computing the different srGW, we do not need an iterative solver to estimate it.

Original: Consequently, it leads to a speed up on CPU of 2 to 3 orders of magnitude compared to our main competitors (see
Revised: Consequently, it leads to a speed up on CPU of 2 to 3 orders of magnitude compared to our main competitors (see

Original: Section 5.3) which DL methods require such iterative scheme.
Revised: Section 5.3) whose DL methods, instead, require such iterative scheme.

Original: 4. DL- BASED MODEL FOR GRAPHS COMPLETION
Revised: 4. DL- BASED MODEL FOR GRAPHS COMPLETION

Original: The structure C estimated on the dataset D can be used to infer/complete a new graph from the dataset being only partially observed.
Revised: The structure C estimated on the dataset D can be used to infer/complete a new graph from the dataset that is only partially observed.

Original: In this setting, we aim at recovering the full structure C ∈ R n × n while only a subset of relations between n obs < n nodes is observed, denoted as C obs ∈ R n obs × n obs .
Revised: In this setting, we aim at recovering the full structure C ∈ R n × n while only a subset of relations between n obs < n nodes is observed, denoted as C obs ∈ R n obs × n obs .

Original: This amounts to solve:
Revised: This amounts to solving:

Original: [Equation]

Original: [Equation]

Original: C , h , C (cid:17) , where (cid:101) C

Original: = (cid:34)

Original: C obs ..

Original: [Equation]
Revised: [Equation]

Original: C

Original: [Equation] (cid:35) [Equation] (9) where only the n 2 − n 2 obs coefﬁcients collected into C imp are optimized (and thus imputed).
Revised: where only the n 2 − n 2 obs coefﬁcients collected into C imp are optimized (and thus imputed).

Original: We solve the optimization problem above by a classical projected gradient descent thanks to the Envelope
Revised: We solve the optimization problem above by a classical projected gradient descent.

Original: Theorem. Indeed the gradient of srGW w.r.t C imp can be obtained as the gradient of the srGW cost function evaluated at a ﬁxed optimal coupling T (cid:63) .
Revised: At each iteration we ﬁnd an optimal coupling T (cid:63) of srGW that is used to calculate the gradient of srGW w.r.t C imp . The latter is obtained as the gradient of the srGW cost function evaluated at the ﬁxed optimal coupling T (cid:63) [Figure]

Revised: by using the Envelope Theorem (Bonnans & Shapiro, 2000).

Original: The projection step is here to enforce known properties of C , such as positivity and symmetry.
Revised: The projection step is here to enforce known properties of C , such as positivity and symmetry.

Original: In practice the estimated C imp will have continuous[Figure] values, so one has to apply a thresholding (with value 0 . 5 ) on C imp to recover a binary adjacency matrix.
Revised: In practice the estimated C imp will have continuous values, so one has to apply a thresholding (with value 0 . 5 ) on C imp to recover a binary adjacency matrix.

Original: The method can be easily extended to labeled graphs by also optimizing the node features of non-observed nodes.
Revised: The method can be easily extended to labeled graphs by also optimizing the node features of non-observed nodes.

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: This section illustrates the behavior of srGW on graph partitioning (a.k.a. nodes clustering), clustering of graphs and graph completion.
Revised: This section illustrates the behavior of srGW on graph partitioning (a.k.a. nodes clustering), clustering of graphs and graph completion.

Original: All the Python implementations in the experiments will be released on Github.
Revised: All the Python implementations in the experiments will be released on Github.

Original: For all experiments we provide more details about validation grids and initialization in the annex.
Revised: For all experiments we provide e.g. validation grids, initializations and complementary metrics in the annex (see sections: partitioning 7.5.1,clustering 7.5.2, completion 7.5.3).

Original: 5. G RAPH PARTITIONING
Revised: 5. G RAPH PARTITIONING

Original: As discussed in Section 2, it is possible to achieve graph partitioning via OT by estimating a GW matching between the graph to partition G =
Revised: As discussed in Section 2, it is possible to achieve graph partitioning via OT by estimating a GW matching between the graph to partition G =

Original: ( C , h ) ( n nodes) and a smaller graph G =
Revised: ( C , h ) ( n nodes) and a smaller graph G =

Original: ( I q , h ) , with q (cid:28) n nodes.
Revised: ( I q , h ) , with q (cid:28) n nodes.

Original: The atom I q is set as the identity matrix to enforce the emergence of densely connected groups (i.e. communities ).
Revised: The atom I q is set as the identity matrix to enforce the emergence of densely connected groups (i.e. communities ).

Original: The distributions h estimates the proportion of nodes in each cluster.
Revised: The distribution h estimates the proportion of nodes in each cluster.

Original: We recall that h must be given to compute the GW distance, whereas it is estimated with srGW.
Revised: We recall that h must be given to compute the GW distance, whereas it is estimated with srGW.

Original: All partitioning performances are measured by Adjusted Mutual Information (AMI, Vinh et al., 2010).
Revised: All partitioning performances are measured by Adjusted Mutual Information (AMI, Vinh et al., 2010).

Original: Simulated data.
Revised: Simulated data.

Original: In Figure 2, we illustrate the behavior of GW and srGW partitioning on a toy graph, simulated according to SBM (Holland et al., 1983) with 3 clusters of different proportions.
Revised: In Figure 2, we illustrate the behavior of GW and srGW partitioning on a toy graph, simulated according to SBM (Holland et al., 1983) with 3 clusters of different proportions.

Original: We see that miss-classiﬁcation occurs either when the proportions h do not ﬁt the true ones or when q (cid:54) = 3 .
Revised: We see that miss-classiﬁcation occurs either when the proportions h do not ﬁt the true ones or when q (cid:54) = 3 .

Original: On the other hand, clustering from srGW can simultaneously recover any cluster proportions (since it estimates them) and  select the actual number of clusters, using the sparse properties of h 
Revised: On the other hand, clustering from srGW can simultaneously recover any cluster proportions (since it estimates them) and can select the actual number of clusters, using the sparse properties of h [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Real-world datasets.
Revised: Real-world datasets.

Original: In order to benchmark the srGW partitioning on real (directed and undirected) graphs, we consider 4 datasets:
Revised: In order to benchmark the srGW partitioning on real (directed and undirected) graphs, we consider 4 datasets (details provided in Table 4 of the annex):

Original: a Wikipedia hyperlink network (Yang & Leskovec, 2015); a directed graph of email interactions between departments of a European research institute (Yin et al., 2017); 
Revised: a Wikipedia hyperlink network (Yang & Leskovec, 2015); a directed graph of email interactions between departments of a European research institute (Yin et al., 2017); an Amazon product network (Yang & Leskovec,

Original: an Amazon product network (Yang & Leskovec, 2015); a network of interactions between indian villages (Banerjee et al., 2013).
Revised: ); a network of interactions between indian villages (Banerjee et al., 2013)

Original: For the directed graphs, we adopt the symmetrization procedure described in Chowdhury & Needham (2021).
Revised: For the directed graphs, we adopt the symmetrization procedure described in Chowdhury & Needham (2021).

Original: Our main competitors are the two GW based partitioning methods proposed by Xu et al.
Revised: Our main competitors are the two GW based partitioning methods proposed by Xu et al.

Original: (2019b)
Revised: (2019b)

Original: and Chowdhury & Needham (2021).
Revised: and Chowdhury & Needham (2021).

Original: The former (GWL) relies on adjacency matrices, the latter (SpecGWL) adopts heat kernels on the graph normalized laplacians (SpecGWL).

Original: TheGW
Revised: The

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: former (GWL) relies on adjacency matrices, the latter (SpecGWL) adopts heat kernels on the graph normalized laplacians (SpecGWL).

Original:  solver of Flamary et al.
Revised: The GW solver of Flamary et al.

Original: (2021) was used for these methods.
Revised: (2021) was used for these methods.

Original: For fairness, we also consider these two representations for srGW partitioning (namely srGW and srSpecGW).
Revised: For fairness, we also consider these two representations for srGW partitioning (namely srGW and srSpecGW).

Original: Finally, three competing methods specialized in graph partitioning are also considered: FastGreedy (Clauset et al., 2004), 
Revised: Finally, three competing methods specialized in graph partitioning are also considered: FastGreedy (Clauset et al., 2004), Louvain (with validation of its resolution parameter,

Original: Louvain (with validation of its resolution parameter, Blondel et al., 2008) and Infomap (Rosvall & Bergstrom, 2008).
Revised: Blondel et al., 2008) and Infomap (Rosvall & Bergstrom, 2008).

Original: The graph partitioning results are reported in Table 1.
Revised: The graph partitioning results are reported in Table 1.

Original: Our method srGW always outperforms the GW based approaches and on this application the entropic regularization seems to improve the performance.
Revised: Our method srGW always outperforms the GW based approaches and on this application the entropic regularization seems to improve the performance.

Original: We want to stress that our general purpose divergence srGW outperforms methods that have been speciﬁcally designed for nodes clustering tasks on 3 out of 6 datasets .
Revised: We want to stress that our general purpose divergence srGW outperforms methods that have been speciﬁcally designed for nodes clustering tasks on 3 out of 6 datasets.

Original: 5. C LUSTERING OF GRAPHS DATASETS
Revised: 5. C LUSTERING OF GRAPHS DATASETS

Original: Datasets and methods.
Revised: Datasets and methods.

Original: We now show how the embeddings ( h (cid:63)k ) k ∈ [[ K ]] provided by the srGW
Revised: We now show how the embeddings ( h (cid:63)k ) k ∈ [[ K ]] provided by the srGW

Original: Dictionary Learning can be particularly useful for the task of graphs clustering.
Revised: Dictionary Learning can be particularly useful for the task of graphs clustering.

Original: We considered here three types of datasets: i) social networks from IMDB-B and IMDB-M (Yanardag & Vishwanathan, 2015); ii) graphs with discrete features representing chemical compounds from MUTAG (Debnath et al., 1991) and cuneiform signs from PTC-MR (Krichene et al., 2015); iii) graphs with continuous features, namely BZR, COX2 (Sutherland et al., 2003) andPROTEINS, ENZYMES (Borgwardt &
Revised: We considered here three types of datasets (details provided in Table 9 of the annex): i) social networks from IMDB-B and IMDB-M (Yanardag & Vishwanathan, 2015); ii) graphs with discrete features representing chemical compounds from MUTAG (Debnath et al., 1991) and cuneiform signs from PTC-MR (Krichene et al., 2015); iii) graphs with continuous features, namely BZR, COX2 (Sutherland et al., 2003) and

Original:  Kriegel, 2005).
Revised: PROTEINS, ENZYMES (Borgwardt & Kriegel, 2005).

Original: Our main competitors are the following OT-based SOTA models: i) GDL (VincentCuaz et al., 2021) and its regularized version, namely GDL λ ; ii) GWF (Xu, 2020a), with both ﬁxed (GWF-f) and random (GWF-r, default setting for the method) atom size; iii) GW kmeans (GW-k), a k-means equipped with GW distances and barycenters (Peyr´e et al., 2016).
Revised: Our main competitors are the following OT-based SOTA models: i) GDL (Vincent-Cuaz et al., 2021) and its regularized version, namely GDL λ ; ii) GWF (Xu, 2020), with both ﬁxed (GWF-f) and random (GWF-r, default setting for the method) atom size; iii) GW kmeans (GW-k), a k-means equipped with GW distances and barycenters (Peyr´e et al., 2016).

Original: Experimental settings.
Revised: Experimental settings.

Original: For all experiments we follow the benchmark proposed in Vincent-Cuaz et al.
Revised: For all experiments we follow the benchmark proposed in Vincent-Cuaz et al.

Original: The clustering performances are measured by means of Rand Index (RI, Rand, 1971).
Revised: The clustering performances are measured by means of Rand Index (RI, Rand, 1971).

Original: The standard Euclidean distance is used to implement k-means over srGW and GWFs embeddings, but we use for GDL the dedicated Mahalanobis distance as described in Vincent-Cuaz et al.
Revised: The standard Euclidean distance is used to implement k-means over srGW and GWFs embeddings, but we use for GDL the dedicated Mahalanobis distance as described in Vincent-Cuaz et al.

Original: GW-k does not use any embedding since it directly computes (a GW) k-means over the input graphs.
Revised: GW-k does not use any embedding since it directly computes (a GW) k-means over the input graphs.

Original: For each parameter conﬁguration (number of atoms, number of nodes and regularization parameter) we run each experiment ﬁve times, independently.
Revised: For each parameter conﬁguration (number of atoms, number of nodes and regularization parameter, detailed in section 7.5.2) we run each experiment ﬁve times, independently.

Original: The mean RI over the ﬁve runs is computed and the dictionary conﬁguration leading to the highest RI for each method is reported.
Revised: The mean RI over the ﬁve runs is computed and the dictionary conﬁguration leading to the highest RI for each method is reported.

Original: Results and discussion.
Revised: Results and discussion.

Original: Clustering performances and running times are reported in Tables 2 and 3, respectively.
Revised: Clustering performances and running times are reported in Tables 2 and 3, respectively.

Original: All variants of srGW DL are at least comparable with the SOTA GW based methods.
Revised: All variants of srGW DL are at least comparable with the SOTA GW based methods.

Original: Remarkably, the sparsity promoting variants always outperform other methods.
Revised: Remarkably, the sparsity promoting variants always outperform other methods.

Original: Notably Table [Figure] shows embedding computation times of the order of the millisecond for srGW, two order of magnitude faster than the competitors.
Revised: Notably Tableshows embedding computation times of the order of the millisecond for srGW, two order of magnitude faster than the competitors.

Original: 5. G RAPHS COMPLETION
Revised: 5. G RAPHS COMPLETION

Original: Finally, we present graph completion results on the real world datasets IMDB-B and MUTAG, using the approach proposed in 4.2.
Revised: Finally, we present graph completion results on the real world datasets IMDB-B and MUTAG, using the approach proposed in 4.2.

Original: Since this completion problem has never been investigated by existing
Revised: Since this completion problem has never been investigated by existing

Original: GW graph methods, we adapted the learning procedure used for srGW to GDL (Vincent-Cuaz et al., 2021). 
Revised: GW graph methods, we adapted the learning procedure used for srGW to GDL (Vincent-Cuaz et al., 2021). [Table] [Figure]

Original: Experimental setting.
Revised: Experimental setting.

Original: Since the datasets do not explicitly contain graphs with missing nodes, we proceed as follow: ﬁrst we split the dataset into a training dataset ( D train ) used to learn the dictionary and a test dataset ( D test ) reserved for the completion tasks.
Revised: Since the datasets do not explicitly contain graphs with missing nodes, we proceed as follow: ﬁrst we split the dataset into a training dataset ( D train ) used to learn the dictionary and a test dataset ( D test ) reserved for the completion tasks.

Original: For each graph of C ∈ D test , we created incomplete graphs C obs by independently removing 10% and 20% of their nodes, uniformly at random.
Revised: For each graph of C ∈ D test , we created incomplete graphs C obs by independently removing 10% and 20% of their nodes, uniformly at random.

Original: The partially observed graphs are then reconstructed using the procedure described in
Revised: The partially observed graphs are then reconstructed using the procedure described in

Original: Section 4.2 and the average performance of each method is computed for 5 different dataset splits.
Revised: Section 4.2 and the average performance of each method is computed for 5 different dataset splits.

Original: Results.
Revised: Results.

Original: The graph completion results are reported in Figure 3.
Revised: The graph completion results are reported in Figure 3.

Original: Our srGW dictionary learning outperforms GDL consistently, when enough data is available to learn the atoms.
Revised: Our srGW dictionary learning outperforms GDL consistently, when enough data is available to learn the atoms.

Original: When the proportion of train/test data varies, we can see that the linear GDL model that maintains the marginal constraints tends to be less sensitive to the scarcity of data.
Revised: When the proportion of train/test data varies, we can see that the linear GDL model that maintains the marginal constraints tends to be less sensitive to the scarcity of data.

Original: This can come form the fact that srGW is more ﬂexible thanks to the optimization of h but can slightly overﬁt when few data is available.
Revised: This can come form the fact that srGW is more ﬂexible thanks to the optimization of h but can slightly overﬁt when few data is available.

Original: Sparsity promoting regularization can clearly compensate this overﬁtting and systematically leads to the best completion performances (high accuracy, low Means Square Error).
Revised: Sparsity promoting regularization can clearly compensate this overﬁtting and systematically leads to the best completion performances (high accuracy, low Means Square Error).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce a new transport based divergence between structured data by relaxing the mass constraint on the second distribution of the Gromov-Wasserstein problem.
Revised: We introduce a new transport based divergence between structured data by relaxing the mass constraint on the second distribution of the Gromov-Wasserstein problem.

Original: After designing efﬁcient solvers to estimate this divergence, called the semi-relaxed Gromov-Wasserstein (srGW), we suggest to learn a unique structure to describe a dataset of graphs in the srGW sense.
Revised: After designing efﬁcient solvers to estimate this divergence, called the semi-relaxed Gromov-Wasserstein (srGW), we suggest to learn a unique structure to describe a dataset of graphs in the srGW sense.

Original: This novel modeling can be seen as a Dictionary Learning approach where graphs are embedded as a subgraph of a single atom.
Revised: This novel modeling can be seen as a Dictionary Learning approach where graphs are embedded as a subgraph of a single atom.

Original: Numerical experiments highlight the interest of our methods for graph partitioning, and unsupervised representation learning whose evaluation is conducted through clustering and completion of graphs.
Revised: Numerical experiments highlight the interest of our methods for graph partitioning, and unsupervised representation learning whose evaluation is conducted through clustering and completion of graphs.

Original: We believe that this new divergence will unlock the potential of GW for graphs with unbalanced proportions of nodes.
Revised: We believe that this new divergence will unlock the potential of GW for graphs with unbalanced proportions of nodes.

Original: The associated fast numerical solvers allow to handle large size graph datasets, which was not possible with current GW solvers.
Revised: The associated fast numerical solvers allow to handle large size graph datasets, which was not possible with current GW solvers.

Original: One interesting future research direction includes an analysis of srGW to perform parameters estimation of stochastic block models.
Revised: One interesting future research direction includes an analysis of srGW to perform parameters estimation of stochastic block models.

Original: Also, as relaxing the second marginal constraint in the original optimization problem gives more degrees of freedom to the underlying problem, one can expect dedicated regularization schemes, over e.g. the level of sparsity of h , to address a variety of application needs.
Revised: Also, as relaxing the second marginal constraint in the original optimization problem gives more degrees of freedom to the underlying problem, one can expect dedicated regularization schemes, over e.g. the level of sparsity of h , to address a variety of application needs.

Original: Finally, our method can be seen as a special relaxation of the subgraph isomorphism problem.
Revised: Finally, our method can be seen as a special relaxation of the subgraph isomorphism problem.

Original: It remains to be understood theoretically in which sense this relaxation holds.
Revised: It remains to be understood theoretically in which sense this relaxation holds.

Original:  A PPENDIX 7. N OTATIONS & DEFINITIONS
Revised:  A PPENDIX 7. N OTATIONS & DEFINITIONS

Original: In this section we recall the notations used in the rest of the appendix.
Revised: In this section we recall the notations used in the rest of the appendix.

Original: Notations.
Revised: Notations.

Original: For a vector h ∈ R m we deﬁne its support as supp( h ) =
Revised: For a vector h ∈ R m we deﬁne its support as supp( h ) =

Original: { i ∈ [[ m ]] | h
Revised: { i ∈ [[ m ]] | h

Original: i (cid:54) = 0 } .
Revised: i (cid:54) = 0 } .

Original: Note that if h ∈ Σ
Revised: Note that if h ∈ Σ

Original: m we have supp( h ) =
Revised: m we have supp( h ) =

Original: { i ∈ [[ m ]] | h i > 0 } .
Revised: { i ∈ [[ m ]] | h i > 0 } .

Original: The cardinal of a discrete set A is denoted | A | .
Revised: The cardinal of a discrete set A is denoted | A | .

Original: For a 4-D tensor L =
Revised: For a 4-D tensor L =

Original: ( L ijkl ) ijkl we denote ⊗ the tensor-matrix multiplication such that for a given matrix M , L ⊗ M is the matrix with entries ( (cid:80) kl L ijkl M kl ) ij .
Revised: ( L ijkl ) ijkl we denote ⊗ the tensor-matrix multiplication such that for a given matrix M , L ⊗ M is the matrix with entries ( (cid:80) kl L ijkl M kl ) ij .

Original: Cosntraints on couplings.
Revised: Cosntraints on couplings.

Original: We introduce U ( h , h ) the set of all admissible couplings between h and h , i.e the set
Revised: We introduce U ( h , h ) the set of all admissible couplings between h and h , i.e the set

Original: U ( h , h ) : = { T ∈ R n × m + | T 1 m = h , T (cid:62) 1 n = h } .
Revised: [Equation]

Original: We also introduce for any h ∈ Σ n and m ∈ N ∗ , U ( h , m ) the set of all admissible couplings between h and any histogram of Σ m , i.e the set
Revised: We also introduce for any h ∈ Σ n and m ∈ N ∗ , U ( h , m ) the set of all admissible couplings between h and any histogram of Σ m , i.e the set

Original: U ( h , m ) := { T ∈ R n × m + | T 1 m = h } , such that ∀ T ∈
Revised: [Equation]

Original:  U ( h , m ) , the second marginal h (= T (cid:62) 1 n ) belongs to Σ m .
Revised: such that ∀ T ∈ U ( h , m ) , the second marginal h (= T (cid:62) 1 n ) belongs to Σ m .

Original: Gromov-Wassrestein distance.
Revised: Gromov-Wasserstein distance.

Original: For any q ≥ 1 , the Gromov-Wasserstein distance of order q between ( C , h ) and ( C , h ) is deﬁned by:
Revised: For any q ≥ 1 , the Gromov-Wasserstein distance of order q between ( C , h ) and ( C , h ) is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Note that this is equivalent to
Revised: Note that this is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where L q ( C , C ) is the 4-D tensor
Revised: where L q ( C , C ) is the 4-D tensor

Original: such that L q ( C , C ) = (cid:0) ( C ij − C kl ) q (cid:1) ijkl .
Revised: such that L q ( C , C ) = (cid:0) ( C ij − C kl ) q (cid:1) ijkl .

Original: Semi-relaxed Gromov-Wasserstein divergence.
Revised: Semi-relaxed Gromov-Wasserstein divergence.

Original: The semi-relaxed Gromov-Wasserstein divergence of order q satisﬁes the following equation:
Revised: The semi-relaxed Gromov-Wasserstein divergence of order q satisﬁes the following equation:

Original: [Equation]
Revised: [Equation]

Original: or equivalently
Revised: or equivalently

Original: [Equation]
Revised: [Equation]

Original: Fused Gromov-Wasserstein distance.
Revised: Fused Gromov-Wasserstein distance.

Original: For attributed graphs i.e with nodes features, we use the
Revised: For attributed graphs i.e with nodes features, we use the

Original: Fused Gromov-Wasserstein distance (Vayer et al., 2019).
Revised: Fused Gromov-Wasserstein distance (Vayer et al., 2019).

Original: Consider two attributed graphs G =
Revised: Consider two attributed graphs G =

Original: ( C , F , h ) and G = (cid:0) C , F , h (cid:1) where F = ( F i ) i ∈
Revised: ( C , F , h ) and G = (cid:0) C , F , h (cid:1) where F = ( F i ) i ∈

Original: [[ n ]] ∈ R n × d and F = ( F j ) j ∈ [[ m ]] ∈ R m × d are their respective matrix of features which rows .
Revised: [[ n ]] ∈ R n × d and F = ( F j ) j ∈ [[ m ]] ∈ R m × d are their respective matrix of features which rows .

Original: Given a trade-off parameter between structures and features denoted α ∈ [0 , 1] , the FGW qq,α distance is deﬁned as the result of the following optimization problem
Revised: Given a trade-off parameter between structures and features denoted α ∈ [0 , 1] , the FGW qq,α distance is deﬁned as the result of the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to
Revised: which is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where M q ( F , F ) is a matrix with entries M q ( F , F ) ij = (cid:107) F i − F j (cid:107) qq .
Revised: where M q ( F , F ) is a matrix with entries M q ( F , F ) ij = (cid:107) F i − F j (cid:107) qq .

Original: Semi-relaxed Gromov-Wasserstein distance.
Revised: Semi-relaxed Gromov-Wasserstein distance.

Original: Then in a similar way than for the GromovWasserstein distance, the semi-relaxed Fused Gromov-Wasserstein of order q is deﬁned for any trade-off parameter α ∈ [0 , 1] as the result of the following optimization problem
Revised: Then in a similar way than for the GromovWasserstein distance, the semi-relaxed Fused Gromov-Wasserstein of order q is deﬁned for any trade-off parameter α ∈ [0 , 1] as the result of the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: Note that these problems come down to regularizing the quadratic problem of GW or srGW with a linear term in T taking into account a similarity measure between features of attributed graphs.
Revised: Note that these problems come down to regularizing the quadratic problem of GW or srGW with a linear term in T taking into account a similarity measure between features of attributed graphs

Original:  7. S EMI - RELAXED (F USED ) G ROMOV -W ASSERSTEIN PROPERTIES 7.2. E QUIVALENCE OF THE OPTIMIZATION PROBLEMS
Revised: . 7. S EMI - RELAXED (F USED ) G ROMOV -W ASSERSTEIN PROPERTIES 7.2. E QUIVALENCE OF THE OPTIMIZATION PROBLEMS

Original: First of all let us begin with proving the equivalence between the optimization problems deﬁning our semi-relaxed Gromov-Wasserstein divergence.
Revised: First of all let us begin with proving the equivalence between the optimization problems deﬁning our semi-relaxed Gromov-Wasserstein divergence.

Original: Consider two observed graph ( C , h ) and ( C , h ) of n and m nodes.
Revised: Consider two observed graph ( C , h ) and ( C , h ) of n and m nodes.

Original: The ﬁrst problem reads as
Revised: The ﬁrst problem reads as

Original: [Equation]
Revised: [Equation]

Revised: m

Original: [Equation] 
Revised: [Equation] [Equation]

Original: The second optimization problem coming from the relaxation of the marginal constraint on admissible couplings T over the target distribution h , reads as
Revised: The second optimization problem coming from the relaxation of the marginal constraint on admissible couplings T over the target distribution h , reads as

Original: [Equation]
Revised: [Equation]

Original: Note that in the following proofs we always assume that graphs correspond to ﬁnite discrete measures and that their geometry is well-deﬁned in the sense that entries of C and C are always ﬁnite.
Revised: Note that in the following proofs we always assume that graphs correspond to ﬁnite discrete measures and that their geometry is well-deﬁned in the sense that entries of C and C are always ﬁnite.

Original: This implies that graphs deﬁne compact spaces ensuring existence of that optimal solutions for both problems.
Revised: This implies that graphs deﬁne compact spaces ensuring existence of that optimal solutions for both problems.

Original: Proposition 2 Problems 15 and 16 are equivalent.
Revised: Lemma 1 Problems 15 and 16 are equivalent.

Original: Proof.
Revised: Proof.

Original: Consider a solution of problem 15 denoted ( h (cid:63) 1 , T 1 ) note that the deﬁnition implies that
Revised: Consider a solution of problem 15 denoted ( h (cid:63) 1 , T 1 ) note that the deﬁnition implies that

Original: T 1 ∈ U ( h , h (cid:63) 1 ) .
Revised: T 1 ∈ U ( h , h (cid:63) 1 ) .

Original: Another observation is that given h (cid:63) 1 , the transport plan T 1 also belongs to arg min T ∈U
Revised: Another observation is that given h (cid:63) 1 , the transport plan T 1 also belongs to arg min T ∈U

Original: ( h , h (cid:63) 1 )
Revised: ( h , h (cid:63) 1 )

Original: L q ( C , C , T ) hence is an optimal solution of GW q ( C , h , C , h (cid:63) 1 ) .
Revised: L q ( C , C , T ) hence is an optimal solution of GW q ( C , h , C , h (cid:63) 1 ) .

Original: Now consider a solution of problem 16 denoted T 2 with second marginal h (cid:63) 2 .
Revised: Now consider a solution of problem 16 denoted T 2 with second marginal h (cid:63) 2 .

Original: By deﬁnition the couple ( h (cid:63) 2 , T 2 ) is suboptimal for problem (15) i.e
Revised: By deﬁnition the couple ( h (cid:63) 2 , T 2 ) is suboptimal for problem (15) i.e

Original: [Equation]
Revised: [Equation]

Original: And the symmetric also holds as T 1 is a suboptimal admissible coupling for problem 16 i.e ,
Revised: And the symmetric also holds as T 1 is a suboptimal admissible coupling for problem 16 i.e ,

Original: [Equation]
Revised: [Equation]

Original: Hence implying that L q ( C , C , T 1 ) =
Revised: Hence implying that L q ( C , C , T 1 ) =

Original: L q ( C , C , T 2 ) .
Revised: L q ( C , C , T 2 ) .

Original: Which implies that T∈ arg min T ∈U ( h ,m )
Revised: Which implies that T∈ arg min T ∈U ( h ,m )

Original: L q ( C , C , T ) , also ( h (cid:63) 2 , T 2 ) ∈ arg min h ∈ Σ m , T ∈U ( h 1 , h )
Revised: L q ( C , C , T ) , also ( h (cid:63) 2 , T 2 ) ∈ arg min h ∈ Σ m , T ∈U ( h 1 , h )

Original: L q ( C , C , T ) and then
Revised: L q ( C , C , T ) and then

Original: GW q ( C , h , C ,h
Revised: GW q ( C , h , C ,

Original: [Equation]
Revised: [Equation]

Original: C ,

Original: [Equation]
Revised: [Equation]

Original: Therefore by double inclusion we have
Revised: Therefore by double inclusion we have

Original: [Equation]
Revised: [Equation]

Original: which is enough to prove that both problems are equivalent.
Revised: which is enough to prove that both problems are equivalent.

Original: The equivalence between analog problems for the semi-relaxed Fused Gromov-Wasserstein divergence can be proved in the exact same way.
Revised: The equivalence between analog problems for the semi-relaxed Fused Gromov-Wasserstein divergence can be proved in the exact same way.

Original: 7.2. Z ERO CONDITIONS OF (F USED ) SR GW
Revised: 7.2. Z ERO CONDITIONS OF (F USED ) SR GW

Original: We prove the following result:
Revised: We prove the following result:

Original: Proposition 3 Let C ∈ R n × n and C ∈ R m × m  and h ∈ Σ n with supp( h ) =  n  .
Revised: Proposition 1 Let C ∈ R n × n and C ∈ R m × m be distance matrices and h ∈ Σ n with supp( h ) = [[ n ]] .

Original: Then srGW 2 ( C , h , C ) = 

Revised: Then srGW q ( C , h , C ) =

Original: iff there exists h ∈ Σ m such that GW 2 ( C , h , C , h ) = 0 .
Revised: iff there exists h ∈ Σ m with card(supp( h ))

Original: [-As a consequence, if C , C are distance srGW 2 ( h , C , C ) = 
 iff there exists T ∈ R n × m + with n ≤ m , with ﬁrst marginal h and whose second marginal h ∈ Σ m is such that (cid:12)(cid:12) supp( h ) (cid:12)(cid:12)-] =
Revised: [-As a consequence, if C , C are distance srGW 2 ( h , C , C ) = 
 iff there exists T ∈ R n × m + with n ≤ m , with ﬁrst marginal h and whose second marginal h ∈ Σ m is such that (cid:12)(cid:12) supp( h ) (cid:12)(cid:12)-] =

Original: n andthere is a bijection σ : supp( h ) →
Revised: n and a bijection σ : supp( h ) →

Original: [[ n ]] which satisﬁes:
Revised: [[ n ]] such that:

Original: [Equation]
Revised: [Equation]

Original: And:
Revised: And:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst point is a direct consequence of the deﬁnition of srGW as minh ∈ Σ m GW qq
Revised: This a direct consequence of the deﬁnition of srGW as min

Original: ( C , h , C , h ) .
Revised: h [Equation]

Original: For the second point: ( ⇒ ) Assume there exists T ∈ U
Revised: m GW

Original: ( h , m )
Revised: [Equation]

Original: such that L q ( C , C , T ) = 0 .Let us denote h
Revised: Assume that srGW q ( C , h , C ) = 0 .

Original: = T (cid:62) 1 n the second marginal of this coupling.

Original: Then we have GW q ( C , h , C , h ) = 0  .
Revised: Then we have GW q ( C , h , C , h ) = 0 for some h .

Original: By virtue to Gromov-Wasserstein properties (Sturm, 2012, Lemma 1.10) there exists a bijection σ between the support of the distributions which is distance preserving.
Revised: By virtue to Gromov-Wasserstein properties (Sturm, 2012, Lemma 1.10) there exists a bijection σ between the support of the distributions which is distance preserving.

Original: In other words, there exists σ : supp( h ) → supp( h ) =
Revised: In other words, there exists σ : supp( h ) → supp( h ) =

Original: [[ n ]] such that h ( j )
Revised: [[ n ]] such that h ( j )

Original: =
Revised: =

Original: h ( σ ( j )) for all 
Revised: h ( σ ( j )) for all j ∈ supp( h ) and C kl = C σ ( k ) σ ( l ) for all k, l ∈ supp( h ) 2 . ( ⇐ ) Consider

Revised: T ∈

Revised: U

Original: Algorithm 3 CG solver for srGW, optionally with a linear regularization term D{+( h , h ) and the induced bijection σ satisfying equations 20 and 21. It is trivial to verify that L ( C , C , T ) = 0 implying that GW q ( C , h , C , h ) = 
 Moreover as T+} ∈ R n × n .
Revised: {+( h , h ) and the induced bijection σ satisfying equations 20 and 21. It is trivial to verify that L ( C , C , T ) = 0 implying that GW q ( C , h , C , h ) = 
 Moreover as T+} ∈ U ( h , m ) since

Revised: U ( h , h ) ⊂

Revised: U ( h , m ) and the because same cost is involved in both transport problems, we have:

Revised: [Equation]

Revised: Algorithm 3 CG solver for srGW, optionally with a linear regularization term D ∈ R n × n .

Original: Note that
Revised: Note that

Original: D = 0 for unregularized version of srGW of equation 25.
Revised: D = 0 for unregularized version of srGW of equation 25.

Original: 1: repeat 2:
Revised: 1: repeat 2:

Original: G ( t ) ← Compute gradient w.r.t T of (24) satisfying equation 26 applied in T ( t ) .
Revised: G ( t ) ← Compute gradient w.r.t T of (24) satisfying equation 26 applied in T ( t ) .

Original: Get direction X ( t ) problem: Solve independent subproblems on rows of G ( t ) thanks to the equivalence stated here
Revised: Get direction X ( t ) problem: Solve independent subproblems on rows of G ( t ) thanks to the equivalence stated here

Original: [Equation]
Revised: [Equation]

Original: Get optimal step size γ (cid:63) for the descent direction X ( t ) − T ( t ) : for any α ∈ [0 , 1] let us denote
Revised: Get optimal step size γ (cid:63) for the descent direction X ( t ) − T ( t ) : for any α ∈ [0 , 1] let us denote

Original: Z (
Revised: Z (

Original: [Equation]
Revised: [Equation]

Original: ) ( γ ) =
Revised: ) ( γ ) =

Original: T ( t ) + γ ( X ( t ) − T ( t ) ) , then γ (cid:63) satisﬁes:
Revised: T ( t ) + γ ( X ( t ) − T ( t ) ) , then γ (cid:63) satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: (23) factored as a second-order polynomial function of the form aγ 2 + bγ + c , by using linearity of the tensor-multiplication ⊗ and of the scalar product.
Revised: (23) factored as a second-order polynomial function of the form aγ 2 + bγ + c , by using linearity of the tensor-multiplication ⊗ and of the scalar product.

Original: Then the closed form is obtained by simple analysis of coefﬁcients a and b as in (Vayer et al., 2019).
Revised: Then the closed form is obtained by simple analysis of coefﬁcients a and b as in (Vayer et al., 2019).

Original: [Equation]
Revised: [Equation]

Original: 6: until convergence decided based on relative variation of the loss between step t and t + 1 .
Revised: 6: until convergence decided based on relative variation of the loss between step t and t + 1 .

Original: j ∈ supp( h ) and C kl =

Original: C σ ( k ) σ ( l ) for all k, l ∈ supp( h ) 2 .

Original: We can conclude by observing that the permutation σ induce an optimal coupling T as deﬁned in (20).

Original: ( ⇐ ) Consider

Original: T ∈

Original: U

Original: ( h , h ) and the induced bijection σ satisfying equations 20 and 21.

Original: It is trivial to verify that L ( C , C , T ) = 0 implying

Original: [-that GW q ( C , h , C , h ) = 
-]
Revised: [-that GW q ( C , h , C , h ) = 
-]

Original: Moreover as T ∈ U ( h , m ) since

Original: U ( h , h ) ⊂

Original: U ( h , m ) and the because same cost is involved in both transport problems, we have:

Original: [Equation]

Original: 7.  A LGORITHMIC DETAILS
Revised:  A LGORITHMIC DETAILS

Original: We provide in this section the algorithmic details completing our explanations given in the main paper (see subsection 3.2).
Revised: We provide in this section the algorithmic details completing our explanations given in the main paper (see subsection 3.2).

Original: Note that for all numerical experiments we considered q = 2 , so the following algorithms are speciﬁc to this scenario.
Revised: Note that for all numerical experiments we considered q = 2 , so the following algorithms are speciﬁc to this scenario.

Original: 7.3. C ONDITIONAL G RADIENT SOLVER FOR SR GW AND SR FGW
Revised: 7.3. C ONDITIONAL G RADIENT SOLVER FOR SR GW AND SR FGW

Original: The optimization problem related to computing srGW 2 2 ( C , h , C ) can be reformulated as
Revised: The optimization problem related to computing srGW 22 ( C , h , C ) can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: (24) where ⊗ K denotes the Kronecker product of two matrices, vec the column-stacking operator and the power operation on C is applied element-wise.
Revised: where ⊗ K denotes the Kronecker product of two matrices, vec the column-stacking operator and the power operation on C is applied element-wise.

Original: For simplicity let us use the other equivalent formulation using the 4-D tensor notation i.e
Revised: For simplicity let us use the other equivalent formulation using the 4-D tensor notation i.e

Original: [Equation]
Revised: [Equation]

Original: (25) where L ( C , C ) = (cid:0) ( C ij − C kl ) 2 (cid:1) ijkl .
Revised: 25) where L ( C , C ) = (cid:0) ( C ij − C kl ) 2 (cid:1) ijkl .

Original: This is a non-convex optimization problem with the same objective and gradient G than GW which can be expressed in the general case as
Revised: This is a non-convex optimization problem with the same objective and gradient G than GW which can be expressed in the general case as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that if C and C are symmetric matrices the gradient can be factored to
Revised: Note that if C and C are symmetric matrices the gradient can be factored to

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: convergence decided on relative variation of the loss between step t and t + 1 .

Original: Note that we implemented both versions as this factorization reduces slightly computation times in practice.
Revised: Note that we implemented both versions as this factorization reduces slightly computation times in practice.

Original: We propose to use a Conditional Gradient (CG) algorithm Jaggi (2013) to solve this problem, provided in Alg. 3.
Revised: We propose to use a Conditional Gradient (CG) algorithm Jaggi (2013) to solve this problem, provided in Alg. 3.

Original: For the sake of conciseness, we also introduce here a linear regularization term of the form (cid:104) D , T (cid:105) which will be used while enforcing sparsity promoting regularization on srGW or for adapting this algorithm to srFGW where D results from distances between features (up to proper scaling with the trade-off parameter α ).
Revised: For the sake of conciseness, we also introduce here a linear regularization term of the form (cid:104) D , T (cid:105) which will be used while enforcing sparsity promoting regularization on srGW or for adapting this algorithm to srFGW where D results from distances between features (up to proper scaling with the trade-off parameter α ).

Original: Then in this general case, our CG algorithm consists in solving at each iteration t a linearization (cid:104) X , G ( t )
Revised: Then in this general case, our CG algorithm consists in solving at each iteration t the following linearization of the problem (24)

Original: + D (cid:105) of the problem (24), aside from the regularization term, where G ( t ) is the gradient at T ( t ) of (24). 
Revised: [Equation] 28) where G ( t ) is the gradient at T ( t ) of (24). The optimization problem above can be very efﬁciently solved as discussed in (Flamary et al., 2016, Equation (8)). Indeed the problem above can clearly be reformulated as n independent linear problems under simplex constraints (each row of X can be solved independently) of the form

Original: This isefﬁciently achieved thanks to the trivial equivalence stated in equation 22.
Revised: [Equation] where g r is the row r of G . Optimizing a linear function over the simplex of dimensionality n can be done in O ( m ) because it consists in ﬁnding the smallest component i (cid:63) = arg min g r in the linear cost, the solution being a scaled dirac vector h r δ i (cid:63) where all the mass is positioned on component i (cid:63) .

Original: The solution of the linearized problem provides a descent direction X ( t ) − T ( t ) , and a line-search is performed to get the optimal step size as described in equation 23.
Revised: The solution of the linearized problem provides a descent direction X ( t ) − T ( t ) , and a line-search is performed to get the optimal step size as described in equation 23.

Original: Note that the gradients at time step
Revised: Note that the gradients at time step

Original: T ( t ) and T ( t +1) have a simple linear relation, which we used in our implementation for conciseness but it is rather equivalent to the more classical implementation literally expressing terms involved in the line-search part.
Revised: T ( t ) and T ( t +1) have a simple linear relation, which we used in our implementation for conciseness but it is rather equivalent to the more classical implementation literally expressing terms involved in the line-search part.

Original: Extension to srFGW.
Revised: Extension to srFGW.

Original: Let us denote two attributed graphs G =
Revised: Let us denote two attributed graphs G =

Original: ( C , F , h ) and ( C , F , h ) , of n and m nodes
Revised: ( C , F , h ) and ( C , F , h ) , of n and m nodes

Original: respectively.
Revised: respectively.

Original: Then the semi-relaxed Fused Gromov-Wasserstein of order 2 is deﬁned for any trade-off parameter α ∈ [0 , 1] as the result of the following optimization problem
Revised: Then the semi-relaxed Fused Gromov-Wasserstein of order 2 is deﬁned for any trade-off parameter α ∈ [0 , 1] as the result of the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: ) where M ( F , F ) denote the matrix of euclidean distances between features of F and F , i.e. M ij = (cid:107) F i − F j (cid:107) 22 .
Revised: ) where M ( F , F ) denote the matrix of euclidean distances between features of F and F , i.e. M ij = (cid:107) F i − F j (cid:107) 22 .

Original: As a side note, one efﬁcient way to compute this matrix in practice is by using the following factorization M ( F , F ) =
Revised: As a side note, one efﬁcient way to compute this matrix in practice is by using the following factorization M ( F , F ) =

Original: ( F (cid:12) F ) 1 d 1 (cid:62) m
Revised: ( F (cid:12) F ) 1 d 1 (cid:62) m

Original: + 1 n 1 (cid:62) d ( F (cid:12) F )
Revised: + 1 n 1 (cid:62) d ( F (cid:12) F )

Original: − 2 F F (cid:62) , where (cid:12) denotes the Hadamard product.
Revised: − 2 F F (cid:62) , where (cid:12) denotes the Hadamard product.

Original: The problem in equation 28 is a nonconvex quadratic problem which gradient w.r.t.
Revised: The problem in equation 14 is a nonconvex quadratic problem which gradient w.r.t.

Original: T reads as :
Revised: T reads as :

Original: [Equation]
Revised: [Equation]

Original: (29) where G corresponds to the gradient of the GW cost satisfying equation 26.
Revised: where G corresponds to the gradient of the GW cost satisfying equation 26.

Original: Therefore we propose to tackle this problem by a straight-forward adaptation of the CG algorithm 3 detailed above by adding the multiplier α to the GW cost and setting D =
Revised: Therefore we propose to tackle this problem by a straight-forward adaptation of the CG algorithm 3 detailed above by adding the multiplier α to the GW cost and setting D =

Original: (1 − α )
Revised: (1 − α )

Original: M
Revised: M

Original: ( F , F ) .
Revised: ( F , F ) .

Original: 7.3. A LGORITHMIC DETAILS AND GUARANTIES ON ENTROPIC SR GW.
Revised: 7.3. A LGORITHMIC DETAILS AND GUARANTIES ON ENTROPIC SR GW.

Original: Recent OT applications have shown the interest of adding an entropic regularization to the exact problem (24) Cuturi (2013); Peyr´e et al.
Revised: Recent OT applications have shown the interest of adding an entropic regularization to the exact problem (24) Cuturi (2013); Peyr´e et al.

Original: We detail here how to design a mirror-descent scheme w.r.t.
Revised: We detail here how to design a mirror-descent scheme w.r.t.

Original: the Kullback-Leibler divergence (KL) to solve (24), in the same vein than Peyr´e et al.
Revised: the Kullback-Leibler divergence (KL) to solve (24), in the same vein than Peyr ´ e et al.

Original: (2016); Xu et al.
Revised: (2016); Xu et al.

Original: (2019b)
Revised: (2019b)

Original: ; Xie et al.
Revised: ; Xie et al.

Original: Algorithmic details.
Revised: Algorithmic details.

Original: Indeed in order to solve the srGW problem stated in 16, we can use a mirror-descent scheme w.r.t.
Revised: Indeed in order to solve the srGW problem stated in 16, we can use a mirror-descent scheme w.r.t.

Original: the KL geometry.
Revised: the KL geometry.

Original: At iteration t , the update the current transport plan
Revised: At iteration t , the update the current transport plan

Original: T ( t ) ∈ U ( h , m ) results from the following optimization problem
Revised: T ( t ) ∈ U ( h , m ) results from the following optimization problem

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Where G satisﬁes 26 and KL( T | T ( t ) )
Revised: Where G satisﬁes 26 and KL( T | T ( t ) )

Original: = (cid:80) ij T i,j log( T i,j T ( t ) i,j ) − T
Revised: = (cid:80) ij T i,j log( T i,j T ( t ) i,j ) − T

Original: i,j + T ( t ) i,j . Let us denote the entropy of any T ∈ R n × m + by H( T ) =
Revised: i,j + T ( t ) i,j . Let us denote the entropy of any T ∈ R n × m + by H( T ) =

Original: − (cid:80) ij T ij (log T ij − 1) , one can simply prove the the following relation:
Revised: − (cid:80) ij T ij (log T ij − 1) , one can simply prove the the following relation:

Original: [Equation]
Revised: [Equation]

Original: Then using this relation, the problem in 30 is equivalent to
Revised: Then using this relation, the problem in 32 is equivalent to

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: Algorithm 4 MD solver for entropic with a term

Revised: Note that D = 0 for unregularized version of srGW of equation 25.

Original: Denoting M ( t ) ( (cid:15) )
Revised: Denoting M ( t ) ( (cid:15) )

Original: =
Revised: =

Original: G ( T ( t ) ) − (cid:15) log T ( t ) , overall we end up with the following iterations 
Revised: G ( T ( t ) ) − (cid:15) log T ( t ) , overall we end up with the following iterations [Equation] min [Equation]

Original: [Equation]
Revised: [Equation]

Original: Which is equivalent thanks to the relation stated above to
Revised: Which is equivalent thanks to the relation stated above to

Revised: T ( t +1) ← arg min

Original: [Equation]
Revised: [Equation]

Original: ) where K(
Revised: KL [Equation] K

Original: [Equation]
Revised: [Equation]

Revised: ( (cid:15) ) | T ) where K ( t ) ( (cid:15) ) :

Revised: = exp {− M (

Revised: [Equation]

Original:  Following the seminal work of (Benamou et al., 2015), the optimal T ( t +1) is given by a simple scaling of the matrix K ( t ) ( (cid:15) ) reading as
Revised: ( (cid:15) [Equation] /(cid:15) } (36) Following the seminal work of (Benamou et al., 2015), the optimal T ( t +1) is given by a simple scaling of the matrix K ( t ) ( (cid:15) ) reading as

Original: [Equation]
Revised: [Equation]

Original: Note that an analog scheme is achievable while penalizing the srGW problem with a linear term of the form (cid:104) D , T (cid:105) .
Revised: Note that an analog scheme is achievable while penalizing the srGW problem with a linear term of the form (cid:104) D , T (cid:105) .

Original: Which would simply result in a modiﬁcation of the matrix M ( t ) ( (cid:15) )
Revised: Which would simply result in a modiﬁcation of the matrix M ( t ) ( (cid:15) )

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: This more general setting is summarized in algorithm 4.
Revised: This more general setting is summarized in algorithm 4.

Original: Similarly than for the CG algorithm 3, it is straight-forward to adapt the MD algorithm 4 to the semi-relaxed Fused Gromov-Wasserstein using FGW cost expressed in 29.
Revised: Similarly than for the CG algorithm 3, it is straight-forward to adapt the MD algorithm 4 to the semi-relaxed Fused Gromov-Wasserstein using FGW cost expressed in 31.

Original: 7.3. S PARSITY PROMOTING REGULARIZATION OF SR GW
Revised: 7.3. S PARSITY PROMOTING REGULARIZATION OF SR GW

Original: In order to promote sparsity of the estimated target distribution while matching ( C , h ) to the structure
Revised: In order to promote sparsity of the estimated target distribution while matching ( C , h ) to the structure

Original: C using srGW, we suggest to use Ω( T ) =
Revised: C using srGW, we suggest to use Ω( T ) =

Original: (cid:80) j | h
Revised: (cid:80) j | h

Original: j | 1 / 2 which deﬁnes a concave function in the positive orthant R + .
Revised: j | 1 / 2 which deﬁnes a concave function in the positive orthant R + .

Original: This results to the following optimization
Revised: This results to the following optimization

Original: [Equation]
Revised: [Equation]

Original: (37) with λ g ∈ R + an hyperparameter.
Revised: (39) with λ g ∈ R + an hyperparameter.

Original: As mentioned in the main paper, equation 37 can be tackled with a majorisation-minimisation (MM) algorithm.
Revised: As mentioned in the main paper, equation 39 can be tackled with a majorisation-minimisation (MM) algorithm.

Original: MM consists of iteratively minimising an upper bound of the objective function which is tight at the current iterate (Hunter & Lange, 2004).
Revised: MM consists of iteratively minimising an upper bound of the objective function which is tight at the current iterate (Hunter & Lange, 2004).

Original: With this procedure, the objective function is guaranteed to decrease at every iteration.
Revised: With this procedure, the objective function is guaranteed to decrease at every iteration.

Original: In our case, we only need to majorize the penalty term Ω( T ) to obtain a tractable function.
Revised: In our case, we only need to majorize the penalty term Ω( T ) to obtain a tractable function.

Original: Denoting h ( t ) =
Revised: Denoting h ( t ) =

Original: T ( t ) (cid:62) 1 n , the estimate at iteration t , one can simply apply the tangent inequality
Revised: T ( t ) (cid:62) 1 n , the estimate at iteration t , one can simply apply the tangent inequality

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using this inequality is equivalent to linearize the regularization term at h ( t ) whose contribution can be absorbed into the inner product as (cid:104) L ( C , C ) ⊗ T + R ( t ) , T (cid:105) where R ( t ) =
Revised: Using this inequality is equivalent to linearize the regularization term at h ( t ) whose contribution can be absorbed into the inner product as (cid:104) L ( C , C ) ⊗ T + R ( t ) , T (cid:105) where R ( t ) =

Original: λ g 2 ( h ( t ) j ) − 1 / 2 ij .
Revised: λ g 2 ( h ( t ) j ) − 1 / 2 ij .

Original: The overall optimization procedure is summarized in 7.
Revised: The overall optimization procedure is summarized in 5.

Original: Note that the same procedure is used for promoting sparsity of srFGW using the adaptation of our CG solver 3 and MD solver 4 to this scenario.[Equation] [Equation] [Equation]
Revised: Note that the same procedure is used for promoting sparsity of srFGW using the adaptation of our CG solver 3 and MD solver 4 to this scenario.

Original: [Equation]

Original: 7. L EARNING THE TARGET STRUCTURE
Revised: 7. L EARNING THE TARGET STRUCTURE

Original: We detail in the following the algorithms for the srGW Dictionary Learning and its application to graphs completion.
Revised: We detail in the following the algorithms for the srGW Dictionary Learning and its application to graphs completion.

Original: 7.4. SR GW D ICTIONARY L EARNING .
Revised: 7.4. SR GW D ICTIONARY L EARNING .

Original: We propose to learn the graph atom C ∈ R m × m from the observed data D = { ( C k , h k ) } k ∈
Revised: We propose to learn the graph atom C ∈ R m × m from the observed data D = { ( C k , h k ) } k ∈

Original: [ K ] , by optimizing
Revised: [ K ] , by optimizing

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This is a nonconvex problem that we propose to tackle thanks to a stochastic gradient algorithm summarized in 6.
Revised: This is a nonconvex problem that we propose to tackle thanks to a stochastic gradient algorithm summarized in 6.

Original: At each iteration it consists in sampling a batch of graphs B = { ( C k , h k ) } k ∈
Revised: At each iteration it consists in sampling a batch of graphs B = { ( C k , h k ) } k ∈

Original: [[ B ]] from the dataset and to embed each graph where h k = T (cid:62) k 1 n by solving independent srGW problems for the current state of the dictionary.
Revised: [[ B ]] from the dataset and to embed each graph where h k = T (cid:62) k 1 n by solving independent srGW problems for the current state of the dictionary.

Original: Then we compute the estimate of the gradient over C reading as
Revised: Then we compute the estimate of the gradient over C reading as

Original: [Equation]
Revised: [Equation]

Original: Note that only the entries ( i, j ) of C such that i and j belongs to ∪ k supp( h k ) can have a non-null gradient.
Revised: Note that only the entries ( i, j ) of C such that i and j belongs to ∪ k supp( h k ) can have a non-null gradient.

Original: Finally depending on the input structures we consider we apply a projection on an adequate set S , for instance if input structures are adjacency matrices we consider S as the set of non-negative symmetric matrices (iterates can preserve symmetry but not necessarily non-negativity depending on the chosen learning rate).
Revised: Finally depending on the input structures we consider we apply a projection on an adequate set S , for instance if input structures are adjacency matrices we consider S as the set of non-negative symmetric matrices (iterates can preserve symmetry but not necessarily non-negativity depending on the chosen learning rate).

Original: Extension to attributed graphs.
Revised: Extension to attributed graphs.

Original: The stochastic algorithm described above can be adapted to a dataset of attributed graph D = { ( C k , F k , h k ) } k ∈
Revised: The stochastic algorithm described above can be adapted to a dataset of attributed graph D = { ( C k , F k , h k ) } k ∈

Original: [[ K ]] with nodes features in R d , by learning an attributed graph atom ( C , F ) .
Revised: [[ K ]] with nodes features in R d , by learning an attributed graph atom ( C , F ) .

Original: The optimization problem would now read as
Revised: The optimization problem would now read as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: for any α ∈ [0 , 1] .
Revised: for any α ∈ [0 , 1] .

Original: The stochastic algorithm 6 is then adapted by ﬁrst computing embeddings based on srFGW instead of srGW .
Revised: The stochastic algorithm 6 is then adapted by ﬁrst computing embeddings based on srFGW instead of srGW .

Original: Then simultaneously updating C following (42) up to the factor α and
Revised: Then simultaneously updating C following (44) up to the factor α and

Original: F thanks to the following equation
Revised: F thanks to the following equation

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: MD λ g 2 (

Original: ( t ) j ) − 1 / the new local linearization of

Original: [Table]

Original: Compute the optimal representations (cid:101) G ( t of the ( C ( t ) , h ) onto the dictionary and optimal transport T s

Original: [[ s ]] w s C s projected

Original: 7.4. DL- BASED MODEL FOR GRAPHS COMPLETION
Revised: DL- BASED MODEL FOR GRAPHS COMPLETION

Original: The estimated structure C , learned on the dataset D can be used to infer/complete a new graph form the dataset being only partially observed.
Revised: The estimated structure C , learned on the dataset D can be used to infer/complete a new graph form the dataset being only partially observed.

Original: Let us say that we want to recover the structure C ∈ R n × n
Revised: Let us say that we want to recover the structure C ∈ R n × n

Original: but we only observed the relations between n obs < n nodes denoted as C obs ∈ R n obs × n obs .
Revised: but we only observed the relations between n obs < n nodes denoted as C obs ∈ R n obs × n obs .

Original: We propose to solve the following optimization problems to recover the full matrix C while modeling graphs with our srGW Dictionary Learning 
Revised: We propose to solve the following optimization problems to recover the full matrix C while modeling graphs with our srGW Dictionary Learning min

Original: [Equation]
Revised: [Equation]

Revised: imp srGW 22 (cid:16) (cid:101) C , h , C (cid:17) , where (cid:101)

Original:  Graph comletion with GDL For the linear dictionary GDL (Vincent-Cuaz et al., 2021) with several atoms denoted { C s } s ∈ [[ S ]] sharing the same weightsh
Revised: C = (cid:34) C obs ... [Equation] Graph completion with GDL For the linear dictionary GDL (Vincent-Cuaz et al., 2021) with several atoms denoted { C s } s ∈ [[ S ]] sharing the same weights

Original:  we adapted our formulation to their model:
Revised: h we adapted our formulation to their model:

Original: [Equation]
Revised: [Equation]

Original: The matrix C has only the n 2 − n 2 obs coefﬁcients collected into C imp are optimized (and thus imputed).
Revised: The matrix C has only the n 2 − n 2 obs coefﬁcients collected into C imp are optimized (and thus imputed).

Original: This way C imp expresses the connections between the imputed nodes, and between these nodes and the observed ones in C obs .
Revised: This way C imp expresses the connections between the imputed nodes, and between these nodes and the observed ones in C obs .

Original: We tackle the non-convex problems above following the procedure described in Algorithm 8.
Revised: We tackle the non-convex problems above following the procedure described in Algorithm 7.

Original: It consists in an alternating scheme where at each iteration t , we embed the current estimated graph
Revised: It consists in an alternating scheme where at each iteration t , we embed the current estimated graph

Original: C ( t ) on the dictionary depending on the chosen model (see (44)).
Revised: C ( t ) on the dictionary depending on the chosen model (see (46)).

Original: Let us denote (cid:101) G ( t ) =
Revised: Let us denote (cid:101) G ( t ) =

Original: ( (cid:101) C ( t ) , (cid:101) h ( t ) )
Revised: ( (cid:101) C ( t ) , (cid:101) h ( t ) )

Original: this representation which respectively reads for srGW as (cid:101) G ( t ) =
Revised: this representation which respectively reads for srGW as (cid:101) G ( t ) =

Original: ( C , h ( t ) ) and for GDL as (cid:101) G ( t )
Revised: ( C , h ( t ) ) and for GDL as (cid:101) G ( t )

Original: =
Revised: =

Original: ( (cid:80) s ∈ [[ S ]] w ( t ) s C s , h ) .
Revised: ( (cid:80) s ∈ [[ S ]] w ( t ) s C s , h ) .

Original: Note that from this embedding step we also get the optimal transport T ( t ) from the GW distance between ( C ( t ) , h ) and (cid:101) G ( t ) .
Revised: Note that from this embedding step we also get the optimal transport T ( t ) from the GW distance between ( C ( t ) , h ) and (cid:101) G ( t ) .

Original: Then we can update the imputed part C ( t ) imp of C ( t ) while keeping ﬁxed C obs thanks to a projected gradient step with gradient w.r.t.
Revised: Then we can update the imputed part C ( t ) imp of C ( t ) while keeping ﬁxed C obs thanks to a projected gradient step with gradient w.r.t.

Original: C reading as
Revised: C reading as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 7. D ETAILS ON THE NUMERICAL EXPERIMENTS . 7.5. G RAPH PARTITIONING
Revised: 7. D ETAILS ON THE NUMERICAL EXPERIMENTS . 7.5. G RAPH PARTITIONING

Revised: Datasets

Revised: [Table]

Revised: Wikipedia

Original: [Table]
Revised: [Table]

Revised: EU-email

Revised: [Table]

Revised: 25 Amazon

Original: [Table]
Revised: [Table]

Original:  We detail here the benchmark between srGW and state-of-the-art methods for graph partitioning on real (directed and undirected) graphs.
Revised: 41 Village  0. [Table] Detailed partitioning benchmark. We detail here the benchmark between srGW and state-ofthe-art methods for graph partitioning on real (directed and undirected) graphs.

Original: We replicated the benchmark from (Chowdhury & Needham, 2021) using the 4 datasets whose preprocessing is detailed in their paper and resulting statistics are provided in Table 4.
Revised: We replicated the benchmark from (Chowdhury & Needham, 2021) using the 4 datasets whose preprocessing is detailed in their paper and resulting statistics are provided in Table 4.

Original: We considered the two GW based partitioning methods proposed by (Xu et al., 2019a) (GWL) and (Chowdhury & Needham, 2021) (SpecGWL).
Revised: We considered the two GW based partitioning methods proposed by (Xu et al., 2019a) (GWL) and (Chowdhury & Needham, 2021) (SpecGWL).

Original: We benchmarked our method denoted srGW and srSpecGW which respectively use adjacency and heat kernels on normalized laplacian matrices as inputs since they were both use by competitors.
Revised: We benchmarked our method denoted srGW and srSpecGW which respectively use adjacency and heat kernels on normalized laplacian matrices as inputs since they were both use by competitors.

Original: All these OT based methods depend on hyperparameters which can be tuned in an unsupervised way ( i.e. without knowing the ground truth partition) based on modularity maximization (Chowdhury & Needham, 2021).
Revised: All these OT based methods depend on hyperparameters which can be tuned in an unsupervised way ( i.e. without knowing the ground truth partition) based on modularity maximization (Chowdhury & Needham, 2021).

Original: Following their numerical experiments, we considered as input distribution h for the observed graph the parameterized power-law transformations of the form h
Revised: Following their numerical experiments, we considered as input distribution h for the observed graph the parameterized power-law transformations of the form h

Original: i = p
Revised: i = p

Original: i (cid:80) i p i
Revised: i (cid:80) i p i

Original: where p i =
Revised: where p i =

Original: ( deg ( i ) + a ) b , with deg ( i ) the i-th node degree and real parameters a ∈ R and b ∈ [0 , 1] .
Revised: ( deg ( i ) + a ) b , with deg ( i ) the i-th node degree and real parameters a ∈ R and b ∈ [0 , 1] .

Original: If the graph has no isolated nodes we chose a = 0 and a = 1 otherwise.
Revised: If the graph has no isolated nodes we chose a = 0 and a = 1 otherwise.

Original: b is validated within thesevalues { 0 , 0 . 0001 , 0 .
Revised: b is validated within these 10 values { 0 , 0 . 0001 , 0 .

Original: 5 , 1 } which progressively transform the input distribution from the uniformone to the normalized degrees distribution.
Revised: 5 , 1 } which progressively transform the input distribution from the uniform to the normalized degree ones.

Original:  The heat parameter for SpecGWL and srSpecGW is tuned for each dataset within the range [1 , 100] by recursively splitting this range into 5 values, ﬁnd the parameter leading to maximum modularity, and repeat the same process on the induced new interval with this best parameter as center.
Revised: An ablation study of this parameter is reported in Table 6. The heat parameter for SpecGWL and srSpecGW is tuned for each dataset within the range [1 , 100] by recursively splitting this range into 5 values, ﬁnd the parameter leading to maximum modularity, and repeat the same process on the induced new interval with this best parameter as center.

Original: This process is stopped based on the relative variation of maximum modularity between two successive iterates of precision 10 − 3 .
Revised: This process is stopped based on the relative variation of maximum modularity between two successive iterates of precision 10 − 3 .

Original: A similar scheme can be used to ﬁne-tune the entropic parameter.
Revised: A similar scheme can be used to ﬁne-tune the entropic parameter.

Original:  (Chowdhury & Needham, 2021) also discussed the sensitivity of GW solvers to the initialization and showed that GW matchings with heat kernels have considerably less spurious local minimum than those using adjacency matrices.
Revised: On our algorithm initialization. (Chowdhury & Needham, 2021) also discussed the sensitivity of GW solvers to the initialization and showed that GW matchings with heat kernels have considerably less spurious local minimum than those using adjacency matrices.

Original: This way it is arguable to compare both methods by using the product hh (cid:62) as default initialization.
Revised: This way it is arguable to compare both methods by using the product hh (cid:62) as default initialization.

Original: We observed for srGW that using an initialization based on the hard assignments of a kmeans on the rows of the input representations (up to the left scaling diag( h ) ), was a better trade-off for both kinds of representations.
Revised: We observed for srGW that using an initialization based on the hard assignments of a kmeans on the rows of the input representations (up to the left scaling diag( h ) ), was a better trade-off for both kinds of representations.

Original: Hence we applied this scheme for our method in this benchmark.
Revised: Hence we applied this scheme for our method in this benchmark.

Original: We illustrate in Table 4 how srGW reﬁnes these hard assignments by soft ones through OT.
Revised: We illustrate in Table 4 how srGW reﬁnes these hard assignments by soft ones through OT.

Revised: Note that for these partitioning tasks we should/can not initialize the transport plan of our srGW solver using the product of h ∈ Σ N with a uniform target distribution h (0)

Revised: = 1 Q 1 Q , leading to

Revised: T (0) =

Revised: 1 Q h 1 (cid:62) Q .

Original: 7.5. E XPERIMENTS BASED ON DICTIONARY LEARNING
Revised: Indeed, for any symmetric input representation C of the graph, the partial derivative of our objective w.r.t the ( p, q ) ∈ [[ N ]] × [[ Q ]] entries of T , satisﬁes [Equation] This expression is independent of q ∈ [[ Q ]] , so taking the minimum value over each row in the direction ﬁnding step of our CG algorithm (see algo 3) will lead to X = T (0) . Then the line-search step involving, for any γ ∈ [0 , 1] , Z (0) ( γ ) =

Revised: T (0) + γ ( X − T (0) ) will be independent of γ as

Revised: Z (0) ( γ ) =

Revised: T (0) .

Revised: This would imply that the algorithm will terminate with optimal solution T (cid:63) = T (0) being a non-informative coupling.

Revised: Parameterized input distributions: Ablation study.

Revised: We reported in Table 6 an ablation study of the parameter b ∈ [0 , 1] introduced on the input graph distributions ﬁrst suggested by (Xu et al., 2019a).

Revised: In most scenario and every benchmarked methods, a parameter b ∈ ]0 , 1[ leads to best

Revised: AMI performances (except for the dataset Village), while the common assumption of uniform

Revised: [Table]

Revised: distribution remains competitive.

Revised: The use of raw normalized degree distributions consequently reduces partitioning performances of all methods.

Revised: Hence these results ﬁrst indicated that further research on the input distributions could be beneﬁcial.

Revised: They also suggest that the commonly used uniform input distributions provide a good compromise in terms of performances while being parameter-free.

Revised: [Table]

Revised: Additional clustering metric.

Revised: In order to complete our partitioning benchmark on real datasets, we report in 11 the Adjusted Rand Index (ARI).

Revised: The comparison between ARI and AMI has been thoroughly investigated in (Romano et al., 2016) and led to the following conclusion: ARI should be used when the reference clustering has large equal sized clusters; AMI should be used when the reference clustering is unbalanced and there exist small clusters.

Revised: Our method srGW always outperforms the GW based approaches for a given type of input structure representations (adjacency vs heat kernels) and on this application the entropic regularization seems to improve the performance.

Revised: Note that this metric also leads to slight variations of rankings, which seems to occur more often for methods explicitly based on modularity maximization.

Revised: We want to stress that for this metric our general purpose divergence srGW outperforms methods that have been speciﬁcally designed for nodes clustering tasks on 4 out of 6 datasets (instead of 3 using the AMI).

Revised: [Figure]

Revised: srGW runtimes: CPU vs GPU for large graphs.

Revised: Partitioning experiments with our methods were run on a

Revised: GPU Tesla K80 as it brought a considerable speed up in terms of computation time compared to using CPUs as large graphs had to be processed.

Revised: To illustrate this matter, we generated 10 graphs following Stochastic Block

Revised: Models with 10 clusters, a varying number of nodes in { 100 , 200 , ..., 2900 , 3000 } and the same connectivity matrix.

Revised: We report in Table the averaged runtimes of one CG iteration depending on the size of the input graphs.

Revised: For small graphs with a few hundreds of nodes, performances on CPUs or a single GPU are comparable.

Revised: However, operating on GPU becomes very beneﬁcial once graphs of a few thousand of nodes are processed.

Revised: Runtimes comparison on real datasets.

Revised: We report in Table 8 the runtimes on CPUs for the partitioning benchmark on real datasets.

Revised: For the sake of comparison, we ran the srGW partitioning experiments using CPUs instead of a GPU.

Revised: We can observe that GW partitioning of raw adjacency matrices (GWL) is the most competitive in terms of computation time (while being

Revised: the

Revised: [Table]

Revised: last in terms of clustering performances)

Revised: , on par with InfoMap.

Revised: Our srGW partitioning methods lack behind the GW based ones in terms of speed which can be explained in many ways.

Revised: [Table]

Revised: We want to stress that as illustrated in the previous paragraph, the use of CPUs for srGW multiplied by 10 to 20 times the computation times observed on a

Revised: GPU.

Revised: First the linear OT problems involved in each

Revised: CG iteration of GW based methods are solved thanks to a C solver (POT’s implementation (Flamary et al., 2021)), whereas our srGW CG solver is fully implemented in Python.

Revised: This can compensate the computational beneﬁts of srGW solver regarding GW solver (see details in section 3.2 of the main paper).

Revised: So the computation time became mostly affected by the number of computed gradients.

Revised: We observed in these experiments that for the same precision level, srGW needed considerably more iterations to converge than GW.

Revised: Let us recall that our srGW partitioning method consists in matching an observed graph to the identity matrix C

Revised: =

Revised: I , whereas GW based partitioning methods use as target structure C =

Revised: diag ( h ) as h is ﬁxed beforehand.

Revised: Our lack of heterogeneity in the chosen target structure could be the reason of these slow convergence patterns.

Revised: hence advocates for further studies regarding srGW based graph partitioning using more informative structures.

Revised: 7.5. D ICTIONARY L EARNING : CLUSTERING EXPERIMENTS

Original: We report in table 6 the statistics of the datasets used for our benchmark on clustering of datasets graphs and graphs completion.
Revised: We report in table 9 the statistics of the datasets used for our benchmark on clustering of datasets graphs and graphs completion.

Original: Detailed settings for the clustering benchmark.
Revised: Detailed settings for the clustering benchmark.

Original: We detail now the experimental setting used in the clustering benchmark derived from the benchmark conducted by Vincent-Cuaz et al.[Table]
Revised: We detail now the experimental setting used in the clustering benchmark derived from the benchmark conducted by Vincent-Cuaz et al.

Original: For the datasets with attributes involving FGW, we validated 15 values of the trade-off parameter α via a logspace search in (0 , 0 . 5) and symmetrically (0 . 5 , 1) .
Revised: For the datasets with attributes involving FGW, we validated 15 values of the trade-off parameter α via a logspace search in (0 , 0 . 5) and symmetrically (0 . 5 , 1) .

Original: For DL based approaches, a ﬁrst step consists into learning the atoms.
Revised: For DL based approaches, a ﬁrst step consists into learning the atoms.

Original: srGW dictionary sizes are tested in M ∈ { 10 , 20 , 30 , 40 , 50 } , the atom is initialized by randomly sampling its entries from N (0 . 5 , 0 . 01) and made symmetric.
Revised: srGW dictionary sizes are tested in M ∈ { 10 , 20 , 30 , 40 , 50 } , the atom is initialized by randomly sampling its entries from N (0 . 5 , 0 . 01) and made symmetric.

Original: The extension of srGW to attributed graphs, namely srFGW, is referred as srGW for conciseness in 2 of the main paper.
Revised: The extension of srGW to attributed graphs, namely srFGW, is referred as srGW for conciseness in 2 of the main paper.

Original: One efﬁcient way to initialize atoms features for minimizing our resulting reconstruction errors is to use a Kmeans algorithm seeking for M clusters on the nodes features observed in the dataset.
Revised: One efﬁcient way to initialize atoms features for minimizing our resulting reconstruction errors is to use a Kmeans algorithm seeking for M clusters on the nodes features observed in the dataset.

Original: For GDL and GWF, a variable number of S = βk atoms is validated, where k denotes the number of classes and β ∈ { 2 , 4 , 6 , 8 } .
Revised: For GDL and GWF, a variable number of S = βk atoms is validated, where k denotes the number of classes and β ∈ { 2 , 4 , 6 , 8 } .

Original: The size of the atoms M is set to the median of observed graph sizes within the dataset for GDL and GWF-f. These methods initialize their atoms by sampling observed graphs within the dataset, with adequate sizes forGDL and GWF-f, while sizes of
Revised: The size of the atoms M is set to the median of observed graph sizes within the dataset for GDL and GWF-f. These methods initialize their atoms by sampling observed graphs within the dataset, with adequate sizes for

Original:  GWF-r are just determined by this random sampling procedure independently of the number of nodes distribution.
Revised: GDL and GWF-f, while sizes of GWF-r are just determined by this random sampling procedure independently of the number of nodes distribution.

Original: For srGW g , srGW e + g and GDL λ , the coefﬁcient of our respective sparsity promoting regularizers is validated within { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 } .
Revised: For srGW g , srGW e + g and GDL λ , the coefﬁcient of our respective sparsity promoting regularizers is validated within { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 } .

Original: Then for srGW e , srGW e + g , GWF-f and GWF-r, the entropic regularization coefﬁcient is validated also within { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 } .
Revised: Then for srGW e , srGW e + g , GWF-f and GWF-r, the entropic regularization coefﬁcient is validated also within { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 } .

Original: Finally, we considered the same settings for the stochastic algorithm hyperparameters across all methods: learning rates are validated within { 0 . 01 , 0 .
Revised: Finally, we considered the same settings for the stochastic algorithm hyperparameters across all methods: learning rates are validated within { 0 . 01 , 0 .

Original: 001 } while the batch size is validated within { 16 , 32 } ; 
Revised: 001 } while the batch size is validated within { 16 , 32 } ; We learn all models ﬁxing a maximum number of epochs of(over convergence requirements) and implemented an ( unsupervised ) early-stopping strategy which [Table] [Table]

Original: We learn all models ﬁxing a maximum number of epochs of 100 (over convergence requirements) and implemented an ( unsupervised ) early-stopping strategy which consists in computing the respective unmixings every 5 epochs and stop the learning process if the cumulated reconstruction error ( Err t ) does not improve anymore over 2 consecutive evaluations ( i.e. Err t ≤ Err t +1 and Err t ≤ Err t +2 ). 
Revised: - - - - - - GW-k - - - - - - - - consists in computing the respective unmixings every 5 epochs and stop the learning process if the cumulated reconstruction error ( Err t ) does not improve anymore over 2 consecutive evaluations ( i.e. Err t ≤ Err t +1 and Err t ≤ Err t +2 ). For the sake of consistency, we report in Table 7 the Averaged Mutual Information (AMI) performances on this benchmark, as we reported the RI in the main paper to be consistent with our main competitors.

Original: Visualization of srGW embeddings.
Revised: Visualizations of srGW embeddings.

Original: We provide in Figure 4 some examples of graphs embeddings from the dataset IMDB-B learned on a srGW dictionary C of 10 nodes.
Revised: We provide in Figure 5 some examples of graphs embeddings from the dataset IMDB-B learned on a srGW dictionary C of 10 nodes.

Original: We assigned different colors to each nodes of the graph atom (forth column) in order to visualize the correspondences recovered by the OT plan T (cid:63) resulting from the projection of the respective sample ( C , h ) onto C in the srGW sense (third column).
Revised: We assigned different colors to each nodes of the graph atom (forth column) in order to visualize the correspondences recovered by the OT plan T (cid:63) resulting from the projection of the respective sample ( C , h ) onto C in the srGW sense (third column).

Original: As the atom has continuous values we set the edges intensity of grey proportionally to the entries of C .
Revised: As the atom has continuous values we set the edges intensity of grey proportionally to the entries of C .

Original: By coloring nodes of the observed graphs based on the OT to its respective embedding ( C ) (second column) we clearly observe that key subgraphs information, such as clusters and hubs are captured within the embedding. 
Revised: By coloring nodes of the observed graphs based on the OT to its respective embedding ( C ) (second column) we clearly observe that key subgraphs information, such as clusters and hubs are captured within the embedding. [Figure]

Original: Note that if we increase the size of the dictionary, our embeddings are reﬁned and can bring complementary structural information at a higher resolution e.g. ﬁnding substructures or variable connectivity between nodes in the same cluster.
Revised: Impact of the graph atom size. Moreover if we increase the size of the dictionary, our embeddings are reﬁned and can bring complementary structural information at a higher resolution e.g. ﬁnding substructures or variable connectivity between nodes in the same cluster.

Revised: We illustrate these resolution patterns in Table 6 for embeddings learned on the

Revised: IMDB-B dataset.

Revised: We represent the embedded graph size distributions depending on the sizes of the graph atom learned thanks to srGW and its sparse variant srGW g .

Revised: For any graph atom size, the mean of each embedded graph size distribution represented with a white dot is below the atom size, hence embeddings are sparse and subparts of the atom are indeed selected.

Revised: Moreover, promoting sparsity of the embeddings (srGW g ) lead to more concentrated embedded graph size distributions with lower averaged sizes than its unregularized counterpart ( srGW ), as expected.

Original: Experiment details on graphs completion.
Revised: Finally, these distributions seem to reach a stable conﬁguration when the graph atom size is Data sample C i , h i Data sample (colored by T ) Projected ̄ C , ̄ h ⋆

Revised: Dictionary ̄ C

Revised: Data sample C i , h i

Revised: Data sample (colored by T )

Revised: Projected ̄ C , ̄ h ⋆

Revised: Dictionary ̄ C

Revised: Data sample C i , h i

Revised: Data sample (colored by T )

Revised: Projected ̄ C , ̄ h ⋆

Revised: Dictionary ̄ C

Revised: Data sample C i , h i

Revised: Data sample (colored by T )

Revised: Projected ̄ C , ̄ h ⋆

Revised: Dictionary ̄ C large enough.

Revised: This argues in favor of the existence of a (rather small) threshold on the atom size where the heterogeneity of all the graphs contained in the dataset is well summarized in the dictionaries.

Revised: 7.5. D ICTIONARY L EARNING : C OMPLETION EXPERIMENTS

Original:  For these completion experiments the exact same scheme, than in our clustering benchmark, is applied for learning srGW and GDL dictionaries on formed datasets D train .
Revised: Experiment details on graphs completion. For these completion experiments the exact same scheme, than in our clustering benchmark, is applied for learning srGW and GDL dictionaries on formed datasets D train .

Original: On interesting speciﬁcity used on dataset IMDB-B was for the initialization.
Revised: On interesting speciﬁcity used on dataset IMDB-B was for the initialization.

Original: Indeed instead of initializing the structure atom iid from N (0 . 5 , 0 . 01) , we initialized the entries of
Revised: Indeed instead of initializing the structure atom iid from N (0 . 5 , 0 . 01) , we initialized the entries of

Original: C imp denoting connections only between the new imputed nodes randomly.
Revised: C imp denoting connections only between the new imputed nodes randomly.

Original: Then we initialized entries representating connections of these nodes to the ones of C obs seeing the degree (scaled by maximum observed degrees) of each observed node as initial probability for a new node to be connected to this observed node.
Revised: Then we initialized entries representating connections of these nodes to the ones of C obs seeing the degree (scaled by maximum observed degrees) of each observed node as initial probability for a new node to be connected to this observed node.

Original: Then for each entries ( i, j ) where x i belongs to C obs and x j to
Revised: Then for each entries ( i, j ) where x i belongs to C obs and x j to

Original: C imp , C ij is initialized as N ( deg ( x i ) max iobs deg ( x i ) , 0 . 01) .
Revised: C imp , C ij is initialized as N ( [Equation] ( x [Equation]

Original:  This initialization led to better performances for both model so might be a ﬁrst good practice to tackle completion of clustered graph using OT.
Revised: [Equation] ( x i ) , 0 . 01) . This initialization led to better performances for both model so might be a ﬁrst good practice to tackle completion of clustered graph using OT.

Original: For
Revised: For

Original:  MUTAG we sticked with the initialization used for our and initialized imputed features randomly in the range of locally observed features.
Revised: [Figure] l e t i o n a cc u r a c [Figure] MUTAG we sticked with the initialization used for our and initialized imputed features randomly in the range of locally observed features.

Original: [Figure]
Revised: Additional experiments on graphs completion.

Original: Dictionary
Revised: We complete the experiments on completion tasks of datasets IMDB-B and MUTAG reported in the subsection 5.3 of the main paper. Let us recall that for these experiments we ﬁxed a percentage of imputed nodes (10 % and 20%) and looked at the evolution of completion performances over the proportion of train/test datasets. Here instead

Original: [Figure]
Revised: [Figure]

Original: C
Revised: we ﬁx the proportion of the test dataset to 10% and make percentage of imputed nodes vary in { 10 , 15 , 20 , 25 , 30 } . A similar benchmark procedure than for experiments of the main paper is conducted.

Original: [Figure]
Revised: [Figure]

Original: Projected
Revised: These graph completion results are reported in Figure 7 for IMDB-B dataset and in Figure

Original: [Figure] C , ̄ h ⋆ Dictionary ̄ C [Figure]
Revised: Our srGW dictionary learning and its regularized variants outperform GDL and GDL λ

Original: Projected [Figure]
Revised: consistently when the percentage of imputed nodes is not too high ( < 20% ), whereas this trend is reversed for high percentage of imputed nodes.

Original: C , ̄
Revised: Indeed, as srGW dictionaries capture subgraph patterns of variable resolutions from the input graphs, the scarcity of prior information in an observed graph leads to a too high number of valid possibilities to complete it. Whereas GDL dictionaries based on GW lead to more steady performances as they keep their focus on global structures. Interestingly, the sparsity promoting regularization can clearly compensate this kind of overﬁtting over subgraphs for higher levels of imputed nodes and systematically leads to better completion performances (high accuracy, low

Original: [Figure] ⋆ Dictionary ̄ C
Revised: Means Square Error).

Original: [Figure]
Revised: Moreover, the entropic regularization of srGW ( srGW e and srGW e + g ) can be favorably used to compensate this overﬁtting pattern for high percentages of imputed nodes ( > 20% ) and also pairs well with the sparse regularization ( srGW e + g ).