Original: I NFO G RAPH : U NSUPERVISED AND S EMI - SUPERVISED G RAPH -L EVEL R EPRESENTATION L EARNING VIA M UTUAL I NFORMATION M AXIMIZATION
Revised: I NFO G RAPH : U NSUPERVISED AND S EMI - SUPERVISED G RAPH -L EVEL R EPRESENTATION L EARNING VIA M UTUAL I NFORMATION M AXIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios.
Revised: This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios.

Original: Graph-level representations are critical in a variety of real-world applications such as predicting the properties of molecules and community analysis in social networks.
Revised: Graph-level representations are critical in a variety of real-world applications such as predicting the properties of molecules and community analysis in social networks.

Original: Traditional graph kernel based methods are simple, yet effective for obtaining ﬁxed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs.
Revised: Traditional graph kernel based methods are simple, yet effective for obtaining ﬁxed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs.

Original: There are also some recent methods based on language models (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives.
Revised: There are also some recent methods based on language models (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives.

Original: Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations.
Revised: Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations.

Original: We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles).
Revised: We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles).

Original: By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures.
Revised: By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures.

Original: Furthermore, we further propose
Revised: Furthermore, we further propose

Original: InfoGraph*, an extension of InfoGraph for semi-supervised scenarios.
Revised: InfoGraph*, an extension of InfoGraph for semi-supervised scenarios.

Original: InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGraph and the representations learned by existing supervised methods.
Revised: InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGraph and the representations learned by existing supervised methods.

Original: As a result, the supervised encoder learns from unlabeled data while preserving the latent semantic space favored by the current supervised task.
Revised: As a result, the supervised encoder learns from unlabeled data while preserving the latent semantic space favored by the current supervised task.

Original: Experimental results on the tasks of graph classiﬁcation and molecular property prediction show that InfoGraph is superior to state-of-the-art baselines and InfoGraph* can achieve performance competitive with state-of-the-art semi-supervised models.
Revised: Experimental results on the tasks of graph classiﬁcation and molecular property prediction show that InfoGraph is superior to state-of-the-art baselines and InfoGraph* can achieve performance competitive with state-of-the-art semi-supervised models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs have proven to be an effective way to represent very diverse types of data including social networks Newman & Girvan (2004), biological reaction networksPavlopoulos et al.
Revised: Graphs have proven to be an effective way to represent very diverse types of data including social networks Newman & Girvan (2004), biological reaction networksPavlopoulos et al.

Original: (2011), proteinprotein interactions Krogan et al.
Revised: (2011), proteinprotein interactions Krogan et al.

Original: (2006), the quantum mechanical properties of individual molecules
Revised: (2006), the quantum mechanical properties of individual molecules

Original: Xie & Grossman (2018); Jin et al.
Revised: Xie & Grossman (2018); Jin et al.

Original: (2018), and many more.
Revised: (2018), and many more.

Original: Graphs provide explicit information about the coupling between individual units in a larger part along with a well deﬁned framework for assigning properties to the nodes and the edges connecting them.
Revised: Graphs provide explicit information about the coupling between individual units in a larger part along with a well deﬁned framework for assigning properties to the nodes and the edges connecting them.

Original: There has been a signiﬁcant amount of previous work done studying many aspects of graphs including link prediction Gao et al.
Revised: There has been a signiﬁcant amount of previous work done studying many aspects of graphs including link prediction Gao et al.

Original: (2011); Wang et al.
Revised: (2011); Wang et al.

Original: and node prediction Blei et al.
Revised: and node prediction Blei et al.

Original: Due to its ﬂexibility, graph-like data structures can capture rich information which is critical in many applications.
Revised: Due to its ﬂexibility, graph-like data structures can capture rich information which is critical in many applications.

Original: At the lowest level, much work has been done on learning node representations– low-dimensional vector embeddings of individual nodes Perozzi et al.
Revised: At the lowest level, much work has been done on learning node representations– low-dimensional vector embeddings of individual nodes Perozzi et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: ; Grover & Leskovec (2016).
Revised: ; Grover & Leskovec (2016).

Original: Another ﬁeld that has attracted a large amount of attention recently is learning representations of entire graphs.
Revised: Another ﬁeld that has attracted a large amount of attention recently is learning representations of entire graphs.

Original: Such a problem is critical in a variety of applications such as predicting the properties of molecular graphs in both drug discovery and material science Chen et al.
Revised: Such a problem is critical in a variety of applications such as predicting the properties of molecular graphs in both drug discovery and material science Chen et al.

Original: (2019b;a).
Revised: (2019b;a).

Original: There has been some recent progress based on neural message passing algorithms Gilmer et al.
Revised: There has been some recent progress based on neural message passing algorithms Gilmer et al.

Original: ; Xie & Grossman (2018), which learn the representations of entire graphs in a supervised way.
Revised: ; Xie & Grossman (2018), which learn the representations of entire graphs in a supervised way.

Original: These methods have been shown achieving state-of-the-art results on a variety of different prediction tasks
Revised: These methods have been shown achieving state-of-the-art results on a variety of different prediction tasks

Original: Kipf et al.
Revised: Kipf et al.

Original: ; Xie & Grossman (2018); Gilmer et al.
Revised: ; Xie & Grossman (2018); Gilmer et al.

Original: ; Chen et al. (2019a).
Revised: ; Chen et al. (2019a).

Original: However, one of the most difﬁcult obstacles for supervised learning on graphs is that it is often very costly or even impossible to collect annotated labels.
Revised: However, one of the most difﬁcult obstacles for supervised learning on graphs is that it is often very costly or even impossible to collect annotated labels.

Original: For example, in the chemical domain labels are typically produced with a costly Density Functional Theory (DFT) calculation.
Revised: For example, in the chemical domain labels are typically produced with a costly Density Functional Theory (DFT) calculation.

Original: One option is to use semi-supervised methods which combine a small handful of labels with a larger, unlabeled, dataset.
Revised: One option is to use semi-supervised methods which combine a small handful of labels with a larger, unlabeled, dataset.

Original: In real-world applications, partially labeled datasets are common, making tools that are able to efﬁciently utilize the present labels particularly useful.
Revised: In real-world applications, partially labeled datasets are common, making tools that are able to efﬁciently utilize the present labels particularly useful.

Original: Coming up with methods that are able to learn unsupervised representations of an entire graph, as opposed to nodes, is an important step in working with unlabeled or partially labeled graphs
Revised: Coming up with methods that are able to learn unsupervised representations of an entire graph, as opposed to nodes, is an important step in working with unlabeled or partially labeled graphs

Original: Narayanan et al.
Revised: Narayanan et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: ; Nguyen et al.
Revised: ; Nguyen et al.

Original: For example, there exists work that explores pre-training techniques for graphs to improve generalization Hu et al.
Revised: For example, there exists work that explores pre-training techniques for graphs to improve generalization Hu et al.

Original: Another common approach to unsupervised representation learning on graphs is through graph kernels Pržulj (2007); Kashima et al.
Revised: Another common approach to unsupervised representation learning on graphs is through graph kernels Pržulj (2007); Kashima et al.

Original: ; Orsini et al.
Revised: ; Orsini et al.

Original: However, many of these methods do not provide explicit graph embeddings which many machine learning algorithms operate on.
Revised: However, many of these methods do not provide explicit graph embeddings which many machine learning algorithms operate on.

Original: Furthermore, the handcrafted features of graph kernels lead to high dimensional, sparse or non-smooth representations and thus result in poor generalization performance, especially on large datasets Narayanan et al.
Revised: Furthermore, the handcrafted features of graph kernels lead to high dimensional, sparse or non-smooth representations and thus result in poor generalization performance, especially on large datasets Narayanan et al.

Original: Unsupervised learning of latent representations is also an important problem in other domains, such as image generation Kingma & Welling (2013);
Revised: Unsupervised learning of latent representations is also an important problem in other domains, such as image generation Kingma & Welling (2013);

Original: Kim & Mnih (2018) and natural language processing
Revised: Kim & Mnih (2018) and natural language processing

Original: Mikolov et al. (2013a).
Revised: Mikolov et al. (2013a).

Original: A recent work introduced Deep Infomax, a method that maximizes the mutual information content between the input data and the learned representation Hjelm et al.
Revised: A recent work introduced Deep Infomax, a method that maximizes the mutual information content between the input data and the learned representation Hjelm et al.

Original: This method outperforms other methods on many unsupervised learning tasks.
Revised: This method outperforms other methods on many unsupervised learning tasks.

Original: Motivated by Deep
Revised: Motivated by Deep

Original: InfoMax Hjelm et al.
Revised: InfoMax Hjelm et al.

Original: (2018), we aim to use mutual information maximization for unsupervised representation learning on the entire graph.
Revised: (2018), we aim to use mutual information maximization for unsupervised representation learning on the entire graph.

Original: Speciﬁcally, our objective is to maximize the mutual information between the representations of entire graphs and the representations of substructures of different granularity.
Revised: Speciﬁcally, our objective is to maximize the mutual information between the representations of entire graphs and the representations of substructures of different granularity.

Original: We name our model InfoGraph .
Revised: We name our model InfoGraph .

Original: We also propose a semi-supervised learning model which we name InfoGraph* .
Revised: We also propose a semi-supervised learning model which we name InfoGraph* .

Original: We employ a student-teacher framework similar to Mean-Teacher method Tarvainen & Valpola (2017).
Revised: We employ a student-teacher framework similar to Mean-Teacher method Tarvainen & Valpola (2017).

Original: We maximize the mutual information between intermediate representations of the two models so that the student model learns from the teacher model.
Revised: We maximize the mutual information between intermediate representations of the two models so that the student model learns from the teacher model.

Original: The student model is trained on the labeled data using a supervised objective function while the teacher model is trained on unlabeled data with InfoGraph .
Revised: The student model is trained on the labeled data using a supervised objective function while the teacher model is trained on unlabeled data with InfoGraph .

Original: Using InfoGraph* , we achieve performance competitive with state-of-the-art methods on molecular property prediction.
Revised: Using InfoGraph* , we achieve performance competitive with state-of-the-art methods on molecular property prediction.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We propose InfoGraph, an unsupervised graph representation learning method based on Deep InfoMax (DIM) Hjelm et al.
Revised: • We propose InfoGraph, an unsupervised graph representation learning method based on Deep InfoMax (DIM) Hjelm et al.

Original: • We show that InfoGraph can be extended to semi-supervised prediction tasks on graphs.
Revised: • We show that InfoGraph can be extended to semi-supervised prediction tasks on graphs.

Original: • We empirically show that InfoGraph surpasses state-of-the-art performance on graph classiﬁcation tasks with unsupervised learning and obtains performance comparable with state-ofart methods on molecular property prediction tasks using semi-supervised learning.
Revised: • We empirically show that InfoGraph surpasses state-of-the-art performance on graph classiﬁcation tasks with unsupervised learning and obtains performance comparable with state-ofart methods on molecular property prediction tasks using semi-supervised learning.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Representation learning for graphs has mainly dealt with supervised learning tasks.
Revised: Representation learning for graphs has mainly dealt with supervised learning tasks.

Original: Recently, however, researchers have proposed algorithms that learn graph-level representations in an unsupervised manner
Revised: Recently, however, researchers have proposed algorithms that learn graph-level representations in an unsupervised manner

Original: Narayanan et al.
Revised: Narayanan et al.

Original: ; Adhikari et al.
Revised: ; Adhikari et al.

Original: Concurrently to this work, information maximizing graph neural networks (IGNN) was introduced which uses mutual information maximization between edge states and transform parameters to achieve state-of-the-art predictions on a variety of supervised molecule property prediction tasks
Revised: Concurrently to this work, information maximizing graph neural networks (IGNN) was introduced which uses mutual information maximization between edge states and transform parameters to achieve state-of-the-art predictions on a variety of supervised molecule property prediction tasks

Original: Chen et al.
Revised: Chen et al.

Original: (2019b).
Revised: (2019b).

Original: In this work, our focus is on unsupervised and semi-supervised scenarios.
Revised: In this work, our focus is on unsupervised and semi-supervised scenarios.

Original: Graph Kernels .
Revised: Graph Kernels .

Original: Constructing graph kernels is a common unsupervised task in learning graph representations.
Revised: Constructing graph kernels is a common unsupervised task in learning graph representations.

Original: These kernels are typically evaluated on node classiﬁcation tasks.
Revised: These kernels are typically evaluated on node classiﬁcation tasks.

Original: In graph kernels, a graph G is decomposed into (possibly different) { G s } sub-structures.
Revised: In graph kernels, a graph G is decomposed into (possibly different) { G s } sub-structures.

Original: The graph kernel K ( G 1 , G 2 ) is deﬁned based on the frequency of each sub-structure appearing in G 1 and G 2 respectively.
Revised: The graph kernel K ( G 1 , G 2 ) is deﬁned based on the frequency of each sub-structure appearing in G 1 and G 2 respectively.

Original: Namely,
Revised: Namely,

Original: K ( G 1 , G 2 ) =
Revised: K ( G 1 , G 2 ) =

Original: (cid:104)
Revised: (cid:104)

Original: f
Revised: f

Original: G s 1 , f G s 2 (cid:105) , where f G s is the vector containing frequencies of { G s } sub-structures, and (cid:104) , (cid:105) is an inner product in an appropriately normalized vector space.
Revised: G s 1 , f G s 2 (cid:105) , where f G s is the vector containing frequencies of { G s } sub-structures, and (cid:104) , (cid:105) is an inner product in an appropriately normalized vector space.

Original: Much work has been devoted to deciding which sub-structures are more suitable than others (refer to appendix A.1).
Revised: Much work has been devoted to deciding which sub-structures are more suitable than others (refer to appendix A.1).

Original: Instead of deﬁning hand crafted similarity measures between substructures, InfoGraph adopts a more principled metric – mutual information.
Revised: Instead of deﬁning hand crafted similarity measures between substructures, InfoGraph adopts a more principled metric – mutual information.

Original: Contrastive methods .
Revised: Contrastive methods .

Original: An important approach for unsupervised representation learning is to train an encoder to be contrastive between representations that capture statistical dependencies of interest and those that do not.
Revised: An important approach for unsupervised representation learning is to train an encoder to be contrastive between representations that capture statistical dependencies of interest and those that do not.

Original: For example, a contrastive approach may employ a scoring function, training the encoder to increase the score on “real” input (a.k.a, positive examples) and decrease the score on “fake” input (a.k.a., negative samples).
Revised: For example, a contrastive approach may employ a scoring function, training the encoder to increase the score on “real” input (a.k.a, positive examples) and decrease the score on “fake” input (a.k.a., negative samples).

Original: For more detailed discussion, refer to appendix A.2.
Revised: For more detailed discussion, refer to appendix A.2.

Original: Deep Graph InfoMax (DGI) Veliˇckovi´c et al.
Revised: Deep Graph InfoMax (DGI) Veliˇckovi´c et al.

Original: (2018) also belongs to this category, which aims to train a node encoder that maximizes mutual information between node representations and the pooled global graph representation.
Revised: (2018) also belongs to this category, which aims to train a node encoder that maximizes mutual information between node representations and the pooled global graph representation.

Original: Although we built upon a similar methodology, our aim is different than theirs as our goal is to obtain embeddings at the whole graph level for unsupervised and semisupervised learning whereas DGI only evaluates node level embeddings.
Revised: Although we built upon a similar methodology, our aim is different than theirs as our goal is to obtain embeddings at the whole graph level for unsupervised and semisupervised learning whereas DGI only evaluates node level embeddings.

Original: In order to differentiate our method with Deep Graph Infomax (Veliˇckovi´c et al.
Revised: In order to differentiate our method with Deep Graph Infomax (Veliˇckovi´c et al.

Original: , we term our model InfoGraph.
Revised: , we term our model InfoGraph.

Original: Semi-supervised Learning .
Revised: Semi-supervised Learning .

Original: A comprehensive overview of semi-supervised learning (SSL) methods is out of the scope of this paper.
Revised: A comprehensive overview of semi-supervised learning (SSL) methods is out of the scope of this paper.

Original: We refer readers to Appendix B for a short overview or Zhu et al.
Revised: We refer readers to Appendix B for a short overview or Zhu et al.

Original: ; Chapelle et al.
Revised: ; Chapelle et al.

Original: ; Oliver et al.
Revised: ; Oliver et al.

Original: (2018) for more comprehensive discussions.
Revised: (2018) for more comprehensive discussions.

Original: Here, we discuss a state-of-the-art method applicable for regression tasks – Mean Teacher Tarvainen &
Revised: Here, we discuss a state-of-the-art method applicable for regression tasks – Mean Teacher Tarvainen &

Original: Valpola (2017).Mean Teacher adds a loss term which encourages the distance between the original network’s output and the teacher’s output to be small.
Revised: Valpola (2017).Mean Teacher adds a loss term which encourages the distance between the original network’s output and the teacher’s output to be small.

Original: The teacher’s predictions are made using an exponential moving average of parameters from previous training steps.
Revised: The teacher’s predictions are made using an exponential moving average of parameters from previous training steps.

Original: Inspired by the “studentteacher” framework in Mean Teacher model, our semi-supervised model (InfoGraph*) deploys two separate encoders but instead of explicitly encouraging the output of the student model to be similar to the teacher model’s output, we enable the student model to learn from the teacher model by maximizing mutual information between intermediate representations learned by two models.
Revised: Inspired by the “studentteacher” framework in Mean Teacher model, our semi-supervised model (InfoGraph*) deploys two separate encoders but instead of explicitly encouraging the output of the student model to be similar to the teacher model’s output, we enable the student model to learn from the teacher model by maximizing mutual information between intermediate representations learned by two models.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Most recent work on graphs focus on supervised learning tasks or learning node representations.
Revised: Most recent work on graphs focus on supervised learning tasks or learning node representations.

Original: However, many graph analytic tasks such as graph classiﬁcation, regression, and clustering require representing entire graphs as ﬁxed-length feature vectors.
Revised: However, many graph analytic tasks such as graph classiﬁcation, regression, and clustering require representing entire graphs as ﬁxed-length feature vectors.

Original: Though graph-level representations can be obtained through the node-level representations implicitly, explicitly extracting the graph can be more straightforward and optimal for graph-oriented tasks.
Revised: Though graph-level representations can be obtained through the node-level representations implicitly, explicitly extracting the graph can be more straightforward and optimal for graph-oriented tasks.

Original: Another scenario that is important, yet attracts comparatively less attention in the graph related literature is semi-supervised learning.
Revised: Another scenario that is important, yet attracts comparatively less attention in the graph related literature is semi-supervised learning.

Original: One of the biggest challenges in prediction tasks in biology Yan et al. (2017); Yang et al.
Revised: One of the biggest challenges in prediction tasks in biology Yan et al. (2017); Yang et al.

Original: (2014) or molecular machine learning Duvenaud et al.
Revised: (2014) or molecular machine learning Duvenaud et al.

Original: ; Gilmer et al.
Revised: ; Gilmer et al.

Original: (2017); Jia & Liang (2017) is the extreme scarcity of labeled data.
Revised: (2017); Jia & Liang (2017) is the extreme scarcity of labeled data.

Original: Therefore, semi-supervised learning, in which a large number of unlabeled samples are incorporated with a small number of labeled samples to enhance accuracy of models, will play a key role in these areas.
Revised: Therefore, semi-supervised learning, in which a large number of unlabeled samples are incorporated with a small number of labeled samples to enhance accuracy of models, will play a key role in these areas.

Original: In this section, we ﬁrst formulate an unsupervised whole graph representation learning problem and a semi-supervised prediction task on graphs.
Revised: In this section, we ﬁrst formulate an unsupervised whole graph representation learning problem and a semi-supervised prediction task on graphs.

Original: Then, we present our method to learn graph-level representations.
Revised: Then, we present our method to learn graph-level representations.

Original: Afterwards we present our proposed model for the semi-supervised learning scenario.
Revised: Afterwards we present our proposed model for the semi-supervised learning scenario.

Original: 3. P ROBLEM D EFINITION
Revised: 3. P ROBLEM D EFINITION

Original: Unsupervised Graph Representation Learning .
Revised: Unsupervised Graph Representation Learning .

Original: Given a set of graphs G = { G 1 , G 2 , ... } and a positive integer δ (the expected embedding size), our goal is to learn a δ -dimensional distributed representation of every graph G i ∈ G .
Revised: Given a set of graphs G = { G 1 , G 2 , ... } and a positive integer δ (the expected embedding size), our goal is to learn a δ -dimensional distributed representation of every graph G i ∈ G .

Original: We denote the number of nodes in G i as | G i | .
Revised: We denote the number of nodes in G i as | G i | .

Original: We denote the matrix of representations of all graphs as Φ ∈ R | G |× δ .
Revised: We denote the matrix of representations of all graphs as Φ ∈ R | G |× δ .

Original: Semi-supervied Graph Prediction Tasks .
Revised: Semi-supervied Graph Prediction Tasks .

Original: Given a set of labeled graphs G L = { G 1 , · · · , G | G L | } with corresponding output { o 1 , · · · , o | G L | } , and a set of unlabeled samples G U =
Revised: Given a set of labeled graphs G L = { G 1 , · · · , G | G L | } with corresponding output { o 1 , · · · , o | G L | } , and a set of unlabeled samples G U =

Original: { G | G L | +1 , · · · , G | G L | + | G U | }
Revised: { G | G L | +1 , · · · , G | G L | + | G U | }

Original: 3. I NFO G RAPH
Revised: 3. I NFO G RAPH

Original: We focus on graph neural networks (GNNs)—a ﬂexible class of embedding architectures which generate node representations by repeated aggregation over local node neighborhoods.
Revised: We focus on graph neural networks (GNNs)—a ﬂexible class of embedding architectures which generate node representations by repeated aggregation over local node neighborhoods.

Original: The representations of nodes are learned by aggregating the features of their neighborhood nodes, so we refer to these as patch representations.
Revised: The representations of nodes are learned by aggregating the features of their neighborhood nodes, so we refer to these as patch representations.

Original: GNNs utilize a READOUT function to summarize all the obtained patch representations into a ﬁxed length graph-level representation.
Revised: GNNs utilize a READOUT function to summarize all the obtained patch representations into a ﬁxed length graph-level representation.

Original: Formally, the k -th layer of a GNN is
Revised: Formally, the k -th layer of a GNN is

Original: [Equation]
Revised: [Equation]

Original: where h ( k ) v is the feature vector of node v at the k -th iteration/layer (or patch representation centered at node i ),
Revised: where h ( k ) v is the feature vector of node v at the k -th iteration/layer (or patch representation centered at node i ),

Original: e uv is the feature vector of the edge between u and v , and N ( v ) are neighborhoods to node v .
Revised: e uv is the feature vector of the edge between u and v , and N ( v ) are neighborhoods to node v .

Original: h (0) v is often initialized as node features.
Revised: h (0) v is often initialized as node features.

Original: READOUT can be a simple permutation invariant function such as averaging or a more sophisticated graph-level pooling function Ying et al.
Revised: READOUT can be a simple permutation invariant function such as averaging or a more sophisticated graph-level pooling function Ying et al.

Original: Zhang et al.
Revised: Zhang et al.

Original: We seek to obtain graph representations by maximizing the mutual information between graph-level and patch-level representations.
Revised: We seek to obtain graph representations by maximizing the mutual information between graph-level and patch-level representations.

Original: By doing so, the graph representations can learn to encode aspects of the data that are shared across all substructures.
Revised: By doing so, the graph representations can learn to encode aspects of the data that are shared across all substructures.

Original: Assume that we are given a set of training samples
Revised: Assume that we are given a set of training samples

Original: G :=
Revised: G :=

Original: { G j ∈ G } Nj =1 with empirical probability distribution P on the input space.
Revised: { G j ∈ G } Nj =1 with empirical probability distribution P on the input space.

Original: Let φ denote the set of parameters of a K -layer graph neural network.
Revised: Let φ denote the set of parameters of a K -layer graph neural network.

Original: After the ﬁrst k layers of the graph neural network, the input graph will be encoded into a set of patch representations { h ( k ) i } Ni =1 .
Revised: After the ﬁrst k layers of the graph neural network, the input graph will be encoded into a set of patch representations { h ( k ) i } Ni =1 .

Original: Next, we summarize feature vectors at all depths of the graph neural network into a single feature vector that captures patch information at different scales centered at every node.
Revised: Next, we summarize feature vectors at all depths of the graph neural network into a single feature vector that captures patch information at different scales centered at every node.

Original: Inspired by Xu et al.
Revised: Inspired by Xu et al.

Original: (2018b), we use concatenation.
Revised: (2018b), we use concatenation.

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: where h iφ is the summarized patch representation centered at node i and H φ ( G ) is the global representation after applying READOUT.
Revised: where h iφ is the summarized patch representation centered at node i and H φ ( G ) is the global representation after applying READOUT.

Original: Note that here we slightly abuse the notation of h .
Revised: Note that here we slightly abuse the notation of h .

Original: We deﬁne our mutual information (MI) estimator on global/local pairs, maximizing the estimated MI over the given dataset G :=
Revised: We deﬁne our mutual information (MI) estimator on global/local pairs, maximizing the estimated MI over the given dataset G :=

Original: { G j ∈ G } N j =1 :
Revised: { G j ∈ G } N j =1 :

Original: [Equation]
Revised: [Equation]

Original: I φ,ψ is the mutual information estimator modeled by discriminator T ψ and parameterized by a neural network with parameters ψ .
Revised: I φ,ψ is the mutual information estimator modeled by discriminator T ψ and parameterized by a neural network with parameters ψ .

Original: We use the Jensen-Shannon MI estimator (following the formulation of
Revised: We use the Jensen-Shannon MI estimator (following the formulation of

Original: Nowozin et al.
Revised: Nowozin et al.

Original: [Equation]
Revised: [Equation]

Original: where x is an input sample, x (cid:48) (negative sample) is an input sampled from ˜ P = P , a distribution identical to the empirical probability distribution of the input space, and sp ( z )
Revised: where x is an input sample, x (cid:48) (negative sample) is an input sampled from ˜ P = P , a distribution identical to the empirical probability distribution of the input space, and sp ( z )

Original: = log(1 + e z ) is the softplus function.
Revised: = log(1 + e z ) is the softplus function.

Original: In practice, we generate negative samples using all possible combinations of global and local patch representations across all graph instances in a batch.
Revised: In practice, we generate negative samples using all possible combinations of global and local patch representations across all graph instances in a batch.

Original: Since H φ ( G ) is encouraged to have high MI with patches that contain information at all scales, this favours encoding aspects of the data that are shared across patches and aspects that are shared across scales.
Revised: Since H φ ( G ) is encouraged to have high MI with patches that contain information at all scales, this favours encoding aspects of the data that are shared across patches and aspects that are shared across scales.

Original: The algorithm is illustrated in Fig.
Revised: The algorithm is illustrated in Fig.

Original: It should be noted that our model is similar to Deep Graph Infomax (DGI) Veliˇckovi´c et al.
Revised: It should be noted that our model is similar to Deep Graph Infomax (DGI) Veliˇckovi´c et al.

Original: (2018), a model proposed for learning unsupervised node embeddings.
Revised: (2018), a model proposed for learning unsupervised node embeddings.

Original: However, there are important design differences due to the different problems that we are focusing on.
Revised: However, there are important design differences due to the different problems that we are focusing on.

Original: First, in DGI they use random sampling to obtain negative samples due to the fact that they are mainly focusing on learning node embeddings on a graph.
Revised: First, in DGI they use random sampling to obtain negative samples due to the fact that they are mainly focusing on learning node embeddings on a graph.

Original: However, contrastive methods require a large number of negative samples to be competitive Hjelm et al.
Revised: However, contrastive methods require a large number of negative samples to be competitive Hjelm et al.

Original: (2018), thus the use of batch-wise generation of negative samples is crucial as we are trying to learn graph embeddings given many graph instances.
Revised: (2018), thus the use of batch-wise generation of negative samples is crucial as we are trying to learn graph embeddings given many graph instances.

Original: Second, the choice of graph convolution encoders is also crucial.
Revised: Second, the choice of graph convolution encoders is also crucial.

Original: We use GIN Xu et al.
Revised: We use GIN Xu et al.

Original: (2018a) while DGI uses GCN Kipf & Welling (2016) as GIN provides a better inductive bias for graph level applications.
Revised: (2018a) while DGI uses GCN Kipf & Welling (2016) as GIN provides a better inductive bias for graph level applications.

Original: Graph neural network designs should be considered carefully so that graph representations can be discriminative towards other graph instances.
Revised: Graph neural network designs should be considered carefully so that graph representations can be discriminative towards other graph instances.

Original: For example, we use sum over mean for READOUT and that can provide important information regarding the size of the graph.
Revised: For example, we use sum over mean for READOUT and that can provide important information regarding the size of the graph.

Original: 3. S EMI - SUPERVISED I NFO G RAPH
Revised: 3. S EMI - SUPERVISED I NFO G RAPH

Original: Based on the previous unsupervised model, a straightforward way to do semi-supervised property prediction on graphs is to combine the purely supervised loss and the unsupervised objective function which acts as a regularization term.
Revised: Based on the previous unsupervised model, a straightforward way to do semi-supervised property prediction on graphs is to combine the purely supervised loss and the unsupervised objective function which acts as a regularization term.

Original: In doing so, the model is trained to predict properties for the labeled dataset while keeping a rich discriminative intermediate representation learned from both the labeled and the unlabeled dataset.
Revised: In doing so, the model is trained to predict properties for the labeled dataset while keeping a rich discriminative intermediate representation learned from both the labeled and the unlabeled dataset.

Original: That is, we try to minimize the following objective function:
Revised: That is, we try to minimize the following objective function:

Original: [Equation]
Revised: [Equation]

Original: where L supervised ( y φ ( G i ) , o i ) is deﬁned as the loss function of graph G i that measures the discrepancy between the classiﬁer output y φ ( G i ) and the true output o i .
Revised: where L supervised ( y φ ( G i ) , o i ) is deﬁned as the loss function of graph G i that measures the discrepancy between the classiﬁer output y φ ( G i ) and the true output o i .

Original: L unsupervised ( h φ ( G j ); H φ ( G j ) is the unsupervised InfoGraph loss term as deﬁned in eq.
Revised: L unsupervised ( h φ ( G j ); H φ ( G j ) is the unsupervised InfoGraph loss term as deﬁned in eq.

Original: equation 4 that can be optimized using both labeled and unlabeled data.
Revised: equation 4 that can be optimized using both labeled and unlabeled data.

Original: The hyper-parameter λ controls the relative weight between the purely supervised and the unsupervised loss.
Revised: The hyper-parameter λ controls the relative weight between the purely supervised and the unsupervised loss.

Original: The intuition behind this is that the model will beneﬁt from learning a good representation from the large amount of unlabeled data while learning to predict the corresponding supervised label.
Revised: The intuition behind this is that the model will beneﬁt from learning a good representation from the large amount of unlabeled data while learning to predict the corresponding supervised label.

Original: However, supervised tasks and unsupervised tasks may favor different information or a different semantic space.
Revised: However, supervised tasks and unsupervised tasks may favor different information or a different semantic space.

Original: Simply combining the two loss functions using the same encoder may lead to “negative transfer” 1 (Pan & Yang, 2009; Rosenstein et al., 2005).
Revised: Simply combining the two loss functions using the same encoder may lead to “negative transfer” 1 (Pan & Yang, 2009; Rosenstein et al., 2005).

Original: We propose a simple way to alleviate this problem: we deploy two encoder models: the encoder on the labelled data (supervised encoder) and the encoder on the unlabelled data (unsupervised encoder).
Revised: We propose a simple way to alleviate this problem: we deploy two encoder models: the encoder on the labelled data (supervised encoder) and the encoder on the unlabelled data (unsupervised encoder).

Original: For transferring the learned representations from the unsupervised encoder to the supervised encoder, we deﬁne a loss term that encourages the representations learned by the two encoders to have high mutual information, at all levels of representations (third term of Eq.
Revised: For transferring the learned representations from the unsupervised encoder to the supervised encoder, we deﬁne a loss term that encourages the representations learned by the two encoders to have high mutual information, at all levels of representations (third term of Eq.

Original: Formally, let ϕ denote the set of parameters of another
Revised: Formally, let ϕ denote the set of parameters of another

Original: K -layered graph neural network, identical to the one parameterized by φ , and let λ be a tunable hyper-parameter, H kφ ( G ) , H kϕ ( G ) be global encoder representations of the graph G at encoder layer k , then total loss function can be deﬁned as follows:
Revised: K -layered graph neural network, identical to the one parameterized by φ , and let λ be a tunable hyper-parameter, H kφ ( G ) , H kϕ ( G ) be global encoder representations of the graph G at encoder layer k , then total loss function can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Notice that this formulation can be seen as a special instance of the student-teacher framework.
Revised: Notice that this formulation can be seen as a special instance of the student-teacher framework.

Original: However, unlike the recent student-teacher methods for semi-supervised learning (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Verma et al., 2019b), which enforce the predictions of the student model to be similar to the teacher model, we enforce the transfer of knowledge from the teacher model to the student model via mutual-information maximization at various levels of representations.
Revised: However, unlike the recent student-teacher methods for semi-supervised learning (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Verma et al., 2019b), which enforce the predictions of the student model to be similar to the teacher model, we enforce the transfer of knowledge from the teacher model to the student model via mutual-information maximization at various levels of representations.

Original: In practice, to reduce the computation overhead introduced by the third term of Eq 8, instead of enforcing the mutual-information maximization over all the layers of the encoders, at each training update, we enforce mutual-information maximization on a randomly chosen layer of the encoder (Verma et al., 2019a).
Revised: In practice, to reduce the computation overhead introduced by the third term of Eq 8, instead of enforcing the mutual-information maximization over all the layers of the encoders, at each training update, we enforce mutual-information maximization on a randomly chosen layer of the encoder (Verma et al., 2019a).

Original: In our semi-supervised experiments, we refer to the naive method using the objective function given in eq.
Revised: In our semi-supervised experiments, we refer to the naive method using the objective function given in eq.

Original: equation 6 as InfoGraph.
Revised: equation 6 as InfoGraph.

Original: We refer to the method that uses two separate encoders and employ the objective function given in eq.
Revised: We refer to the method that uses two separate encoders and employ the objective function given in eq.

Original: equation 8 as InfoGraph*.
Revised: equation 8 as InfoGraph*.

Original: InfoGraph* is fully summarized in
Revised: InfoGraph* is fully summarized in

Original: Figure 2.
Revised: Figure 3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the effectiveness of the graph-level representation learned by InfoGraph on downstream graph classiﬁcation tasks and on semi-supervised molecular property prediction tasks.
Revised: We evaluate the effectiveness of the graph-level representation learned by InfoGraph on downstream graph classiﬁcation tasks and on semi-supervised molecular property prediction tasks.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: For graph classiﬁcation, we conduct experiments on 6 well-known benchmark datasets: MUTAG,
Revised: For graph classiﬁcation, we conduct experiments on 6 well-known benchmark datasets: MUTAG,

Original: PTC, REDDIT-BINARY, REDDIT-MULTI-5K, IMDB-BINARY, and IMDB-MULTI (Yanardag &
Revised: PTC, REDDIT-BINARY, REDDIT-MULTI-5K, IMDB-BINARY, and IMDB-MULTI (Yanardag &

Original: Vishwanathan (2015)).
Revised: Vishwanathan (2015)).

Original: For semi-supervised learning tasks, we use the publicly available QM9 dataset
Revised: For semi-supervised learning tasks, we use the publicly available QM9 dataset

Original: Ramakrishnan et al.
Revised: Ramakrishnan et al.

Original: Additional details of the datasets can be found in Appendix B.
Revised: Additional details of the datasets can be found in Appendix B.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: For graph classiﬁcation, we used 6 state-of-the-art graph kernels for comparison: Random Walk (RW)
Revised: For graph classiﬁcation, we used 6 state-of-the-art graph kernels for comparison: Random Walk (RW)

Original: Gärtner et al.
Revised: Gärtner et al.

Original: Shortest Path Kernel (SP) Borgwardt & Kriegel (2005), Graphlet Kernel (GK)
Revised: Shortest Path Kernel (SP) Borgwardt & Kriegel (2005), Graphlet Kernel (GK)

Original: Shervashidze et al.
Revised: Shervashidze et al.

Original: , Weisfeiler-Lehman Sub-tree Kernel (WL) Shervashidze et al.
Revised: , Weisfeiler-Lehman Sub-tree Kernel (WL) Shervashidze et al.

Original: Deep Graph Kernels (DGK) Yanardag & Vishwanathan (2015), and Multi-Scale Laplacian Kernel (MLG) Kondor & Pan (2016).
Revised: Deep Graph Kernels (DGK) Yanardag & Vishwanathan (2015), and Multi-Scale Laplacian Kernel (MLG) Kondor & Pan (2016).

Original: Aside from graph kernels, we also compare with 3 unsupervised graph-level representation learning methods: node2vec Grover & Leskovec (2016), sub2vec Adhikari et al.
Revised: Aside from graph kernels, we also compare with 3 unsupervised graph-level representation learning methods: node2vec Grover & Leskovec (2016), sub2vec Adhikari et al.

Original: (2018), and graph2vec Narayanan et al.
Revised: (2018), and graph2vec Narayanan et al.

Original: Node2vec is a neural embedding framework that learns feature representations of individual nodes in graphs and we aggregate node embeddings to obtain graph embeddings.
Revised: Node2vec is a neural embedding framework that learns feature representations of individual nodes in graphs and we aggregate node embeddings to obtain graph embeddings.

Original: For semi-supervised tasks, aside from comparing the results with the fully supervised results, we also compare our results with a state-of-the-art semi-supervised method: Mean Teachers Tarvainen &
Revised: For semi-supervised tasks, aside from comparing the results with the fully supervised results, we also compare our results with a state-of-the-art semi-supervised method: Mean Teachers Tarvainen &

Original: Valpola (2017).
Revised: Valpola (2017).

Original: 4. E XPERIMENT C ONFIGURATION
Revised: 4. E XPERIMENT C ONFIGURATION

Original: For graph classiﬁcation tasks, we adopt the same procedure of previous works Niepert et al.
Revised: For graph classiﬁcation tasks, we adopt the same procedure of previous works Niepert et al.

Original: Verma & Zhang (2017); Yanardag & Vishwanathan (2015); Zhang et al.
Revised: Verma & Zhang (2017); Yanardag & Vishwanathan (2015); Zhang et al.

Original: (2018) to make a fair comparison and used 10-fold cross validation accuracy to report the classiﬁcation performance.
Revised: (2018) to make a fair comparison and used 10-fold cross validation accuracy to report the classiﬁcation performance.

Original: Experiments are repeated 5 times.
Revised: Experiments are repeated 5 times.

Original: We report results from previous papers with the same experimental setup if available.
Revised: We report results from previous papers with the same experimental setup if available.

Original: If results are not previously reported, we implement them and conduct a hyperparameter search according to the original paper.
Revised: If results are not previously reported, we implement them and conduct a hyperparameter search according to the original paper.

Original: For node2vec Grover & Leskovec (2016), we took the result from Narayanan et al.
Revised: For node2vec Grover & Leskovec (2016), we took the result from Narayanan et al.

Original: but we did not run it on all datasets as the implementation details are not clear in the paper.
Revised: but we did not run it on all datasets as the implementation details are not clear in the paper.

Original: For Deep Graph Kernels, we report the best result out of Deep WL
Revised: For Deep Graph Kernels, we report the best result out of Deep WL

Original: Kernels, Deep GK Kernels, and Deep RW Kernels.
Revised: Kernels, Deep GK Kernels, and Deep RW Kernels.

Original: For sub2vec, we report the best result out of its two variants: sub2vec-N and sub2vec-S .
Revised: For sub2vec, we report the best result out of its two variants: sub2vec-N and sub2vec-S .

Original: For all methods, the embedding dimension is set toand parameters of downstream classiﬁers are independently tuned using cross validation on training folds of data.
Revised: For all methods, the embedding dimension is set toand parameters of downstream classiﬁers are independently tuned using cross validation on training folds of data.

Original: The best average classiﬁcation accuracy is reported for each method.
Revised: The best average classiﬁcation accuracy is reported for each method.

Original: The classiﬁcation accuracies are computed using LIBSVM Chang & Lin (2011), and the C parameter was selected from { 10 − 3 , 10 − 2 , . .
Revised: The classiﬁcation accuracies are computed using LIBSVM Chang & Lin (2011), and the C parameter was selected from { 10 − 3 , 10 − 2 , . .

Original: The QM9 dataset has 130462 molecules in it.
Revised: The QM9 dataset has 130462 molecules in it.

Original: We adopt similar experimental settings as traditional semi-supervised methods Tarvainen & Valpola (2017); Laine & Aila (2016); Miyato et al.
Revised: We adopt similar experimental settings as traditional semi-supervised methods Tarvainen & Valpola (2017); Laine & Aila (2016); Miyato et al.

Original: We randomly chose 5000 samples as labeled samples for training and another 10000 as validation samples, 10000 samples for testing, and use the rest as unlabeled training samples.
Revised: We randomly chose 5000 samples as labeled samples for training and another 10000 as validation samples, 10000 samples for testing, and use the rest as unlabeled training samples.

Original: Note that we use the exact same split when running the supervised model and the semi-supervised model.
Revised: Note that we use the exact same split when running the supervised model and the semi-supervised model.

Original: We use the validation set to do model selection and we report scores on the test set.
Revised: We use the validation set to do model selection and we report scores on the test set.

Original: All targets were normalized to have mean 0 and variance 1.
Revised: All targets were normalized to have mean 0 and variance 1.

Original: We minimize the mean squared error between the model output and the target, although we evaluate mean absolute error.
Revised: We minimize the mean squared error between the model output and the target, although we evaluate mean absolute error.

Original: 4. M ODEL C ONFIGURATION
Revised: 4. M ODEL C ONFIGURATION

Original: For the unsupervised experiments, we use the Graph Isomorphism Network (GIN)
Revised: For the unsupervised experiments, we use the Graph Isomorphism Network (GIN)

Original: Xu et al.
Revised: Xu et al.

Original: (2018a).
Revised: (2018a).

Original: For the semi-supervised experiments, we adopt the same model as in Gilmer et al.
Revised: For the semi-supervised experiments, we adopt the same model as in Gilmer et al.

Original: (2017) (enn-s2s).
Revised: (2017) (enn-s2s).

Original: As recommended in Oliver et al.
Revised: As recommended in Oliver et al.

Original: (2018), we use the exact same underlying model architecture when comparing semi-supervised learning approaches as our goal is not to produce state-of-the-art results, but instead to provide a rigorous comparative analysis in a common framework.
Revised: (2018), we use the exact same underlying model architecture when comparing semi-supervised learning approaches as our goal is not to produce state-of-the-art results, but instead to provide a rigorous comparative analysis in a common framework.

Original: In both scenarios, models were trained using SGD with the Adam optimizer.
Revised: In both scenarios, models were trained using SGD with the Adam optimizer.

Original: We use Pytorch Paszke et al.
Revised: We use Pytorch Paszke et al.

Original: (2017) and the Pytorch Geometric Fey & Lenssen (2019) libraries for all our experiments.
Revised: (2017) and the Pytorch Geometric Fey & Lenssen (2019) libraries for all our experiments.

Original: For detailed hyper-parameter settings and architecture detail of the discriminator, see Appendix C.
Revised: For detailed hyper-parameter settings and architecture detail of the discriminator, see Appendix C.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: to supervised result) of the semi-supervised models using the same underlying model.
Revised: to supervised result) of the semi-supervised models using the same underlying model.

Original: Lower scores are better and values less than 1.0 indicate better performance than the supervised baseline.
Revised: Lower scores are better and values less than 1.0 indicate better performance than the supervised baseline.

Original:  R ESULTS
Revised:  R ESULTS

Original: The results of evaluating unsupervised graph level representations using downstream graph classiﬁcation tasks are presented in Table 1.
Revised: The results of evaluating unsupervised graph level representations using downstream graph classiﬁcation tasks are presented in Table 1.

Original: We show results from six methods including three state-of-the-art graph kernel methods: WL Shervashidze et al.
Revised: We show results from six methods including three state-of-the-art graph kernel methods: WL Shervashidze et al.

Original: (2011), DGK Yanardag & Vishwanathan (2015), and
Revised: (2011), DGK Yanardag & Vishwanathan (2015), and

Original: MLG Kondor & Pan (2016).
Revised: MLG Kondor & Pan (2016).

Original: While these kernel methods perform well on individual datasets, none of them are competitive across all of the datasets.
Revised: While these kernel methods perform well on individual datasets, none of them are competitive across all of the datasets.

Original: Additionally, MLG suffers from a long run time and take more than 24 hours to run on the two larger benchmark datasets.
Revised: Additionally, MLG suffers from a long run time and take more than 24 hours to run on the two larger benchmark datasets.

Original: We ﬁnd that InfoGraph outperforms all of these baselines on 4 out of 6 of the datasets.
Revised: We ﬁnd that InfoGraph outperforms all of these baselines on 4 out of 6 of the datasets.

Original: In the other 2 datasets, InfoGraph still has very competitive performance.
Revised: In the other 2 datasets, InfoGraph still has very competitive performance.

Original: The results of the semi-supervised learning experiments on the molecular property prediction task are presented in Table 2.
Revised: The results of the semi-supervised learning experiments on the molecular property prediction task are presented in Table 2.

Original: We observe that by simply combining the supervised objective with the unsupervised infomax objective (InfoGraph) obtains better performance compared to the purely supervised models on 7 out of 12 of the targets.
Revised: We observe that by simply combining the supervised objective with the unsupervised infomax objective (InfoGraph) obtains better performance compared to the purely supervised models on 7 out of 12 of the targets.

Original: However, in 1 out of 12 targets it does not obtain better performance and in 4 out of 12 targets, it results in poorer performance.
Revised: However, in 1 out of 12 targets it does not obtain better performance and in 4 out of 12 targets, it results in poorer performance.

Original: This “negative transfer” effect may be caused by the fact that the supervised objective and the unsupervised objective favor different information or different latent semantic space.
Revised: This “negative transfer” effect may be caused by the fact that the supervised objective and the unsupervised objective favor different information or different latent semantic space.

Original: This effect is alleviated with InfoGraph*, our modiﬁed version of InfoGraph for semi-supervised learning.
Revised: This effect is alleviated with InfoGraph*, our modiﬁed version of InfoGraph for semi-supervised learning.

Original: InfoGraph* improves over the supervised model in all the 12 targets.
Revised: InfoGraph* improves over the supervised model in all the 12 targets.

Original: InfoGraph* obtains the best result on 11 targets while the
Revised: InfoGraph* obtains the best result on 11 targets while the

Original: Mean Teacher method obtains the best results on 2 targets (with one overlap).
Revised: Mean Teacher method obtains the best results on 2 targets (with one overlap).

Original: However, the Mean
Revised: However, the Mean

Original: Teacher model yields worse performance on 2 targets when compared to the supervised result.
Revised: Teacher model yields worse performance on 2 targets when compared to the supervised result.

Original:  C ONCLUSION AND F UTURE WORK
Revised:  C ONCLUSION AND F UTURE WORK

Original: In this paper we propose InfoGraph to learn unsupervised graph-level representations and InfoGraph* for semi-supervised learning.
Revised: In this paper we propose InfoGraph to learn unsupervised graph-level representations and InfoGraph* for semi-supervised learning.

Original: We conduct experiments on graph classiﬁcation and molecular property prediction tasks to evaluate these two methods.
Revised: We conduct experiments on graph classiﬁcation and molecular property prediction tasks to evaluate these two methods.

Original: Experimental results show that InfoGraph and
Revised: Experimental results show that InfoGraph and

Original: InfoGraph* are both very competitive with state-of-the-art methods.
Revised: InfoGraph* are both very competitive with state-of-the-art methods.

Original: There are many research works on semi-supervised learning on image data, but few of them focus on semi-supervised learning for graph structured data.
Revised: There are many research works on semi-supervised learning on image data, but few of them focus on semi-supervised learning for graph structured data.

Original: In the future, we aim to explore semi-supervised frameworks designed speciﬁcally for graphs
Revised: In the future, we aim to explore semi-supervised frameworks designed speciﬁcally for graphs

Original: A R ELATED W ORK A. G RAPH K ERNELS
Revised: A R ELATED W ORK A. G RAPH K ERNELS

Original: Popular graph kernels are graphlets Pržulj (2007);
Revised: Popular graph kernels are graphlets Pržulj (2007);

Original: Shervashidze et al.
Revised: Shervashidze et al.

Original: (2009), random walk and shortest path kernels Kashima et al.
Revised: (2009), random walk and shortest path kernels Kashima et al.

Original: ; Borgwardt & Kriegel (2005), and the Weisfeiler-Lehman subtree kernel Shervashidze et al.
Revised: ; Borgwardt & Kriegel (2005), and the Weisfeiler-Lehman subtree kernel Shervashidze et al.

Original: Furthermore, deep graph kernels Yanardag & Vishwanathan (2015), graph invariant kernels Orsini et al.
Revised: Furthermore, deep graph kernels Yanardag & Vishwanathan (2015), graph invariant kernels Orsini et al.

Original: (2015), optimal assignment graph kernels Kriege et al.
Revised: (2015), optimal assignment graph kernels Kriege et al.

Original: (2016) and multiscale Laplacian graph kernels Kondor & Pan (2016) have been proposed with the goal to redeﬁne kernel functions to appropriately capture sub-structural similarity at different levels.
Revised: (2016) and multiscale Laplacian graph kernels Kondor & Pan (2016) have been proposed with the goal to redeﬁne kernel functions to appropriately capture sub-structural similarity at different levels.

Original: Another line of research in this area focuses on efﬁciently computing these kernels either through exploiting certain structural dependencies, or via approximations/randomization Feragen et al.
Revised: Another line of research in this area focuses on efﬁciently computing these kernels either through exploiting certain structural dependencies, or via approximations/randomization Feragen et al.

Original: (2013); de Vries (2013); Neumann et al.
Revised: (2013); de Vries (2013); Neumann et al.

Original: A. C ONTRASTIVE M ETHODS
Revised: A. C ONTRASTIVE M ETHODS

Original: Contrastive methods are central many popular word-embedding methods Collobert & Weston (2008)
Revised: Contrastive methods are central many popular word-embedding methods Collobert & Weston (2008)

Original: Mnih & Kavukcuoglu (2013); Mikolov et al.
Revised: Mnih & Kavukcuoglu (2013); Mikolov et al.

Original: (2013b).
Revised: (2013b).

Original: Word2vec Mikolov et al.
Revised: Word2vec Mikolov et al.

Original: (2013a) is an unsupervised algorithm which obtains word representations by using the representations to predict context words (the words that surround it).
Revised: (2013a) is an unsupervised algorithm which obtains word representations by using the representations to predict context words (the words that surround it).

Original: Doc2vec Le & Mikolov (2014) is an extension of the continuous Skip-gram model that predicts representations of words from that of a document containing them.
Revised: Doc2vec Le & Mikolov (2014) is an extension of the continuous Skip-gram model that predicts representations of words from that of a document containing them.

Original: Researchers extended many of these unsupervised language models to learn representations of graph-structured input Adhikari et al.
Revised: Researchers extended many of these unsupervised language models to learn representations of graph-structured input Adhikari et al.

Original: ; Narayanan et al.
Revised: ; Narayanan et al.

Original: For example, graph2vec
Revised: For example, graph2vec

Original: Narayanan et al.
Revised: Narayanan et al.

Original: (2017) extends Doc2vec to arbitrary graphs.
Revised: (2017) extends Doc2vec to arbitrary graphs.

Original: Intuitively, for graph2vec a graph and the rooted subgraphs in it correspond to a document and words in a paragraph vector, respectively.
Revised: Intuitively, for graph2vec a graph and the rooted subgraphs in it correspond to a document and words in a paragraph vector, respectively.

Original: One of the technical contributions of the paper is using the Weisfeiler-Lehman relabelling algorithm
Revised: One of the technical contributions of the paper is using the Weisfeiler-Lehman relabelling algorithm

Original: Weisfeiler & Lehman (1968); Shervashidze et al.
Revised: Weisfeiler & Lehman (1968); Shervashidze et al.

Original: (2011) to enumerate all rooted subgraphs up to a speciﬁed depth.
Revised: (2011) to enumerate all rooted subgraphs up to a speciﬁed depth.

Original: AWE (Anonymous Walk Embeddings) Ivanov & Burnaev (2018) is another method based on CBOW framework.
Revised: AWE (Anonymous Walk Embeddings) Ivanov & Burnaev (2018) is another method based on CBOW framework.

Original: instead of using rooted subgraphs as words like graph2vec, AWE considers anonymous walk embeddings for the same source node as co-occurring words.
Revised: instead of using rooted subgraphs as words like graph2vec, AWE considers anonymous walk embeddings for the same source node as co-occurring words.

Original: InfoGraph has the two advantages when compared with these methods.
Revised: InfoGraph has the two advantages when compared with these methods.

Original: First, InfoGraph learns representations directly from data instead of utilizing hand-crafted procedures (i.e. Weisfeiler-Lehman relabelling algorithm in graph2vec and random walkw in AWE).
Revised: First, InfoGraph learns representations directly from data instead of utilizing hand-crafted procedures (i.e. Weisfeiler-Lehman relabelling algorithm in graph2vec and random walkw in AWE).

Original: Second, InfoGraph has a clear objective that can be easily combined with other objectives.
Revised: Second, InfoGraph has a clear objective that can be easily combined with other objectives.

Original: For example, InfoGraph*, the semi-supervised method that we proposed.
Revised: For example, InfoGraph*, the semi-supervised method that we proposed.

Original: B S EMI -S UPERVISED L EARNING
Revised: B S EMI -S UPERVISED L EARNING

Original: Here we discuss the most common class of SSL methods which involve adding an additional loss term to the training of a neural network as they are pragmatic and are currently the state-of-the-art on image classiﬁcation datasets.
Revised: Here we discuss the most common class of SSL methods which involve adding an additional loss term to the training of a neural network as they are pragmatic and are currently the state-of-the-art on image classiﬁcation datasets.

Original: Entropy Minimization (EntMin): EntMin Grandvalet & Bengio (2005) adds a loss term applied that encourages the network to make “conﬁdent” (low-entropy) predictions for all unlabeled examples, regardless of their class.
Revised: Entropy Minimization (EntMin): EntMin Grandvalet & Bengio (2005) adds a loss term applied that encourages the network to make “conﬁdent” (low-entropy) predictions for all unlabeled examples, regardless of their class.

Original: Pseudo-Labeling: Pseudo-labeling Lee (2013) proceeds by producing “pseudo-labels” for unlabeled input data points using the prediction function itself over the course of training.
Revised: Pseudo-Labeling: Pseudo-labeling Lee (2013) proceeds by producing “pseudo-labels” for unlabeled input data points using the prediction function itself over the course of training.

Original: Pseudo-labels which have a corresponding class probability that is larger than a predeﬁned threshold are used as targets for a the standard supervised loss function.
Revised: Pseudo-labels which have a corresponding class probability that is larger than a predeﬁned threshold are used as targets for a the standard supervised loss function.

Original: Π -Model: Neural networks can produce different outputs for the same input while common regularization techniques such as data augmentation, dropout, and adding noise are applied.
Revised: Π -Model: Neural networks can produce different outputs for the same input while common regularization techniques such as data augmentation, dropout, and adding noise are applied.

Original: Π -Model
Revised: Π -Model

Original: Laine & Aila (2016); Sajjadi et al.
Revised: Laine & Aila (2016); Sajjadi et al.

Original: (2016) adds a loss term which encourages the distance between a network’s output for different passes of unlabeled data through the network to be small.
Revised: (2016) adds a loss term which encourages the distance between a network’s output for different passes of unlabeled data through the network to be small.

Original: Virtual Adversarial Training: Instead of relying on the built-in stochasticity as in Π -Model, Virtual
Revised: Virtual Adversarial Training: Instead of relying on the built-in stochasticity as in Π -Model, Virtual

Original: Adversarial Training (VAT) Miyato et al.
Revised: Adversarial Training (VAT) Miyato et al.

Original: (2018) directly approximates a tiny perturbation to add to the input which would most signiﬁcantly affect the output of the prediction function.
Revised: (2018) directly approximates a tiny perturbation to add to the input which would most signiﬁcantly affect the output of the prediction function.

Original: This pertubation can be approximated with an extra back-propagation for each optimization step.
Revised: This pertubation can be approximated with an extra back-propagation for each optimization step.

Original: Mean Teacher: A difﬁculty with the Π -model approach is that it relies on a potentially unstable “target” prediction, namely the second stochastic network prediction which can rapidly change over the course of training.
Revised: Mean Teacher: A difﬁculty with the Π -model approach is that it relies on a potentially unstable “target” prediction, namely the second stochastic network prediction which can rapidly change over the course of training.

Original: As a result, Tarvainen & Valpola (2017) proposed to obtain a more stable target output for unlabeled data by setting the target to predictions made using an exponential moving average of parameters from previous training steps.
Revised: As a result, Tarvainen & Valpola (2017) proposed to obtain a more stable target output for unlabeled data by setting the target to predictions made using an exponential moving average of parameters from previous training steps.

Original: C D ATASETS C. G RAPH C LASSIFICATION D ATASETS
Revised: C D ATASETS C. G RAPH C LASSIFICATION D ATASETS

Original: MUTAG contains 188 mutagenic aromatic and heteroaromatic nitro compounds with 7 different discrete labels.
Revised: MUTAG contains 188 mutagenic aromatic and heteroaromatic nitro compounds with 7 different discrete labels.

Original: PTC is a dataset of 344 different chemical compounds that have been tested for carcinogenicity in male and female rats.
Revised: PTC is a dataset of 344 different chemical compounds that have been tested for carcinogenicity in male and female rats.

Original: This dataset has 19 discrete labels.
Revised: This dataset has 19 discrete labels.

Original: IMDB-BINARY and
Revised: IMDB-BINARY and

Original: IMDB-MULTI are movie collaboration datasets.
Revised: IMDB-MULTI are movie collaboration datasets.

Original: Each graph corresponds to an ego-network for each actor/actress, where nodes correspond to actors/actresses and an edge is drawn between two actors/actresses if they appear in the same movie.
Revised: Each graph corresponds to an ego-network for each actor/actress, where nodes correspond to actors/actresses and an edge is drawn between two actors/actresses if they appear in the same movie.

Original: Each graph is derived from a pre-speciﬁed genre of movies, and the task is to classify the genre graph it is derived from.
Revised: Each graph is derived from a pre-speciﬁed genre of movies, and the task is to classify the genre graph it is derived from.

Original: REDDIT-BINARY and
Revised: REDDIT-BINARY and

Original: REDDIT-MULTI5K are balanced datasets where each graph corresponds to an online discussion thread and nodes correspond to users.
Revised: REDDIT-MULTI5K are balanced datasets where each graph corresponds to an online discussion thread and nodes correspond to users.

Original: An edge was drawn between two nodes if at least one of them responded to another’s comment.
Revised: An edge was drawn between two nodes if at least one of them responded to another’s comment.

Original: The task is to classify each graph to the community or subreddit that it belongs to.
Revised: The task is to classify each graph to the community or subreddit that it belongs to.

Original: C. QM 
Revised: C. QM

Original: All molecules in the dataset consist of Hydrogen (H), Carbon (C), Oxygen (O), Nitrogen (N), and
Revised: All molecules in the dataset consist of Hydrogen (H), Carbon (C), Oxygen (O), Nitrogen (N), and

Original: Flourine (F) atoms and contain up to 9 non-Hydrogen atoms.
Revised: Flourine (F) atoms and contain up to 9 non-Hydrogen atoms.

Original: In all, this results in about 134, drug-like organic molecules that span a wide range of chemical compositions and properties.
Revised: In all, this results in about 134, drug-like organic molecules that span a wide range of chemical compositions and properties.

Original: A total of 12 interesting and fundamental chemical properties are pre-computed for each molecule.
Revised: A total of 12 interesting and fundamental chemical properties are pre-computed for each molecule.

Original: For a detailed description of the properties in the QM9 dataset, see section 10.2 of Gilmer et al.
Revised: For a detailed description of the properties in the QM9 dataset, see section 10.2 of Gilmer et al.

Original: D M ODEL C ONFIGURATION
Revised: D M ODEL C ONFIGURATION

Original: For the unsupervised experiments, we use the Graph Isomorphism Network (GIN)
Revised: For the unsupervised experiments, we use the Graph Isomorphism Network (GIN)

Original: Xu et al.
Revised: Xu et al.

Original: (2018a).
Revised: (2018a).

Original: GNN layers are chosen from { 4 , 8 , 12 } .
Revised: GNN layers are chosen from { 4 , 8 , 12 } .

Original: Initial learning rate is chosen from the set { 10 − 2 , 10 − 3 , 10 − 4 } .
Revised: Initial learning rate is chosen from the set { 10 − 2 , 10 − 3 , 10 − 4 } .

Original: The number of epochs are chosen from { 10 , 20 , 100 } .
Revised: The number of epochs are chosen from { 10 , 20 , 100 } .

Original: The batch size is set to 128.
Revised: The batch size is set to 128.

Original: For the semi-supervised experiments, the number of set2set computations is set to 3.
Revised: For the semi-supervised experiments, the number of set2set computations is set to 3.

Original: Model were trained with an initial learning rate 0.001 for 500 epochs with a batch size 20.
Revised: Model were trained with an initial learning rate 0.001 for 500 epochs with a batch size 20.

Original: For the supervised case, the weight decay is chosen from { 0 , 10 − 3 , 10 − 4 } .
Revised: For the supervised case, the weight decay is chosen from { 0 , 10 − 3 , 10 − 4 } .

Original: For InfoGraph and InfoGraph*, λ is chosen from { 10 − 3 , 10 − 4 , 10 − 5 } .
Revised: For InfoGraph and InfoGraph*, λ is chosen from { 10 − 3 , 10 − 4 , 10 − 5 } .

Original: The discriminator scores global-patch representation pairs by passing two representations to different non-linear transformations and then takes the dot product of the two transformed representations.
Revised: The discriminator scores global-patch representation pairs by passing two representations to different non-linear transformations and then takes the dot product of the two transformed representations.

Original: Both non-linear transformations are parameterized by 3-layered feed-forward neural networks with jumping connections.
Revised: Both non-linear transformations are parameterized by 3-layered feed-forward neural networks with jumping connections.

Original: Following each linear layer is a ReLU activation function.
Revised: Following each linear layer is a ReLU activation function.

Revised: E C ONVERGENCE P LOT

Revised: To prove that the objective Eq.8 with multiple loss terms can be optimized, we provide a convergence plot of InfoGraph*.

Revised: We can see that the three loss terms all converge after around 150 epochs of training.