Original: E MBEDDING MODELS THROUGH THE LENS OF STABLE COLORING
Revised: E MBEDDING MODELS THROUGH THE LENS OF STABLE COLORING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Embedding-based approaches ﬁnd the semantic meaning of tokens in structured data such as natural language, graphs, and even images.
Revised: Embedding-based approaches ﬁnd the semantic meaning of tokens in structured data such as natural language, graphs, and even images.

Original: To a great degree, these approaches have developed independently in different domains.
Revised: To a great degree, these approaches have developed independently in different domains.

Original: However, we ﬁnd a common principle underlying these formulations, and it is rooted in solutions to the stable coloring problem in graphs (Weisfeiler-Lehman isomorphism test).
Revised: However, we ﬁnd a common principle underlying these formulations, and it is rooted in solutions to the stable coloring problem in graphs (Weisfeiler-Lehman isomorphism test).

Original: For instance, we ﬁnd links between stable coloring, distribution hypothesis in natural language processing, and non-local-means denoising algorithm in image signal processing.
Revised: For instance, we ﬁnd links between stable coloring, distribution hypothesis in natural language processing, and non-local-means denoising algorithm in image signal processing.

Original: We even ﬁnd that stable coloring has strong connections to a broad class of unsupervised embedding models which is surprising at ﬁrst since stable coloring is generally applied for combinatorial problems.
Revised: We even ﬁnd that stable coloring has strong connections to a broad class of unsupervised embedding models which is surprising at ﬁrst since stable coloring is generally applied for combinatorial problems.

Original: To establish this connection concretely we deﬁne a mathematical framework that deﬁnes continuous stable coloring on graphs and develops optimization problems to search for them.
Revised: To establish this connection concretely we deﬁne a mathematical framework that deﬁnes continuous stable coloring on graphs and develops optimization problems to search for them.

Original: Grounded on this framework, we show that many algorithms ranging across different domains are, in fact, searching for continuous stable coloring solutions of an underlying graph corresponding to the domain.
Revised: Grounded on this framework, we show that many algorithms ranging across different domains are, in fact, searching for continuous stable coloring solutions of an underlying graph corresponding to the domain.

Original: We show that popular and widely used embedding models such as Word2Vec, AWE, BERT, Node2Vec, and
Revised: We show that popular and widely used embedding models such as Word2Vec, AWE, BERT, Node2Vec, and

Original: Vis-Transformer can be understood as instantiations of our general algorithm that solves the problem of continuous stable coloring.
Revised: Vis-Transformer can be understood as instantiations of our general algorithm that solves the problem of continuous stable coloring.

Original: These instantiations offer useful insights into the workings of state-of-the-art models like BERT stimulating new research directions.
Revised: These instantiations offer useful insights into the workings of state-of-the-art models like BERT stimulating new research directions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Embedding models are ubiquitous in wide range of real-world applications such as information retrieval (Zuccon et al., 2015), natural language processing (NLP)
Revised: Embedding models are ubiquitous in wide range of real-world applications such as information retrieval (Zuccon et al., 2015), natural language processing (NLP)

Original: (Mikolov et al., 2013a;b), graph classiﬁcation (Grover & Leskovec, 2016; Hamilton et al., 2017) and many more.
Revised: (Mikolov et al., 2013a;b), graph classiﬁcation (Grover & Leskovec, 2016; Hamilton et al., 2017) and many more.

Original: These models map categorical entities to continuous dense representations (typically in R d ) which provide a continuous measure of semantic similarity across categorical entities.
Revised: These models map categorical entities to continuous dense representations (typically in R d ) which provide a continuous measure of semantic similarity across categorical entities.

Original: Nowadays, there is a heavy dependence on unsupervised pre-trained embedding models across domains like Transformers in NLP (Devlin et al., 2019), Visual Transformers (ViT) in Computer Vision (Dosovitskiy et al., 2020), Graph Neural
Revised: Nowadays, there is a heavy dependence on unsupervised pre-trained embedding models across domains like Transformers in NLP (Devlin et al., 2019), Visual Transformers (ViT) in Computer Vision (Dosovitskiy et al., 2020), Graph Neural

Original: Networks (Hamilton et al., 2017; Xu et al., 2019b) since they learn rich semantic representations of entities from massive amounts of unlabelled data.
Revised: Networks (Hamilton et al., 2017; Xu et al., 2019b) since they learn rich semantic representations of entities from massive amounts of unlabelled data.

Original: With little ﬁnetuning, these models achieve stateof-the-art results on most of the supervised downstream tasks like sentiment analysis (Xu et al., 2019a), object detection (Beal et al., 2020), and graph classiﬁcation (Xu et al., 2019b).
Revised: With little ﬁnetuning, these models achieve stateof-the-art results on most of the supervised downstream tasks like sentiment analysis (Xu et al., 2019a), object detection (Beal et al., 2020), and graph classiﬁcation (Xu et al., 2019b).

Original: Historically, embedding models were developed almost independently across structured domains such as NLP, graphs, images, and so on.
Revised: Historically, embedding models were developed almost independently across structured domains such as NLP, graphs, images, and so on.

Original: These algorithms use the neighborhood structure around an entity to obtain the embedding for the entity.
Revised: These algorithms use the neighborhood structure around an entity to obtain the embedding for the entity.

Original: Interestingly, a popular hypothesis in NLP - Distributional Hypothesis states that the “meaning” of the word is determined by its context (neighbors) (Harris, 1954; Sahlgren, 2008).
Revised: Interestingly, a popular hypothesis in NLP - Distributional Hypothesis states that the “meaning” of the word is determined by its context (neighbors) (Harris, 1954; Sahlgren, 2008).

Original: This hypothesis forms the basis of most unsupervised embedding learning models in NLP (Mikolov et al., 2013a;b; Pennington et al., 2014; Bojanowski et al., 2017;
Revised: This hypothesis forms the basis of most unsupervised embedding learning models in NLP (Mikolov et al., 2013a;b; Pennington et al., 2014b; Bojanowski et al., 2017;

Original: Sonkar et al., 2020).
Revised: Sonkar et al., 2020).

Original: Similarly, non-local-means, a denoising algorithm in signal processing, tries to ﬁnd pixels that should be the same based on the similarity of its neighborhood structure (patch of the image around the pixel in this case)
Revised: Similarly, non-local-means, a denoising algorithm in signal processing, tries to ﬁnd pixels that should be the same based on the similarity of its neighborhood structure (patch of the image around the pixel in this case)

Original: (Awate & Whitaker, 2006; Buades et al., 2005).
Revised: (Awate & Whitaker, 2006; Buades et al., 2005).

Original: Even
Revised: Even

Original: Graph neural network (GNN) architectures ensure that information of the surrounding neighbors is systematically incorporated in the embedding of a node, even in supervised settings (Hamilton et al., 2017; Maron et al., 2019; Xu et al., 2019b).
Revised: Graph neural network (GNN) architectures ensure that information of the surrounding neighbors is systematically incorporated in the embedding of a node, even in supervised settings (Hamilton et al., 2017; Maron et al., 2019; Xu et al., 2019b).

Original: Thus somehow various communities working across diverse domains have narrowed down on an entity’s neighborhood structure to deﬁne the entity’s meaning.
Revised: Thus somehow various communities working across diverse domains have narrowed down on an entity’s neighborhood structure to deﬁne the entity’s meaning.

Original: In this paper, we try to establish this common principle on mathematically robust grounds.
Revised: In this paper, we try to establish this common principle on mathematically robust grounds.

Original: Structured domains can be easily represented as graphs with relations between entities as edges in the graphs.
Revised: Structured domains can be easily represented as graphs with relations between entities as edges in the graphs.

Original: For example in NLP words can be treated as nodes of some graph, and co-occurrence relation between words can be represented as an edge.
Revised: For example in NLP words can be treated as nodes of some graph, and co-occurrence relation between words can be represented as an edge.

Original: As mentioned before that graph embedding architecturescaptures capture the topological structure around the node in the node embeddings, and if structured domains can be represented as graphs, this raises the question do embedding models from structured domains like NLP and Vision also operate on some domain induced graph and capture neighborhood structural properties in their entity embeddings since we have seen that embedding models across these domains tend to capture “neighborhood” information?
Revised: As mentioned before that graph embedding architectures capture the topological structure around the node in the node embeddings, and if structured domains can be represented as graphs, this raises the question do embedding models from structured domains like NLP and Vision also operate on some domain induced graph and capture neighborhood structural properties in their entity embeddings since we have seen that embedding models across these domains tend to capture “neighborhood” information?

Original: To answer this question, we dive into combinatorial graph theory to understand how to deﬁne the notion of structural equivalence a.k.a.
Revised: To answer this question, we dive into combinatorial graph theory to understand how to deﬁne the notion of structural equivalence a.k.a.

Original: isomorphic structures.
Revised: isomorphic structures.

Original: Weisfeiler-Lehman (1-WL) algorithm (or color reﬁnement algorithm) is the most popular heuristic used to identify graph isomorphism (Weisfeiler & Leman, 1968) and can distinguish a broad class of graphs (Babai & Kucera, 1979).
Revised: Weisfeiler-Lehman (1-WL) algorithm (or color reﬁnement algorithm) is the most popular heuristic used to identify graph isomorphism (Weisfeiler & Leman, 1968) and can distinguish a broad class of graphs (Babai & Kucera, 1979).

Original: The ﬁxed point solution of 1-WL is called a stable-coloring and has the property that any two nodes with the same color have the same multi-set of colors in their neighborhood.
Revised: The ﬁxed point solution of 1-WL is called a stable-coloring and has the property that any two nodes with the same color have the same multi-set of colors in their neighborhood.

Original: In essence it means if two nodes have the same color, the graph looks structurally identical from these nodes.
Revised: In essence it means if two nodes have the same color, the graph looks structurally identical from these nodes.

Original: Finally to answer the question if embedding models from structured domains capture the domain induced graph’s topological properties in their entity embeddings, one can ﬁnd links between the mechanics of these models and stable coloring / 1-WL algorithm.
Revised: Finally to answer the question if embedding models from structured domains capture the domain induced graph’s topological properties in their entity embeddings, one can ﬁnd links between the mechanics of these models and stable coloring / 1-WL algorithm.

Original: In this paper, we establish this connection by providing a general framework linking existing algorithms to stable coloring.
Revised: In this paper, we establish this connection by providing a general framework linking existing algorithms to stable coloring.

Original: We propose a more ﬂexible version of stable coloring (SC) called continuous stable coloring (CSC ) — a strict generalization of SC.
Revised: We propose a more ﬂexible version of stable coloring (SC) called continuous stable coloring (CSC ) — a strict generalization of SC.

Original: CSC states that the similarity of their neighborhoods determines the similarity of two nodes.
Revised: CSC states that the similarity of their neighborhoods determines the similarity of two nodes.

Original: Based on this notion, we deﬁne a series of optimization problems to solve the problem of CSC .
Revised: Based on this notion, we deﬁne a series of optimization problems to solve the problem of CSC .

Original: We show that various algorithms in NLP like word2vec, AWE, BERT (Devlin et al., 2019), images processing like Visual Transformer (Dosovitskiy et al., 2020), graphs like Node2Vec (Grover & Leskovec, 2016), etc. are essentially solving different instantiations of this common optimization problem.
Revised: We show that various algorithms in NLP like word2vec, AWE, BERT (Devlin et al., 2019), images processing like Visual Transformer (Dosovitskiy et al., 2020), graphs like Node2Vec (Grover & Leskovec, 2016), etc. are essentially solving different instantiations of this common optimization problem.

Original: Current research already establishes the link between the 1-WL algorithm (Grohe, 2020; Morris et al., 2021; Shervashidze et al., 2011; Morris et al., 2017) and GNN architectures, which has sparked a new line of research in improving GNN architectures (Hamilton et al., 2017; Xu et al., 2019b; Maron et al., 2019; Morris et al., 2020a;b).
Revised: Current research already establishes the link between the 1-WL algorithm (Grohe, 2020; Morris et al., 2021; Shervashidze et al., 2011; Morris et al., 2017) and GNN architectures, which has sparked a new line of research in improving GNN architectures (Hamilton et al., 2017; Xu et al., 2019b; Maron et al., 2019; Morris et al., 2020a;b).

Original: We hope that the new link we establish between stable coloring and unsupervised embedding algorithms will also stimulate new exciting research in embeddings for other structured domains of NLP and Vision.
Revised: We hope that the new link we establish between stable coloring and unsupervised embedding algorithms will also stimulate new exciting research in embeddings for other structured domains of NLP and Vision.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we deﬁne a stable colored graph, provide an outline for 1-WL (Weisfeiler-Lehman) graph isomorphism test and General Aggregate and Update (GAU ) for Graph Neural Networks (GNNs).
Revised: In this section, we deﬁne a stable colored graph, provide an outline for 1-WL (Weisfeiler-Lehman) graph isomorphism test and General Aggregate and Update (GAU ) for Graph Neural Networks (GNNs).

Original: We also discuss how domains of NLP and images can be seen as graphs.
Revised: We also discuss how domains of NLP and images can be seen as graphs.

Original: 2. S TABLE
Revised: 2. S TABLE

Original: Let a coloring function C be an overloaded function deﬁned on the vertices as well on set of vertices of G =
Revised: Let a coloring function C be an overloaded function deﬁned on the vertices as well on set of vertices of G =

Original: ( V , E ) , i.e., C : V → N and C : 2 V → {{ N }} where N is a set of natural numbers representing colors, and {{ . }} is a multiset with overloading deﬁned as C ( A )
Revised: ( V , E ) , i.e., C : V → N and C : 2 V → {{ N }} where N is a set of natural numbers representing colors, and {{ . }} is a multiset with overloading deﬁned as C ( A )

Original: =
Revised: =

Original: {{C ( v ) | v ∈ A ⊂
Revised: {{C ( v ) | v ∈ A ⊂

Original: V}} .
Revised: V}} .

Original: We denote neighborhood of a node u ∈ V as N ( v ) =
Revised: We denote neighborhood of a node u ∈ V as N ( v ) =

Original: { u | ( v, u ) ∈ E} .
Revised: { u | ( v, u ) ∈ E} .

Original: Deﬁnition 2.1 (Stable Coloring) .
Revised: Deﬁnition 2.1 (Stable Coloring) .

Original: An undirected graph G =
Revised: An undirected graph G =

Original: ( V , E ) is stable colored w.r.t coloring function C if it holds that C ( u )
Revised: ( V , E ) is stable colored w.r.t coloring function C if it holds that C ( u )

Original: =
Revised: =

Original: C ( v ) if and only if C ( N ( u )) =
Revised: C ( v ) if and only if C ( N ( u )) =

Original: C ( N ( v ) .
Revised: C ( N ( v )) .

Original: We can extend the above deﬁnition to directed graphs and graphs with labels.
Revised: We can extend the above deﬁnition to directed graphs and graphs with labels.

Original: A directed graph
Revised: A directed graph

Original: G =
Revised: G =

Original: ( V , E ) is stable colored w.r.t a coloring function C : V → N if it holds that C ( u )
Revised: ( V , E ) is stable colored w.r.t a coloring function C : V → N if it holds that C ( u )

Original: =
Revised: =

Original: C ( v ) if and only if C ( N in ( u )) =
Revised: C ( v ) if and only if C ( N in ( u )) =

Original: C ( N in ( v )) and C ( N out ( u ))
Revised: C ( N in ( v )) and C ( N out ( u ))

Original: =
Revised: =

Original: C ( N out ( v )) where N in ( u ) = { w | ( w, u ) ∈ E} and N out ( u ) =
Revised: C ( N out ( v )) where N in ( u ) = { w | ( w, u ) ∈ E} and N out ( u ) =

Original: { w | ( u, w ) ∈ E} .
Revised: { w | ( u, w ) ∈ E} .

Original: Likewise, an edge-labelled undirected graph G =
Revised: Likewise, an edge-labelled undirected graph G =

Original: ( V , E ) is stable colored w.r.t a coloring function C : V → N if it holds that C ( u )
Revised: ( V , E ) is stable colored w.r.t a coloring function C : V → N if it holds that C ( u )

Original: =
Revised: =

Original: C ( v ) if and only if ∀ l, C ( N l ( u )) =
Revised: C ( v ) if and only if ∀ l, C ( N l ( u )) =

Original: C ( N l ( v ) where N l ( u ) =
Revised: C ( N l ( v )) where N l ( u ) =

Original: { w | ( u, w ) ∈
Revised: { w | ( u, w ) ∈

Original: E l } where E l ⊂ E for an edge label l .
Revised: E l } where E l ⊂ E for an edge label l .

Original: We also deﬁne a weak -stable coloring for a graph G =
Revised: We also deﬁne a weak -stable coloring for a graph G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: An undirected graph G =
Revised: An undirected graph G =

Original: ( V , E ) is weak -stable colored w.r.t coloring function C if it holds that C ( u )
Revised: ( V , E ) is weak -stable colored w.r.t coloring function C if it holds that C ( u )

Original: =
Revised: =

Original: C ( v ) if C ( N ( u )) =
Revised: C ( v ) if C ( N ( u )) =

Original: C ( N ( v ) where N ( u ) =
Revised: C ( N ( v )) where N ( u ) =

Original: { w | ( u, w ) ∈ E} .
Revised: { w | ( u, w ) ∈ E} .

Original: WL ALGORITHM AND G ENERAL A GGREGATE AND U PDATE F RAMEWORK (GAU )
Revised: 2. 1-WL ALGORITHM AND G ENERAL A GGREGATE AND U PDATE F RAMEWORK (GAU )

Original: 1-WL algorithm: 1-WL is an iterative algorithm to achieve a stable coloring C for G =
Revised: 1-WL algorithm: 1-WL is an iterative algorithm to achieve a stable coloring C for G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: Let C i denote the coloring at iteration i .
Revised: Let C i denote the coloring at iteration i .

Original: It starts with a coloring scheme C 0 such that C 0 ( v ) is same for all v ∈ V .
Revised: It starts with a coloring scheme C 0 such that C 0 ( v ) is same for all v ∈ V .

Original: In each iteration, it assigns a different color to u and v if C i ( N ( u ))
Revised: In each iteration, it assigns a different color to u and v if C i ( N ( u ))

Original: (cid:54) =
Revised: (cid:54) =

Original: C  ( N ( u ) until a stable coloring C is reached.
Revised: C i ( N ( u )) until a stable coloring C is reached.

Original: General Aggregate and Update Framework (GAU )
Revised: General Aggregate and Update Framework (GAU )

Original: For a general multi-layer GNN, the General
Revised: For a general multi-layer GNN, the General

Original: Aggregate and Update framework to compute node/vertex embeddings (corresponding to colors) of
Revised: Aggregate and Update framework to compute node/vertex embeddings (corresponding to colors) of

Original: G =
Revised: G =

Original: ( V , E ) is given iteratively by:
Revised: ( V , E ) is given iteratively by:

Original: [Equation]
Revised: [Equation]

Original: (1) where f ( k ) agg ( . ) and f ( k ) update ( . ) map vertex multiset embeddings to a metric space such as R d .
Revised: (1) where f ( k ) agg ( . ) and f ( k ) update ( . ) map vertex multiset embeddings to a metric space such as R d .

Original: In GraphSage (Hamilton et al., 2017), authors showed that the iterative procedure in 1-WL algorithm is analogous to General Aggregate and Update procedure in GNNs.
Revised: In GraphSage (Hamilton et al., 2017), authors showed that the iterative procedure in 1-WL algorithm is analogous to General Aggregate and Update procedure in GNNs.

Original: This connection has lead to the research direction where 1-WL is being used as a standard to which GNN architectures are being compared to (Xu et al., 2019b; Maron et al., 2019; Morris et al., 2020b).
Revised: This connection has lead to the research direction where 1-WL is being used as a standard to which GNN architectures are being compared to (Xu et al., 2019b; Maron et al., 2019; Morris et al., 2020b).

Original: Xu et al.
Revised: Xu et al.

Original: (2019b) prove that GAU is as powerful as 1-WL if the functions f update and f agg are injective.
Revised: (2019b) prove that GAU is as powerful as 1-WL if the functions f update and f agg are injective.

Original: Various domains such as NLP, Images and Graphs can be viewed as a graph on their elementary tokens.
Revised: Various domains such as NLP, Images and Graphs can be viewed as a graph on their elementary tokens.

Original: How we construct graphs is explained in section 6.1 and examples are given in appendix ??
Revised: How we construct graphs is explained in section 6.1 and examples are given in appendix B.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Word embeddings have been popular in NLP since decades (Deerwester et al., 1989; Morin & Bengio, 2005; Mikolov et al., 2013b; Bojanowski et al., 2017).
Revised: Word embeddings have been popular in NLP since decades (Deerwester et al., 1989; Morin & Bengio, 2005; Mikolov et al., 2013b; Bojanowski et al., 2017).

Original: A lot of work has been done to understand the mathematical underpinning of these models, for instance, relation of embedding models to co-occurrence statistics (Levy & Goldberg, 2014; Hashimoto et al., 2016; Allen et al., 2019).
Revised: A lot of work has been done to understand the mathematical underpinning of these models, for instance, relation of embedding models to co-occurrence statistics (Levy & Goldberg, 2014; Hashimoto et al., 2016; Allen et al., 2019).

Original: Study of empirical properties of these embedding models (e.g. analogies) has also attracted theoretical research (Allen & Hospedales, 2019; Ethayarajh et al., 2019).
Revised: Study of empirical properties of these embedding models (e.g. analogies) has also attracted theoretical research (Allen & Hospedales, 2019; Ethayarajh et al., 2019).

Original: Recently, graph community has also seen a surge in learning node and graph embeddings.
Revised: Recently, graph community has also seen a surge in learning node and graph embeddings.

Original: The notion of capturing the structural neighborhood around a node inside the node embedding has been the driving principle of these semi-supervised node embedding algorithms like node2vec (Grover &
Revised: The notion of capturing the structural neighborhood around a node inside the node embedding has been the driving principle of these semi-supervised node embedding algorithms like node2vec (Grover &

Original: Leskovec, 2016), and GraphSAGE (Hamilton et al., 2017).
Revised: Leskovec, 2016), and GraphSAGE (Hamilton et al., 2017).

Original: Hamilton et al.
Revised: Hamilton et al.

Original: (2017) pointed out that their GraphSAGE node embedding algorithm mimics the aggregate and update procedure of 1-WL algorithm.
Revised: (2017) pointed out that their GraphSAGE node embedding algorithm mimics the aggregate and update procedure of 1-WL algorithm.

Original: Subsequently, these neighborhood informative node embeddings found their applications in constructing graph embeddings, and thereby used for classiﬁcation of structurally equivalent or isomorphic graphs (Chen et al., 2019).
Revised: Subsequently, these neighborhood informative node embeddings found their applications in constructing graph embeddings, and thereby used for classiﬁcation of structurally equivalent or isomorphic graphs (Chen et al., 2019).

Original: Chen et al.
Revised: Xu et al.

Original: (2019) in their GIN (Graph Isomorphism Network) model modiﬁed the aggregate procedure of GraphSAGE to construct graph embedders which were provably as powerful as 1-WL algorithm in distinguishing non-isomorphic graphs.
Revised: (2019b) in their GIN (Graph Isomorphism Network) model modiﬁed the aggregate procedure of GraphSAGE to construct graph embedders which were provably as powerful as 1-WL algorithm in distinguishing non-isomorphic graphs.

Original: This redirected the research into designing more powerful variants of graph embedders like PPGN (Maron et al., 2019) and k -GNN (Morris et al., 2019) which were provably as powerful as 3-WL and k -WL test respectively.
Revised: This redirected the research into designing more powerful variants of graph embedders like PPGN (Maron et al., 2019) and k -GNN (Morris et al., 2019) which were provably as powerful as 3-WL and k -WL test respectively.

Original: Grohe (2020) discusses about these phenomenal works in increasing the expressivity of graph embeddings for supervised graph classiﬁcation and regression tasks.
Revised: Grohe (2020) discusses about these phenomenal works in increasing the expressivity of graph embeddings for supervised graph classiﬁcation and regression tasks.

Original: While this line of research focuses on improving expressivity and generalizability of Graph networks based on its connection to 1-WL, we explore and formalize the unsupervised algorithms under the light of stable coloring / 1-WL algorithm and show that all the current models stem from the common principle that a tokens meaning is derived from its neighbours.
Revised: While this line of research focuses on improving expressivity and generalizability of Graph networks based on its connection to 1-WL, we explore and formalize the unsupervised algorithms under the light of stable coloring / 1-WL algorithm and show that all the current models stem from the common principle that a tokens meaning is derived from its neighbours.

Original: The rest of the paper is organised as follows.
Revised: The rest of the paper is organised as follows.

Original: We begin with a discussion of connections between discrete stable coloring and non-local means algorithm in Image processing and distributional hypothesis in NLP.
Revised: We begin with a discussion of connections between discrete stable coloring and non-local means algorithm in Image processing and distributional hypothesis in NLP.

Original: We then deﬁne a continuous version of SC and develop optimization problems to solve for CSC in section 5.
Revised: We then deﬁne a continuous version of SC and develop optimization problems to solve for CSC in section 5.

Original: In section 6 we show how current state-of-the art embedding models are solving the CSC problem in disguise.
Revised: In section 6 we show how current state-of-the art embedding models are solving the CSC problem in disguise.

Original:  A LGORITHMS WITH R OOTS IN D ISCRETE S TABLE C OLORING
Revised:  A LGORITHMS WITH R OOTS IN D ISCRETE S TABLE C OLORING

Original: Distributional hypothesis in NLP states that words that occur in similar contexts are semantically similar.
Revised: Distributional hypothesis in NLP states that words that occur in similar contexts are semantically similar.

Original: Analogously, the non-local means image denoising algorithm in CV literature assigns similar intensity values to pixels that have similar patches surrounding it.
Revised: Analogously, the non-local means image denoising algorithm in CV literature assigns similar intensity values to pixels that have similar patches surrounding them.

Original: The essence of both ideas is that the value of an entity is determined by its neighborhood.
Revised: The essence of both ideas is that the value of an entity is determined by its neighborhood.

Original: In this section, we connect these two ideas to the idea of stable coloring which states that if two nodes in a stable colored graph have same colored neighborhood, they have the same color.
Revised: In this section, we connect these two ideas to the idea of stable coloring which states that if two nodes in a stable colored graph have same colored neighborhood, they have the same color.

Original: The proofs of the following theorems are provided in appendix
Revised: The proofs of the following theorems are provided in appendix.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: (Distributional Hypothesis(DH) in NLP encodes SC) Let { w 1 , w 2 , .., w n } be words in vocabulary, and M ∈ R n × n is a co-occurrence matrix with entries e ij ∈ N containing the number of times w i co-occurs with w j within a ﬁxed context window.
Revised: (Distributional Hypothesis(DH) in NLP encodes SC) Let { w 1 , w 2 , .., w n } be words in vocabulary, and M ∈ R n × n is a co-occurrence matrix with entries e ij ∈ N containing the number of times w i co-occurs with w j within a ﬁxed context window.

Original: Let there be a function f : w → N which takes a word and maps it to a color c ∈ N (assigns meaning in accordance with Distributional
Revised: Let there be a function f : w → N which takes a word and maps it to a color c ∈ N (assigns meaning in accordance with Distributional

Original: Hypothesis), such that f ( w i ) =
Revised: Hypothesis), such that f ( w i ) =

Original: f
Revised: f

Original: ( w j ) only if row i is same as row j in matrix M .
Revised: ( w j ) only if row i is same as row j in matrix M .

Original: Construct a graph
Revised: Construct a graph

Original: G DH with words as nodes and its adjacency matrix given by M .
Revised: G DH with words as nodes and edge labels given by M .

Original: Then, function f deﬁnes a stable coloring on graph G DH .
Revised: Then, function f deﬁnes a stable coloring on graph G DH .

Original: DH deﬁnes words to be semantically same if they are substitutions of each other.
Revised: DH deﬁnes words to be semantically same if they are substitutions of each other.

Original: In a true graph ( not the one created from samples) their co-occurrence frequencies with other words is exactly the same.
Revised: In a true graph ( not the one created from samples) their co-occurrence frequencies with other words is exactly the same.

Original: Equating color assigned by f to represent meaning of a word as per DH, it is easy to see that words that end with same color are semantically same.
Revised: Equating color assigned by f to represent meaning of a word as per DH, it is easy to see that words that end with same color are semantically same.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Consider a discrete signal y ( x ) sampled at n points x i ( i = 1 , ..., n ) , and let the sequence p ( x i ) =
Revised: Consider a discrete signal y ( x ) sampled at n points x i ( i = 1 , ..., n ) , and let the sequence p ( x i ) =

Original: ( y ( x i − t ) , .., y ( x i − 1 ) , y ( x i +1 ) , .., y
Revised: ( y ( x i − t ) , .., y ( x i − 1 ) , y ( x i +1 ) , .., y

Original: ( x i + t )) be a patch of neighborhood values around each x
Revised: ( x i + t )) be a patch of neighborhood values around each x

Original: i for some context window length t .
Revised: i for some context window length t .

Original: NLM denoises the signal y ( x i ) , i = 1 , ..., n with iterative updates.
Revised: NLM denoises the signal y ( x i ) , i = 1 , ..., n with iterative updates.

Original: The ﬁxed point denoised version of the signal y d can be written as follows:
Revised: The ﬁxed point denoised version of the signal y d can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where D ( x i ) =
Revised: where D ( x i ) =

Original: (cid:80) nj =1 K ( y d ( x i ) , y d ( x j )) and K is an arbitrary kernel function.
Revised: (cid:80) nj =1 K ( y d ( x i ) , y d ( x j )) and K is an arbitrary kernel function.

Original: Let the graph
Revised: Let the graph

Original: G nlm =
Revised: G nlm =

Original: ( V nlm , E nlm ) where each x i is a node u i in V nlm
Revised: ( V nlm , E nlm ) where each x i is a node u i in V nlm

Original: and each pair ( u i , u j ) with | i − j | ≤ t is represented as a directed edge in E nlm with label ( i − j ) .
Revised: and each pair ( u i , u j ) with | i − j | ≤ t is represented as a directed edge in E nlm with label ( i − j ) .

Original: Then the ﬁxed point solution of NLM with Kronecker delta kernel K δ , y d : N rrowR deﬁnes a weak -stable coloring over the graph G nlm 
Revised: Then the ﬁxed point solution of NLM with Kronecker delta kernel K δ , y d : N → R deﬁnes a weak -stable coloring over the graph G nlm .

Original: An image is an example of a discrete signal with pixel intensities as signal values.
Revised: An image is an example of a discrete signal with pixel intensities as signal values.

Original: With Kronecker delta kernel, NLM terminates with the intensity value of y d ( x i ) =
Revised: With Kronecker delta kernel, NLM terminates with the intensity value of y d ( x i ) =

Original: y d ( x j ) only if p d ( x i ) =
Revised: y d ( x j ) only if p d ( x i ) =

Original: p d ( x j ) .
Revised: p d ( x j ) .

Original: Assigning color to node n i as x i ’s ﬁnal denoised intensity value, one can observe that NLM only terminates when a weak -stable colored graph is deﬁned over the ﬁnal image.
Revised: Assigning color to node n i as x i ’s ﬁnal denoised intensity value, one can observe that NLM only terminates when a weak -stable colored graph is deﬁned over the ﬁnal image.

Original: Note that if the pixel values have high variety, then the probability to get a stable-colored graph is high.
Revised: Note that if the pixel values have high variety, then the probability to get a stable-colored graph is high.

Original: From the above two examples, it is interesting to observe that a stable colored graph G emerges from the underlying principles used by two different domains.
Revised: From the above two examples, it is interesting to observe that a stable colored graph G emerges from the underlying principles used by two different domains.

Original: Both in DH and NLM, the words/pixels that end up with the same color have same colored neighborhood around them.
Revised: Both in DH and NLM, the words/pixels that end up with the same color have same colored neighborhood around them.

Original: In order to ﬁnd more connections to stable coloring, especially that of embedding models, we need amore continuous representation of color and hence in next section we start building on a novel notion of color.
Revised: In order to ﬁnd more connections to stable coloring, especially that of embedding models, we need a continuous representation of color and hence in next section we start building on a novel notion of color.

Original:  G ENERAL A GGREGATE L EARNING FRAMEWORK FOR CSC
Revised:  G ENERAL A GGREGATE L EARNING FRAMEWORK FOR CSC

Original: As shown in the earlier sections, even discrete stable coloring can be connected to various concepts in NLP and image processing.
Revised: As shown in the earlier sections, even discrete stable coloring can be connected to various concepts in NLP and image processing.

Original: We ﬁnd even more deep rooted links between stable coloring and unsupervised learning algorithms in structured domains.
Revised: We ﬁnd even more deep rooted links between stable coloring and unsupervised learning algorithms in structured domains.

Original: In fact, we can view embeddings of each node as a continuous ‘color’ assigned to each node.
Revised: In fact, we can view embeddings of each node as a continuous ‘color’ assigned to each node.

Original: In order to show these links, we ﬁrst need to generalize the idea of SC to a setting where we can talk about continuously comparable colors.
Revised: In order to show these links, we ﬁrst need to generalize the idea of SC to a setting where we can talk about continuously comparable colors.

Original: This section is organised as follows.
Revised: This section is organised as follows.

Original: We ﬁrst deﬁne a CSC problem analogous to SC .
Revised: We ﬁrst deﬁne a CSC problem analogous to SC .

Original: We show that this is a strict generalization of SC (discrete) problem.
Revised: We show that this is a strict generalization of SC (discrete) problem.

Original: Then, inspired by the GAU for stable coloring, we propose a series of learning problems having GAU at their core (called General Aggregate Learning , GAL in short) .
Revised: Then, inspired by the GAU for stable coloring, we propose a series of learning problems having GAU at their core (called General Aggregate Learning , GAL in short) .

Original: We end this section with a generalized algorithm whose parameters, as we show in section 6, can be initialized in various ways to obtain algorithms in varied domains that have been a de facto standard in those domains since a long time.
Revised: We end this section with a generalized algorithm whose parameters, as we show in section 6, can be initialized in various ways to obtain algorithms in varied domains that have been a de facto standard in those domains since a long time.

Original: 5. C ONTINUOUS S TABLE C OLORING (CSC )
Revised: 5. C ONTINUOUS S TABLE C OLORING (CSC )

Original: Let us consider graph G =
Revised: Let us consider graph G =

Original: ( V , E ) representing the structured domain under consideration.
Revised: ( V , E ) representing the structured domain under consideration.

Original: In SC , we assign categorical colors to the nodes of the graph.
Revised: In SC , we assign categorical colors to the nodes of the graph.

Original: However, for most analytical tasks, including machine learning, we need to assign continuous labels (or embedding) to the nodes.
Revised: However, for most analytical tasks, including machine learning, we need to assign continuous labels (or embedding) to the nodes.

Original: Hence, we introduce the problem of CSC .
Revised: Hence, we introduce the problem of CSC .

Original: Let the domain of colors assigned be a metric space L (eg. R d ) associated with a distance metric D .
Revised: Let the domain of colors assigned be a metric space L (eg. R d ) associated with a distance metric D .

Original: There are various ways to deﬁne similarity metrics in literature based on D .
Revised: There are various ways to deﬁne similarity metrics in literature based on D .

Original: For the sake of discussion in this section, we would use a simple deﬁnition of
Revised: For the sake of discussion in this section, we would use a simple deﬁnition of

Original: S ( x, y ) =
Revised: S ( x, y ) =

Original: e −D ( x,y ) .
Revised: e −D ( x,y ) .

Original: While the theorems with other deﬁnitions will change in appearance, they will still maintain the spirit of analysis.
Revised: While the theorems with other deﬁnitions will change in appearance, they will still maintain the spirit of analysis.

Original:  LetC :
Revised: We denote the neighbourhood N ( u ) = { v | ( u, v ) ∈ E} . Let

Original:  V → L denote the coloring of nodes.
Revised: C : V → L denote the coloring of nodes.

Original: We overload the function C , C : 2 V → N L , to operate on subset of nodes as given in the following equation.
Revised: We overload the function C , C : 2 V → N L , to operate on subset of nodes as given in the following equation.

Original: We use the notation N L to denote all multi-subsets of L .
Revised: We use the notation N L to denote all multi-subsets of L .

Original: [Equation]
Revised: [Equation]

Original: We deﬁne S N : N L × N L → R as the similarity metric over the multi-subsets of L via the same similarity S over L and a permutation invariant and injective aggregator function f agg : N L → L as[Equation] , A, B [Equation]
Revised: We deﬁne S N : N L × N L → R as the similarity metric over the multi-subsets of L via the same similarity S over L and a permutation invariant and injective aggregator function f agg : N L → L as

Original: L

Original: [Equation]
Revised: [Equation]

Original: We refer to this embedding as the continuous color under CSC .
Revised: We refer to this embedding as the continuous color under CSC .

Original: We use this terminology interchangeably as is best for the context.
Revised: We use this terminology interchangeably as is best for the context.

Original: We denote nodes by small case letters ( u, v, ... ) and the subset/multi-subsets of nodes by upper case letters ( A, B, .. ) etc.
Revised: We denote nodes by small case letters ( u, v, ... ) and the subset/multi-subsets of nodes by upper case letters ( A, B, .. ) etc.

Original: Let us now take a look at the continuous stable coloring formulation (CSC ).
Revised: Let us now take a look at the continuous stable coloring formulation (CSC ).

Original: Deﬁnition 5.1.
Revised: Deﬁnition 5.1.

Original: (Continuous Stable Coloring ( L , S , f agg ) ).
Revised: (Continuous Stable Coloring ( L , S , f agg ) ).

Original: The coloring C : V → L of nodes in graph G =
Revised: The coloring C : V → L of nodes in graph G =

Original: ( V , E ) is called continuous stable coloring parameterised by similarity kernel S :
Revised: ( V , E ) is called continuous stable coloring parameterised by similarity kernel S :

Original: L × L → R for some metric space L and an injective aggregator function f agg : N L → L if the following holds:
Revised: L × L → R for some metric space L and an injective aggregator function f agg : N L → L if the following holds:

Original: [Equation]
Revised: [Equation]

Original: In most applications, we look at L = R d for some d > 0 .
Revised: In most applications, we look at L = R d for some d > 0 .

Original: Essentially, CSC states that the similarity between embeddings of two nodes should be equal to the similarity between the neighborhoods of the two nodes.
Revised: Essentially, CSC states that the similarity between embeddings of two nodes should be equal to the similarity between the neighborhoods of the two nodes.

Original: Thus CSC relaxes the Kronecker delta function of comparison over categories in SC to a general similarity metric over L .
Revised: Thus CSC relaxes the Kronecker delta function of comparison over categories in SC to a general similarity metric over L .

Original: In fact CSC is a strict generalization of SC which we showthis in the next theorem.
Revised: In fact CSC is a strict generalization of SC which we show in the next theorem.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: (SC is a special case of CSC ).
Revised: (SC is a special case of CSC ).

Original: Stable coloring (discrete) problem is an instance of continuous stable coloring problem with L = N , S
Revised: Stable coloring (discrete) problem is an instance of continuous stable coloring problem with L = N , S

Original: ( i, j )
Revised: ( i, j )

Original: = 1 ( i = j ) and S N ( s 1 , s 2 )
Revised: = 1 ( i = j ) and S N ( s 1 , s 2 )

Original: =
Revised: =

Original: 1 ( s 1 = s 2 ) where i, j ∈ N and s 1 , s 2 ∈ N N .
Revised: 1 ( s 1 = s 2 ) where i, j ∈ N and s 1 , s 2 ∈ N N .

Original: In this case f agg : N N → N function isessentially an injective hash function which maps multi-subsets of N to N .
Revised: In this case f agg : N N → N function is an injective hash function which maps multi-subsets of N to N .

Original: It is easy to verify the validity of this theorem by using correct values for L , S and f agg as mentioned in the theorem.
Revised: It is easy to verify the validity of this theorem by using correct values for L , S and f agg as mentioned in the theorem.

Original: Next, we deﬁne a learning problem GAL which solves the CSC problem.
Revised: Next, we deﬁne a learning problem GAL which solves the CSC problem.

Original: 5. G ENERAL A GGREGATE L EARNING (GAL )
Revised: 5. G ENERAL A GGREGATE L EARNING (GAL )

Original: Consider G =
Revised: Consider G =

Original: ( V , E ) representing the structured domain under consideration.
Revised: ( V , E ) representing the structured domain under consideration.

Original: In (Xu et al., 2019b), authors showed that GAU framework (equation 1) with  injective f merge and f agg operations is equivalent to 1-WL which solves SC problem.
Revised: In (Xu et al., 2019b), authors showed that GAU framework (equation 1) with an injective f merge and f agg operations is equivalent to 1-WL which solves SC problem.

Original: We deﬁne a solution to the CSC problem that is similar in spirit of GAU .
Revised: We deﬁne a solution to the CSC problem that is similar in spirit of GAU .

Original: However, instead of providing an iterative algorithm like GAU , we pose an optimization objective which, in essence, learns the stable solution directly.
Revised: However, instead of providing an iterative algorithm like GAU , we pose an optimization objective which, in essence, learns the stable solution directly.

Original: First, we deﬁne the notations.
Revised: First, we deﬁne the notations.

Original: Recall that the task is to assign continuous labels to graph nodes in metric space L with distance metric D and the similarity metric is given as S ( x, y )
Revised: Recall that the task is to assign continuous labels to graph nodes in metric space L with distance metric D and the similarity metric is given as S ( x, y )

Original: =
Revised: =

Original: e −D ( x,y ) .
Revised: e −D ( x,y ) .

Original: Notation for the embedding matrix E is deﬁned in the same way as that of a coloring function C , such that E : V → L and E : 2 V → {{ L }} :
Revised: Notation for the embedding matrix E is deﬁned in the same way as that of a coloring function C , such that E : V → L and E [Equation]

Original: [Equation]
Revised: [Equation]

Original: E stores the color assignments of all nodes in V and is learned in our setting.
Revised: E stores the color assignments of all nodes in V and is learned in our setting.

Original: Let us look at our ﬁrst optimization objective which follows naturally from the deﬁnition of CSC .
Revised: Let us look at our ﬁrst optimization objective which follows naturally from the deﬁnition of CSC .

Original: Deﬁnition 5.2.
Revised: Deﬁnition 5.2.

Original: (Global GAL formulation) Let matrix E ∈ R |V|× d store embeddings (or colors) of nodes in graph G =
Revised: (Global GAL formulation) Let matrix E ∈ R |V|× d store embeddings (or colors) of nodes in graph G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: Then the optimization objective of Global GAL (G-GAL ) parameterized by an injective function f agg : N L → L (where L = R d ) is to learn an embedding matrix E that minimizes the following
Revised: Then the optimization objective of Global GAL (G-GAL ) parameterized by an injective function f agg : N L → L (where L = R d ) is to learn an embedding matrix E that minimizes the following

Original: [Equation]
Revised: [Equation]

Original: The function f agg can be as simple as a sum operation or as complex as a neural network architecture with learn-able parameters.
Revised: The function f agg can be as simple as a sum operation or as complex as a neural network architecture with learnable parameters.

Original: The global formulation follows naturally from the deﬁnition of CSC ( R d , S , f agg ) problem.
Revised: The global formulation follows naturally from the deﬁnition of CSC ( R d , S , f agg ) problem.

Original: Note that we project neighborhood embeddings( or coloring) into the same space as node embeddings.
Revised: Note that we project neighborhood embeddings( or coloring) into the same space as node embeddings.

Original: This will be important for subsequent formulations.
Revised: This will be important for subsequent formulations.

Original: Research Question: Can an effective learning strategy be formulated for minimizing G-GAL loss?
Revised: Research Question: Can an effective learning strategy be formulated for minimizing G-GAL loss?

Original: An effective algorithm for G-GAL formulation can stimulate further research.
Revised: An effective algorithm for G-GAL formulation can stimulate further research.

Original: To the best of our knowledge, we do not know of an algorithm in any domain which uses this formulation.
Revised: To the best of our knowledge, we do not know of an algorithm in any domain which uses this formulation.

Original: We think a possible reason why it is difﬁcult to learn is that in most applications, we only have access to subgraph samples of the true underlying graph.
Revised: We think a possible reason why it is difﬁcult to learn is that in most applications, we only have access to subgraph samples of the true underlying graph.

Original: Working with a sample of neighbourhood ˆ N ( u ) instead of the complete neighborhood implies we are estimating ˆ f agg ( E ( N ( u ))) .
Revised: Working with a sample of neighbourhood ˆ N ( u ) instead of the complete neighborhood implies we are estimating ˆ f agg ( E ( N ( u ))) .

Original: The errors in S ( ˆ f agg ( E ( N ( u ))) , ˆ f agg ( E ( N ( v )))) increase super-linearly with error in ˆ f agg ( E ( N ( u )) .
Revised: The errors in S ( ˆ f agg ( E ( N ( u ))) , ˆ f agg ( E ( N ( v )))) increase super-linearly with error in ˆ f agg ( E ( N ( u )) .

Original: Nonetheless, this formulation can be of independent interest.
Revised: Nonetheless, this formulation can be of independent interest.

Original: The above mentioned issue of multiplying noisy neighborhood estimates motivates us to ﬁnd a simpler learning problem.
Revised: The above mentioned issue of multiplying noisy neighborhood estimates motivates us to ﬁnd a simpler learning problem.

Original: We formulate a different learning problem and show in theorem 5.
Revised: We formulate a different learning problem and show in theorem 5.

Original: that this simpler problem, which we call Node-local GAL (L-GAL ) problem also solves G-GAL problem.
Revised: that this simpler problem, which we call Node-local GAL (L-GAL ) problem also solves G-GAL problem.

Original: Deﬁnition 5.3.
Revised: Deﬁnition 5.3.

Original: (L-GAL formulation).
Revised: (L-GAL formulation).

Original: Let matrix E ∈ R |V|× d store embeddings (or colors) of nodes in graph G =
Revised: Let matrix E ∈ R |V|× d store embeddings (or colors) of nodes in graph G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: Then the optimization objective of L-GAL parameterized by an injective function f agg : N L → L (where L = R d ) is to learn an embedding matrix E that minimizes the following
Revised: Then the optimization objective of L-GAL parameterized by an injective function f agg : N L → L (where L = R d ) is to learn an embedding matrix E that minimizes the following

Original: [Equation]
Revised: [Equation]

Original: In the above L-GAL formulation, we uncoupled the G-GAL formulation and thus eliminated the issue of multiplying noisy neighborhood estimates.
Revised: In the above L-GAL formulation, we uncoupled the G-GAL formulation and thus eliminated the issue of multiplying noisy neighborhood estimates.

Original: However, we would like to emphasize that the solution to L-GAL is also a good solution to G-GAL .
Revised: However, we would like to emphasize that the solution to L-GAL is also a good solution to G-GAL .

Original: We quantify this relation in the next theorem.
Revised: We quantify this relation in the next theorem.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: (L-GAL solves G-GAL ).
Revised: (L-GAL solves G-GAL ).

Original: Let the solution E to L-GAL upper-bounds each term in the summation of loss in equation 8 by some (cid:15) > 0 ; thus it upper-bounds total loss by |V| (cid:15) .
Revised: Let the solution E to L-GAL upper-bounds each term in the summation of loss in equation 8 by some (cid:15) > 0 ; thus it upper-bounds total loss by |V| (cid:15) .

Original: Then the same solution matrix E is a solution to G-GAL with each term in summation upper-bounded by 2 (cid:15) and thus upper-bounding the total loss by |V| 2 (cid:15) .
Revised: Then the same solution matrix E is a solution to G-GAL with each term in summation upper-bounded by 2 (cid:15) and thus upper-bounding the total loss by |V| 2 (cid:15) .

Original: The proof of theorem 5.2 can be found in appendix.
Revised: The proof of theorem 5.2 can be found in appendix.

Original: Uncoupling works because if you alternately bring E ( u ) and E ( v ) closer to f agg ( E ( N ( u ))) and f agg ( E ( N ( v ))) respectively, it forces the distances between ( E ( u ) , E ( v ) ) and ( f agg ( E ( N ( u ))) ,
Revised: Uncoupling works because if you alternately bring E ( u ) and E ( v ) closer to f agg ( E ( N ( u ))) and f agg ( E ( N ( v ))) respectively, it forces the distances between ( E ( u ) , E ( v ) ) and ( f agg ( E ( N ( u ))) ,

Original: f agg
Revised: f agg

Original: ( E ( N ( v ))) ) to be nearly equal.
Revised: ( E ( N ( v ))) ) to be nearly equal.

Original: 5. L-GAL WITH SAMPLES FROM G = ( V , E )
Revised: 5. L-GAL WITH SAMPLES FROM G = ( V , E )

Original: Often when trying to solve the L-GAL problem, one will be forced to work on sub-graph samples of the graph instead of the entire graph.
Revised: Often when trying to solve the L-GAL problem, one will be forced to work on sub-graph samples of the graph instead of the entire graph.

Original: This can happen for multiple practical reasons : (1) True graph is not known and we have access to the graph only through instances of sub-graphs.
Revised: This can happen for multiple practical reasons : (1) True graph is not known and we have access to the graph only through instances of sub-graphs.

Original: For example, true NLP graph is not known.
Revised: For example, true NLP graph is not known.

Original: But we have access to NLP text which hints at the graph.
Revised: But we have access to NLP text which hints at the graph.

Original: Performing gradient descent on the entire graph is computationally prohibitive.
Revised: Performing gradient descent on the entire graph is computationally prohibitive.

Original: For example social networking graphs are massive.
Revised: For example social networking graphs are massive.

Original: In this case one need to sample nodes from V and then sample the neighborhood of these nodes to estimate f agg ( N ( u )) for all sampled nodes u .
Revised: In this case one need to sample nodes from V and then sample the neighborhood of these nodes to estimate f agg ( N ( u )) for all sampled nodes u .

Original: To account for this practical scenario, we propose an optimization objective for working with sub-graphs.
Revised: To account for this practical scenario, we propose an optimization objective for working with sub-graphs.

Original: Let X be the data available for a particular domain, G x = V x , E x be the sub-graph corresponding to the example x ∈ X .
Revised: Let X be the data available for a particular domain, G x = V x , E x be the sub-graph corresponding to the example x ∈ X .

Original: Let the neighborhood function restricted to this sub-graph be N x .
Revised: Let the neighborhood function restricted to this sub-graph be N x .

Original: We deﬁne the optimization objective as,
Revised: We deﬁne the optimization objective as,

Original: [Equation]
Revised: [Equation]

Original: where ˆ N ( u ) is neighborhood induced by the sub-graph.
Revised: where ˆ N ( u ) is neighborhood induced by the sub-graph.

Original: Essentially, we consider each node in the sub-graph and its induced neighbours ˆ N from the sub-graph as an example and optimize the loss.
Revised: Essentially, we consider each node in the sub-graph and its induced neighbours ˆ N from the sub-graph as an example and optimize the loss.

Original: 5. CSC WITH CONSTRAINTS
Revised: 5. CSC WITH CONSTRAINTS

Original: The problem of CSC , as are most other learning algorithms, is under-speciﬁed (particularly when f agg is a highly expressive function).
Revised: The problem of CSC , as are most other learning algorithms, is under-speciﬁed (particularly when f agg is a highly expressive function).

Original: Also, sometimes there is additional information that one wants to induct in the loss function which is not present in graph structure.
Revised: Also, sometimes there is additional information that one wants to induct in the loss function which is not present in graph structure.

Original: For example, in commercial product search settings, “nike” and “adidas” get similar embeddings due to similar neighborhood graph structure but one might want that the embeddings learned distinguish between these entities.
Revised: For example, in commercial product search settings, “nike” and “adidas” get similar embeddings due to similar neighborhood graph structure but one might want that the embeddings learned distinguish between these entities.

Original: In such cases, a constraint needs to be imposed in CSC to minimize similarity between such pairs.
Revised: In such cases, a constraint needs to be imposed in CSC to minimize similarity between such pairs.

Original: This concept is generalized by using a negative sampling function NS : V → 2 V which deﬁnes a set of nodes in 2 V on which a constraint needs to be imposed for any given node in V .
Revised: This concept is generalized by using a negative sampling function NS : V → 2 V which deﬁnes a set of nodes in 2 V on which a constraint needs to be imposed for any given node in V .

Original: We write the problem of constrained CSC as follows:
Revised: We write the problem of constrained CSC as follows:

Original: [Equation]
Revised: [Equation]

Original: ) subject to the CSC condition:
Revised: ) subject to the CSC condition:

Original: [Equation]
Revised: [Equation]

Original: Using the Lagrangian multiplier we can re-write the constrained optimization objective of L-GAL as
Revised: Using the Lagrangian multiplier we can re-write the constrained optimization objective of L-GAL as

Original: [Equation]
Revised: [Equation]

Original: In most applications, λ is set to 1.
Revised: In most applications, λ is set to 1.

Original: There are variants of inducing negative loss into a system, e.g., softmax (or sampled softmax ) is one such popular variant.
Revised: There are variants of inducing negative loss into a system, e.g., softmax (or sampled softmax ) is one such popular variant.

Original: 5. O PTIMIZATION ALGORITHM TO SOLVE L-GAL AND ITS VARIANTS
Revised: 5. O PTIMIZATION ALGORITHM TO SOLVE L-GAL AND ITS VARIANTS

Original: We have developed a series of optimization objectives to solve the problem posed by CSC in the previous subsections.
Revised: We have developed a series of optimization objectives to solve the problem posed by CSC in the previous subsections.

Original: We close this section by discussing the algorithms used to solve these optimization objectives.
Revised: We close this section by discussing the algorithms used to solve these optimization objectives.

Original: We can use any of the standard optimization algorithms: ﬁrst order algorithms like gradient descent (Courant et al., 1994), stochastic gradient descent (Kiefer & Wolfowitz, 1952) or second order algorithms like Adam (Kingma & Ba, 2015), Adagrad (Duchi et al., 2011), etc.
Revised: We can use any of the standard optimization algorithms: ﬁrst order algorithms like gradient descent (Courant et al., 1994), stochastic gradient descent (Kiefer & Wolfowitz, 1952) or second order algorithms like Adam (Kingma & Ba, 2015), Adagrad (Duchi et al., 2011), etc.

Original: When solving problem on complete graph, we denote the algorithm as A ( G =
Revised: When solving problem on complete graph, we denote the algorithm as A ( G =

Original: ( V , E ) , S , f agg , NS) which is parameterized by underlying graph G =
Revised: ( V , E ) , S , f agg , NS) which is parameterized by underlying graph G =

Original: ( V , E ) , similarity metric S , aggregation function f agg , and negative sampling function NS :
Revised: ( V , E ) , similarity metric S , aggregation function f agg , and negative sampling function NS :

Original: V → 2 V .
Revised: V → 2 V .

Original: Whenever working with sub-graph samples of the true graph G , we denote the algorithm as as A s ( X, S , f agg , NS) where X is the data, each element x of which, induces a sub-graph G x
Revised: Whenever working with sub-graph samples of the true graph G , we denote the algorithm as as A s ( X, S , f agg , NS) where X is the data, each element x of which, induces a sub-graph G x

Original: ( The precise deﬁnition of this in provided in section 6.1)
Revised: ( The precise deﬁnition of this in provided in section 6.1)

Original:  I NSTANTIATIONS OF L-GAL IN THE L ITERATURE
Revised:  I NSTANTIATIONS OF L-GAL IN THE L ITERATURE

Original: Firstly,  describe the construction of the graph for a particular domain and sub-graph induction based on the sample in the data.
Revised: Firstly, we describe the construction of the graph for a particular domain and sub-graph induction based on samples from the data.

Original: Secondly we summarize the graph construction and instantiations of LGAL optimization objective for different embedding models in table 1 and discuss some aspects of it in section 6.2.
Revised: Secondly we summarize the graph construction and instantiations of L-GAL optimization objective for different embedding models in table 1 and discuss some aspects of it in section 6.2.

Original: 6. G RAPH C ONSTRUCTION .
Revised: 6. G RAPH C ONSTRUCTION .

Original: We provide a generic recipe of graph construction on structured domains.
Revised: We provide a generic recipe of graph construction on structured domains.

Original: Consider for example a domain with token set T .
Revised: Consider for example a domain with token set T .

Original: Our graph will have these tokens as nodes.
Revised: Our graph will have these tokens as nodes.

Original: However, we can introduce even higher order tokens by combining tokens.
Revised: However, we can introduce even higher order tokens by combining tokens.

Original: For example, the set T 2 will denote a set of all bi-gram tokens in the domain.
Revised: For example, the set T 2 will denote a set of all bi-gram tokens in the domain.

Original: We can extend this idea to n -grams by considering the set T n .
Revised: We can extend this idea to n -grams by considering the set T n .

Original: Thus, the set of all nodes in the graph of domain can be written as
Revised: Thus, the set of all nodes in the graph of domain can be written as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: where k is the gram-depth.
Revised: where k is the gram-depth.

Original: The edges in this graph of gram-depth k depend on the application.
Revised: The edges in this graph of gram-depth k depend on the application.

Original: This is illustrated in ﬁgure 1a.
Revised: This is illustrated in ﬁgure 1a.

Original: In most cases, including the models discussed in this paper, the edges are added on basis of co-occurrence statistics in the data.
Revised: In most cases, including the models discussed in this paper, the edges are added on basis of co-occurrence statistics in the data.

Original: For example, let us consider the case of NLP.
Revised: For example, let us consider the case of NLP.

Original: The sentence “The tree fell” in a graph with a gram-depth of 3 will include the nodes { the, tree, fell, { the, tree } , { the, fell } , { tree, fell } , { the, tree, fell }} .
Revised: The sentence “The tree fell” in a graph with a gram-depth of 3 will include the nodes { the, tree, fell, { the, tree } , { the, fell } , { tree, fell } , { the, tree, fell }} .

Original: If the edges are added on the basis of cooccurrence, then based on the above sentence we add the edges between “The” and “fell,” “The tree” and “fell”, “The fell” and “tree,” and so on.
Revised: If the edges are added on the basis of cooccurrence, then based on the above sentence we add the edges between “The” and “fell,” “The tree” and “fell”, “The fell” and “tree,” and so on.

Original: Additionally, edges can have weights corresponding to the frequency of co-occurrence.
Revised: Additionally, edges can have weights corresponding to the frequency of co-occurrence.

Original: While the graph on tokens represents the true graph, the algorithm A s which solves the sampled version of L-GAL problem uses the sub-graph of G =
Revised: While the graph on tokens represents the true graph, the algorithm A s which solves the sampled version of L-GAL problem uses the sub-graph of G =

Original: ( V , E ) which is induced by the sample.
Revised: ( V , E ) which is induced by the sample.

Original: Considering the sample x ∈ X as a set of tokens x ⊆ T , the graph induced by this set of tokens,
Revised: Considering the sample x ∈ X as a set of tokens x ⊆ T , the graph induced by this set of tokens,

Original: G x = ( V x , E x ) with neighborhood function N x where 
Revised: G x = ( V x , E x ) with neighborhood function N x where [Equation]

Original: [Equation]
Revised: [Equation]

Original: 6. C URRENT S TATE - OF - THE - ART EMBEDDING MODELS ARE INSTANTIATIONS OF L-GAL
Revised: 6. C URRENT S TATE - OF - THE - ART EMBEDDING MODELS ARE INSTANTIATIONS OF L-GAL

Original: Now, we summarize the instantiations of A s for variety of algorithms in domains of NLP, images, and graphs.
Revised: Now, we summarize the instantiations of A s for variety of algorithms in domains of NLP, images, and graphs.

Original: For details on working of each algorithm, we direct the reader to the corresponding original papers.
Revised: For details on working of each algorithm, we direct the reader to the corresponding original papers.

Original: We want to point out the usage of exp (cid:104) x , y (cid:105) (cid:0) or 1 1+exp( −(cid:104) x , y (cid:105) ) (cid:1) as similarity metrics.
Revised: We want to point out the usage of exp (cid:104) x , y (cid:105) (cid:0) or 1 1+exp( −(cid:104) x , y (cid:105) ) (cid:1) as similarity metrics.

Original: Under the unit-norm assumption on x and y , a standard assumption for analysis of embeddings, exp (cid:104) x , y (cid:105) = exp (1 − D 2 ( x , y ) / 2) which is only a function of the l 2 norm D and inversely proportional to it.
Revised: Under the unit-norm assumption on x and y , a standard assumption for analysis of embeddings, exp (cid:104) x , y (cid:105) = exp (1 − D 2 ( x , y ) / 2) which is only a function of the l 2 norm D and inversely proportional to it.

Original: Thus, exp (cid:104) x , y (cid:105) ( or 1 1+exp( −(cid:104) x , y (cid:105) ) ) is a valid similarity metric under unit-norm assumption.
Revised: Thus, exp (cid:104) x , y (cid:105) ( or 1 1+exp( −(cid:104) x , y (cid:105) ) ) is a valid similarity metric under unit-norm assumption.

Original: 6.2. W ORD 2V EC , N ODE 2V EC , AWE
Revised: 6.2. W ORD 2V EC , N ODE 2V EC , AWE

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: The algorithm A s for L-GAL initialized with parameters from table 1 for Word2Vec,
Revised: The algorithm A s for L-GAL initialized with parameters from table 1 for Word2Vec,

Original: Node2Vec and AWE leads to exactly the algorithms (with possibly minor variations) as proposed in the original papers of Word2Vec, Node2Vec and AWE
Revised: Node2Vec and AWE leads to exactly the algorithms (with possibly minor variations) as proposed in the original papers of Word2Vec, Node2Vec and AWE

Original: The proof of the above theorem is quite straightforward and is provided in the appendix.
Revised: The proof of the above theorem is quite straightforward and is provided in the appendix.

Original: The formulation of sampled L-GAL problem provides a natural explanation of why AWE performs better than Word2Vec.
Revised: The formulation of sampled L-GAL problem provides a natural explanation of why AWE performs better than Word2Vec.

Original: Both Word2Vec and AWE operate on the same graph G =
Revised: Both Word2Vec and AWE operate on the same graph G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: However, they compute their estimates of f agg ( ˆ N ( u )) differently.
Revised: However, they compute their estimates of f agg ( ˆ N ( u )) differently.

Original: While word2vec performs simple sum, AWE performs weighted sum.
Revised: While word2vec performs simple sum, AWE performs weighted sum.

Original: One can prove that the MSE error with simple sum is larger than with weighted sum when weights are equal to the co-occurrence frequency.
Revised: One can prove that the MSE error with simple sum is larger than with weighted sum when weights are equal to the co-occurrence frequency.

Original: AWE models these weights as these weights are not known apriori and the authors in the paper hint at the learned weights being representative of the these co-occurrence frequencies.
Revised: AWE models these weights as these weights are not known apriori and the authors in the paper hint at the learned weights being representative of the these co-occurrence frequencies.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 6.2. BERT AND V I T
Revised: 6.2. BERT AND V I T

Original: Theorem 6.2.
Revised: Theorem 6.2.

Original: Considering BERT and ViT as stacks of attention layers without any non-linearity and initializing the algorithm A s with the parameters mentioned in the table 1 leads to the algorithms (with possibly minor variations) as proposed in the original papers of BERT and ViT.
Revised: Considering BERT and ViT as stacks of attention layers without any non-linearity and initializing the algorithm A s with the parameters mentioned in the table 1 leads to the algorithms (with possibly minor variations) as proposed in the original papers of BERT and ViT.

Original: The proof of the theorem is provided in the appendix.
Revised: The proof of the theorem is provided in the appendix.

Original: For this analysis, we will assume that BERT is a stack of attention layers without any non-linearity.
Revised: For this analysis, we will assume that BERT is a stack of attention layers without any non-linearity.

Original: BERT and ViT have similar architecture and hence we provide a single proof for them.
Revised: BERT and ViT have similar architecture and hence we provide a single proof for them.

Original: Let us consider the sub-graph induced for BERT by a sample x , which means it essentially has the nodes ∪ ki =1 x i .
Revised: Let us consider the sub-graph induced for BERT by a sample x , which means it essentially has the nodes ∪ ki =1 x i .

Original: The sub-graph will have every node connect with every other node.
Revised: The sub-graph will have every node connect with every other node.

Original: Thus when we look at the embedding of a particular masked word, according to deﬁnition of f agg for BERT we will get,
Revised: Thus when we look at the embedding of a particular masked word, according to deﬁnition of f agg for BERT we will get,

Original: [Equation]
Revised: [Equation]

Original: Now if this is exactly the computation that BERT (or ViT) performs then with softmax style negative sampling loss, BERT will follow the L-GAL optimization.
Revised: Now if this is exactly the computation that BERT (or ViT) performs then with softmax style negative sampling loss, BERT will follow the L-GAL optimization.

Original: We claim that BERT does indeed compute this ˆ f agg .
Revised: We claim that BERT does indeed compute this ˆ f agg .

Original: Ideally, BERT should learn the embedding matrix for all the nodes including tokens and higher order tokens.
Revised: Ideally, BERT should learn the embedding matrix for all the nodes including tokens and higher order tokens.

Original: However, this is computationally expensive.
Revised: However, this is computationally expensive.

Original: Hence, BERT actually models the higher order tokens in terms of its component tokens.
Revised: Hence, BERT actually models the higher order tokens in terms of its component tokens.

Original: It can be seen that if the modelling is a weighted linear sum of the components, then BERT essentially ends up computing equation 14.
Revised: It can be seen that if the modelling is a weighted linear sum of the components, then BERT essentially ends up computing equation 14.

Original: More details can be found in appendix.
Revised: More details can be found in appendix.

Original: Also, as in AWE it models the weights in the summation of f agg (via the attention mechanism).
Revised: Also, as in AWE it models the weights in the summation of f agg (via the attention mechanism).

Original: The information propagated through BERT can be visualized as shown in ﬁgure 1b.
Revised: The information propagated through BERT can be visualized as shown in ﬁgure 1b.

Original: At each layer i ∈ { 1 , .., k } , BERT computes the token embeddings of order i and this information from each layer ﬂows to the masked word node and gets aggregated.
Revised: At each layer i ∈ { 1 , .., k } , BERT computes the token embeddings of order i and this information from each layer ﬂows to the masked word node and gets aggregated.

Original:  I MPLICATIONS AND C ONCLUSIONS
Revised:  I MPLICATIONS AND C ONCLUSIONS

Original: In this paper, we deﬁne a notion of CSC to understand the internals of wide range of embedding models across diverse structured domains.
Revised: In this paper, we deﬁne a notion of CSC to understand the internals of wide range of embedding models across diverse structured domains.

Original: Grounded on this notion, we propose a generalized LGAL optimization problem that solves for CSC on graphs induced by any structured domain.
Revised: Grounded on this notion, we propose a generalized LGAL optimization problem that solves for CSC on graphs induced by any structured domain.

Original: We thereby prove equivalence between loss functions of popular NLP, image, and graph embedding models and our proposed constrained L-GAL optimization loss operating on domain-speciﬁc graphs.
Revised: We thereby prove equivalence between loss functions of popular NLP, image, and graph embedding models and our proposed constrained L-GAL optimization loss operating on domain-speciﬁc graphs.

Original: The methodology of construction of these graphs is also presented for each structured domain.
Revised: The methodology of construction of these graphs is also presented for each structured domain.

Original: Our proposed framework with its robust mathematical founding on CSC us graph theoretical perspective to these unsupervised embedding models.
Revised: Our proposed framework with its robust mathematical founding on CSC us graph theoretical perspective to these unsupervised embedding models.

Original: We have already seen great improvements in supervised models based on the connection between 1-WL and GNNs.
Revised: We have already seen great improvements in supervised models based on the connection between 1-WL and GNNs.

Original: We believe that our formulation will stimulate further research in embedding models.
Revised: We believe that our formulation will stimulate further research in embedding models.

Original: Insights from the underlying graph view of the computation in embedding models can give us new directions to improve quality and scalability and training efﬁciency of these models.
Revised: Insights from the underlying graph view of the computation in embedding models can give us new directions to improve quality and scalability and training efﬁciency of these models.

Revised: A A PPENDIX A. A WALK THROUGH POPULAR EMBEDDING MODELS AND UNDERSTANDING THEIR DESIGN PRINCIPLES AGAINST THE BACKDROP OF OUR FRAMEWORK

Revised: The proposed L-GAL formulation abstracts out different components of embedding models and thus enables design of better embedding models by focusing on these components.

Revised: The components are as follows,

Revised: • Domain Graphs • f agg Neighborhood aggregation function.

Original: A. T HEOREM 4.
Revised: • NS : Negative Sampling set. • Estimation of f agg ( N ( u )) via sampling. One can think of the strategies focusing on these components as , Better sampling/ estimates of a node’s neighborhoods.

Revised: Better design of f agg function.

Revised: Construction of more expressive graphs that embed higher order tokens.

Revised: Importance based negative sampling to inject knowledge that best negates the neighbor information given by our proposed domain graph.

Revised: With these four strategies in place, next we understand the development of NLP models over time by choosing word2vec, one of the most widely-used embedding model, as our starting reference.

Revised: We demonstrate how our proposed strategies have manifested in different embedding models that have improved over word2vec in last decade.

Revised: Recall from table 1, that word2vec uses

Revised: • G =

Revised: ( V, E ) where V is the set of all single words as nodes and E is the edge between words, say u and v, with co-occurrence as the weight.

Revised: • f agg ( N ( u )) weighted sum of neighbour embeddings • NS(u): All words except u • Sampling technique : sentences

Revised: Strategy 1 : Better estimation of f agg ( N ( u )) - Glove vs Word2vec Word2vec is an iterative algorithm over sentences.

Revised: It uses sentence as a sample to guess word’s neighborhood which induces a lot of noise.

Revised: Our framework would suggest to use the entire computation of f agg instead of sampling if possible for a given f agg and graph.

Revised: Glove does exactly this.

Revised: Glove (Pennington et al., 2014a) improves upon word2vec by using global co-occurrence statistics to eliminate noise in these neighborhood estimates.

Revised: Glove’s optimization objective (given by equation 12 which is simpliﬁed to equation 15 in original paper) is as follows:

Revised: [Equation]

Revised: where V is the vocabulary, X ij is the co-occurrence count of word i with word j in the corpus, and Q ij = exp w Ti w j (cid:80)

Revised: Vk =1 exp w Ti w k .

Revised: Note that Glove simpliﬁes Q ij = exp w Ti w j which is identical to the similarity kernel we use in our analysis with unit norm assumption.

Revised: Strategy 1, thus, helps us to understand why Glove improves upon word2vec.

Revised: Strategy 2 : Better f agg - AWE vs Word2vec Let us now consider another word embedding model, AWE (Attention Word Embedding), that improves upon the CBOW model of word2vec.

Revised: CBOW model weighs each word equally to compute the context embedding, however, AWE weighs each context word differently as some context words are more important for prediction of the masked word than others.

Revised: The context vector in AWE is given by:

Revised: [Equation]

Revised: where where b is the size of the context window and w i is the index of each word ( w 0 is the index of the masked word; the rest are the indices of the context words), a w i is the attention weight of each context word vector u w i calculated using the key matrix K and the query matrix Q .

Revised: Note that in word2vec a w i = 1 .

Revised: Strategy 2 helps us understand why AWE outperforms Word2vec.

Revised: AWE formulates a better f agg function as compared to Word2Vec.

Revised: Strategy 3 : More informative domain graphs - BERT vs Word2vec As discussed in section ?? ,

Revised: BERT uses a graph with higher order tokens.

Revised: In fact, the highest order of token can be interpreted as the number of layers in ther BERT.

Revised: It also uses the AWE style weighting of the neighbours.

Revised: Strategy 3 (along with strategy 2) can help us understand how BERT improves over Word2vec embedding model.

Revised: Strategy 3 can also help us understand why (Le & Mikolov, 2014; Tang et al., 2015; Liu et al., 2015) can improve over Word2Vec.

Revised: Strategy 4 : Better NS - WSD-W2V vs Word2Vec

Revised: Strategy 4 helps to understand yet another shortcoming of word2vec.

Revised: Word2vec struggles with modeling antonyms and often brings antonyms close to each other (Thalenberg, 2016).

Revised: This is not surprising since antonyms tend to occur in similar contexts, i.e., identical neighborhoods.

Revised: With our proposed framework, one can arrive at this conclusion from a graph theoretic perspective.

Revised: Antonym subgraphs are isomorphic, and thus antonyms get assigned similar continuous stable colors.

Revised: To overcome this, one needs to introduce information into the setup using variants of negative sampling.

Revised: Thus external datasets like wordnet (Orkphol & Yang, 2019) need to complement word2vec’s training strategy for efﬁcient modeling of antonyms.

Revised: To conclude, the model design changes that our framework suggest (strategies 1-4) can be found in existing works.

Revised: A. T HEOREM 4.

Revised: Let { w 1 , w 2 , .., w n } be words in vocabulary, and M ∈ R n × n is a co-occurrence matrix with entries e ij ∈ N containing the number of times w i co-occurs with w j within a ﬁxed context window.

Revised: Let there be a function f : w → N which takes a word and maps it to a color c ∈ N (assigns meaning in accordance with Distributional Hypothesis), such that f ( w i ) =

Revised: f ( w j ) only if row i is same as row j in matrix M .

Revised: Construct a graph G DH with words as nodes and its adjacency matrix given by M .

Original:   Consider a discrete signal y ( x ) sampled at n points x i ( i = 1 , ..., n ) , and let the sequence p ( x i ) =
Revised: Then, function f deﬁnes a stable coloring on graph G DH . Proof: The proof follows trivially from the deﬁnition of f . A. T HEOREM 4.  Consider a discrete signal y ( x ) sampled at n points x i ( i = 1 , ..., n ) , and let the sequence p ( x i ) =

Original: ( y ( x i − t ) , .., y ( x i − 1 ) , y ( x i +1 ) , .., y
Revised: ( y ( x i − t ) , .., y ( x i − 1 ) , y ( x i +1 ) , .., y

Original: ( x i + t )) be a patch of neighborhood values around each x
Revised: ( x i + t )) be a patch of neighborhood values around each x

Original: i for some context window length t .
Revised: i for some context window length t .

Original: NLM denoises the signal y ( x i ) , i = 1 , ..., n with iterative updates.
Revised: NLM denoises the signal y ( x i ) , i = 1 , ..., n with iterative updates.

Original: The ﬁxed point denoised version of the signal y d can be written as follows:
Revised: The ﬁxed point denoised version of the signal y d can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where D ( x i ) =
Revised: where D ( x i ) =

Original: (cid:80) nj =1 K ( y d ( x i ) , y d ( x j )) and K is an arbitrary kernel function.
Revised: (cid:80) nj =1 K ( y d ( x i ) , y d ( x j )) and K is an arbitrary kernel function.

Original: Let the graph
Revised: Let the graph

Original: G nlm =
Revised: G nlm =

Original: ( V nlm , E nlm ) where each x i is a node u i in V nlm
Revised: ( V nlm , E nlm ) where each x i is a node u i in V nlm

Original: and each pair ( u i , u j ) with | i − j | ≤ t is represented as a directed edge in E nlm with label ( i − j ) .
Revised: and each pair ( u i , u j ) with | i − j | ≤ t is represented as a directed edge in E nlm with label ( i − j ) .

Original: Then the ﬁxed point solution of NLM with Kronecker delta kernel K δ , y d : N → R deﬁnes a weak -stable coloring over the graph G nlm
Revised: Then the ﬁxed point solution of NLM with Kronecker delta kernel K δ , y d : N → R deﬁnes a weak -stable coloring over the graph G nlm

Original: Proof:
Revised: Proof:

Original: In order to prove a weak coloring, we need to prove that if the neighbourhoods of a pixel are equal then its value is equal.
Revised: In order to prove a weak coloring, we need to prove that if the neighbourhoods of a pixel are equal then its value is equal.

Original: Consider any two pixels x i and x j with same neighbourhoods, Then
Revised: Consider any two pixels x i and x j with same neighbourhoods, Then

Original: K δ ( p d ( x i ) , p d ( x j ) = 1 .
Revised: K δ ( p d ( x i ) , p d ( x j ) = 1 .

Original: Let { x k 1 , x k 2 , x k 3 , ... } be the set of size L of pixels whose neighbourhoods that match with neighbourhood of x i and x j .
Revised: Let { x k 1 , x k 2 , x k 3 , ... } be the set of size L of pixels whose neighbourhoods that match with neighbourhood of x i and x j .

Original: Thus, by equation of ﬁnal stable solution, both the pixel values are equal to y d ( x i ) =
Revised: Thus, by equation of ﬁnal stable solution, both the pixel values are equal to y d ( x i ) =

Original: y d ( x j ) =
Revised: y d ( x j ) =

Original: 1 L (cid:80) Ll
Revised: 1 L (cid:80) Ll

Original: =1 K δ ( p d ( x i ) , p d ( x k l ))
Revised: =1 K δ ( p d ( x i ) , p d ( x k l ))

Original: y d ( x k l )
Revised: y d ( x k l )

Original: A. T HEOREM 5.
Revised: [Equation]

Original: (SC is a special case of CSC ).
Revised: (SC is a special case of CSC ).

Original: Stable coloring (discrete) problem is an instance of continuous stable coloring problem with L = N , S
Revised: Stable coloring (discrete) problem is an instance of continuous stable coloring problem with L = N , S

Original: ( i, j )
Revised: ( i, j )

Original: = 1 ( i = j ) and S N ( s 1 , s 2 )
Revised: = 1 ( i = j ) and S N ( s 1 , s 2 )

Original: =
Revised: =

Original: 1 ( s 1 = s 2 ) where i, j ∈ N and s 1 , s 2 ∈ N N .
Revised: 1 ( s 1 = s 2 ) where i, j ∈ N and s 1 , s 2 ∈ N N .

Original: In this case f agg : N N → N function is essentially an injective hash function which maps multi-subsets of N to N .
Revised: In this case f agg : N N → N function is essentially an injective hash function which maps multi-subsets of N to N .

Original: Proof: The proof of the theorem trivially follows by using the parameters of CSC as mentioned in the theorem.
Revised: Proof: The proof of the theorem trivially follows by using the parameters of CSC as mentioned in the theorem.

Original: A. T HEOREM 5. 
Revised: A. T HEOREM 5.

Original: Let the solution E to L-GAL upper-bounds each term in the summation of loss in equation 8 by some (cid:15) > 0 ; thus it upper-bounds total loss by |V| (cid:15) .
Revised: Let the solution E to L-GAL upper-bounds each term in the summation of loss in equation 8 by some (cid:15) > 0 ; thus it upper-bounds total loss by |V| (cid:15) .

Original: Then the same solution matrix E is a solution to
Revised: Then the same solution matrix E is a solution to

Original: G-GAL with each term in summation upper-bounded by 2 (cid:15) and thus upper-bounding the total loss by |V| 2 (cid:15) .
Revised: G-GAL with each term in summation upper-bounded by 2 (cid:15) and thus upper-bounding the total loss by |V| 2 (cid:15) .

Original: Proof:
Revised: Proof:

Original: [Equation]
Revised: [Equation]

Original: As D is a metric, we have

Original: [Equation]
Revised: [Equation]

Original:  D [Equation] f
Revised: As D is a metric, we have

Original: [Equation]
Revised: [Equation]

Original: N

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]

Original: f agg

Original: [Equation]
Revised: [Equation]

Original: N

Original: [Equation]

Original: v )

Original: [Equation]

Original: −D

Original: [Equation]

Original: E

Original: [Equation]

Original: u
Revised: u

Original: [Equation]

Original: E ( v )

Original: [Equation]

Original: ≤ D ( f agg ( E ( N ( u )

Original: [Equation]

Original: E ( u )

Original: [Equation]

Original: D

Original: [Equation]

Original: E

Original: [Equation]

Original: v )

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: N ( v ))))
Revised: N ( v ))))

Original: By hypothesis, that the each term in the loss of L-GAL is bound by (cid:15)
Revised: By hypothesis, that the each term in the loss of L-GAL is bound by (cid:15)

Original: [Equation]
Revised: [Equation]

Original: Thus ,
Revised: Thus ,

Original: [Equation]
Revised: [Equation]

Original: A. T HEOREM 6. 
Revised: A. T HEOREM 6.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: The algorithm A s for L-GAL initialized with parameters from table 1 for Word2Vec,
Revised: The algorithm A s for L-GAL initialized with parameters from table 1 for Word2Vec,

Original: Node2Vec and AWE leads to exactly the algorithms (with possibly minor variations) as proposed in the original papers of Word2Vec, Node2Vec and AWE proof It is easy to check that the parameters speciﬁed in the table 1 exactly provides the algorithms of corresponding models.
Revised: Node2Vec and AWE leads to exactly the algorithms (with possibly minor variations) as proposed in the original papers of Word2Vec, Node2Vec and AWE proof It is easy to check that the parameters speciﬁed in the table 1 exactly provides the algorithms of corresponding models.

Original: Some more details can be found in appendix B
Revised: Some more details can be found in appendix B

Original: A. T HEOREM 6. 
Revised: A. T HEOREM 6.

Original: Considering BERT and ViT as stacks of attention layers without any non-linearity and initializing the algorithm A s with the parameters mentioned in the table ?? leads to the algorithms (with possibly minor variations) as proposed in the original papers of BERT and ViT with the linear modelling on embedding of higher order tokens.
Revised: Considering BERT and ViT as stacks of attention layers without any non-linearity and initializing the algorithm A s with the parameters mentioned in the table 1 leads to the algorithms (with possibly minor variations) as proposed in the original papers of BERT and ViT with the linear modelling on embedding of higher order tokens.

Original: Proof : The parameters of BERT / ViT such as loss used, tokens can be veriﬁed from the table.
Revised: Proof : The parameters of BERT / ViT such as loss used, tokens can be veriﬁed from the table.

Original: It only remains to prove that BERT computation is same as that of f agg mentioned.
Revised: It only remains to prove that BERT computation is same as that of f agg mentioned.

Original: This is done under two assumptions, (1) BERT is just a stack of attention layers.
Revised: This is done under two assumptions, (1) BERT is just a stack of attention layers.

Original: (2) each higher order token’s embedding is modelled as a linear model of it constituent embeddings.
Revised: (2) each higher order token’s embedding is modelled as a linear model of it constituent embeddings.

Original: Let us consider the token ( u, i ) to be masked.
Revised: Let us consider the token ( u, i ) to be masked.

Original: Without loss of generality let the position in the token be last position.
Revised: Without loss of generality let the position in the token be last position.

Original: Let each path reaching the level k at masked word be denoted with the locations of the words the path takes at each level.
Revised: Let each path reaching the level k at masked word be denoted with the locations of the words the path takes at each level.

Original: So a path of (0,1,0,2) says that it takes the route of words w 0 , w 1 , w 0 , w 2 to reach the masked word at the ﬁnal level.
Revised: So a path of (0,1,0,2) says that it takes the route of words w 0 , w 1 , w 0 , w 2 to reach the masked word at the ﬁnal level.

Original: Then we can write the new computation of reaching BERT at level k as
Revised: Then we can write the new computation of reaching BERT at level k as

Original: [Equation]
Revised: [Equation]

Original: This can be seen as the embedding of the path as BERT models it (linear model assumption).
Revised: This can be seen as the embedding of the path as BERT models it (linear model assumption).

Original: It is easy to see that each path of length k maps one-to-one to a higher order token in T k .
Revised: It is easy to see that each path of length k maps one-to-one to a higher order token in T k .

Original: Under this relation PATHS(k) is exactly the set of all nodes in
Revised: Under this relation PATHS(k) is exactly the set of all nodes in

Original: S k
Revised: S k

Original: The information also ﬂows from masked words which aggregates the embeddings of S i , i < k .
Revised: The information also ﬂows from masked words which aggregates the embeddings of S i , i < k .

Original: Thus the ﬁnal computation of bert at level K looks like
Revised: Thus the ﬁnal computation of bert at level K looks like

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for some β s
Revised: for some β s

Original: B E XAMPLES OF GRAPH CONSTRUCTION B. V ARIOUS DOMAINS AS GRAPHS
Revised: B E XAMPLES OF GRAPH CONSTRUCTION B. V ARIOUS DOMAINS AS GRAPHS

Original: Any general set of tokens say T and associated relation R ⊆
Revised: Any general set of tokens say T and associated relation R ⊆

Original: T ×T can give us the graph G =
Revised: T ×T can give us the graph G =

Original: ( V , E ) with V = T and ( u, v ) ∈
Revised: ( V , E ) with V = T and ( u, v ) ∈

Original: E iff R ( u, v ) = 1 .
Revised: E iff R ( u, v ) = 1 .

Original: While the underlying graph of NLP is not explicitly known some examples of relations that can be drawn from the sentence structure are as follows.
Revised: While the underlying graph of NLP is not explicitly known some examples of relations that can be drawn from the sentence structure are as follows.

Original: • Let V be the set of all words { w 1 , w 2 , ...w |V| } .
Revised: • Let V be the set of all words { w 1 , w 2 , ...w |V| } .

Original: ( w 1 , w 2 ) ∈
Revised: ( w 1 , w 2 ) ∈

Original: E iff the words w 1 and w 2 co-occur in a sentence.
Revised: E iff the words w 1 and w 2 co-occur in a sentence.

Original: Additionally, we can assign a weight to the edge equal to the frequency of co-occurrence observed in some natural language corpus.
Revised: Additionally, we can assign a weight to the edge equal to the frequency of co-occurrence observed in some natural language corpus.

Original: • Let V be a set of all subsets of T of size less than or equal to k .
Revised: • Let V be a set of all subsets of T of size less than or equal to k .

Original: Let t 1 , t 2 ∈ V then the relation ( t 1 , t 2 )
Revised: Let t 1 , t 2 ∈ V then the relation ( t 1 , t 2 )

Original: ∈
Revised: ∈

Original: E iff t 1 ∪ t 2 appear together in some sentence in natural language.
Revised: E iff t 1 ∪ t 2 appear together in some sentence in natural language.

Original: Similarly we can also view the entire image collection as a graph G =
Revised: Similarly we can also view the entire image collection as a graph G =

Original: ( V , E ) where V is a set of all k × k patches observed in all images ( or alternatively images of a particular category ).
Revised: ( V , E ) where V is a set of all k × k patches observed in all images ( or alternatively images of a particular category ).

Original: Some examples of relations might be,
Revised: Some examples of relations might be,

Original: • R ( p 1 , p 2 ) =
Revised: • R ( p 1 , p 2 ) =

Original: 1 iff p 1 occurs in a larger patch K × K around p 2 in some images • R ( p 1 , p 2 )
Revised: 1 iff p 1 occurs in a larger patch K × K around p 2 in some images • R ( p 1 , p 2 )

Original: =
Revised: =

Original: 1 iff p 1 adjacent to p 2 in four directions (left, right, top, down)
Revised: 1 iff p 1 adjacent to p 2 in four directions (left, right, top, down)

Original: Graph domain naturally maps to the graph data structure.
Revised: Graph domain naturally maps to the graph data structure.

Original: However, we can construct more informative graphs from the basic graph structure.
Revised: However, we can construct more informative graphs from the basic graph structure.

Original: For example, we can add hyper-edges to add hyper-graph structure to the same graph.
Revised: For example, we can add hyper-edges to add hyper-graph structure to the same graph.

Original: [Equation]
Revised: [Equation]

Original: Word2vec is one of the standard word embedding models (Mikolov et al., 2013a;b).
Revised: Word2vec is one of the standard word embedding models (Mikolov et al., 2013a;b).

Original: The continuous bag-of-words (CBOW) model of word2vec predicts a masked word using its context.
Revised: The continuous bag-of-words (CBOW) model of word2vec predicts a masked word using its context.

Original: Let E =
Revised: Let E =

Original: [ v 1 , ..., v N ] T ∈ R N ×
Revised: [ v 1 , ..., v N ] T ∈ R N ×

Original: d be the word embedding matrix where N is the size of the vocabulary, and d is the size of the word vector.
Revised: d be the word embedding matrix where N is the size of the vocabulary, and d is the size of the word vector.

Original: CBOW looks at the set of sentences drawn from NLP corpus X and uses it to learn E .
Revised: CBOW looks at the set of sentences drawn from NLP corpus X and uses it to learn E .

Original: For each sentence x of size n , CBOW creates n examples of ( u, ctx( u ) = { v | v ∈ x, v (cid:54) = u } ) pairs by choosing one word and using rest of the words as context ctx .
Revised: For each sentence x of size n , CBOW creates n examples of ( u, ctx( u ) = { v | v ∈ x, v (cid:54) = u } ) pairs by choosing one word and using rest of the words as context ctx .

Original: It uses the following formulation,
Revised: It uses the following formulation,

Original: [Equation]
Revised: [Equation]

Original: where E (ctx( u )) =
Revised: where E (ctx( u )) =

Original: (cid:80) v ∈ ctx( u ) E ( v ) .
Revised: (cid:80) v ∈ ctx( u ) E ( v ) .

Original: Note that original formulation of CBOW uses softmax which is a variant of the negative sampling loss formulation.
Revised: Note that original formulation of CBOW uses softmax which is a variant of the negative sampling loss formulation.

Original: We will show in theorem ??
Revised: We will show in theorem ??

Original: that Word2Vec is indeed same as A s using some speciﬁc initialization of its parameters.
Revised: that Word2Vec is indeed same as A s using some speciﬁc initialization of its parameters.

Original: We ﬁrst deﬁne the parameters.
Revised: We ﬁrst deﬁne the parameters.

Original: • Graph G w 2 v .
Revised: • Graph G w 2 v .

Original: Let { w 1 , w 2 , .., w | V | } be words in vocabulary V , and M ∈ R | V |×| V | is a cooccurrence matrix with entries e ij ∈ N containing the number
Revised: Let { w 1 , w 2 , .., w | V | } be words in vocabulary V , and M ∈ R | V |×| V | is a cooccurrence matrix with entries e ij ∈ N containing the number

Original: of times w i co-occurs with w j within a ﬁxed context window.
Revised: of times w i co-occurs with w j within a ﬁxed context window.

Original: Construct a graph G w 2 v with words as nodes and its adjacency matrix given by M • Data X w 2 v .
Revised: Construct a graph G w 2 v with words as nodes and its adjacency matrix given by M • Data X w 2 v .

Original: Every example is a set of words (sentence) and the graph induced by these words is a sub-graph in G w 2 v with every node connecting every other node.
Revised: Every example is a set of words (sentence) and the graph induced by these words is a sub-graph in G w 2 v with every node connecting every other node.

Original: • Aggregation function f w 2 v .
Revised: • Aggregation function f w 2 v .

Original: Let f agg (ctx( u ))
Revised: Let f agg (ctx( u ))

Original: = (cid:80) v ∈ ctx( u ) E ( v ) .
Revised: = (cid:80) v ∈ ctx( u ) E ( v ) .

Original: • Similarity Metric S sgm .
Revised: • Similarity Metric S sgm .

Original: Let S ( x, y ) be σ ( x, y ) .
Revised: Let S ( x, y ) be σ ( x, y ) .

Original: Under unit-norm assumption, this is a valid similarity metric derived from l 2 norm distance metric.
Revised: Under unit-norm assumption, this is a valid similarity metric derived from l 2 norm distance metric.

Original: B. A TTENTION W ORD E MBEDDINGS (AWE)
Revised: B. A TTENTION W ORD E MBEDDINGS (AWE)

Original: As can be observed in f w 2 v , CBOW model equally weights the context words when making a prediction.
Revised: As can be observed in f w 2 v , CBOW model equally weights the context words when making a prediction.

Original: To address this limitation of CBOW, AWE augments CBOW model with an attention mechanism to attend to context words that are most relevant for prediction of masked word.
Revised: To address this limitation of CBOW, AWE augments CBOW model with an attention mechanism to attend to context words that are most relevant for prediction of masked word.

Original: AWE also uses the same formulation as word2vec given by equation 25, except that context vector is a weighted sum of context word embeddings.
Revised: AWE also uses the same formulation as word2vec given by equation 25, except that context vector is a weighted sum of context word embeddings.

Original: In AWE, E (ctx( u )) =
Revised: In AWE, E (ctx( u )) =

Original: (cid:80) v ∈ ctx( u ) w uv E ( v ) , where w uv = exp ( < k u , q v > ) models the importance of context word v for predicting the masked word u using key and query word embedding matrices given by K =
Revised: (cid:80) v ∈ ctx( u ) w uv E ( v ) , where w uv = exp ( < k u , q v > ) models the importance of context word v for predicting the masked word u using key and query word embedding matrices given by K =

Original: [ k 1 , ..., k N ] T ∈ R N × d and Q = [ q 1 , ..., q N ] T ∈ R N × d respectively.
Revised: [ k 1 , ..., k N ] T ∈ R N × d and Q = [ q 1 , ..., q N ] T ∈ R N × d respectively.

Original: As we did for word2vec, we will show in theorem ??
Revised: As we did for word2vec, we will show in theorem ??

Original: that AWE is indeed same as A s using some speciﬁc initialization of its parameters.
Revised: that AWE is indeed same as A s using some speciﬁc initialization of its parameters.

Original: We ﬁrst deﬁne the parameters.
Revised: We ﬁrst deﬁne the parameters.

Original: • Graph G awe .
Revised: • Graph G awe .

Original: G awe =
Revised: G awe =

Original: G w 2 v .
Revised: G w 2 v .

Original: • Data X awe .
Revised: • Data X awe .

Original: Similar to word2vec, every example is a set of words (sentence) but unlike the case of word2vec, the graph induced by these words is a weighted sub-graph in G awe .
Revised: Similar to word2vec, every example is a set of words (sentence) but unlike the case of word2vec, the graph induced by these words is a weighted sub-graph in G awe .

Original: • Aggregation function f awe .
Revised: • Aggregation function f awe .

Original: Let f agg (ctx( u ))
Revised: Let f agg (ctx( u ))

Original: = (cid:80) v ∈ ctx( u ) w uv E ( v ) .
Revised: = (cid:80) v ∈ ctx( u ) w uv E ( v ) .

Original: • Similarity Metric S sgm .
Revised: • Similarity Metric S sgm .

Original: Let S ( x, y ) be σ ( x, y ) .
Revised: Let S ( x, y ) be σ ( x, y ) .

Original: Under unit-norm assumption, this is a valid similarity metric derived from l 2 norm distance metric.
Revised: Under unit-norm assumption, this is a valid similarity metric derived from l 2 norm distance metric.