Original: H YPEREDGE 2 VEC : D ISTRIBUTED R EPRESENTATIONS FOR H YPEREDGES
Revised: H YPEREDGE 2 VEC : D ISTRIBUTED R EPRESENTATIONS FOR H YPEREDGES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Data structured in form of overlapping or non-overlapping sets is found in a variety of domains, sometimes explicitly but often subtly.
Revised: Data structured in form of overlapping or non-overlapping sets is found in a variety of domains, sometimes explicitly but often subtly.

Original: For example, teams, which are of prime importance in social science studies are “sets of individuals”; “item sets” in pattern mining are sets; and for various types of analysis in language studies a sentence can be considered as a “set or bag of words”.
Revised: For example, teams, which are of prime importance in social science studies are “sets of individuals”; “item sets” in pattern mining are sets; and for various types of analysis in language studies a sentence can be considered as a “set or bag of words”.

Original: Although building models and inference algorithms for structured data has been an important task in the ﬁelds of machine learning and statistics, research on “set-like” data still remains less explored.
Revised: Although building models and inference algorithms for structured data has been an important task in the ﬁelds of machine learning and statistics, research on “set-like” data still remains less explored.

Original: Relationships between pairs of elements can be modeled as edges in a graph.
Revised: Relationships between pairs of elements can be modeled as edges in a graph.

Original: However, modeling relationships that involve all members of a set, a hyperedge is a more natural representation for the set.
Revised: However, modeling relationships that involve all members of a set, a hyperedge is a more natural representation for the set.

Original: In this work, we focus on the problem of embedding hyperedges in a hypergraph (a network of overlapping sets) to a low dimensional vector space.
Revised: In this work, we focus on the problem of embedding hyperedges in a hypergraph (a network of overlapping sets) to a low dimensional vector space.

Original: We propose a probabilistic deep-learning based method as well as a tensor-based algebraic model, both of which capture the hypergraph structure in a principled manner without loosing set-level information.
Revised: We propose a probabilistic deep-learning based method as well as a tensor-based algebraic model, both of which capture the hypergraph structure in a principled manner without loosing set-level information.

Original: Our central focus is to highlight the connection between hypergraphs (topology), tensors (algebra) and probabilistic models.
Revised: Our central focus is to highlight the connection between hypergraphs (topology), tensors (algebra) and probabilistic models.

Original: We present a number of interesting baselines, some of which adapt existing node-level embedding models to the hyperedge-level, as well as sequence based language techniques which are adapted for set structured hypergraph topology.
Revised: We present a number of interesting baselines, some of which adapt existing node-level embedding models to the hyperedge-level, as well as sequence based language techniques which are adapted for set structured hypergraph topology.

Original: The performance is evaluated with a network of social groups and a network of word phrases.
Revised: The performance is evaluated with a network of social groups and a network of word phrases.

Original: Our experiments show that accuracy wise our methods perform similar to those of baselines which are not designed for hypergraphs.
Revised: Our experiments show that accuracy wise our methods perform similar to those of baselines which are not designed for hypergraphs.

Original: Moreover, our tensor based method is quiet efﬁcient as compared to deep-learning based auto-encoder method.
Revised: Moreover, our tensor based method is quiet efﬁcient as compared to deep-learning based auto-encoder method.

Original: We therefore, argue that we have proposed more general methods which are suited for hypergraphs (and therefore also for graphs) while maintaining accuracy and efﬁciency.
Revised: We therefore, argue that we have proposed more general methods which are suited for hypergraphs (and therefore also for graphs) while maintaining accuracy and efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The Internet has made it possible to record as well as share data about various entities, which may be related or interacting in a sophisticated manner.
Revised: The Internet has made it possible to record as well as share data about various entities, which may be related or interacting in a sophisticated manner.

Original: Complex networks Strogatz (2001); Roy et al. (2017) are often used to study such multi-faceted interactions among objects.
Revised: Complex networks (Strogatz, 2001; Roy et al., 2017) are often used to study such multi-faceted interactions among objects.

Original: Among the various models for complex networks, graphs are the most commonly used representations (Strogatz, 2001).
Revised: Among the various models for complex networks, graphs are the most commonly used representations (Strogatz, 2001).

Original: Graphs consist of set of nodes representing the entities under study, and edges between node-pairs capture the relationship between the entity pair Singhal et al.
Revised: Graphs consist of set of nodes representing the entities under study, and edges between node-pairs capture the relationship between the entity pair Singhal et al.

Original: (2013a;b).
Revised: (2013a;b).

Original: However, data in many applications has more complex structure, involving relations between multiple entities simultaneously Singhal et al. (2014), unlike simple pair-wise relationships modeled by graphs.
Revised: However, data in many applications has more complex structure, involving relations between multiple entities simultaneously (Singhal et al., 2014), unlike simple pair-wise relationships modeled by graphs.

Original: In fact, such data is more abundantly found in the real world than has been usually studied (Estrada & RodriguezVelazquez, 2005).
Revised: In fact, such data is more abundantly found in the real world than has been usually studied (Estrada & Rodriguez-Velazquez, 2005).

Original: Figure 1 shows such examples for networks of groups, sentences, and item sets.
Revised: Figure 1 shows such examples for networks of groups, sentences, and item sets.

Original: Hypergraph (Berge, 1984), which is a generalization of graphs, is a popular model to naturally capture higher-order relationships between sets of objects (Figure 2).
Revised: Hypergraph (Berge, 1984), which is a generalization of graphs, is a popular model to naturally capture higher-order relationships between sets of objects (Figure 2) (Estrada & Rodriguez-Velazquez, 2005).

Original: A specialization of this model is the Simplicial complex (Munkres, 1984) (Figure 2), in which additionally each hyperedge has the subset closure property, i.e., each subset of hyperedge is also a valid hyperedge.
Revised: A specialization of this model is the Simplicial complex (Munkres, 1984) (Figure 2), in which additionally each hyperedge has the subset closure property, i.e., each subset of hyperedge is also a valid hyperedge.

Original: Within machine learning, algorithms guided by the structure of such higher order networks (Zhou et al., 2006) have found applications in a variety of domains (Gao et al., 2013; Tian et al., 2009; Li & Li, 2013; Sharma et al., 2015).
Revised: Within machine learning, algorithms guided by the structure of such higher order networks (Zhou et al., 2006) have found applications in a variety of domains (Gao et al., 2013; Tian et al., 2009; Li & Li, 2013; Sharma et al., 2015; 2017).

Original: More recently, the success of feature (or representation) learning in natural language processing or NLP (Mikolov et al., 2013a) has stimulated interest in its application for network analysis by learning low dimensional representation for each network node (Grover & Leskovec, 2016; Tang et al., 2015b; Perozzi et al., 2014).
Revised: More recently, the success of feature (or representation) learning in natural language processing or NLP (Mikolov et al., 2013a) has stim- ulated interest in its application for network analysis by learning low dimensional representation for each network node (Grover & Leskovec, 2016; Tang et al., 2015b; Perozzi et al., 2014).

Original: These learned representations can then be provided as input to supervised learning algorithms to perform various prediction tasks related to link prediction or node classiﬁcation.
Revised: These learned representations can then be provided as input to supervised learning algorithms to perform various prediction tasks related to link prediction or node classiﬁcation.

Original: However, this new line of research is limited to simple graphs, as they extract features for the graph nodes or edges by designing an objective function that only takes into account the dyadic relationship modeled by a simple edge.
Revised: However, this new line of research is limited to simple graphs, as they extract features for the graph nodes or edges by designing an objective function that only takes into account the dyadic relationship modeled by a simple edge.

Original: In this paper we focus on developing methods to learn representations for not just nodes but also for set of nodes (hyperedges or simplex faces).
Revised: In this paper we focus on developing methods to learn representations for not just nodes but also for set of nodes (hyperedges or simplex faces).

Original: The proposed methods take into account the higherorder network structure while learning the embeddings.
Revised: The proposed methods take into account the higherorder network structure while learning the embeddings without loosing the set-level information.

Original: These learned features (embeddings) can then be employed by a supervised or semi-supervised algorithm to perform various predictive tasks pertaining to the set of nodes.
Revised: These learned features (embeddings) can then be employed by a supervised or semi-supervised algorithm to perform various predictive tasks pertaining to the set of nodes.

Original: For example performance prediction of a team (set of individuals) engaged in a collaborative task or sentiment analysis of a sentence (set of words).
Revised: For example performance prediction of a team (set of individuals) engaged in a collaborative task or sentiment analysis of a sentence (set of words).

Original: Traditional graph embedding methods such as spectral clustering have been extended to hypergraphs (Zhou et al., 2006).
Revised: Traditional graph embedding methods such as spectral clustering have been extended to hypergraphs (Zhou et al., 2006).

Original: However, these embeddings are learned for nodes, not for hyperedges, although the algorithm takes into account the hypergraph structure.
Revised: However, these embeddings are learned for nodes, not for hyperedges, although the algorithm takes into account the hypergraph structure.

Original: Also, (Agarwal et al., 2006) have criticized that such representations can be learned by constructing graphs, which are proxies for the hypergraph structure.
Revised: Also, (Agarwal et al., 2006) have criticized that such representations can be learned by constructing graphs, which are proxies for the hypergraph structure and loose the set-level information.

Original: In fact, we are unaware of any work within network analysis that considers embeddings at the hyperedge-level.
Revised: In fact, we are unaware of any work within network analysis that considers embeddings at the hyperedge-level.

Original: On the contrary, in the natural language processing (NLP) literature, methods for learning embeddings for higher-order entities like sentences and paragraphs have been proposed (Le & Mikolov, 2014).
Revised: On the contrary, in the natural language processing (NLP) literature, methods for learning embeddings for sequences like sentences and paragraphs have been proposed (Le & Mikolov, 2014).

Original: Also more recently, there has been interest in modeling ”setlike” structures within deep-learning community (Vinyals et al., 2016; Rezatoﬁghi et al., 2016).
Revised: However recently, there has been interest in modeling “set-like” structures within deep-learning community (Vinyals et al., 2016; Rezatoﬁghi et al., 2016).

Original: But they do not consider the hypergraph structure between the sets and therefore, not model hypergraphs in a principled manner.
Revised: But they do not consider the hypergraph structure between the sets and therefore, not model hypergraphs in a principled manner.

Original: Our paper, therefore, aims to ﬁll this gap by learning representations at the set-level itself, while taking into account the topological connectivity between the various sets in a hypergraph.
Revised: Our paper, therefore, aims to ﬁll this gap by learning representations at the set-level itself, while taking into account the topological connectivity between the various sets in a hypergraph as well as focusing on retaining the set-level information.

Original: Another distinguishing feature of our work is that we leverage the existing structure within the network data.
Revised: Another distinguishing feature of our work is that we leverage the existing structure within the network data.

Original: Recent attempts along these lines in network literature (Grover & Leskovec, 2016;
Revised: Recent attempts along these lines in network literature (Grover & Leskovec, 2016;

Original: Tang et al., 2015b; Perozzi et al., 2014) argue that language models have ready-made context in the form of sentences or paragraphs to train the model, which are not available in networks, and therefore, they propose different ways to generate this context.
Revised: Tang et al., 2015b; Perozzi et al., 2014) argue that language models have ready-made context in the form of sentences or paragraphs to train the model, which are not available in networks, and therefore, they propose different ways to generate this context.

Original: By contrast, we focus on networks where the context is already present, e.g. collaboration networks where collaborative teams are hyperedges, or language hyper-networks where sentences are hyperedges.
Revised: By contrast, we focus on networks where the context is already present, e.g. collaboration networks where collaborative teams are hyperedges, or language hyper-networks where sentences are hyperedges.

Original: We propose several methods, some of which use the  existing node-level method to generate set-level embeddings, and other novel models optimization criterion which directly learn hyperedge-level representations.
Revised: We propose two methods both of which capture the hypergraph structure in a principled manner. First, is an auto-encoder based deep-learning based method and second, is a tensor-based algebraic method. We also highlight the connection between hypergraphs (topology), tensors (algebra) and probabilistic models. A number of interesting baselines are also developed, some of which adapt existing node-level embedding models to hypergraph setting and also sequence based NLP

Revised: tech- niques are adapted for set structured hypergraphs.

Original: We have experimented with social group dataset (network of teams as a hypergraph) from an online game for team (hyperedge) performance prediction as well as language networks (sentence/phrase hypergraphs) for phrase-level sentiment analysis.
Revised: We have experimented with social group dataset (network of teams as a hypergraph) from an online game for team (hyperedge) performance prediction as well as language networks (sentence/phrase hypergraphs) for phrase-level sentiment analysis.

Original: Our results demonstrate the effectiveness of  our approach.
Revised: Our experiments show that accuracy wise our methods perform similar to those of baselines which are not designed for hypergraphs. Moreover, our tensor based method is quiet efﬁcient as compared to deep-learning based auto-encoder method. We therefore, argue that we have proposed more general methods which are suited for hypergraphs (and therefore also for graphs) while maintaining accuracy and efﬁciency.

Original: In summary the main contributions of our paper are:
Revised: In summary the main contributions of our paper are:

Original: • We develop a novel hypergraph based auto-encoder for building hyperedge representations and highlight how the extra structure within hypergraphs can naturally generate regularization inducing noise.
Revised: • We develop a novel hypergraph based auto-encoder for building hyperedge representations and highlight how the extra structure within hypergraphs can naturally generate regularization inducing noise.

Original: • We also propose the novel concept of a dual tensor corresponding to the hypergraph dual.
Revised: • We also propose the novel concept of a dual tensor corresponding to the hypergraph dual.

Original: In general, we consider using factors from decomposition of N -way tensors corresponding to hypergraph, for node & hyperedge embeddings, as a novel approach.
Revised: In general, we consider using factors from decomposition of N -way tensors corresponding to hypergraph, for node & hyperedge embeddings, as a novel approach.

Original: We are unaware of any such works or applications employing this approach.
Revised: We are unaware of any such works or applications employing this approach.

Original: • We propose learning network representation by leveraging the existing structure present in network data as context, in contrast to existing context generating techniques.
Revised: • We propose learning network representation by leveraging the existing structure present in network data as context, in contrast to existing context generating techniques.

Original: • We highlight the connection between higher order tensor methods and higher order probabilistic models in NLP.
Revised: • We highlight the connection between higher order tensor methods and higher order probabilistic models in NLP.

Original: • We argue that tensor/matrix based methods can be used to construct embeddings which have a natural graph/hypergraph theoretic interpretation.
Revised: • We argue that tensor/matrix based methods can be used to construct embeddings which have a natural graph/hypergraph theoretic interpretation.

Original: Therefore, the embedding can be better interpreted unlike the neural network based black-box approaches.
Revised: Therefore, the embedding can be better interpreted unlike the neural network based black-box approaches

Original: Following is the outline for the rest of the paper.
Revised: Following is the outline for the rest of the paper.

Original: In section 2 we describe the problem deﬁnition and statement followed by Section 3 where we describe in detail the various methods proposed in this paper.
Revised: In section 2 we describe the problem deﬁnition and statement followed by Section 3 where we describe in detail the various methods proposed in this paper.

Original: Section 4 describes the datasets, experimental tasks & settings for the models.
Revised: Section 4 describes the datasets, experimental tasks & settings for the models.

Original: Sectionprovides an overview of the related literature followed by conclusion.
Revised: Sectionprovides an overview of the related literature followed by conclusion and appendix.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this paper we consider the scenario where we have a collection of elements .
Revised: In this paper we consider the scenario where we have a collection of elements .

Original: These elements can represent individual actors in case of social groups or words in sentences or items in item-sets within a transaction database.
Revised: These elements can represent individual actors in case of social groups or words in sentences or items in itemsets within a transaction database.

Original: In other words a social group or a sentence or an item-set are sets which contain these elements .
Revised: In other words a social group or a sentence or an item-set are sets which contain these elements .

Original: Let V = { v 1 , v 2 , ..., v n } represents n elements and we have m sets deﬁned over these elements, denoted by G = { g 1 , g 2 , ..., g m } , where g i ⊆
Revised: Let V = { v 1 , v 2 , ..., v n } represents n elements and we have m sets deﬁned over these elements, denoted by G = { g 1 , g 2 , ..., g m } , where g i ⊆

Original: V represents the i th set.
Revised: V represents the i th set.

Original: The cardinality | g i | represents the number of elements in the set.
Revised: The cardinality | g i | represents the number of elements in the set.

Original: Also each set g i ∈ G has an occurrence number R ( g i ) , which denotes the number of times it has occurred.
Revised: Also each set g i ∈ G has an occurrence number R ( g i ) , which denotes the number of times it has occurred.

Original: Such overlapping or non-overlapping sets can be modeled as a hypergraph Berge (1984), where the nodes and hyperedges, represent the elements and sets, respectively.
Revised: Such overlapping or non-overlapping sets can be modeled as a hypergraph Berge (1984), where the nodes and hyperedges, represent the elements and sets, respectively.

Original: This hypergraph is represented as N g =
Revised: This hypergraph is represented as N g =

Original: ( V, G ) with G as the collection of hyperedges over the nodes V .
Revised: ( V, G ) with G as the collection of hyperedges over the nodes V .

Original: The incidence matrix H ∈ { 0 , 1 } | G |×| V | for N g represents the presence of nodes in different hyperedges with H ( g i , v )=
Revised: The incidence matrix H ∈ { 0 , 1 } | G |×| V | for N g represents the presence of nodes in different hyperedges with H ( g i , v )

Original:  if v ∈ g i else 0 .
Revised: = 1 if v ∈ g i else 0 .

Original: 2. P ROBLEM S TATEMENT
Revised: Problem Statement:

Original: Given this setting, our goal is to learn the mapping φ : G → R d from hyperedges to feature representations (i.e., embeddings) that can be used to build predictive models involving sets.
Revised: Given this setting, our goal is to learn the mapping φ : G → R d from hyperedges to feature representations (i.e., embeddings) that can be used to build predictive models involving sets.

Original: Here d is a parameter specifying the number of dimensions of the embedding vector.
Revised: Here d is a parameter specifying the number of dimensions of the embedding vector.

Original: Equivalently, φ can be thought of as a look-up matrix of size | G | × d , where | G | is the total number of sets or hyperedges.
Revised: Equivalently, φ can be thought of as a look-up matrix of size | G | × d , where | G | is the total number of sets or hyperedges.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In the following subsections, we describe our methods for hyperedge embedding.

Original: As we had mentioned before there are several methods to learn representation of nodes in a graph.
Revised: As we had mentioned before there are several methods to learn representation of nodes in a graph.

Original: Given that a hyperedge is a set of nodes, a natural question arises is that, can we combine the node level embeddings (learned using existing methods) within a given hyperedge to ﬁnd a suitable representation of that hyperedge?
Revised: Given that a hyperedge is a set of nodes, a natural question arises is that, can we combine the node level embeddings (learned using existing methods) within a given hyperedge to ﬁnd a suitable representation of that hyperedge?

Original: However, there are large number of possible ways one can combine the node embeddings.
Revised: However, there are large number of possible ways one can combine the node embeddings.

Original: Therefore,  we should expect thata method for hyperedge embedding should learn the embeddings for hyperedges directly in a more principled manner.In fact, in each of the methods below we shall ﬁnd both node as well as hyperedge level embeddings. In our experiments we will evaluate the various combinations of both types of embeddings. 3. H YPEREDGE 2V EC U SING D ENOISING A UTOENCODER
Revised: Therefore, in the following two subsections, we develop two methods that learn the embeddings for hyperedges directly in a more principled manner.

Original: An autoencoder (Bengio et al., 2009), takes an input vector x ∈ [0 , 1] n and maps it to a latent representation y ∈ [0 , 1] d .

Original: This is typically done using an afﬁne mapping followed by a nonlinearity (more so when the input, like in our case, is binary (Vincent et al., 2010)): f en ( x ) =

Original: s ( Wx + b ) , with parameters θ = { W , b } . Here, W is a n × d weight matrix and b is  the offset. This latent representation is reconstructed into a vector z = f de ( y ) =
Revised: Our tensor based method is focused on retaining the set-level information intact while harnessing the hypergraph network structure.

Original: s ( W (cid:48)

Original: y + b (cid:48) )

Original: , in the input space, z ∈ [0 , 1] n and parameters θ (cid:48) = { W (cid:48) , b (cid:48) } . The mappings f en and f de are referred as the encoder and decoder , respectively. The representation y is learned by minimizing the following reconstruction error:
Revised: Where the auto-encoder based method harnesses the non-linearity of representation offered by deep learning to generate crisp and powerful embeddings.

Original: [Equation]

Original: where L is a loss function, which in case of binary or bit probabilities is often chosen as the crossentropy loss :

Original: [Equation]

Original: In their paper, Vincent et al.

Original: (2010) have shown that minimizing reconstruction amounts to maximizing the lower bound on the mutual information between input ( x ) and the representation y .

Original: However, they have further argued (Vincent et al.

Original: that y retaining information about input x is insufﬁcient.

Original: They further, propose the idea that the learned representation should be able to recover ( denoising ) the original input even after being trained with corrupted input ( adding noise ).

Original: They generate the corrupted input ( ˜x ), using a stochastic mapping q

Original: ( ˜x | x ) .

Original: Choice of noise is usually either Gaussian for real inputs and Salt-and-pepper noise for discrete inputs.

Original: The denoising autoencoder then learns the representation for each input, x , same as Equation 1, but with the following modiﬁed loss function:

Original: L ( x ( i ) , g θ (cid:48)

Original: ( f θ ( ˜x ( i )))) . We leverage the denoising autoencoder for learning representation for j th hyperedge, by treating each hyperedge as an input, x = H ( j, :) . The size of this input vector for each hyperedge is n , which is the number of vertices in the hypergraph. In most natural hypergraphs, specially social networks, n can be quiet high ranging from thousands to millions or even billions (like Facebook for example).
Revised: Tensor based method has a natural hasse diagram based topological interpretation and can generate both node and hyperedge embeddings.

Original: Therefore, randomly using a discrete noise like salt-and-pepper , might not be reasonable, as there are large number of possible permutations (as size n is large) and not all of them are related.

Original: Random addition of 1 s or deletion of existing 1 s from x , amounts to randomly adding or deleting vertices to the hyperedge corresponding to x .

Original: This might end up in new hyperedges that are completely unrelated to the given hyperedge ( x ).

Original: For example, users (nodes) in a social network from completely different regions of the network suddenly form a group (hyperedge).

Original: Such anomalous scenarios rarely happen in practice andsocial groups evolve in a gradual fashion via simple processes (Sharma et al., 2017; 2015). Rather, we take advantage of the hypergraph structure to systematically guide us in generating this noise. A hypergraph canbe deﬁned by its corresponding hasse lattice (Sharma et al., 2017). For a given hyperedge ( x ), we consider the sub-lattice consisting of only those hyperedges that are a distance h from it in the complete lattice.
Revised: Whereas deep learning model is more black-box based approach and can only generate hyperedge embeddings.

Original: On this sub-lattice we sample hyperedges (nodes in sub-lattice) by performing random walk starting at the given hyperedge’s node (see Figure 3).

Original: Our stochastic mapping q

Original: ( ˜x | x ) is therefore, a random walk on the sub-lattice of hyperdge ( x ) containing hyperedges at distance h from it.

Original: Intuitively, the hyperedges coming within a reasonable distance will affect each others representations and will have more similar representations.

Original: We will refer to the hyperedge representations learned by the above autoencoder technique, as h2v-auto .

Original: 3. H YPEREDGE 2 VEC U SING N- WAY T ENSOR D ECOMPOSITION
Revised: 3. H YPEREDGE 2 VEC U SING N- WAY T ENSOR D ECOMPOSITION

Original: In this section, we develop tensor (higher-order matrices) based linear algebraic methods that learn node as well as hyperedge embedding by taking into account the joint probability over a hyperedge.
Revised: In this section, we develop tensor (higher-order matrices) based linear algebraic methods that learn node as well as hyperedge embedding by taking into account the joint probability over a hyperedge.

Original: The idea behind using tensors is that they retain the set-level information contained in a hypergraph, unlike the proxy graphs (corresponding to hypergraphs) based techniques (used as baselines in our experiments), which approximate hyperedge or set-level information with dyadic edge-level information.
Revised: The idea behind using tensors is that they retain the set-level information contained in a hypergraph, unlike the proxy graphs (corresponding to hypergraphs) based techniques (used as baselines in our experiments), which approximate hyperedge or set-level information with dyadic edge-level information.

Original: Therefore, we design an algorithm which is principally suited for hypergraph structured data.
Revised: Therefore, we design an algorithm which is principally suited for hypergraph structured data.

Original: For a given hypergraph we can extract a sub-hypergraph that only consists of the hyperedges with cardinality k .
Revised: For a given hypergraph we can extract a sub-hypergraph that only consists of the hyperedges with cardinality k .

Original: This sub-hypergraph is a k -uniform hypergraph or k -graph Cooper & Dutle (2012).
Revised: This sub-hypergraph is a k -uniform hypergraph or k -graph Cooper & Dutle (2012).

Original: Corresponding to this k -uniform hypergraph, we can deﬁne a k th order n -dimensional symmetric tensor Qi (2005)
Revised: Corresponding to this k -uniform hypergraph, we can deﬁne a k th order n -dimensional symmetric tensor Qi (2005)

Original: A khyp =
Revised: A khyp =

Original: ( a p 1 ,p 2 ,..,p k ) ∈ R [ k,n ] whose elements are initialized as follows:
Revised: ( a p 1 ,p 2 ,..,p k ) ∈ R [ k,n ] whose elements are initialized as follows:

Original: [Equation]
Revised: [Equation]

Original: where { v p 1 , v p 2 , ..., v p k } ∈ g i and | g i |
Revised: where { v p 1 , v p 2 , ..., v p k } ∈ g i and | g i |

Original: =
Revised: =

Original: k, ∀ i ∈ { 1 , ..., m } .
Revised: k, ∀ i ∈ { 1 , ..., m } .

Original: Note that symmetry here implies that value of element a p 1 ,p 2 ,..,p k is invariable under any permutation of its indices ( p 1 , p 2 , .., p k ) .
Revised: Note that symmetry here implies that value of element a p 1 ,p 2 ,..,p k is invariable under any permutation of its indices ( p 1 , p 2 , .., p k ) .

Original: Rest all the elements in the tensor are zeros.
Revised: Rest all the elements in the tensor are zeros.

Original: In a similar manner we can also deﬁne a dual tensor , corresponding to hypergraph dual where the roles of nodes and hyperedges are interchanged.
Revised: In a similar manner we can also deﬁne a dual tensor , corresponding to hypergraph dual where the roles of nodes and hyperedges are interchanged.

Original: We consider all the hyperedges in the hypergraph dual that are of cardinality k .
Revised: We consider all the hyperedges in the hypergraph dual that are of cardinality k .

Original: This basically corresponds to all the vertices in the original hypergraph which have a degree of k , i.e., they are part of exactly k hyperedges in the original hypergraph.
Revised: This basically corresponds to all the vertices in the original hypergraph which have a degree of k , i.e., they are part of exactly k hyperedges in the original hypergraph.

Original: Corresponding to this k -uniform hypergraph dual, we can deﬁne a k th order m -dimensional symmetric dual tensor A kdual =
Revised: Corresponding to this k -uniform hypergraph dual, we can deﬁne a k th order m -dimensional symmetric dual tensor A kdual =

Original: ( a q 1 , [Equation] .,q k ) ∈ R [ k,m ] whose elements are initialized as follows:
Revised: ( a q 1 ,q 2 ,..,q k ) ∈ R [ k,m ] whose elements are initialized as follows:

Original: [Equation]
Revised: [Equation]

Original: where { g q 1 , g q 2 , ..., g q k } (cid:51) v j and |{ r | H ( r, j ) = 1 }| = k, ∀ j ∈ { 1 , ..., n } .
Revised: where { g q 1 , g q 2 , ..., g q k } (cid:51) v j and |{ r | H ( r, j ) = 1 }| = k, ∀ j ∈ { 1 , ..., n } .

Original: Note that this tensor is also symmetric and rest all the elements in the tensor are zeros.
Revised: Note that this tensor is also symmetric and rest all the elements in the tensor are zeros.

Original: To realize our aim of learning node and hyperedge embeddings we perform joint CP Tensor Decomposition Kolda & Bader (2009) (of the tensors we just described) across different cardinality hyperedges simultaneously.
Revised: To realize our aim of learning node and hyperedge embeddings we perform joint CP Tensor Decomposition Kolda & Bader (2009) (of the tensors we just described) across different cardinality hyperedges simultaneously.

Original: Speciﬁcally, for the node embeddings we solve the following optimization problem:
Revised: Speciﬁcally, for the node embeddings we solve the following optimiza- tion problem:

Original: [Equation]
Revised: [Equation]

Original: where,
Revised: where,

Original: [Equation]
Revised: [Equation]

Original: with z r ∈ R n , λ r ∈ R + , Z ∈ R n × d and Z (: , r ) =
Revised: with z r ∈ R n , λ r ∈ R + , Z ∈ R n × d and Z (: , r ) =

Original: z r .
Revised: z r .

Original: Notice, that equation 6 is the standard symmetric CP decomposition but equation 5 is summation of reconstruction errors in different tensor decompositions for different cardinality hyperedges.
Revised: Notice, that equation 4 is the standard symmetric CP decomposition but equation 3 is summation of reconstruction errors in different tensor decompositions for different cardinality hyperedges.

Original: Each error term learns a common latent factor matrix Z using the empirical observed k -uniform sub-hypergraph stored in A khyp tensor.
Revised: Each error term learns a common latent factor matrix Z using the empirical observed k -uniform sub-hypergraph stored in A khyp tensor.

Original: We perform this joint decomposition by augmenting the standard CP Decomposition into the
Revised: We perform this joint decomposition by augmenting the standard CP Decomposition into the

Original: Hypergraph-CP-ALS algorithm 1.
Revised: Hypergraph-CP-ALS algorithm 1.

Original: Lines (4-8) is the standard ( non-symmetric ) CP Decomposition
Revised: Lines (4-8) is the standard ( non-symmetric ) CP Decomposition

Original: (see reference (Kolda & Bader, 2009) for details), which solves the following optimization:
Revised: (see reference (Kolda & Bader, 2009) for details), which solves the following optimization:

Original: [Equation]
Revised: [Equation]

Original: with k u ( j ) r ∈ R n , λ r ∈ R + , k U ( j ) ∈ R n × d , k U ( j ) (: , r ) =
Revised: with k u ( j ) r ∈ R n , λ r ∈ R + , k U ( j ) ∈ R n × d , k U ( j ) (: , r ) =

Original: k u ( j ) r and j ∈ { 1 , .., k } .
Revised: k u ( j ) r and j ∈ { 1 , .., k } .

Original: As we want to learn common representations ( Z ) for all the nodes, we add additional constraint that enforce that k U ( j ) are same for all j ∈ { 1 , .., k } and for all k ∈ [ c min , c max ] .
Revised: As we want to learn common representations ( Z ) for all the nodes, we add additional constraint that enforce that k U ( j ) are same for all j ∈ { 1 , .., k } and for all k ∈ [ c min , c max ] .

Original: Lines (9-15) is where we force all representations for all cardinalities to be average of the k U ( j ) achieved by the last decomposition iteration that occurred in lines (4-8).
Revised: Lines (9-15) is where we force all representations for all cardinalities to be average of the k U ( j ) achieved by the last decomposition iteration that occurred in lines (4-8).

Original: Averaging can be interpreted as equal contribution from the latent factors learned from different cardinality (uniform) sub-hypergraphs.
Revised: Averaging can be interpreted as equal contribution from the latent factors learned from different cardinality (uniform) sub-hypergraphs.

Original: We make repeated pass through the entire hypergraph (by learning via different k -uniform sub-hypergraph (line 3)) until the objective (equation 5) converges.
Revised: We make repeated pass through the entire hypergraph (by learning via different k -uniform sub-hypergraph (line 3)) until the objective (equation 3) converges.

Original: In our implementation we empirically observe that using the above mentioned unbiased averaging heuristic, our algorithm converges successfully.
Revised: In our implementation we empirically observe that using the above mentioned unbiased averaging heuristic, our algorithm converges successfully.

Original: The same algorithm 1 is used to get the hyperedge embeddings, by just passing A dual instead of A hyp .
Revised: The same algorithm 1 is used to get the hyperedge embeddings, by just passing A dual instead of A hyp .

Original: We shall jointly refer to the embeddings achieved for nodes and hyperedges via the above tensor decomposition techniques as t2v .
Revised: We shall jointly refer to the embeddings achieved for nodes and hyperedges via the above tensor decomposition techniques as t2v .

Original: We would like to highlight a few points regarding the tensor methods.

Original: The tensors that we have employed are super-symmetric and hence able to capture distribution over sets rather than sequence.

Original: But in general we can employ a k -way tensor which is not symmetric to even capture sequence.

Original: In this sense tensors are more general purpose.

Original: Another point one can observe that when we initialize the hypergraph tensor A khyp , we have initialized all the permutations of vertices corresponding to a given hyperedge (Eq. 3).

Original: Moreover, we initialize it by the repetition counts.

Original: 3. H YPEREDGE 2V EC U SING S EN 2V EC

Original: As mentioned before, the most commonly used model for studying complex interactions in networks is graphs, where each edge represents a dyadic interaction between nodes Strogatz (2001). Therefore, even if the original interaction is not dyadic like a set of researchers (3 or more) collaborating on a publication, we shall break down this joint interaction into dyadic interactions. Even though a large number of complex network data naturally occurs as hypergraphs Estrada & RodriguezVelazquez (2005), the popularity of ”think like dyadic edges” and not like sets seems to us as a hindrance, speciﬁcally when the end aim is to model the joint distribution at the level of sets. In this paper, as it is obvious, we go by ”think like set” paradigm, speciﬁcally, when the data is naturally
Revised: Algorithm

Original: 1: randomly initialize k U ( j ) , ∀ k ∈ [ c min , c max ] , ∀ j ∈ { 1 , ..., k } 2: repeat 3: for k = c min to c max do 4: for j = 1 to k do 5:
Revised: 1: randomly initialize k U ( j ) , ∀ k ∈ [ c min , c max ] , ∀ j ∈ { 1 , ..., k } 2: repeat 3: for k = c min to c max do 4: for j = 1 to k do 5:

Original: V ← k U ( 1 ) (cid:124) k U ( 1 ) ∗ ... ∗ k U ( j − 1 ) (cid:124) k U ( j − 1 ) ∗ k U ( j + 1 ) (cid:124) k
Revised: V ← k U ( 1 ) (cid:124) k U ( 1 ) ∗ ... ∗ k U ( j − 1 ) (cid:124) k U ( j − 1 ) ∗ k U ( j + 1 ) (cid:124) k

Original: U ( j + 1 ) ∗ ...
Revised: U ( j + 1 ) ∗ ...

Original: ∗ k U ( j ) (cid:124) k
Revised: ∗ k U ( j ) (cid:124) k

Original: U ( j ) 6: k U ( j ) ← ( A khyp )
Revised: U ( j ) 6: k U ( j ) ← ( A khyp )

Original: ( j ) ( k U ( k ) (cid:12) ...
Revised: ( j ) ( k U ( k ) (cid:12) ...

Original: (cid:12) k
Revised: (cid:12) k

Original: U ( j + 1 ) (cid:12) k U ( j − 1 ) (cid:12) ...
Revised: U ( j + 1 ) (cid:12) k U ( j − 1 ) (cid:12) ...

Original: (cid:12) k
Revised: (cid:12) k

Original: U ( 1 ) )
Revised: U ( 1 ) )

Original: V † 7: normalize columns of k U ( j ) (and store norms as λ ) 8: end for 9:
Revised: V † 7: normalize columns of k U ( j ) (and store norms as λ ) 8: end for 9:

Original: Z ← 1 k (cid:80) kj =1 k U (j ) 10: for p= c min to cmax do 11: for j = 1 to p do 12: p U( j) ← Z 13: end for 14: end for 15: end for 16: until ﬁt criteria achieved or maximum number of iterations exceeded 17: return Zoccurring as network of sets (i.e., a hypergraph).
Revised: Z ← 1 k (cid:80) kj =1 k U ( ) 10: for p c min to c do 11: for j = 1 to p do 12: p U j ← Z 13: end for 14: end for 15: end for 16: until ﬁt criteria achieved or maximum number of iterations exceeded 17: return Z

Original: Therefore, if we think of hyperedge as a set, a natural question arises is that are there any existing techniques that learn embeddings at set level? We, henceforth, explore the most popular discrete data on which representation learning has been applied is text data. Recently, Le & Mikolov (2014) proposed two representation learning methods for sentences as shown in Figure 4: a) a distributed memory ( DM ) model, and b) a distributed bag of words ( DBOW ) model.
Revised: We would like to highlight a few points regarding the tensor methods.

Original: In the DM model, every sentence in the dataset G is represented by a d dimensional vector in a shared lookup matrix S ∈ | G |× d . Similarly, every word in the vocabulary Ω is represented by a d dimensional vector in another shared lookup matrix L ∈ | Ω |× d . Given an input sentence v = ( v 1 , v 2 · · · v m ) , the corresponding sentence vector from S and the word vectors from L are averaged to predict the next word in a context. More formally, let φ denote the mapping from sentence and word ids to their respective vectors in S and L , the DM model minimizes the following objective:
Revised: The tensors that we have employed are super-symmetric and hence able to capture distribution over sets rather than sequence.

Original: [Equation] where z is the average of φ ( v ) , φ ( v t − k +1 ) , · · · , φ ( v t − 1 ) input vectors, and ω ( v t ) is the output vector representation of word v t ∈ Ω . The sentence vector φ ( v ) is shared across all (sliding window) contexts extracted from the same sentence, thus acts as a distributed memory. Instead of predicting the next word in the context, the DBOW model predicts the words in the context independently given the sentence id as input.
Revised: But in general we can employ a k -way tensor which is not symmetric to even capture sequence.

Original: More formally, DBOW minimizes the following objective: [Equation] Both the methods take sentences as input and return embeddings for words as well as the sentence. To apply the Sen2Vec models directly to hyperedges, as our ﬁrst method we generate a proxy sentence for each hyperedge by leveraging the contextual information in the hyperedge.
Revised: In this sense tensors are more general purpose.

Original:  A proxy sentence v i is formed for each hyperedge g i ∈ G as a sequence made by concatenating all the permutations of the nodes (as words) in the hyperedge and further repeat- ing this sequence as many time this hyperedge occurred. For example, for a three node hyperedge g i = { 1 , 4 , 7 } which has occurred two times we make the following sentence v i : { 1,4,7,1,7,4,7,1,4,7,4,1,4,1,7,4,7,1,1,4,7,1,7,4,7,1,4,7,4,1, 4,1,7,4,7,1 } of length 12 (6 permutations times 2 occurrence). In this scheme we have permutations and repetitions.
Revised: Another point one can observe that when we initialize the hypergraph tensor A khyp , we have initialized all the permutations of vertices corresponding to a given hyperedge (Eq. 1).

Original: We take all the permutations hoping that it should fool the sequence based model to get a sequence independent embedding. On theother hand, repetition of the permutations acts similarly to observing the same sequence of words several times in the text corpus. Of course one can think of other alternative schemes, but the point we are trying to make is that naturally observed hyperedges captures important contextual information that can be leveraged to achieve better representations. We refer to the node-hyperedge embedding pairs resulting from DM , DBOW models as h2v-dm and h2v-dbow , respectively.
Revised: Moreover, we initialize it by the repetition counts.

Original: 3. H YPEREDGE2 VEC U SING N ODE 2V EC 
Revised: 3. H YPEREDGE 2V EC U SING D ENOISING A UTOENCODER

Original: Grover & Leskovec (2016) propose a  representation method for nodes in a graph called Node2Vec, which uses the skip-gram model Mikolov et al. (2013b) with the intuition that neighborhood nodes within a graph should have similar representations. The objective of the skip-gram model for graphs can be deﬁned as:
Revised: An autoencoder (Bengio et al., 2009), takes an input vector x ∈ [0 , 1] n and maps it to a latent representation y ∈ [0 , 1] d . This is typically done using an afﬁne mapping followed by a nonlinearity (more so when the input, like in our case, is binary (Vincent et al., 2010)): f en ( x ) =

Original: [Equation]
Revised: s ( Wx + b ) , with parameters θ = { W , b } .

Original: where as before, φ and ω denote the input and  the output vector representations of the nodes. The neighboring nodes N ( v ) form the context for node v .
Revised: Here, W is a n × d weight matrix and b is the offset. This latent representation is reconstructed into a vector z = f de ( y ) =

Original: Node2Vec uses a biased random walk which adaptively combines breadth ﬁrst search (BFS) and depth ﬁrst search (DFS) to ﬁnd the neighborhood of a node. The walk attempts to capture two properties of a graph often used for prediction tasks in networks: i) homophily and ii) structural equivalence. According to homophily, nodes in the same group or community should have similar representations.
Revised: s ( W (cid:48)

Original: Structural equivalence suggests that nodes with similar structural roles (hub, bridge) should have similar representations.
Revised: y + b (cid:48) )

Original: In a real-world network, nodes exhibit mixed properties.
Revised: , in the input space, z ∈ [0 , 1] n and parameters θ (cid:48) =

Original: Our aim is to ﬁnd both node and hyperedge level embeddings by taking into account the hypergraph structure.
Revised: { W (cid:48) , b (cid:48) } . The mappings f en and f de are referred as the encoder and decoder , respectively.

Original: For the former, we leverage the adjacency matrix associated with a hypergraph Zhou et al. (2006), which is deﬁned as:
Revised: The representation y is learned by minimizing the following reconstruction error:

Original: [Equation]
Revised: [Equation]

Original: (15) where W e is a diagonal matrix containing the weights of each hyperedge and D v is a diagonal matrix containing the degree of each vertex.
Revised: where L is a loss function, which in case of binary or bit probabilities is often chosen as the crossentropy loss : [Equation]

Original: We take the weight of a hyperedge as its occurrence number, i.e. W e ( i, i ) =
Revised: In their paper, Vincent et al. (2010) have shown that minimizing reconstruction amounts to maximizing the lower bound on the mutual information between input ( x ) and the representation y . However, they have further argued (Vincent et al. that y retaining information about input x is insufﬁcient.

Original: R ( g i ) , ∀{ i ∈ { 1 , ..., n }} . The adjacency matrix ( A hyp ∈ | V |×| V | )associates a weight between a pair of nodes while taking into account the weights of all the hyperedges that encompass a pair of nodes.
Revised: They further, propose the idea that the learned representation should be able to recover ( denoising ) the original input even after being trained with corrupted input ( adding noise ). They generate the corrupted input ( ˜x ), using a stochastic mapping q ( ˜x | x ) .

Original: The weighted graph associated with A hyp in some sense serves as a proxy to the actual hypergraph structure. We can provide A hyp as input to Node2Vec and hypothesize that the random walk over this proxy hypergraph should allow the skip-gram model to learn more meaningful node level embeddings which can be combined to construct hyperedge level embeddings. We refer to these node level embeddings as N2V-hyp . We still have not met our second objective to learn hyperedge embeddings directly.
Revised: Choice of noise is usually either Gaussian for real inputs and Salt-and-pepper noise for discrete inputs.

Original: We again wish to leverage Node2Vec for our purpose. However, Node2Vec only works for graphs by performing random walk over nodes. Therefore, we ask ourselves the question, that can we treat hyperedges as nodes? There can be other ways of doing so, but here we suggest two techniques.
Revised: The denoising autoencoder then learns the representation for each input, x , same as Equation 6, but with the following modiﬁed loss function:

Original: In the ﬁrst technique, we simply consider the hypergraph dual Berge (1984), whose incidence matrix is H dual = H T . The adjacency matrix associated with the hypergraph dual is: [Equation]
Revised: L ( x ( i ) , g θ (cid:48)

Original: 16) where W v is a diagonal matrix containing the weights of each node and D e is a diagonal matrix containing the degree of each hyperedge. We assume no weights on the nodes and take W v = I .The matrix A dual ∈ | G |×| G | represents another hypergraph, but the roles of nodes and hyperedges have now switched. For example, in case of words and sentences, the hyperedges were sentences, but in the hypergraph dual the words become hyperedges and the nodes within a word’s hyperedge represent all the sentences in which the word has appeared.
Revised: ( f θ ( ˜x ( i )))) .

Original: We again give A dual (i.e., a graph proxy for the dual hypergraph) as input to Node2Vec, but this time we get output as the embeddings for the hyperedges associated with the nodes in dual. We refer to these hyperedgeembeddings as h2v-dual .In the second technique, we consider the following adjacency matrix A inv ∈ | G |×| G | : [Equation]
Revised: We leverage the denoising autoencoder for learning representation for j th hyperedge, by treating each hyperedge as an input, x = H ( j, :) .

Original: 17) associated with what we refer to as the inverted hypergraph . This inverted hypergraph is a graph (unlike the dual which is a hypergraph) and there is an edge between two nodes if the hyperedges corresponding to the nodes in the original hypergraph have nodes in common. Weight of this edge is the number of common nodes. We again give A inv as input to Node2Vec to get embeddings for the hyperedges associated with the nodes in inverted hypergraph.
Revised: The size of this input vector for each hyperedge is n , which is the number of vertices in the hypergraph.

Original: We refer to these hyperedge embeddings as h2v-inv .
Revised: In most natural hypergraphs, specially social networks, n can be quiet high ranging from thousands to millions or even billions (like Facebook for example).

Original: These set of methods extract embeddings as the eigenvectors associated with Laplacian matrices corresponding to the various adjacency matrices discussed in the previous section. We consider the following graph Laplacians: [Equation]
Revised: Therefore, randomly using a discrete noise like salt-and-pepper , might not be reasonable, as there are large number of possible permutations (as size n is large) and not all of them are related.

Original: (18) where A graph = H T W e H is the weighted graph associated with the graph corresponding to adjacency matrix A , [Equation] [Equation]
Revised: Random addition of 1 s or deletion of existing 1 s from x , amounts to randomly adding or deleting vertices to the hyperedge corresponding to x .

Original: [Equation] We get the d eigenvectors associated with the smallest d eigenvalues of the above graph Laplacians as the embeddings. We get vertex embeddings using Eq. 19 and hyperedge embedding using Eq.
Revised: This might end up in new hyperedges that are completely unrelated to the given hyperedge ( x ).

Original: 21, and we refer to them together as e2v-hyp . Similarly, we get another pair ofvertex embeddings (using Eq. 18) and hyperedge embedding (using Eq. 21). We refer to the later pair as e2v .
Revised: For example, users (nodes) in a social network from completely different regions of the network suddenly form a group (hyperedge).

Original: Many methods that we have developed so far make use of only pair-wise information. For example, the methods developed using Node2Vec in Section 3.4 are based on the skip-gram model, which learns embedding of nodes while maximizing the conditional probability of a node given another node in a context (Eq. 14).
Revised: Such anomalous scenarios rarely happen in practice and social groups evolve in a gradual fashion via simple processes (Sharma et al., 2017; 2015).

Original: Similarly, the spectral methods (Section 3.5) are inherently two dimensional as they are based on matrix. Same is the case with skip-gram based
Revised: Rather, we take advantage of the hypergraph structure to systematically guide us in generating this noise.

Original: Sen2Vec ( h2v-dbow ). However, Sen2Vec based on the DM architecture ( h2v-dm ) maximizes the conditional probability of a word given the previous set of words ( context ) without breaking this context (by concatenating or averaging the embedding of the previous words). However, the tensors principally capture the joint distribution over various cardinality hyperedges unlike conditional distribution like Sen2vec ( h2v-dm ) (which are more appropriate for sequences). Limitations There are a few limitations we can see with various approaches.
Revised: A hypergraph can be deﬁned by its corresponding hasse lattice (Sharma et al., 2017).

Original: In tensor approach when we extract a sub-hypergraph of a given k size hyperedges, it is possible that a signiﬁcant part of nodes areleft out (as they might not be a part of any k size hyperedge). This can possibly cause a cold start problem in the corresponding tensor decomposition.
Revised: For a given hyperedge ( x ), we consider the sub-lattice consisting of only those hyperedges that are a distance h from it in the complete lattice. On this sub-lattice we sample hyperedges (nodes in sub-lattice) by performing random walk starting at the given hyperedge’s node (see Figure 3).

Original: However, this does not mean that we will not get embedding for those nodes. It simply means that we might not achieve embeddings for these nodes from the decomposition of k -way tensors. But we can get embedding for these nodes from other tensors for a different k .
Revised: Our stochastic mapping q

Original: However, we can mitigate the cold start problem in a principled manner using a graph regularization term ( k U ( j ) T L k U ( j ) )in our tensor decomposition objective function 5. Here, L is  the graph laplacian of our choice.
Revised: ( ˜x | x ) is therefore, a random walk on the sub-lattice of hyperdge ( x ) containing hyperedges at distance h from it.

Original: For example we can use L hyp or L graph when we perform tensor decomposition for node embeddings and use L inv or L dual for hyperedge embeddings. Lastly, the hypergraph autoencoder ( h2v-auto ) can only produce hyperedge embeddings and not the node embeddings. In this sense it is feeble to other methods.
Revised: Intuitively, the hyperedges coming within a reasonable distance will affect each others representations and will have more similar representations. We will refer to the hyperedge representations learned by the above autoencoder technique, as h2v-auto .

Original:  E XPERIMENTS 4. D ATASET D ESCRIPTION
Revised:  E XPERIMENTS 4. D ATASET D ESCRIPTION

Original: As the ﬁrst dataset, we use group interaction log-data of the Sony’s
Revised: As the ﬁrst dataset, we use group interaction log-data of the Sony’s

Original: Online multi-player game EverQuest II ( EQ II ) (www.everquest2.com) for a time period of nine months.
Revised: Online multi-player game EverQuest II ( EQ II ) (www.everquest2.com) for a time period of nine months.

Original: In this game, several players work together as a team to perform various tasks.
Revised: In this game, several players work together as a team to perform various tasks.

Original: Each team earn points after completion of each task, and as the teams progress by earning points, they are escalated to different levels of game play.
Revised: Each team earn points after completion of each task, and as the teams progress by earning points, they are escalated to different levels of game play.

Original: The interestingness of the game increases with each level.
Revised: The interestingness of the game increases with each level.

Original: The points earned by the teams are treated as a measure of group performance.
Revised: The points earned by the teams are treated as a measure of group performance.

Original: Each set of players who played together is treated as a hyperedge.
Revised: Each set of players who played together is treated as a hyperedge.

Original: When the same set of players play together again, we treat it as hyperedge re-occurrence ( R ( g i ) ).
Revised: When the same set of players play together again, we treat it as hyperedge re-occurrence ( R ( g i ) ).

Original: Players can participate in several teams over time, therefore, resulting in a hypergraph with overlapping hyperedges.
Revised: Players can participate in several teams over time, therefore, resulting in a hypergraph with overlapping hyperedges.

Original: We consider hyperedges of cardinality ∈ [2 , 6] .
Revised: We consider hyperedges of cardinality ∈ [2 , 6] .

Original: The resulting dataset contains a total of 5964 hyperedges (teams) among 6536 nodes (players).
Revised: The resulting dataset contains a total of 5964 hyperedges (teams) among 6536 nodes (players).

Original: Second dataset, is the fully labeled and publicly available sentiment analysis corpus of Stanford
Revised: Second dataset, is the fully labeled and publicly available sentiment analysis corpus of Stanford

Original: Sentiment Treebank ( LangNet ) (Socher et al., 2013).
Revised: Sentiment Treebank ( LangNet ) (Socher et al., 2013).

Original: This dataset is based on the reviews from a movie review website (www.rottentomatoes.com) and contains 215,154 unique phrases.
Revised: This dataset is based on the reviews from a movie review website (www.rottentomatoes.com) and contains 215,154 unique phrases.

Original: Each of the phrases are labeled with a sentiment score (a real number ∈ [0 , 1] , larger value indicates positive sentiment) by human annotators.
Revised: Each of the phrases are labeled with a sentiment score (a real number ∈ [0 , 1] , larger value indicates positive sentiment) by human annotators.

Original: Each phrase is a set or hyperedge of words.
Revised: Each phrase is a set or hyperedge of words.

Original: Given that words are shared across various phrases, these common words connect the phrase hyperedges, resulting in a phrase hypergraph with overlapping phrase hyperedges.
Revised: Given that words are shared across various phrases, these common words connect the phrase hyperedges, resulting in a phrase hypergraph with overlapping phrase hyperedges.

Original: Again, we only consider hyperedges of cardinality ∈ [2 , 6] .
Revised: Again, we only consider hyperedges of cardinality ∈ [2 , 6] .

Original: After applying this cardinality ﬁlter we are left with 141,410 hyperedges (phrases) and 21,122 nodes (words).
Revised: After applying this cardinality ﬁlter we are left with 141,410 hyperedges (phrases) and 21,122 nodes (words).

Original: 4. E VALUATION M ETHODOLOGY AND E XPERIMENTAL S ETUP 4.2. M ETHODS C OMPARED
Revised: 4. E VALUATION M ETHODOLOGY AND E XPERIMENTAL S ETUP 4.2. M ETHODS C OMPARED

Revised: As mentioned before we refer to our methods as t2v and h2v-auto , for tensor based and auto-encoder method respectively.

Original: We seven different methods: h2v-DM ,  h2v-DBOW ,  h2v-inv , h2v-dual , e2v, e2v-hyp , t2v and h2vauto .
Revised: We have six baselines: (1) h2v-DM , (2) h2v-DBOW , (3) h2v-inv (4) h2v-dual (5) e2v and (6) e2v-hyp .

Revised: Methods (1-2) are adapted from sequence based language models for set structured data.

Revised: Methods (3-6) are various kind of dyadic graph based embedding methods.

Revised: The details of these baseline can be found in Appendix 6.

Original: Except for h2v-auto , each of these methods result in node as well as hyperedge embeddings of dimension d = 128 .
Revised: Except for h2v-auto , each of these methods result in node as well as hyperedge embeddings of dimension d = 128 .

Original: We further combine the node and hyperedge embedding using ﬁve different strategies: i) node embedding summation (dimension d = 128 ), ii) node embedding summation and concatenation with hyperedge embedding (dimension 2 × d = 256 ), iii) node embedding averaging (dimension d = 128 ), iv) node embedding averaging and concatenation with hyperedge embedding (dimension 2 × d = 256 ), and v) only hyperedge embedding (dimension d = 128 ).
Revised: We further combine the node and hyperedge embedding using ﬁve different strategies: i) node embedding summation (dimension d = 128 ), ii) node embedding summation and concatenation with hyperedge embedding (dimension 2 × d = 256 ), iii) node embedding averaging (dimension d = 128 ), iv) node embedding averaging and concatenation with hyperedge embedding (dimension 2 × d = 256 ), and v) only hyperedge embedding (dimension d = 128 ).

Original: h2v-auto only produces hyperedge embeddings of dimension d = 128 .
Revised: h2v-auto only produces hyperedge embeddings of dimension d = 128 .

Original: But it builds the embeddings using three different scenarios as mentioned in next section.
Revised: But it builds the embeddings using three different scenarios as mentioned in next section.

Original: Therefore, in total we have 38(35+3) different scenarios each resulting in a different hyperedge embedding.
Revised: Therefore, in total we have 38(35+3) different scenarios each resulting in a different hyperedge embedding.

Original: 4.2. E VALUATION T ASKS AND S ETUP
Revised: 4.2. E VALUATION T ASKS AND S ETUP

Original: We perform two regression based tasks for the two datasets.
Revised: We perform two regression based tasks for the two datasets.

Original: In EQII dataset each team (hyperedge) has a team performance score associated with it.
Revised: In EQII dataset each team (hyperedge) has a team performance score associated with it.

Original: This team performance score is a real number, equal to the number of points earned by the team while performing one or more tasks within a gaming session.
Revised: This team performance score is a real number, equal to the number of points earned by the team while performing one or more tasks within a gaming session.

Original: We treat the embedding learned for a given team (hyperedge) as its feature vector which is associated with a real number (team performance).
Revised: We treat the embedding learned for a given team (hyperedge) as its feature vector which is associated with a real number (team performance).

Original: We therefore, perform on regression over all the hyperedges (teams) with team performance as the dependent variable.
Revised: We therefore, perform on regression over all the hyperedges (teams) with team performance as the dependent variable.

Original: Similarly, in LangNet dataset each phrase (hyperedge) has a sentiment score associated with it, which again is a real number.
Revised: Similarly, in LangNet dataset each phrase (hyperedge) has a sentiment score associated with it, which again is a real number.

Original: Similar to the team dataset above, we treat the embedding learned for a given phrase (hyperedge) as its feature vector which is associated with a real number (sentiment score).
Revised: Similar to the team dataset above, we treat the embedding learned for a given phrase (hyperedge) as its feature vector which is associated with a real number (sentiment score).

Original: We therefore, treat this as a regression task with sentiment score as the dependent variable and perform regression using the feature matrix containing embeddings of all the phrases.
Revised: We therefore, treat this as a regression task with sentiment score as the dependent variable and perform regression using the feature matrix containing embeddings of all the phrases.

Original: For both the tasks we just described, we perform several evaluation runs.
Revised: For both the tasks we just described, we perform several evaluation runs.

Original: In each run we randomly choose 30 % of hyperedges (teams or phrases) as the test set and learn ridge regression parameters using the remaining 70 % training hyperedges for each of the 38 different embedding scenarios.
Revised: In each run we randomly choose 30 % of hyperedges (teams or phrases) as the test set and learn ridge regression parameters using the remaining 70 % training hyperedges for each of the 38 different embedding scenarios.

Original: Root mean squared error (RMSE) was chosen as the evaluation metric (the lower, the better).
Revised: Root mean squared error (RMSE) was chosen as the evaluation metric (the lower, the better).

Original: RMSE was calculated for each of the 35 scenarios and for each run.
Revised: RMSE was calculated for each of the 35 scenarios and for each run.

Original: Final RMSE score was taken as the average RMSE score across ﬁve runs.
Revised: Final RMSE score was taken as the average RMSE score across ﬁve runs.

Original: Ridge regression’s hyper-parameter was chosen by 5-fold cross-validation.
Revised: Ridge regression’s hyper-parameter was chosen by 5-fold cross-validation.

Original: For the autoencoder method ( h2v-auto ) we consider three scenarios: (1) single hidden layer ( L 1 ) of d = 128 ; (2) two hidden layers ( L 1 & L 2 ) with size of L 1 : d = 96 and of L 2 : d = 32 .
Revised: For the autoencoder method ( h2v-auto ) we consider three scenarios: (1) single hidden layer ( L 1 ) of d = 128 ; (2) two hidden layers ( L 1 & L 2 ) with size of L 1 : d = 96 and of L 2 : d = 32 .

Original: We concatenate these embedding to get a single d = 128 size embedding; and (3) two hidden layers ( L 1 & L 2 ) with size of L 1 : d = 512 and of L 2 : d = 128 .
Revised: We concatenate these embedding to get a single d = 128 size embedding; and (3) two hidden layers ( L 1 & L 2 ) with size of L 1 : d = 512 and of L 2 : d = 128 .

Original: We use the output of L 2 , which is of dimension d = 128 , as the embedding.
Revised: We use the output of L 2 , which is of dimension d = 128 , as the embedding.

Original: For sampling, we use the distance parameter h
Revised: For sampling, we use the distance parameter h

Original: =for generating the sub-lattice for both datasets.
Revised: =for generating the sub-lattice for both datasets.

Original: Also, p = 10 & p = 5 number of hyperedges are sampled (corresponding to each hyperedge) from EQII and LangNet , respectively.
Revised: Also, p = 10 & p = 5 number of hyperedges are sampled (corresponding to each hyperedge) from EQII and LangNet , respectively.

Original: 4. R ESULTS AND D ISCUSSION
Revised: 4. R ESULTS AND D ISCUSSION

Original: Tables 1 &2 show the RMSE scores for the tasks of team performance prediction and sentiment score prediction, respectively.
Revised: Tables 1 &2 show the RMSE scores for the tasks of team performance prediction and sentiment score prediction, respectively.

Original: These tables contain scores for all the 38 different scenarios: columns represent 7 different models while rows represent combination strategies.
Revised: These tables contain scores for all the 38 different scenarios: columns represent 7 different models while rows represent combination strategies.

Original: The scores for Autoencoder are shown in the separate Table 3 as autoencoder only generates hyperedge embeddings.
Revised: The scores for Autoencoder are shown in the separate Table 3 as autoencoder only generates hyperedge embeddings.

Original: As we can observe that for our dataset and for both the regression tasks, almost all the embeddings are performing very similarly in terms of accuracy.
Revised: As we can observe that for our dataset and for both the regression tasks, almost all the embeddings are performing very similarly in terms of accuracy.

Original: However, we can observer in Table 4 that simple spectral methods based on the hypergraph and its dual ( e2v-hyp ) have signiﬁcantly less running times for both the datasets.
Revised: However, we can observer in Table 4 that tensor based methods have signiﬁcant less time than the expensive auto-encoder technique (Table 3).

Original: This makes these simple methods basedon well understood eigen value decomposition techniques more attractive than the sophisticated and complex Sen2Vec or Node2vec based probabilistic models.
Revised: This makes tensor based well understood and interpretable decomposition techniques more attractive than the black-box type deep learning based auto-encoder method.

Original: Over all, we observe that e2v , e2v-hgraph and t2v , all of which are matrix or tensor based algebraic techniques, perform comparable to state of the art embedding techniques from NLP.
Revised: Over all, we observe that e2v , e2v-hgraph and t2v , all of which are matrix or tensor based algebraic techniques, perform comparable to state of the art embedding techniques from NLP.

Original: Therefore, it provides motivation to further explore algebraic techniques, which are easily interpretable in graph theoretic terms, as a reasonable alternative for learning representations. 
Revised: Therefore, it provides motivation to further explore algebraic techniques, which are easily interpretable in graph theoretic terms, as a reasonable alternative for learning representations. [Table]

Original:  Another thing we notice, is that in case of LangNet dataset, that Node2Vec based h2v-inv and h2vdual methods are simply unable to run, as the A inv and A dual are quiet large (see N/A in Table 2).
Revised: [Table] [Table] [Table] Another thing we notice, is that in case of LangNet dataset, that Node2Vec based h2v-inv and h2vdual methods are simply unable to run, as the A inv and A dual are quiet large (see N/A in Table 2).

Original: It seems that in LangNet and possibly in text data in general the hypergraph dual, which contain phrase to phrase edges turns out be containing signiﬁcantly more edges than the number of node to node edges in A hyp .
Revised: It seems that in LangNet and possibly in text data in general the hypergraph dual, which contain phrase to phrase edges turns out be containing signiﬁcantly more edges than the number of node to node edges in A hyp .

Original: Performing context generation over A inv and A dual graphs turns out be very costly and we were unable to get hyperedge embeddings for h2v-inv and h2v-dual methods.
Revised: Performing context generation over A inv and A dual graphs turns out be very costly and we were unable to get hyperedge embeddings for h2v-inv and h2v-dual methods.

Original: Note, we however do get vertex embedding (as mentioned with a * mark below Table 4).
Revised: Note, we however do get vertex embedding (as mentioned with a * mark below Table 4).

Original: But as you can see that even the time time taken for vertex embedding is similar or more than the total time taken by
Revised: But as you can see that even the time time taken for vertex embedding is similar or more than the total time taken by

Original: Spectral methods for learning both hyperedge as well as vertex embedding, together.
Revised: Spectral methods for learning both hyperedge as well as vertex embedding, together.

Original: This indicates

Original: [Table]

Original: [Table] [Table] [Table] the robustness of the Spectral methods, specially the e2v-hyp for different kinds of hypergraph data and edge densities (sparsity).
Revised: This indicates the robustness of the Spectral methods, specially the e2v-hyp for different kinds of hypergraph data and edge densities (sparsity).

Original: Furthermore, we also observe that even simple element-wise summation or averaging of node embeddings for the nodes (in a given hyperdge) also perform comparably when compared to hyperedge embedding alone.
Revised: Furthermore, we also observe that even simple element-wise summation or averaging of node embeddings for the nodes (in a given hyperdge) also perform comparably when compared to hyperedge embedding alone.

Original: From this we can infer that depending upon the dataset, if we have less hyperedges and more nodes, than we would rather prefer to simply learn the hyperedge embedding directly rather than learning node embeddings and then performing aggregating operation over them.
Revised: From this we can infer that depending upon the dataset, if we have less hyperedges and more nodes, than we would rather prefer to simply learn the hyperedge embedding directly rather than learning node embeddings and then performing aggregating operation over them.

Original: Aggregation might turn out be costly specially if average hyperedge size is large and the choice of aggregation function is an issue.
Revised: Aggregation might turn out be costly specially if average hyperedge size is large and the choice of aggregation function is an issue.

Original: Therefore, learning hyperedge embeddings directly seems to be escape the problem of choosing the aggregation function all together.
Revised: Therefore, learning hyperedge embeddings directly seems to be escape the problem of choosing the aggregation function all together.

Revised: Information Loss Many methods that we have developed so far make use of only pair-wise information.

Revised: For example, the methods developed using Node2Vec in Section 7.2 are based on the skip-gram model, which learns embedding of nodes while maximizing the conditional probability of a node given another node in a context (Eq. 14).

Revised: Similarly, the spectral methods (Section 7.3) are inherently two dimensional as they are based on matrix.

Revised: Same is the case with skip-gram based

Revised: Sen2Vec ( h2v-dbow ).

Original: R ELATED W ORKS
Revised: However, Sen2Vec based on the DM architecture ( h2v-dm ) maximizes the conditional probability of a word given the previous set of words ( context ) without breaking this context (by concatenating or averaging the embedding of the previous words). However, the tensors principally capture the joint distribution over various cardinality hyperedges unlike conditional distribution like Sen2vec ( h2v-dm ) (which are more appropriate for sequences). Limitations There are a few limitations we can see with various approaches. In tensor approach when we extract a sub-hypergraph of a given k size hyperedges, it is possible that a signiﬁcant part of nodes are left out (as they might not be a part of any k size hyperedge).

Revised: This can possibly cause a cold start problem in the corresponding tensor decomposition.

Revised: However, this does not mean that we will not get embedding for those nodes.

Revised: It simply means that we might not achieve embeddings for these nodes from the decomposition of k -way tensors.

Revised: But we can get embedding for these nodes from other tensors for a different k .

Revised: However, we can mitigate the cold start problem in a principled manner using a graph regularization term ( k U ( j ) T L k U ( j ) ) in our tensor decomposition objective function 3.

Revised: Here, L is the graph laplacian of our choice.

Revised: For example we can use L hyp or L graph when we perform tensor decomposition for node embeddings and use L inv or L dual for hyperedge embeddings.

Revised: Lastly, the hypergraph autoencoder ( h2v-auto ) can only produce hyperedge embeddings and not the node embeddings.

Revised: In this sense it is feeble to other methods.

Revised:  R ELATED W ORKS

Original: In Representation learning (Bengio et al., 2013) features or (geometric) representations of the data (topology) are used to build models.
Revised: In Representation learning (Bengio et al., 2013) features or (geometric) representations of the data (topology) are used to build models.

Original: Machine learning algorithms make use of these features for prediction.
Revised: Machine learning algorithms make use of these features for prediction.

Original: Traditionally these features are readily available within the data-set or are engineered manually.
Revised: Traditionally these features are readily available within the data-set or are engineered manually.

Original: However, this is a tedious labor-intensive process.
Revised: However, this is a tedious labor-intensive process.

Original: Alternatively, the representations can be learned by the algorithm automatically in a task-dependent supervised, or a task-independent unsupervised manner.
Revised: Alternatively, the representations can be learned by the algorithm automatically in a task-dependent supervised, or a task-independent unsupervised manner.

Original: In this work we focus on unsupervised algorithms for learning network embeddings.
Revised: In this work we focus on unsupervised algorithms for learning network embeddings.

Original: Unsupervised feature learning using latent models has been done across the ﬁeld of machine learning, e.g. LSA, LDA [3] for language models and Matrix Factorization (Ahmed et al.
Revised: Unsupervised feature learning using latent models has been done across the ﬁeld of machine learning, e.g. LSA, LDA [3] for language models and Matrix Factorization (Ahmed et al.

Original: Community detection (Tang & Liu, 2011) based techniques for networks (Roweis & Saul, 2000;
Revised: Community detection (Tang & Liu, 2011) based techniques for networks (Roweis & Saul, 2000;

Original: Belkin & Niyogi, 2001;
Revised: Belkin & Niyogi, 2001;

Original: Tenenbaum et al., 2000; Cox & Cox, 2000).
Revised: Tenenbaum et al., 2000; Cox & Cox, 2000).

Original: In each case there is a vector of features learned for a word or node, each of whose entries reﬂects association with some latent dimension or community in the case of networks.
Revised: In each case there is a vector of features learned for a word or node, each of whose entries reﬂects association with some latent dimension or community in the case of networks.

Original: This type of latent vector is learned for each node from the graphs and reﬂect, in a sense, the node’s association with the various communities existing within the network.
Revised: This type of latent vector is learned for each node from the graphs and reﬂect, in a sense, the node’s association with the various communities existing within the network.

Original: For words the input is a word co-occurence matrix of words residing in the same document.
Revised: For words the input is a word co-occurence matrix of words residing in the same document.

Original: Word feature vector learned reﬂects the latent topics or classes of words.
Revised: Word feature vector learned reﬂects the latent topics or classes of words.

Original: However, these embeddings reﬂect representation for node or words globally rather than locally .
Revised: However, these embeddings reﬂect representation for node or words globally rather than locally .

Original: However, they are based on the spectral properties of the graph/ afﬁnity matrix and mostly involve some form of eigen-decomposition, which can be costly.
Revised: However, they are based on the spectral properties of the graph/ afﬁnity matrix and mostly involve some form of eigen-decomposition, which can be costly.

Original: A number of embedding methods have been proposed in the neural network literature.
Revised: A number of embedding methods have been proposed in the neural network literature.

Original: Unsupervised models learn embeddings by predicting neighbors in a context.
Revised: Unsupervised models learn embeddings by predicting neighbors in a context.

Original: The context can be deﬁned by nodes in a graph (Grover & Leskovec, 2016) or by words in a sentence (Mikolov et al., 2013b; Le &
Revised: The context can be deﬁned by nodes in a graph (Grover & Leskovec, 2016) or by words in a sentence (Mikolov et al., 2013b; Le &

Original: Mikolov, 2014).
Revised: Mikolov, 2014).

Original: Supervised algorithms learn embeddings which are optimal for the speciﬁc task at hand.
Revised: Supervised algorithms learn embeddings which are optimal for the speciﬁc task at hand.

Original: This results in high accuracy but incurs signiﬁcant computational cost for training.
Revised: This results in high accuracy but incurs signiﬁcant computational cost for training.

Original: Recently, several supervised learning algorithms have been proposed for network analysis (Tian et al., 2014;
Revised: Recently, several supervised learning algorithms have been proposed for network analysis (Tian et al., 2014;

Original: Xiaoyi et al., 2013) and for text networks in a semi-supervised setting (Tang et al., 2015a).
Revised: Xiaoyi et al., 2013) and for text networks in a semi-supervised setting (Tang et al., 2015a).

Original: Representation learning for sets using neural networks has been proposed recently (Vinyals et al., 2016), where a memory network is used to compose features sequentially but in an order invariant manner.
Revised: Representation learning for sets using neural networks has been proposed recently (Vinyals et al., 2016), where a memory network is used to compose features sequentially but in an order invariant manner.

Original: In their very recent paper, Rezatoﬁghi et al.
Revised: In their very recent paper, Rezatoﬁghi et al.

Original: (2016) have tried to answer this set ordering issue by the use of random set theory .
Revised: (2016) have tried to answer this set ordering issue by the use of random set theory .

Original: However, both of these works, do not consider the hypergraph structure of overlapping sets which is the main focus of this paper.
Revised: However, both of these works, do not consider the hypergraph structure of overlapping sets which is the main focus of this paper.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we have proposed several methods to generate higher-order representations for both hyperedges (representing sets of nodes) and hypergraph nodes (that also take into account the hypergraph structure).
Revised: In this paper we have proposed several methods to generate higher-order representations for both hyperedges (representing sets of nodes) and hypergraph nodes (that also take into account the hypergraph structure).

Original: We build feature learning models that leverage the existing structure present in network data as context.
Revised: We build feature learning models that leverage the existing structure present in network data as context.

Original: We highlight the connection between higher order tensor methods and higher order probabilistic models in NLP.
Revised: We highlight the connection between higher order tensor methods and higher order probabilistic models in NLP.

Original: We argue that tensor/matrix based methods can be used to construct embeddings which have a natural graph/hypergraph theoretic interpretation.
Revised: We argue that tensor/matrix based methods can be used to construct embeddings which have a natural graph/hypergraph theoretic interpretation.

Original: Therefore, the embedding can be better interpreted, unlike neural network based black-box approaches.
Revised: Therefore, the embedding can be better interpreted, unlike neural network based black-box approaches.

Original: While introducing a new idea of a dual tensors corresponding to the hypergraph dual, we also highlight the novel approach of using factors from decomposition of N -way tensors corresponding to hypergraph, as generic node & hyperedge representations.  
Revised: While introducing a new idea of a dual tensors corresponding to the hypergraph dual, we also highlight the novel approach of using factors from decomposition of N -way tensors corresponding to hypergraph, as generic node & hyperedge representations.  A PPENDIX 7. H YPEREDGE 2V EC U SING S EN 2V EC

Revised: As mentioned before, the most commonly used model for studying complex interactions in networks is graphs, where each edge represents a dyadic interaction between nodes Strogatz (2001).

Revised: Therefore, even if the original interaction is not dyadic like a set of researchers (3 or more) collaborating on a publication, we shall break down this joint interaction into dyadic interactions.

Revised: Even though a large number of complex network data naturally occurs as hypergraphs Estrada & RodriguezVelazquez (2005), the popularity of ”think like dyadic edges” and not like sets seems to us as a hindrance, speciﬁcally when the end aim is to model the joint distribution at the level of sets.

Revised: In this paper, as it is obvious, we go by ”think like set” paradigm, speciﬁcally, when the data is naturally occurring as network of sets (i.e., a hypergraph).

Revised: Therefore, if we think of hyperedge as a set, a natural question arises is that are there any existing techniques that learn embeddings at set level?

Revised: We, henceforth, explore the most popular discrete data on which representation learning has been applied is text data.

Revised: Recently, Le & Mikolov (2014) proposed two representation learning methods for sentences as shown in Figure 4: a) a distributed memory ( DM ) model, and b) a distributed bag of words ( DBOW ) model.

Revised: In the DM model, every sentence in the dataset G is represented by a d dimensional vector in a shared lookup matrix S ∈ | G |× d .

Revised: Similarly, every word in the vocabulary Ω is represented by a d dimensional vector in another shared lookup matrix L ∈ | Ω |× d .

Revised: Given an input sentence v = ( v 1 , v 2 · · · v m ) , the corresponding sentence vector from S and the word vectors from L are averaged to predict the next word in a context.

Revised: More formally, let φ denote the mapping from sentence and word ids to their respective vectors in S and L , the DM model minimizes the following objective:

Revised: [Equation]

Revised: where z is the average of φ ( v ) , φ ( v t − k +1 ) , · · · , φ ( v t − 1 ) input vectors, and ω ( v t ) is the output vector representation of word v t ∈ Ω .

Revised: The sentence vector φ ( v ) is shared across all (sliding window) contexts extracted from the same sentence, thus acts as a distributed memory.

Revised: Instead of predicting the next word in the context, the DBOW model predicts the words in the context independently given the sentence id as input.

Revised: More formally, DBOW minimizes the following objective:

Revised: [Equation]

Revised: Both the methods take sentences as input and return embeddings for words as well as the sentence.

Revised: To apply the Sen2Vec models directly to hyperedges, as our ﬁrst method we generate a proxy sentence for each hyperedge by leveraging the contextual information in the hyperedge.

Revised: A proxy sentence v i is formed for each hyperedge g i ∈ G as a sequence made by concatenating all the permutations of the nodes (as words) in the hyperedge and further repeating this sequence as many time this hyperedge occurred.

Revised: For example, for a three node hyperedge g i = { 1 , 4 , 7 } which has occurred two times we make the following sentence v i : { 1,4,7,1,7,4,7,1,4,7,4,1,4,1,7,4,7,1,1,4,7,1,7,4,7,1,4,7,4,1, 4,1,7,4,7,1 } of length 12 (6 permutations times 2 occurrence).

Revised: In this scheme we have permutations and repetitions.

Revised: We take all the permutations hoping that it should fool the sequence based model to get a sequence independent embedding.

Revised: On the other hand, repetition of the permutations acts similarly to observing the same sequence of words several times in the text corpus.

Revised: Of course one can think of other alternative schemes, but the point we are trying to make is that naturally observed hyperedges captures important contextual information that can be leveraged to achieve better representations.

Revised: We refer to the node-hyperedge embedding pairs resulting from DM , DBOW models as h2v-dm and h2v-dbow , respectively.

Revised: 7. H YPEREDGE 2 VEC U SING N ODE 2V EC

Revised: Grover & Leskovec (2016) propose a representation method for nodes in a graph called Node2Vec, which uses the skip-gram model Mikolov et al.

Revised: (2013b) with the intuition that neighborhood nodes within a graph should have similar representations.

Revised: The objective of the skip-gram model for graphs can be deﬁned as:

Revised: [Equation]

Revised: where as before, φ and ω denote the input and the output vector representations of the nodes.

Revised: The neighboring nodes N ( v ) form the context for node v .

Revised: Node2Vec uses a biased random walk which adaptively combines breadth ﬁrst search (BFS) and depth ﬁrst search (DFS) to ﬁnd the neighborhood of a node.

Revised: The walk attempts to capture two properties of a graph often used for prediction tasks in networks: i) homophily and ii) structural equivalence.

Revised: According to homophily, nodes in the same group or community should have similar representations.

Revised: Structural equivalence suggests that nodes with similar structural roles (hub, bridge) should have similar representations.

Revised: In a real-world network, nodes exhibit mixed properties.

Revised: Our aim is to ﬁnd both node and hyperedge level embeddings by taking into account the hypergraph structure.

Revised: For the former, we leverage the adjacency matrix associated with a hypergraph Zhou et al.

Revised: (2006), which is deﬁned as:

Revised: [Equation]

Revised: (15) where W e is a diagonal matrix containing the weights of each hyperedge and D v is a diagonal matrix containing the degree of each vertex.

Revised: We take the weight of a hyperedge as its occurrence number, i.e. W e ( i, i ) =

Revised: R ( g i ) , ∀{ i ∈ { 1 , ..., n }} .

Revised: The adjacency matrix ( A hyp ∈ | V |×| V | ) associates a weight between a pair of nodes while taking into account the weights of all the hyperedges that encompass a pair of nodes.

Revised: The weighted graph associated with A hyp in some sense serves as a proxy to the actual hypergraph structure.

Revised: We can provide A hyp as input to Node2Vec and hypothesize that the random walk over this proxy hypergraph should allow the skip-gram model to learn more meaningful node level embeddings which can be combined to construct hyperedge level embeddings.

Revised: We refer to these node level embeddings as N2V-hyp .

Revised: We still have not met our second objective to learn hyperedge embeddings directly.

Revised: We again wish to leverage Node2Vec for our purpose.

Revised: However, Node2Vec only works for graphs by performing random walk over nodes.

Revised: Therefore, we ask ourselves the question, that can we treat hyperedges as nodes?

Revised: There can be other ways of doing so, but here we suggest two techniques.

Revised: In the ﬁrst technique, we simply consider the hypergraph dual Berge (1984), whose incidence matrix is

Revised: H dual = H T .

Revised: The adjacency matrix associated with the hypergraph dual is:

Revised: [Equation]

Revised: 16) where W v is a diagonal matrix containing the weights of each node and D e is a diagonal matrix containing the degree of each hyperedge.

Revised: We assume no weights on the nodes and take W v = I .

Revised: The matrix A dual ∈ | G |×| G | represents another hypergraph, but the roles of nodes and hyperedges have now switched.

Revised: For example, in case of words and sentences, the hyperedges were sentences, but in the hypergraph dual the words become hyperedges and the nodes within a word’s hyperedge represent all the sentences in which the word has appeared.

Revised: We again give A dual (i.e., a graph proxy for the dual hypergraph) as input to Node2Vec, but this time we get output as the embeddings for the hyperedges associated with the nodes in dual.

Revised: We refer to these hyperedge embeddings as h2v-dual .

Revised: In the second technique, we consider the following adjacency

Revised: [Equation]

Revised: [Equation]

Revised: associated with what we refer to as the inverted hypergraph .

Revised: This inverted hypergraph is a graph (unlike the dual which is a hypergraph) and there is an edge between two nodes if the hyperedges corresponding to the nodes in the original hypergraph have nodes in common.

Revised: Weight of this edge is the number of common nodes.

Revised: We again give A inv as input to Node2Vec to get embeddings for the hyperedges associated with the nodes in inverted hypergraph.

Revised: We refer to these hyperedge embeddings as h2v-inv .

Revised: 7. H YPEREDGE 2 VEC U SING S PECTRAL E MBEDDINGS

Revised: These set of methods extract embeddings as the eigenvectors associated with Laplacian matrices corresponding to the various adjacency matrices discussed in the previous section.

Revised: We consider the following graph Laplacians:

Revised: [Equation]

Revised: where A graph = H T W e H is the weighted graph associated with the graph corresponding to adjacency matrix A ,

Revised: [Equation]

Revised: We get the d eigenvectors associated with the smallest d eigenvalues of the above graph Laplacians as the embeddings.

Revised: We get vertex embeddings using Eq.

Revised: 19 and hyperedge embedding using Eq.

Revised: 21, and we refer to them together as e2v-hyp .

Revised: Similarly, we get another pair of vertex embeddings (using Eq. 18) and hyperedge embedding (using Eq. 21).

Revised: We refer to the later pair as e2v .