Original: Y OU ARE A LL S ET : A M ULTISET L EARNING F RAMEWORK FOR H YPERGRAPH N EURAL N ETWORKS
Revised: Y OU ARE A LL S ET : A M ULTISET L EARNING F RAMEWORK FOR H YPERGRAPH N EURAL N ETWORKS

Original: A BSTRACT
Revised: A BSTRACT

Original: Hypergraphs are used to model higher-order interactions amongst agents and there exist many practically relevant instances of hypergraph datasets.
Revised: Hypergraphs are used to model higher-order interactions amongst agents and there exist many practically relevant instances of hypergraph datasets.

Original: To enable the efﬁcient processing of hypergraph data, several hypergraph neural network platforms have been proposed for learning hypergraph properties and structure, with a special focus on node classiﬁcation tasks.
Revised: To enable the efﬁcient processing of hypergraph data, several hypergraph neural network platforms have been proposed for learning hypergraph properties and structure, with a special focus on node classiﬁcation tasks.

Original: However, almost all existing methods use heuristic propagation rules and offer suboptimal performance on benchmarking datasets.
Revised: However, almost all existing methods use heuristic propagation rules and offer suboptimal performance on benchmarking datasets.

Original: We propose AllSet, a new hypergraph neural network paradigm that represents a highly general framework for (hyper)graph neural networks and for the ﬁrst time implements hypergraph neural network layers as compositions of two multiset functions that can be efﬁciently learned for each task and each dataset.
Revised: We propose AllSet, a new hypergraph neural network paradigm that represents a highly general framework for (hyper)graph neural networks and for the ﬁrst time implements hypergraph neural network layers as compositions of two multiset functions that can be efﬁciently learned for each task and each dataset.

Original: The proposed AllSet framework also for the ﬁrst time integrates Deep Sets and Set
Revised: The proposed AllSet framework also for the ﬁrst time integrates Deep Sets and Set

Original: Transformers with hypergraph neural networks for the purpose of learning multiset functions and therefore allows for signiﬁcant modeling ﬂexibility and high expressive power.
Revised: Transformers with hypergraph neural networks for the purpose of learning multiset functions and therefore allows for signiﬁcant modeling ﬂexibility and high expressive power.

Original: To evaluate the performance of AllSet, we conduct the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent signiﬁcant challenges for hypergraph node classiﬁcation.
Revised: To evaluate the performance of AllSet, we conduct the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent signiﬁcant challenges for hypergraph node classiﬁcation.

Original: The results demonstrate that our method has the unique ability to either match or outperform all other hypergraph neural networks across the tested datasets: As an example, the performance improvements over existing methods and a new method based on heterogeneous graph neural networks are close to 4% on the Yelp and Zoo datasets, and 3% on the Walmart dataset.
Revised: The results demonstrate that our method has the unique ability to either match or outperform all other hypergraph neural networks across the tested datasets: As an example, the performance improvements over existing methods and a new method based on heterogeneous graph neural networks are close to 4% on the Yelp and Zoo datasets, and 3% on the Walmart dataset.

Original: Our  implementation is available online.
Revised: Our AllSet network implementation is available online 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph-centered machine learning, and especially graph neural networks (GNNs), have attracted great interest in the machine learning community due to the ubiquity of graph-structured data and the importance of solving numerous real-world problems such as semi-supervised node classiﬁcation and graph classiﬁcation (Zhu, 2005; Shervashidze et al., 2011; L¨u & Zhou, 2011).
Revised: Graph-centered machine learning, and especially graph neural networks (GNNs), have attracted great interest in the machine learning community due to the ubiquity of graph-structured data and the importance of solving numerous real-world problems such as semi-supervised node classiﬁcation and graph classiﬁcation (Zhu, 2005; Shervashidze et al., 2011; L¨u & Zhou, 2011).

Original: Graphs model pairwise interactions between entities, but fail to capture more complex relationships.
Revised: Graphs model pairwise interactions between entities, but fail to capture more complex relationships.

Original: Hypergraphs, on the other hand, involve hyperedges that can connect more than two nodes, and are therefore capable of representing higher-order structures in datasets.
Revised: Hypergraphs, on the other hand, involve hyperedges that can connect more than two nodes, and are therefore capable of representing higher-order structures in datasets.

Original: There exist many machine learning and data mining applications for which modeling high-order relations via hypergraphs leads to better learning performance when compared to graph-based models (Benson et al., 2016).
Revised: There exist many machine learning and data mining applications for which modeling high-order relations via hypergraphs leads to better learning performance when compared to graph-based models (Benson et al., 2016).

Original: For example, in subspace clustering, in order to ﬁt a d -dimensional subspace, we need at least d +1 data points (Agarwal et al., 2005); in hierarchical species classiﬁcation of a FoodWeb, a carbon-ﬂow unit based on four species is signiﬁcantly more predictive than that involving two or three entities (Li & Milenkovic, 2017).
Revised: For example, in subspace clustering, in order to ﬁt a d -dimensional subspace, we need at least d +1 data points (Agarwal et al., 2005); in hierarchical species classiﬁcation of a FoodWeb, a carbon-ﬂow unit based on four species is signiﬁcantly more predictive than that involving two or three entities (Li & Milenkovic, 2017).

Original: Hence, it is desirable to generalize GNN concepts to hypergraphs.
Revised: Hence, it is desirable to generalize GNN concepts to hypergraphs.

Original: One straightforward way to generalize graph algorithms for hypergraphs is to convert hypergraphs to graphs via clique-expansion (CE) (Agarwal et al., 2005;
Revised: One straightforward way to generalize graph algorithms for hypergraphs is to convert hypergraphs to graphs via clique-expansion (CE) (Agarwal et al., 2005;

Original: Zhou et al., 2006).
Revised: Zhou et al., 2006).

Original: CE replaces hyperedges by (possibly weighted) cliques.
Revised: CE replaces hyperedges by (possibly weighted) cliques.

Original: Many recent attempts to generalize GNNs to hypergraphs can be viewed as redeﬁning hypergraph propagation schemes based on CE or its variants (Yadati et al.
Revised: Many recent attempts to generalize GNNs to hypergraphs can be viewed as redeﬁning hypergraph propagation schemes based on CE or its variants (Yadati et al.

Original: , ∗ Equal contribution.
Revised: , ∗ Equal contribution.

Original: ; Feng et al., 2019; Bai et al., 2021), which was also originally pointed out in (Dong et al., 2020).
Revised: ; Feng et al., 2019; Bai et al., 2021), which was also originally pointed out in (Dong et al., 2020).

Original: Despite the simplicity of CE, it is well-known that CE causes distortion and leads to undesired losses in learning performance (Hein et al., 2013; Li & Milenkovic, 2018; Chien et al., 2019b).
Revised: Despite the simplicity of CE, it is well-known that CE causes distortion and leads to undesired losses in learning performance (Hein et al., 2013; Li & Milenkovic, 2018; Chien et al., 2019b).

Original: In parallel, more sophisticated propagation rules directly applicable on hypergraphs, and related to tensor eigenproblems, have been studied as well.
Revised: In parallel, more sophisticated propagation rules directly applicable on hypergraphs, and related to tensor eigenproblems, have been studied as well.

Original: One such example, termed Multilinear PageRank (Gleich et al., 2015), generalizes PageRank techniques (Page et al., 1999; Jeh & Widom, 2003) directly to hypergraphs without resorting to the use of CE.
Revised: One such example, termed Multilinear PageRank (Gleich et al., 2015), generalizes PageRank techniques (Page et al., 1999; Jeh & Widom, 2003) directly to hypergraphs without resorting to the use of CE.

Original: Its propagation scheme is closely related to the Z eigenproblem which has been extensively investigated in tensor analysis and spectral hypergraph theory (Li et al., 2013;
Revised: Its propagation scheme is closely related to the Z eigenproblem which has been extensively investigated in tensor analysis and spectral hypergraph theory (Li et al., 2013;

Original: He & Huang, 2014; Qi & Luo, 2017; Pearson & Zhang, 2014; Gautier et al., 2019).
Revised: He & Huang, 2014; Qi & Luo, 2017; Pearson & Zhang, 2014; Gautier et al., 2019).

Original: An important result of Benson et al.
Revised: An important result of Benson et al.

Original: (2017) shows that tensor-based propagation outperforms a CE-based scheme on several tasks.
Revised: (2017) shows that tensor-based propagation outperforms a CE-based scheme on several tasks.

Original: The pros and cons of these two types of propagation rule in statistical learning frameworks were examined in Chien et al.
Revised: The pros and cons of these two types of propagation rule in statistical learning frameworks were examined in Chien et al.

Original: (2021a).
Revised: (2021a).

Original: More recently, it was shown in Tudisco et al.
Revised: More recently, it was shown in Tudisco et al.

Original: (2020) that label propagation based on CE of hypergraphs does not always lead to acceptable performance.
Revised: (2020) that label propagation based on CE of hypergraphs does not always lead to acceptable performance.

Original: Similarly to Chien et al.
Revised: Similarly to Chien et al.

Original: (2021a), Benson (2019)
Revised: (2021a), Benson (2019)

Original: identiﬁed positive traits of CE eigenvectors but argued in favor of using Z eigenvectors due to their more versatile nonlinear formulation compared to that of the eigenvectors of CE graphs.
Revised: identiﬁed positive traits of CE eigenvectors but argued in favor of using Z eigenvectors due to their more versatile nonlinear formulation compared to that of the eigenvectors of CE graphs.

Original: We address two natural questions pertaining to learning on hypergraphs: “Is there a general framework that includes CE-based, Z-based and other propagations on hypergraphs?”
Revised: We address two natural questions pertaining to learning on hypergraphs: “Is there a general framework that includes CE-based, Z-based and other propagations on hypergraphs?”

Original: and, “Can we learn propagation schemes for hypergraph neural networks suitable for different datasets and different learning tasks?”
Revised: and, “Can we learn propagation schemes for hypergraph neural networks suitable for different datasets and different learning tasks?”

Original: We give afﬁrmative answers to both questions.
Revised: We give afﬁrmative answers to both questions.

Original: We propose a general framework,
Revised: We propose a general framework,

Original: AllSet, which includes both CE-based and tensor-based propagation rules as special cases.
Revised: AllSet, which includes both CE-based and tensor-based propagation rules as special cases.

Original: We also propose two powerful hypergraph neural network layer architectures that can learn adequate propagation rules for hypergraphs using multiset functions.
Revised: We also propose two powerful hypergraph neural network layer architectures that can learn adequate propagation rules for hypergraphs using multiset functions.

Original: Our speciﬁc contributions are as follows.
Revised: Our speciﬁc contributions are as follows.

Original: We show that using AllSet, one can not only model CE-based and tensor-based propagation rules, but also cover propagation methods of most existing hypergraph neural networks, including HyperGCN (Yadati et al., 2019), HGNN (Feng et al., 2019), HCHA (Bai et al., 2021), HNHN (Dong et al., 2020) and HyperSAGE (Arya et al., 2020).
Revised: We show that using AllSet, one can not only model CE-based and tensor-based propagation rules, but also cover propagation methods of most existing hypergraph neural networks, including HyperGCN (Yadati et al., 2019), HGNN (Feng et al., 2019), HCHA (Bai et al., 2021), HNHN (Dong et al., 2020) and HyperSAGE (Arya et al., 2020).

Original: Most importantly, we show that all these propagation rules can be described as a composition of two multiset functions (leading to the proposed method name AllSet).
Revised: Most importantly, we show that all these propagation rules can be described as a composition of two multiset functions (leading to the proposed method name AllSet).

Original: Furthermore, we also show that AllSet is a hypergraph generalization of Message Passing Neural Networks (MPNN) (Gilmer et al., 2017), a powerful graph learning framework encompassing many GNNs such as GCN (Kipf & Welling, 2017) and
Revised: Furthermore, we also show that AllSet is a hypergraph generalization of Message Passing Neural Networks (MPNN) (Gilmer et al., 2017), a powerful graph learning framework encompassing many GNNs such as GCN (Kipf & Welling, 2017) and

Original: GAT (Veli ˇ ckovi ´ c et al., 2018).
Revised: GAT (Veli ˇ ckovi ´ c et al., 2018).

Original: Inspired by Deep Sets (Zaheer et al., 2017) and Set Transformer (Lee et al., 2019), we propose AllDeepSets and AllSetTransformer layers which are end-to-end trainable.
Revised: Inspired by Deep Sets (Zaheer et al., 2017) and Set Transformer (Lee et al., 2019), we propose AllDeepSets and AllSetTransformer layers which are end-to-end trainable.

Original: They can be plugged into most types of graph neural networks to enable effortless generalizations to hypergraphs.
Revised: They can be plugged into most types of graph neural networks to enable effortless generalizations to hypergraphs.

Original: Notably, our work represents the ﬁrst attempt to connect the problem of learning multiset function with hypergraph neural networks, and to leverage the powerful Set Transformer model in the design of these specialized networks.
Revised: Notably, our work represents the ﬁrst attempt to connect the problem of learning multiset function with hypergraph neural networks, and to leverage the powerful Set Transformer model in the design of these specialized networks.

Original: We report, to the best of
Revised: We report, to the best of

Original: our
Revised: our

Original: knowledge, the
Revised: knowledge, the

Original: most
Revised: most

Original: extensive experiments in
Revised: extensive experiments in

Original: the hypergraph neural networks literature
Revised: the hypergraph neural networks literature

Original: pertaining
Revised: pertaining

Original: to semi-supervised node classiﬁcation.
Revised: to semi-supervised node classiﬁcation.

Original: Experimental results against ten baseline methods on ten benchmark datasets and three newly curated and challenging datasets demonstrate the superiority and consistency of our AllSet approach
Revised: Experimental results against ten baseline methods on ten benchmark datasets and three newly curated and challenging datasets demonstrate the superiority and consistency of our AllSet approach

Original: As an example, AllSetTransformer outperforms the best baseline method by close to 4% in accuracy on Yelp and Zoo datasets and 3% on the Walmart dataset; furthermore, AllSetTransformer matches or outperforms the best baseline models on nine out of ten datasets.
Revised: As an example, AllSetTransformer outperforms the best baseline method by close to 4% in accuracy on Yelp and Zoo datasets and 3% on the Walmart dataset; furthermore, AllSetTransformer matches or outperforms the best baseline models on nine out of ten datasets.

Original: Such improvements are not possible with modiﬁcations of HAN (Wang et al., 2019b), a heterogeneous GNN, adapted to hypergraphs or other specialized approaches that do not use Set Transformers.
Revised: Such improvements are not possible with modiﬁcations of HAN (Wang et al., 2019b), a heterogeneous GNN, adapted to hypergraphs or other specialized approaches that do not use Set Transformers.

Original: As another practical
Revised: As another practical

Original: contribution,
Revised: contribution,

Original: we
Revised: we

Original: also
Revised: also

Original: provide a
Revised: provide a

Original: succinct pipeline
Revised: succinct pipeline

Original: for standardization of
Revised: for standardization of

Original: the hypergraph
Revised: the hypergraph

Original: neural
Revised: neural

Original: networks evaluation
Revised: networks evaluation

Original: process based on Pytorch Geometric
Revised: process based on Pytorch Geometric

Original: Fey & Lenssen
Revised: Fey & Lenssen

Original: The pipeline is built in a fashion similar to that proposed in recent benchmarking GNNs papers (Hu et al., 2020a; Lim et al., 2021).
Revised: The pipeline is built in a fashion similar to that proposed in recent benchmarking GNNs papers (Hu et al., 2020a; Lim et al., 2021).

Original: The newly introduced datasets, along with our reported testbed, may be viewed as an initial step toward benchmarking hypergraph neural networks.
Revised: The newly introduced datasets, along with our reported testbed, may be viewed as an initial step toward benchmarking hypergraph neural networks.

Original: All proofs and concluding remarks are relegated to the Appendix.
Revised: All proofs and concluding remarks are relegated to the Appendix.

Original: [Figure]
Revised: [Figure]

Original: 6     b a
Revised: 6     b a

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Notation.
Revised: Notation.

Original: A hypergraph is an ordered pair of sets G ( V , E ) , where V = { 1 , 2 , . . . , n } is the set of nodes while E is the set of hyperedges.
Revised: A hypergraph is an ordered pair of sets G ( V , E ) , where V = { 1 , 2 , . . . , n } is the set of nodes while E is the set of hyperedges.

Original: Each hyperedge e ∈
Revised: Each hyperedge e ∈

Original: E is a subset of V , i.e., e ⊆ V .
Revised: E is a subset of V , i.e., e ⊆ V .

Original: Unlike a graph edge, a hyperedge e may contain more than two nodes.
Revised: Unlike a graph edge, a hyperedge e may contain more than two nodes.

Original: If ∀ e ∈ E one has | e | = d ∈ N , the hypergraph G is termed d -uniform.
Revised: If ∀ e ∈ E one has | e | = d ∈ N , the hypergraph G is termed d -uniform.

Original: A d -uniform hypergraph can be represented by a d -dimensional supersymmetric tensor such that for all distinct collections i 1 , . .
Revised: A d -uniform hypergraph can be represented by a d -dimensional supersymmetric tensor such that for all distinct collections i 1 , . .

Original: , i d ∈ V , A
Revised: , i d ∈ V , A

Original: i 1 ,...,i d = 1 ( d − 2)!
Revised: i 1 ,...,i d = 1 ( d − 2)!

Original: if e = { i 1 , ..., i d } ∈ E , and A i 1 ,...,i d = 0 otherwise.
Revised: if e = { i 1 , ..., i d } ∈ E , and A i 1 ,...,i d = 0 otherwise.

Original: Henceforth, A : ,...,i d is used to denote the slice of A along the ﬁrst coordinate.
Revised: Henceforth, A : ,...,i d is used to denote the slice of A along the ﬁrst coordinate.

Original: A hypergraph can alternatively be represented by its incidence matrix
Revised: A hypergraph can alternatively be represented by its incidence matrix

Original: H , where H ve = 1 if v ∈ e and H ve = 0 otherwise.
Revised: H , where H ve = 1 if v ∈ e and H ve = 0 otherwise.

Original: We use the superscript ( t ) to represent the functions or variables at the t -th step of propagation and (cid:107) to denote concatenation.
Revised: We use the superscript ( t ) to represent the functions or variables at the t -th step of propagation and (cid:107) to denote concatenation.

Original: Furthermore, Θ and b are reserved for a learnable weight matrix and bias of a neural network, respectively.
Revised: Furthermore, Θ and b are reserved for a learnable weight matrix and bias of a neural network, respectively.

Original: Finally, we use σ ( · ) to denote a nonlinear activation function (such as ReLU, eLU or LeakyReLU), which depends on the model used.
Revised: Finally, we use σ ( · ) to denote a nonlinear activation function (such as ReLU, eLU or LeakyReLU), which depends on the model used.

Original: CE-based propagation on hypergraphs.
Revised: CE-based propagation on hypergraphs.

Original: The CE of a hypergraph G ( V , E ) is a weighted graph with the same set of nodes V .
Revised: The CE of a hypergraph G ( V , E ) is a weighted graph with the same set of nodes V .

Original: It can be described in terms of the associated adjacency or incidence matrices which we write with a slight abuse of notation as A ( CE ) ij =
Revised: It can be described in terms of the associated adjacency or incidence matrices which we write with a slight abuse of notation as A ( CE ) ij =

Original: (cid:80)
Revised: (cid:80)

Original: i 3 ,...,i d ∈V A i,j,i 3 ,...,i d and
Revised: i 3 ,...,i d ∈V A i,j,i 3 ,...,i d and

Original: H ( CE ) =
Revised: H ( CE ) =

Original: HH T , respectively.
Revised: HH T , respectively.

Original: It is obvious that these two matrices only differ in their diagonal entries ( 0 s versus node degrees, respectively).
Revised: It is obvious that these two matrices only differ in their diagonal entries ( 0 s versus node degrees, respectively).

Original: One step of propagation of a F -dimensional node feature matrix X ∈ R n × F is captured by A ( CE ) X or H ( CE ) X ; alternatively, in terms of node feature updates, we have
Revised: One step of propagation of a F -dimensional node feature matrix X ∈ R n × F is captured by A ( CE ) X or H ( CE ) X ; alternatively, in terms of node feature updates, we have

Original: [Equation]
Revised: [Equation]

Original: Many existing hypergraph convolutional layers actually perform CE-based propagation, potentially with further degree normalization and nonlinear hyperedge weights.
Revised: Many existing hypergraph convolutional layers actually perform CE-based propagation, potentially with further degree normalization and nonlinear hyperedge weights.

Original: For example, the propagation rule of HGNN (Feng et al., 2019) takes the following node-wise form:
Revised: For example, the propagation rule of HGNN (Feng et al., 2019) takes the following node-wise form:

Original: [Equation]
Revised: [Equation]

Original: where d v denotes the degree of node v , w e is a predeﬁned weight of hyperedge e and σ is the ReLU activation function.
Revised: where d v denotes the degree of node v , w e is a predeﬁned weight of hyperedge e and σ is the ReLU activation function.

Original: The hypergraph convolution in HCHA (Bai et al., 2021) uses different degree normalizations and attention weights, with the attention weights depending on node features and the hyperedge features.
Revised: The hypergraph convolution in HCHA (Bai et al., 2021) uses different degree normalizations and attention weights, with the attention weights depending on node features and the hyperedge features.

Original: If datasets do not contain hyperedge feature information or if the features come from a different domain compared to the node features, one cannot use their attention module (Bai et al., 2021).
Revised: If datasets do not contain hyperedge feature information or if the features come from a different domain compared to the node features, one cannot use their attention module (Bai et al., 2021).

Original: HyperGCN replaces each hyperedge by an incomplete clique via so-called mediators (Yadati et al., 2019).
Revised: HyperGCN replaces each hyperedge by an incomplete clique via so-called mediators (Yadati et al., 2019).

Original: When the hypergraph is 3 -uniform, the aforementioned approach becomes a standard weighted CE.
Revised: When the hypergraph is 3 -uniform, the aforementioned approach becomes a standard weighted CE.

Original: Hence, all the described hypergraph neural networks adapt propagation rules based on CE or its variants, potentially with the addition of nonlinear hyperedge weights.
Revised: Hence, all the described hypergraph neural networks adapt propagation rules based on CE or its variants, potentially with the addition of nonlinear hyperedge weights.

Original: The described methods achieve reasonable good performance on standard cocitation and coauthor benchmarking datasets.
Revised: The described methods achieve reasonable good performance on standard cocitation and coauthor benchmarking datasets.

Original: Tensor-based propagations.
Revised: Tensor-based propagations.

Original: As mentioned in the introduction, there exist more elaborate tensorbased propagation schemes which in some cases outperform CE-based methods.
Revised: As mentioned in the introduction, there exist more elaborate tensorbased propagation schemes which in some cases outperform CE-based methods.

Original: The propagation rules related to Z eigenproblems such as multilinear PageRank (Gleich et al., 2015) and spacey random walks (Benson et al., 2017) are two such examples.
Revised: The propagation rules related to Z eigenproblems such as multilinear PageRank (Gleich et al., 2015) and spacey random walks (Benson et al., 2017) are two such examples.

Original: The Z eigenproblem for an adjacency tensor A of a d -uniform hypergraph is deﬁned as:
Revised: The Z eigenproblem for an adjacency tensor A of a d -uniform hypergraph is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Here, A x d − 1 equals (cid:80) i 2 ,...,i
Revised: Here, A x d − 1 equals (cid:80) i 2 ,...,i

Original: d A : ,i 2 ,...,i d x i 2 . . .
Revised: d A : ,i 2 ,...,i d x i 2 . . .

Original: x i d , an entity frequently used in the tensor analysis literature.
Revised: x i d , an entity frequently used in the tensor analysis literature.

Original: The Z eigenproblem has been extensively studied both in the context of tensor analysis and network sciences; the problem is also known as the l 2 eigenproblem (Lim, 2005; Gautier et al., 2019).
Revised: The Z eigenproblem has been extensively studied both in the context of tensor analysis and network sciences; the problem is also known as the l 2 eigenproblem (Lim, 2005; Gautier et al., 2019).

Original: We refer the interested readers to Qi & Luo (2017) for a more detailed theoretical analysis of the Z eigenproblems and Benson (2019) for its application in the study of hypergraph centralities.
Revised: We refer the interested readers to Qi & Luo (2017) for a more detailed theoretical analysis of the Z eigenproblems and Benson (2019) for its application in the study of hypergraph centralities.

Original: By ignoring the norm constraint, one can deﬁne the following tensor-based propagation rule based on (3) according to:
Revised: By ignoring the norm constraint, one can deﬁne the following tensor-based propagation rule based on (3) according to:

Original: [Equation]
Revised: [Equation]

Original: Despite its interesting theoretical properties, Zprop is known to have what is termed the “unit problem” (Benson, 2019).
Revised: Despite its interesting theoretical properties, Zprop is known to have what is termed the “unit problem” (Benson, 2019).

Original: In practice, the product can cause numerical instabilities for large hyperedges.
Revised: In practice, the product can cause numerical instabilities for large hyperedges.

Original: Furthermore, Zprop has only been studied for the case of d -uniform hypergraphs, which makes it less relevant for general hypergraph learning tasks.
Revised: Furthermore, Zprop has only been studied for the case of d -uniform hypergraphs, which makes it less relevant for general hypergraph learning tasks.

Original: Clearly, CEprop and Zprop have different advantages and disadvantages for different dataset structures.
Revised: Clearly, CEprop and Zprop have different advantages and disadvantages for different dataset structures.

Original: This motivates ﬁnding a general framework that encompasses these two and other propagation rules.
Revised: This motivates ﬁnding a general framework that encompasses these two and other propagation rules.

Original: In this case, we aim to learn the suitable propagation scheme under such a framework for hypergraph neural networks.
Revised: In this case, we aim to learn the suitable propagation scheme under such a framework for hypergraph neural networks.

Original:  A LL S ET : O NE M ETHOD TO B IND T HEM A LL
Revised:  A LL S ET : O NE M ETHOD TO B IND T HEM A LL

Original: We show that all the above described propagation methods can be uniﬁed within one setting, termed
Revised: We show that all the above described propagation methods can be uniﬁed within one setting, termed

Original: AllSet.
Revised: AllSet.

Original: The key observation is that all propagation rules equal a composition of two multiset functions, deﬁned below.
Revised: The key observation is that all propagation rules equal a composition of two multiset functions, deﬁned below.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: A function f is permutation invariant if and only if ∀ π ∈ S n , where S n denotes the symmetric group of order n ! , f ( x π (1) , . . .
Revised: A function f is permutation invariant if and only if ∀ π ∈ S n , where S n denotes the symmetric group of order n ! , f ( x π (1) , . . .

Original: , x π ( n ) )
Revised: , x π ( n ) )

Original: =
Revised: =

Original: f
Revised: f

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n ) .
Revised: , x n ) .

Original: Deﬁnition 3.2.
Revised: Deﬁnition 3.2.

Original: We say that a function f is a multiset function if it is permutation invariant.
Revised: We say that a function f is a multiset function if it is permutation invariant.

Original: Next, let V e, X
Revised: Next, let V e, X

Original: =
Revised: =

Original: X u, : : u ∈ e } denote the multiset of hidden node representations contained in the hyperedge e .
Revised: X u, : : u ∈ e } denote the multiset of hidden node representations contained in the hyperedge e .

Original: Also, let Z ∈ R |E|× F (cid:48) denote the hidden hyperedge representations.
Revised: Also, let Z ∈ R |E|× F (cid:48) denote the hidden hyperedge representations.

Original: Similarly, let
Revised: Similarly, let

Original: E v, Z = { Z e, : : v ∈ e } be the multiset of hidden representations of hyperedges that contain the node v .
Revised: E v, Z = { Z e, : : v ∈ e } be the multiset of hidden representations of hyperedges that contain the node v .

Original: The AllSet framework uses the update rules
Revised: The AllSet framework uses the update rules

Original: [Equation]
Revised: [Equation]

Original: where f V→E and f E→V are two multiset functions with respect to their ﬁrst input.
Revised: where f V→E and f E→V are two multiset functions with respect to their ﬁrst input.

Original: For the initial condition, we choose Z (0) e, : and X (0) v, : to be hyperedge features and node features, respectively (if available).
Revised: For the initial condition, we choose Z (0) e, : and X (0) v, : to be hyperedge features and node features, respectively (if available).

Original: If these are not available, we set both entities to be all-zero matrices.
Revised: If these are not available, we set both entities to be all-zero matrices.

Original: Note that we make the tacit assumption that both functions f V→E and f E→V also include the hypergraph G as an input.
Revised: Note that we make the tacit assumption that both functions f V→E and f E→V also include the hypergraph G as an input.

Original: This allows degree normalization to be part of our framework.
Revised: This allows degree normalization to be part of our framework.

Original: As one can also distinguish the aggregating node v from the multiset V e, X ( t ) , we also have the following AllSet variant:
Revised: As one can also distinguish the aggregating node v from the multiset V e, X ( t ) , we also have the following AllSet variant:

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we omit the last input argument X ( t ) v, : for f V→E in (6), unless explicitly needed (as in the proof pertaining to HyperGCN).
Revised: For simplicity, we omit the last input argument X ( t ) v, : for f V→E in (6), unless explicitly needed (as in the proof pertaining to HyperGCN).

Original: The formulation (5) lends itself to a signiﬁcantly more computationally- and memory-efﬁcient pipeline.
Revised: The formulation (5) lends itself to a signiﬁcantly more computationally- and memory-efﬁcient pipeline.

Original: This is clearly the case since for each hyperedge e , the expression in (5) only uses one hyperedge hidden representation while the expression (6) uses | e | distinct hidden representations.
Revised: This is clearly the case since for each hyperedge e , the expression in (5) only uses one hyperedge hidden representation while the expression (6) uses | e | distinct hidden representations.

Original: This difference can be substantial when the hyperedge sizes are large.
Revised: This difference can be substantial when the hyperedge sizes are large.

Original: Hence, we only use (6) for theoretical analysis purposes and (5) for all experimental veriﬁcations.
Revised: Hence, we only use (6) for theoretical analysis purposes and (5) for all experimental veriﬁcations.

Original: The next theorems establish the universality of the AllSet framework.
Revised: The next theorems establish the universality of the AllSet framework.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: CEpropH of (1) is a special case of AllSet (5) .
Revised: CEpropH of (1) is a special case of AllSet (5) .

Original: Furthermore, CEpropA of (1) and
Revised: Furthermore, CEpropA of (1) and

Original: Zprop of (4) are also special cases of AllSet (6) .
Revised: Zprop of (4) are also special cases of AllSet (6) .

Original: Sketch of proof.
Revised: Sketch of proof.

Original: If we ignore the second input and choose both f V→E and f E→V to be sums over their input multisets, we recover the CE-based propagation rule CEpropH of (1) via (5).
Revised: If we ignore the second input and choose both f V→E and f E→V to be sums over their input multisets, we recover the CE-based propagation rule CEpropH of (1) via (5).

Original: For CEpropA, the same observation is true with respect to (6).
Revised: For CEpropA, the same observation is true with respect to (6).

Original: If one instead chooses f V→E to be the product over its input multiset, multiplied by the scalar | V e, X ( t ) | − 1 , one recovers the tensor-based propagation
Revised: If one instead chooses f V→E to be the product over its input multiset, multiplied by the scalar | V e, X ( t ) | − 1 , one recovers the tensor-based propagation

Original: Zprop of (4) via (6).
Revised: Zprop of (4) via (6).

Original: Next, we show that many state-of-the-art hypergraph neural network layers also represent special instances of AllSet and are strictly less expressive than AllSet.
Revised: Next, we show that many state-of-the-art hypergraph neural network layers also represent special instances of AllSet and are strictly less expressive than AllSet.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: The hypergraph neural network layers of HGNN (2) , HCHA (Bai et al., 2021),
Revised: The hypergraph neural network layers of HGNN (2) , HCHA (Bai et al., 2021),

Original: HNHN (Dong et al., 2020) and HyperSAGE (Arya et al., 2020) are all special cases of AllSet (5) .
Revised: HNHN (Dong et al., 2020) and HyperSAGE (Arya et al., 2020) are all special cases of AllSet (5) .

Original: The HyperGCN layer (Yadati et al., 2019) is a special instance of AllSet (6) .
Revised: The HyperGCN layer (Yadati et al., 2019) is a special instance of AllSet (6) .

Original: Furthermore, all the above methods are strictly less expressive than AllSet (5) and (6) .
Revised: Furthermore, all the above methods are strictly less expressive than AllSet (5) and (6) .

Original: More precisely, there exists a combination of multiset functions f V→E and f E→V for AllSet (5) and (6) that cannot be modeled by any of the aforementioned hypergraph neural network layers.
Revised: More precisely, there exists a combination of multiset functions f V→E and f E→V for AllSet (5) and (6) that cannot be modeled by any of the aforementioned hypergraph neural network layers.

Original: The ﬁrst half of the proof is by direct construction.
Revised: The ﬁrst half of the proof is by direct construction.

Original: The second half of the proof consists of counterexamples.
Revised: The second half of the proof consists of counterexamples.

Original: Intuitively, none of the listed hypergraph neural network layers can model Zprop (3) while we established in the previous result that Zprop is a special case of AllSet.
Revised: Intuitively, none of the listed hypergraph neural network layers can model Zprop (3) while we established in the previous result that Zprop is a special case of AllSet.

Original: The last result shows that the MPNN (Gilmer et al., 2017) framework is also a special instance of our AllSet for graphs, which are (clearly) special cases of hypergraphs.
Revised: The last result shows that the MPNN (Gilmer et al., 2017) framework is also a special instance of our AllSet for graphs, which are (clearly) special cases of hypergraphs.

Original: Hence, AllSet may also be viewed as a hypergraph generalization of MPNN.
Revised: Hence, AllSet may also be viewed as a hypergraph generalization of MPNN.

Original: Note that MPNN itself generalizes many wellknown GNNs, such as GCN (Kipf & Welling, 2017), Gated Graph Neural Networks (Li et al., 2015) and GAT (Veli ˇ ckovi ´ c et al., 2018).
Revised: Note that MPNN itself generalizes many wellknown GNNs, such as GCN (Kipf & Welling, 2017), Gated Graph Neural Networks (Li et al., 2015) and GAT (Veli ˇ ckovi ´ c et al., 2018).

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: MPNN is a special case of AllSet (6) when applied to graphs.
Revised: MPNN is a special case of AllSet (6) when applied to graphs.

Original:  H OW TO L EARN A LL S ET L AYERS
Revised:  H OW TO L EARN A LL S ET L AYERS

Original: The key idea behind AllSet is to learn the multiset functions f V→E and f E→V on the ﬂy for each dataset and task.
Revised: The key idea behind AllSet is to learn the multiset functions f V→E and f E→V on the ﬂy for each dataset and task.

Original: To facilitate this learning process, we ﬁrst have to properly parametrize the multiset functions.
Revised: To facilitate this learning process, we ﬁrst have to properly parametrize the multiset functions.

Original: Ideally, the parametrization should represent a universal approximator for a multiset function that allows one to retain the higher expressive power of our architectures when compared to that of the hypergraph neural networks described in Theorem 3.4.
Revised: Ideally, the parametrization should represent a universal approximator for a multiset function that allows one to retain the higher expressive power of our architectures when compared to that of the hypergraph neural networks described in Theorem 3.4.

Original: For simplicity, we focus on the multiset inputs of f V→E and f E→V and postpone the discussion pertaining to the second arguments of the functions to the end of this section.
Revised: For simplicity, we focus on the multiset inputs of f V→E and f E→V and postpone the discussion pertaining to the second arguments of the functions to the end of this section.

Original: Under the assumption that the multiset size is ﬁnite, the authors of Zaheer et al.
Revised: Under the assumption that the multiset size is ﬁnite, the authors of Zaheer et al.

Original: and
Revised: and

Original: Wagstaff et al.
Revised: Wagstaff et al.

Original: (2019) proved that any multiset functions f can be parametrized as f ( S )
Revised: (2019) proved that any multiset functions f can be parametrized as f ( S )

Original: =
Revised: =

Original: ρ (cid:0)(cid:80) s ∈ S φ ( s ) (cid:1) , where ρ and φ are some bijective mappings (Theorem 4.4 in Wagstaff et al.
Revised: ρ (cid:0)(cid:80) s ∈ S φ ( s ) (cid:1) , where ρ and φ are some bijective mappings (Theorem 4.4 in Wagstaff et al.

Original: In practice, these mappings can be replaced by any universal approximator such as a multilayer perceptron (MLP) (Zaheer et al., 2017).
Revised: In practice, these mappings can be replaced by any universal approximator such as a multilayer perceptron (MLP) (Zaheer et al., 2017).

Original: This leads to the purely MLP AllSet layer for hypergraph neural networks, termed AllDeepSets.
Revised: This leads to the purely MLP AllSet layer for hypergraph neural networks, termed AllDeepSets.

Original: [Equation]
Revised: [Equation]

Original: The authors of Lee et al.
Revised: The authors of Lee et al.

Original: (2019) argued that the unweighted sum in Deep Set makes it hard to learn the importance of each individual contributing term.
Revised: (2019) argued that the unweighted sum in Deep Set makes it hard to learn the importance of each individual contributing term.

Original: Thus, they proposed the Set Transformer paradigm which was shown to offer better performance than Deep Sets as a learnable multiset function architecture.
Revised: Thus, they proposed the Set Transformer paradigm which was shown to offer better performance than Deep Sets as a learnable multiset function architecture.

Original: Based on this result, we also propose an attention-based AllSet layer for hypergraph neural networks, termed AllSetTransformer.
Revised: Based on this result, we also propose an attention-based AllSet layer for hypergraph neural networks, termed AllSetTransformer.

Original: Given the matrix S ∈ R | S |× F which represents the multiset S of F -dimensional real vectors, the deﬁnition of AllSetTransformer is
Revised: Given the matrix S ∈ R | S |× F which represents the multiset S of F -dimensional real vectors, the deﬁnition of AllSetTransformer is

Original: AllSetTransformer: f V→E
Revised: AllSetTransformer: f V→E

Original: [Equation]
Revised: [Equation]

Original: , where 
Revised: , where Y =

Original: [Equation]
Revised: [Equation]

Original: Here, LN represents the layer normalization (Ba et al., 2016), (cid:107) denotes concatenation and θ ∈ R 1 × hF h is a learnable weight; in addition, MH h,ω is a multihead attention mechanism with h heads and activation function ω
Revised: Here, LN represents the layer normalization (Ba et al., 2016), (cid:107) denotes concatenation and θ ∈ R 1 × hF h is a learnable weight; in addition, MH h,ω is a multihead attention mechanism with h heads and activation function ω

Original: (Vaswani et al., 2017).
Revised: (Vaswani et al., 2017).

Original: Note that the dimension of the output of MH h,ω is 1 × hF h where F h is the hidden dimension of V ( i ) ∈ R | S |× F h .
Revised: Note that the dimension of the output of MH h,ω is 1 × hF h where F h is the hidden dimension of V ( i ) ∈ R | S |× F h .

Original: In our experimental setting, we choose ω to be the softmax function which is robust in practice.
Revised: In our experimental setting, we choose ω to be the softmax function which is robust in practice.

Original: Our formulation of projections in
Revised: Our formulation of projections in

Original: MH h,ω is slight more general than standard linear projections.
Revised: MH h,ω is slight more general than standard linear projections.

Original: If one restricts MLP K,i and MLP V,i to one-layer perceptrons without a bias term (i.e., including only a learnable weight matrix), we obtain standard linear projections in the multihead attention mechanism.
Revised: If one restricts MLP K,i and MLP V,i to one-layer perceptrons without a bias term (i.e., including only a learnable weight matrix), we obtain standard linear projections in the multihead attention mechanism.

Original: The output dimensions of
Revised: The output dimensions of

Original: MLP K,i and MLP V,i are both equal to R | S |× F h .
Revised: MLP K,i and MLP V,i are both equal to R | S |× F h .

Original: It is worth pointing out that all the MLP modules operate row-wise, which means they are applied to each multiset element (a real vector) independently and in an identical fashion.
Revised: It is worth pointing out that all the MLP modules operate row-wise, which means they are applied to each multiset element (a real vector) independently and in an identical fashion.

Original: This directly implies that MLP modules are permutation equivariant.
Revised: This directly implies that MLP modules are permutation equivariant.

Original: Based on the above discussion and Proposition 2 of Lee et al.
Revised: Based on the above discussion and Proposition 2 of Lee et al.

Original: we have the following result.
Revised: we have the following result.

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: The functions f V→E and f E→V deﬁned in AllSetTransformer (8) are permutation invariant.
Revised: The functions f V→E and f E→V deﬁned in AllSetTransformer (8) are permutation invariant.

Original: Furthermore, the functions f V→E and f E→V deﬁned in (8) are universal approximators of multiset functions when the size of the input multiset is ﬁnite.
Revised: Furthermore, the functions f V→E and f E→V deﬁned in (8) are universal approximators of multiset functions when the size of the input multiset is ﬁnite.

Original: Note that in practice, both the size of hyperedges and the node degrees are ﬁnite.
Revised: Note that in practice, both the size of hyperedges and the node degrees are ﬁnite.

Original: Thus, the ﬁnite size multiset assumption is satisﬁed.
Revised: Thus, the ﬁnite size multiset assumption is satisﬁed.

Original: Therefore, the above results directly imply that the AllDeepSets (7) and AllSetTransformer (8) layers have the same expressive power as the general AllSet framework.
Revised: Therefore, the above results directly imply that the AllDeepSets (7) and AllSetTransformer (8) layers have the same expressive power as the general AllSet framework.

Original: Together with the result of Theorem 3.4, we arrive at the conclusion that AllDeepSets and AllSetTransformer are both more expressive than any other aforementioned hypergraph neural network.
Revised: Together with the result of Theorem 3.4, we arrive at the conclusion that AllDeepSets and AllSetTransformer are both more expressive than any other aforementioned hypergraph neural network.

Original: Conceptually, one can also incorporate the Set Attention Block and the two steps pooling strategy of Lee et al.
Revised: Conceptually, one can also incorporate the Set Attention Block and the two steps pooling strategy of Lee et al.

Original: (2019) into AllSet.
Revised: (2019) into AllSet.

Original: However, it appears hard to make such an implementation efﬁcient and we hence leave this investigation as future work.
Revised: However, it appears hard to make such an implementation efﬁcient and we hence leave this investigation as future work.

Original: In practice, we ﬁnd that even our simpliﬁed design (8) already outperforms all baseline hypergraph neural networks, as demonstrated by our extensive experiments.
Revised: In practice, we ﬁnd that even our simpliﬁed design (8) already outperforms all baseline hypergraph neural networks, as demonstrated by our extensive experiments.

Original: Also, it is possible to utilize the information of the second argument of f V→E and f E→V via concatenation, another topic relegated to future work.
Revised: Also, it is possible to utilize the information of the second argument of f V→E and f E→V via concatenation, another topic relegated to future work.

Original: As a concluding remark, we observe that the multiset functions in both AllDeepSets (7) and AllSetTransformer (8) are universal approximators for general multiset functions.
Revised: As a concluding remark, we observe that the multiset functions in both AllDeepSets (7) and AllSetTransformer (8) are universal approximators for general multiset functions.

Original: In contrast, the other described hypergraph neural network layers fail to retain the universal approximation property for general multiset functions, as shown in Theorem 3.4.
Revised: In contrast, the other described hypergraph neural network layers fail to retain the universal approximation property for general multiset functions, as shown in Theorem 3.4.

Original: This implies that all these hypergraph neural network layers have strictly weaker expressive power compared to AllDeepSets (7) and AllSetTransformer (8).
Revised: This implies that all these hypergraph neural network layers have strictly weaker expressive power compared to AllDeepSets (7) and AllSetTransformer (8).

Original: We also note that any other universal approximators for multiset functions can easily be combined with our AllSet framework (as we already demonstrated by our Deep Sets and Set
Revised: We also note that any other universal approximators for multiset functions can easily be combined with our AllSet framework (as we already demonstrated by our Deep Sets and Set

Original: Transformer examples).
Revised: Transformer examples).

Original: One possible candidate is Janossy pooling (Murphy et al., 2019) which will be examined as part of our future work.
Revised: One possible candidate is Janossy pooling (Murphy et al., 2019) which will be examined as part of our future work.

Original: Note that more expressive models do not necessarily have better performance than less expressive models, as many other factors inﬂuence system performance (as an example, AllDeepSet performs worse than AllSetTransformer albeit both have the same theoretical expressive power; this is in agreement with the observation from Lee et al.
Revised: Note that more expressive models do not necessarily have better performance than less expressive models, as many other factors inﬂuence system performance (as an example, AllDeepSet performs worse than AllSetTransformer albeit both have the same theoretical expressive power; this is in agreement with the observation from Lee et al.

Original: (2019) that Set Transformer can learn multiset functions better than Deep Set.).
Revised: (2019) that Set Transformer can learn multiset functions better than Deep Set.).

Original: Nevertheless, we demonstrate in the experimental veriﬁcation section that our AllSetTransformer indeed has consistently better performance compared to other baseline methods.
Revised: Nevertheless, we demonstrate in the experimental veriﬁcation section that our AllSetTransformer indeed has consistently better performance compared to other baseline methods.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Due to space limit, the complete discussion on related works is relegated to the Appendix A.
Revised: Due to space limitations, a more comprehensive discussion of related work is relegated to the Appendix A.

Original: Hypergraph learning.
Revised: Hypergraph learning.

Original: Learning on hypergraphs has attracted signiﬁcant attention due to its ability to capture higher-order structural information (Li et al., 2017; 2019).
Revised: Learning on hypergraphs has attracted signiﬁcant attention due to its ability to capture higher-order structural information (Li et al., 2017; 2019).

Original: In the statistical learning and information theory literature, researchers have studied community detection problem for hypergraph stochastic block models (Ghoshdastidar & Dukkipati, 2015; Chien et al., 2018; 2019a).
Revised: In the statistical learning and information theory literature, researchers have studied community detection problem for hypergraph stochastic block models (Ghoshdastidar & Dukkipati, 2015; Chien et al., 2018; 2019a).

Original: In the machine learning community, methods that mitigate drawbacks of CE have been reported in (Li &
Revised: In the machine learning community, methods that mitigate drawbacks of CE have been reported in (Li &

Original: Milenkovic, 2017; Chien et al., 2019b; Hein et al., 2013; Chan et al., 2018).
Revised: Milenkovic, 2017; Chien et al., 2019b; Hein et al., 2013; Chan et al., 2018).

Original: Despite these advances, hypergraph neural network developments are still limited beyond the works covered by the AllSet paradigm.
Revised: Despite these advances, hypergraph neural network developments are still limited beyond the works covered by the AllSet paradigm.

Original: Exceptions include Hyper-SAGNN (Zhang et al., 2019), LEGCN (Yang et al., 2020) and
Revised: Exceptions include Hyper-SAGNN (Zhang et al., 2019), LEGCN (Yang et al., 2020) and

Original: MPNN-R (Yadati, 2020), the only three hypergraph neural networks that do not represent obvious special instances of AllSet.
Revised: MPNN-R (Yadati, 2020), the only three hypergraph neural networks that do not represent obvious special instances of AllSet.

Original: But, Hyper-SAGNN focuses on hyperedge prediction while we focus on semi-supervised node classiﬁcation.
Revised: But, Hyper-SAGNN focuses on hyperedge prediction while we focus on semi-supervised node classiﬁcation.

Original: LEGCN ﬁrst transforms hypergraphs into graphs via line expansion and then applies a standard GCN.
Revised: LEGCN ﬁrst transforms hypergraphs into graphs via line expansion and then applies a standard GCN.

Original: Although the transformation used in LEGCN does not cause a large distortion as CE, it signiﬁcantly increases the number of nodes and edges.
Revised: Although the transformation used in LEGCN does not cause a large distortion as CE, it signiﬁcantly increases the number of nodes and edges.

Original: Hence, it is not memory and computationally efﬁcient (more details about this approach can be found in the
Revised: Hence, it is not memory and computationally efﬁcient (more details about this approach can be found in the

Original: Appendix G).
Revised: Appendix G).

Original: MPNN-R focuses on recursive hypergraphs (Joslyn & Nowak, 2017), which is not the topic of this work.
Revised: MPNN-R focuses on recursive hypergraphs (Joslyn & Nowak, 2017), which is not the topic of this work.

Original: It is an interesting future work to investigate the relationship between AllSet and Hyper-SAGNN or MPNN-R when extending AllSet to hyperedge prediction tasks or recursive hypergraph implementations.
Revised: It is an interesting future work to investigate the relationship between AllSet and Hyper-SAGNN or MPNN-R when extending AllSet to hyperedge prediction tasks or recursive hypergraph implementations.

Original: Notably, Tudisco & Higham (2021);
Revised: Furthermore, Tudisco & Higham (2021);

Original: Tudisco et al.
Revised: Tudisco et al.

Original: (2021) also aim to develop and analyze hypergraph propagations that are “in-between” CE-prop and Z-prop.
Revised: (2021) also proposed to develop and analyze hypergraph propagations that operate “between” CE-prop and Z-prop rules.

Original: These works focus on the theoretical analysis of special types of propagation rules, which can again be modeled by our AllSet frameworks.
Revised: These works focus on the theoretical analysis of special types of propagation rules, which can also be modeled by our AllSet frameworks.

Original: In contrast, our AllSet layers aims to learn complex propagation rules  in an adaptive way.
Revised: Importantly, our AllSet layers do not ﬁx the rules but instead learn them in an adaptive manner.

Original: One can view Tudisco & Higham (2021);
Revised: Also, one can view Tudisco & Higham (2021);

Original: Tudisco et al.
Revised: Tudisco et al.

Original: (2021) as the hypergraph version of SGC, which can be helpful with regard to the interpretability of AllSet layers.
Revised: (2021) as the hypergraph version of SGC, which can help with interpreting the functionalities of AllSet layers.

Original: Star expansion, UniGNN and Heterogeneous GNNs.
Revised: Star expansion, UniGNN and Heterogeneous GNNs.

Original: A very recent work concurrent to ours,
Revised: A very recent work concurrent to ours,

Original: UniGNN (Huang & Yang, 2021), also proposes to unify hypergraph and GNN models.
Revised: UniGNN (Huang & Yang, 2021), also proposes to unify hypergraph and GNN models.

Original: Both Huang & Yang (2021) and our work can be related to hypergraph star expansion (Agarwal et al., 2006;
Revised: Both Huang & Yang (2021) and our work can be related to hypergraph star expansion (Agarwal et al., 2006;

Original: Yang et al., 2020), which results in a bipartite graph (see Figure 1).
Revised: Yang et al., 2020), which results in a bipartite graph (see Figure 1).

Original: One particular variant, UniGIN, is related to our AllDeepSets model, as both represent a hypergraph generalization of GIN (Xu et al., 2019).
Revised: One particular variant, UniGIN, is related to our AllDeepSets model, as both represent a hypergraph generalization of GIN (Xu et al., 2019).

Original: However, UniGNN does not make use of deep learning methods for learning multiset functions, which is crucial for identifying the most appropriate propagation and aggregation rules for individual datasets and learning tasks as done by AllSetTransformer (see Section 6 for supporting information).
Revised: However, UniGNN does not make use of deep learning methods for learning multiset functions, which is crucial for identifying the most appropriate propagation and aggregation rules for individual datasets and learning tasks as done by AllSetTransformer (see Section 6 for supporting information).

Original: Also, the most advanced variant of UniGNN, UniGCNII, is not comparable to AllSetTransformers.
Revised: Also, the most advanced variant of UniGNN, UniGCNII, is not comparable to AllSetTransformers.

Original: Furthermore, one could also naively try to apply heterogeneous GNNs, such as HAN (Wang et al., 2019b), to hypergraph datasets converted into bipartite graphs.
Revised: Furthermore, one could also naively try to apply heterogeneous GNNs, such as HAN (Wang et al., 2019b), to hypergraph datasets converted into bipartite graphs.

Original: This approach was not previously examined in the literature, but we experimentally tested it nevertheless to show that the performance improvements in this case are signiﬁcantly smaller than ours and that the method does not scale well even for moderately sized hypergraphs.
Revised: This approach was not previously examined in the literature, but we experimentally tested it nevertheless to show that the performance improvements in this case are signiﬁcantly smaller than ours and that the method does not scale well even for moderately sized hypergraphs.

Original: Another very recent work (Xue et al., 2021) appears at ﬁrst glance similar to ours, but it solves a very different problem by transforming a heterogeneous bipartite graph into a hypergraph and then apply standard GCN layers to implement f V→E and f E→V , which is more related to HNHN (Dong et al., 2020) and
Revised: Another very recent work (Xue et al., 2021) appears at ﬁrst glance similar to ours, but it solves a very different problem by transforming a heterogeneous bipartite graph into a hypergraph and then apply standard GCN layers to implement f V→E and f E→V , which is more related to HNHN (Dong et al., 2020) and

Original: UniGCN (Huang & Yang, 2021) and very different from AllSetTransformer.
Revised: UniGCN (Huang & Yang, 2021) and very different from AllSetTransformer.

Original: Learning (multi)set functions.
Revised: Learning (multi)set functions.

Original: (Multi)set functions, which are also known as pooling architectures for (multi)sets, have been used in numerous problems including causality discovery (Lopez-Paz et al., 2017), few-shot image classiﬁcation (Snell et al., 2017) and conditional regression and classiﬁcation (Garnelo et al., 2018).
Revised: (Multi)set functions, which are also known as pooling architectures for (multi)sets, have been used in numerous problems including causality discovery (Lopez-Paz et al., 2017), few-shot image classiﬁcation (Snell et al., 2017) and conditional regression and classiﬁcation (Garnelo et al., 2018).

Original: The authors of Zaheer et al.
Revised: The authors of Zaheer et al.

Original: (2017) provide a universal way to parameterize the (multi)set functions under some mild assumptions.
Revised: (2017) provide a universal way to parameterize the (multi)set functions under some mild assumptions.

Original: Similar results have been proposed independently by the authors of Qi et al.
Revised: Similar results have been proposed independently by the authors of Qi et al.

Original: (2017) for computer vision applications.
Revised: (2017) for computer vision applications.

Original: The authors of Lee et al.
Revised: The authors of Lee et al.

Original: (2019) propose to learn multiset functions via attention mechanisms.
Revised: (2019) propose to learn multiset functions via attention mechanisms.

Original: This further inspired the work in Baek et al.
Revised: This further inspired the work in Baek et al.

Original: (2021), which adopted the idea for graph representation learning.
Revised: (2021), which adopted the idea for graph representation learning.

Original: The authors of Wagstaff et al.
Revised: The authors of Wagstaff et al.

Original: (2019) discuss the limitation of Zaheer et al.
Revised: (2019) discuss the limitation of Zaheer et al.

Original: (2017), and speciﬁcally focus on continuous functions.
Revised: (2017), and speciﬁcally focus on continuous functions.

Original: A more general treatment of how to capture complex dependencies among multiset elements is available in Murphy et al.
Revised: A more general treatment of how to capture complex dependencies among multiset elements is available in Murphy et al.

Original: To the best of our knowledge, this work is the ﬁrst to build the connection between learning multiset functions with propagations on hypergraph.
Revised: To the best of our knowledge, this work is the ﬁrst to build the connection between learning multiset functions with propagations on hypergraph.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We focus on semi-supervised node classiﬁcation in the transductive setting.
Revised: We focus on semi-supervised node classiﬁcation in the transductive setting.

Original: We randomly split the data into training/validation/test samples using (50% / 25% / 25%) splitting percentages.
Revised: We randomly split the data into training/validation/test samples using (50% / 25% / 25%) splitting percentages.

Original: We aggregate the results of 20 experiments using multiple random splits and initializations.
Revised: We aggregate the results of 20 experiments using multiple random splits and initializations.

Original: Methods used for comparative studies.
Revised: Methods used for comparative studies.

Original: We compare our AllSetTransformer and AllDeepSets with ten baseline models, MLP, CE + GCN, CE + GAT, HGNN (Feng et al., 2019), HCHA (Bai et al., 2021), HyperGCN (Yadati et al., 2019), HNHN (Dong et al., 2020), UniGCNII (Huang & Yang, 2021) and HAN (Wang et al., 2019b), with both full batch and mini-batch settings.
Revised: We compare our AllSetTransformer and AllDeepSets with ten baseline models, MLP, CE + GCN, CE + GAT, HGNN (Feng et al., 2019), HCHA (Bai et al., 2021), HyperGCN (Yadati et al., 2019), HNHN (Dong et al., 2020), UniGCNII (Huang & Yang, 2021) and HAN (Wang et al., 2019b), with both full batch and mini-batch settings.

Original: All architectures are implemented using the Pytorch Geometric library (PyG) (Fey & Lenssen, 2019) except for HAN, in which case the implementation was retrieved from Deep Graph Library (DGL) (Wang et al., 2019a).
Revised: All architectures are implemented using the Pytorch Geometric library (PyG) (Fey & Lenssen, 2019) except for HAN, in which case the implementation was retrieved from Deep Graph Library (DGL) (Wang et al., 2019a).

Original: The implementation of HyperGCN 2 is used in the same form as reported in the ofﬁcial repository.
Revised: The implementation of HyperGCN 2 is used in the same form as reported in the ofﬁcial repository.

Original: We adapted the implementation of HCHA from PyG.
Revised: We adapted the implementation of HCHA from PyG.

Original: We also reimplemented HNHN and
Revised: We also reimplemented HNHN and

Original: HGNN using the PyG framework.
Revised: HGNN using the PyG framework.

Original: Note that in the original implementation of HGNN 3 , propagation is performed via matrix multiplication which is far less memory and computationally efﬁcient when compared to our implementation.
Revised: Note that in the original implementation of HGNN 3 , propagation is performed via matrix multiplication which is far less memory and computationally efﬁcient when compared to our implementation.

Original: MLP, GCN and GAT are executed directly from PyG.
Revised: MLP, GCN and GAT are executed directly from PyG.

Original: The
Revised: The

Original: UniGCNII code is taken from the ofﬁcial site 4 .
Revised: UniGCNII code is taken from the ofﬁcial site 4 .

Original: For HAN, we tested both the full batch training setting and a mini-batch setting provided by DGL.
Revised: For HAN, we tested both the full batch training setting and a mini-batch setting provided by DGL.

Original: We treated the vertices ( V ) and hyperedges ( E ) as two heterogeneous types of nodes, and used two metapaths: {V → E → V , E → V → E} in its semantic attention.
Revised: We treated the vertices ( V ) and hyperedges ( E ) as two heterogeneous types of nodes, and used two metapaths: {V → E → V , E → V → E} in its semantic attention.

Original: In the mini-batch setting, we used the DGL’s default neighborhood sampler with a number of neighbors set to 32 during training and 64 for validation and testing purposes.
Revised: In the mini-batch setting, we used the DGL’s default neighborhood sampler with a number of neighbors set to 32 during training and 64 for validation and testing purposes.

Original: In both settings, due to the speciﬁc conversion of hyperedges into nodes, our evaluations only involve the original labeled vertices in the hypergraph and ignore the hyperedge node-proxies.
Revised: In both settings, due to the speciﬁc conversion of hyperedges into nodes, our evaluations only involve the original labeled vertices in the hypergraph and ignore the hyperedge node-proxies.

Original: More details regarding the experimental settings and the choices of hyperparameters are given in the Appendix J.
Revised: More details regarding the experimental settings and the choices of hyperparameters are given in the Appendix J.

Original: Benchmarking and new datasets.
Revised: Benchmarking and new datasets.

Original: We use ten available datasets from the existing hypergraph neural networks literature and three newly curated datasets from different application domains.
Revised: We use ten available datasets from the existing hypergraph neural networks literature and three newly curated datasets from different application domains.

Original: The benchmark datasets include cocitation networks Cora, Citeseer and Pubmed, downloaded from Yadati et al.
Revised: The benchmark datasets include cocitation networks Cora, Citeseer and Pubmed, downloaded from Yadati et al.

Original: The coauthorship networks Cora-CA and DBLP-CA were adapted from Yadati et al.
Revised: The coauthorship networks Cora-CA and DBLP-CA were adapted from Yadati et al.

Original: We also tested datasets from the UCI Categorical Machine Learning Repository (Dua & Graff, 2017), including 20Newsgroups, Mushroom and Zoo.
Revised: We also tested datasets from the UCI Categorical Machine Learning Repository (Dua & Graff, 2017), including 20Newsgroups, Mushroom and Zoo.

Original: Datasets from the area of computer vision and computer graphics include ModelNet40 (Wu et al., 2015) and NTU2012 (Chen et al., 2003).
Revised: Datasets from the area of computer vision and computer graphics include ModelNet40 (Wu et al., 2015) and NTU2012 (Chen et al., 2003).

Original: The hypergraph construction follows the recommendations from Feng et al.
Revised: The hypergraph construction follows the recommendations from Feng et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: The three newly introduced datasets are adaptations of Yelp (Yelp), House (Chodrow et al., 2021) and Walmart (Amburg et al., 2020).
Revised: The three newly introduced datasets are adaptations of Yelp (Yelp), House (Chodrow et al., 2021) and Walmart (Amburg et al., 2020).

Original: Since there are no node features in the original datasets of House and Walmart, we use Gaussian random vectors instead, in a fashion similar to what was proposed for Contextual stochastic block models (Deshpande et al., 2018).
Revised: Since there are no node features in the original datasets of House and Walmart, we use Gaussian random vectors instead, in a fashion similar to what was proposed for Contextual stochastic block models (Deshpande et al., 2018).

Original: We use postﬁx ( x ) to indicate the standard deviation of the added Gaussian features.
Revised: We use postﬁx ( x ) to indicate the standard deviation of the added Gaussian features.

Original: The partial statistics of all datasets are provided in Table 1, and more details are available in the Appendix I.
Revised: The partial statistics of all datasets are provided in Table 1, and more details are available in the Appendix I.

Original: Results.
Revised: Results.

Original: We use accuracy (micro-F1 score) as the evaluation metric, along with its standard deviation.
Revised: We use accuracy (micro-F1 score) as the evaluation metric, along with its standard deviation.

Original: The relevant ﬁndings are summarized in Table 2.
Revised: The relevant ﬁndings are summarized in Table 2.

Original: The results show that AllSetTransformer is the most robust hypergraph neural network model and that it has the best overall performance when compared to state-of-the art and new models such a hypergraph HAN.
Revised: The results show that AllSetTransformer is the most robust hypergraph neural network model and that it has the best overall performance when compared to state-of-the art and new models such a hypergraph HAN.

Original: In contrast, all baseline methods perform poorly on at least two datasets.
Revised: In contrast, all baseline methods perform poorly on at least two datasets.

Original: For example, UniGCNII and HAN are two of the best performing baseline models according to our experiments.
Revised: For example, UniGCNII and HAN are two of the best performing baseline models according to our experiments.

Original: However, UniGCNII performs poorly on Zoo, Yelp and Walmart when compared to our AllSetTransformer.
Revised: However, UniGCNII performs poorly on Zoo, Yelp and Walmart when compared to our AllSetTransformer.

Original: More precisely, AllSetTransformer outperforms UniGCNII by 11 .
Revised: More precisely, AllSetTransformer outperforms UniGCNII by 11 .

Original: 01% in accuracy on the Walmart(1) dataset.
Revised: 01% in accuracy on the Walmart(1) dataset.

Original: HAN has poor performance when compared AllSetTransformer on Mushroom, NTU2012 and ModelNet40.
Revised: HAN has poor performance when compared AllSetTransformer on Mushroom, NTU2012 and ModelNet40.

Original: Fur-
Revised: Fur-

Original: [Table]
Revised: [Table]

Original: thermore, it experiences memory overload issues on Yelp and Walmart with a full batch setting and performs poorly with a mini-batch setting on the same datasets.
Revised: thermore, it experiences memory overload issues on Yelp and Walmart with a full batch setting and performs poorly with a mini-batch setting on the same datasets.

Original: Numerically, AllSetTransformer outperforms HAN with full batch setting by 9 .
Revised: Numerically, AllSetTransformer outperforms HAN with full batch setting by 9 .

Original: 14% on the Mushroom and HAN with mini-batch setting and by 16 .
Revised: 14% on the Mushroom and HAN with mini-batch setting and by 16 .

Original: 89% on Walmart(1).
Revised: 89% on Walmart(1).

Original: These ﬁndings emphasize the importance of including diverse datasets from different application domains when trying to test hypergraph neural networks fairly.
Revised: These ﬁndings emphasize the importance of including diverse datasets from different application domains when trying to test hypergraph neural networks fairly.

Original: Testing standard benchmark datasets such as Cora, Citeseer and Pubmed alone is insufﬁcient and can lead to biased conclusions.
Revised: Testing standard benchmark datasets such as Cora, Citeseer and Pubmed alone is insufﬁcient and can lead to biased conclusions.

Original: Indeed, our experiments show that all hypergraph neural networks work reasonably well on these three datasets.
Revised: Indeed, our experiments show that all hypergraph neural networks work reasonably well on these three datasets.

Original: Our results also demonstrate the power of Set
Revised: Our results also demonstrate the power of Set

Original: Transformer when applied to hypergraph learning via the AllSet framework.
Revised: Transformer when applied to hypergraph learning via the AllSet framework.

Original: The execution times of all methods are available in the Appendix J: They show that the complexity of
Revised: The execution times of all methods are available in the Appendix J: They show that the complexity of

Original: AllSet is comparable to that of the baseline hypergraph neural networks.
Revised: AllSet is comparable to that of the baseline hypergraph neural networks.

Original: This further strengthens the case for AllSet, as its performance gains do not come at the cost of high computational complexity.
Revised: This further strengthens the case for AllSet, as its performance gains do not come at the cost of high computational complexity.

Original: Note that in HAN’s mini-batch setting, the neighborhood sampler is performed on CPU instead of
Revised: Note that in HAN’s mini-batch setting, the neighborhood sampler is performed on CPU instead of

Original: GPU, hence its training time is signiﬁcantly higher than that needed by other methods, due to the frequent I/O operation between CPU and GPU.
Revised: GPU, hence its training time is signiﬁcantly higher than that needed by other methods, due to the frequent I/O operation between CPU and GPU.

Original: On some larger datasets such as Yelp and Walmart, 20 runs take more than 24 hours so we only recorded the results for ﬁrst 10 runs.
Revised: On some larger datasets such as Yelp and Walmart, 20 runs take more than 24 hours so we only recorded the results for ﬁrst 10 runs.

Original: The performance of AllDeepSets is not on par with that of AllSetTransformer, despite the fact that both represent universal approximators of the general AllSet formalism.
Revised: The performance of AllDeepSets is not on par with that of AllSetTransformer, despite the fact that both represent universal approximators of the general AllSet formalism.

Original: This result conﬁrms the assessment of Lee et al.
Revised: This result conﬁrms the assessment of Lee et al.

Original: that attention mechanisms are crucial for learning multiset functions in practice.
Revised: that attention mechanisms are crucial for learning multiset functions in practice.

Original: Furthermore, we note that combining CE with existing GNNs such as GCN and GAT leads to suboptimal performance.
Revised: Furthermore, we note that combining CE with existing GNNs such as GCN and GAT leads to suboptimal performance.

Original: CE-based approaches are also problematic in terms of memory efﬁciency when large hyperedges are present.
Revised: CE-based approaches are also problematic in terms of memory efﬁciency when large hyperedges are present.

Original: This is due to the fact the CE of a hyperedge e leads to Θ( | e | 2 ) edges in the resulting graph.
Revised: This is due to the fact the CE of a hyperedge e leads to Θ( | e | 2 ) edges in the resulting graph.

Original: Note that as indicated in Table 2, CE-based methods went out-of-memory (OOM) for 20Newsgroups and Yelp.
Revised: Note that as indicated in Table 2, CE-based methods went out-of-memory (OOM) for 20Newsgroups and Yelp.

Original: These two datasets have the largest maximum hyperedge size, as can be see from Table 1.
Revised: These two datasets have the largest maximum hyperedge size, as can be see from Table 1.

Original: HAN encounters the OOM issue on even more datasets when used in the full batch setting, and its mini-batch setting mode perform poorly on Yelp and
Revised: HAN encounters the OOM issue on even more datasets when used in the full batch setting, and its mini-batch setting mode perform poorly on Yelp and

Original: Walmart.
Revised: Walmart.

Original: This shows that a naive application of standard heterogeneous GNNs on large hypergraphs often fails and is thus not as robust as our AllSetTransformer.
Revised: This shows that a naive application of standard heterogeneous GNNs on large hypergraphs often fails and is thus not as robust as our AllSetTransformer.

Original: The mediator HyperGCN approach exhibits numerical instabilities on some datasets (i.e., Zoo and House) and may hence not be suitable for learning on general hypergraph datasets.
Revised: The mediator HyperGCN approach exhibits numerical instabilities on some datasets (i.e., Zoo and House) and may hence not be suitable for learning on general hypergraph datasets.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: The authors would like to thank  Pan Li  for helpful discussions.
Revised: The authors would like to thank Prof. Pan Li at Purdue University for helpful discussions.

Original: The authors would  like to thank Chaoqi Yang for answering questions about the LEGCN paper.
Revised: The authors would also like to thank Chaoqi Yang for answering questions regarding the LEGCN method.

Original: This work was funded by the NSF grant 1956384.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: We do not aware of any potential ethical issues regarding our work.
Revised: We do not aware of any potential ethical issues regarding our work.

Original: For the newly curated three datasets, there is no private personally identiﬁable information.
Revised: For the newly curated three datasets, there is no private personally identiﬁable information.

Original: Note that although nodes in House datasets represent congresspersons and hyperedges represent members serving on the same committee, this information is public and not subject to privacy constraints.
Revised: Note that although nodes in House datasets represent congresspersons and hyperedges represent members serving on the same committee, this information is public and not subject to privacy constraints.

Original: For the Walmart dataset, the nodes are products and hyperedges are formed by sets of products purchased together.
Revised: For the Walmart dataset, the nodes are products and hyperedges are formed by sets of products purchased together.

Original: We do not include any personal information regarding the buyers (customers) in the text.
Revised: We do not include any personal information regarding the buyers (customers) in the text.

Original: In the Yelp dataset the nodes represent restaurants and hyperedges are collections of restaurants visited by the same user.
Revised: In the Yelp dataset the nodes represent restaurants and hyperedges are collections of restaurants visited by the same user.

Original: Similar to the Walmart case, there is no user information included in the dataset.
Revised: Similar to the Walmart case, there is no user information included in the dataset.

Original: More details of how the features in each datasets are generated can be found in Appendix I.
Revised: More details of how the features in each datasets are generated can be found in Appendix I.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We have tried our best to ensure reproducibility of our results.
Revised: We have tried our best to ensure reproducibility of our results.

Original: We built a succinct pipeline for standardization of the hypergraph neural networks evaluation process based on Pytorch Geometric.
Revised: We built a succinct pipeline for standardization of the hypergraph neural networks evaluation process based on Pytorch Geometric.

Original: This implementation can be checked by referring to our supplementary material.
Revised: This implementation can be checked by referring to our supplementary material.

Original: We include all dataset in our supplementary material and integrate all tested methods in our code.
Revised: We include all dataset in our supplementary material and integrate all tested methods in our code.

Original: Hence, one can simply reproduce all experiment results effortlessly (by just running a one-line command).
Revised: Hence, one can simply reproduce all experiment results effortlessly (by just running a one-line command).

Original: All details regarding how the datasets were prepared are stated in Appendix I. All choices of hyperparameters and the process of selecting them are available in Appendix J. Our experimental settings including the speciﬁcations of our machine and environment, the training/validation/test split and the evaluation metric.
Revised: All details regarding how the datasets were prepared are stated in Appendix I. All choices of hyperparameters and the process of selecting them are available in Appendix J. Our experimental settings including the speciﬁcations of our machine and environment, the training/validation/test split and the evaluation metric.

Original: Once again, refer ro the Appendix J and Section 6.
Revised: Once again, refer ro the Appendix J and Section 6.

Original: We also clearly specify all the package dependencies used in our code in the Supplementary material.
Revised: We also clearly specify all the package dependencies used in our code in the Supplementary material.

Original: A PPENDIX A A DDITIONAL R ELATED WORKS
Revised: A PPENDIX A A DDITIONAL R ELATED WORKS

Original: We further include more related works for completeness.
Revised: We further discuss related works for completeness.

Original: Unfortunately, we can only include them in the Appendix due to space limit.

Original: Graph neural networks on graphs.
Revised: Graph neural networks on graphs.

Original: GNNs have received signiﬁcant attention in the past, often under the umbrella of Geometric Deep Learning (Bronstein et al., 2017).
Revised: GNNs have received signiﬁcant attention in the past, often under the umbrella of Geometric Deep Learning (Bronstein et al., 2017).

Original: Many successful architectures have been proposed for various tasks on graphs, including GCN (Kipf & Welling, 2017),
Revised: Many successful architectures have been proposed for various tasks on graphs, including GCN (Kipf & Welling, 2017),

Original: GAT (Vaswani et al., 2017), GraphSAGE (Hamilton et al., 2017), GIN (Xu et al., 2019) and many others (Klicpera et al., 2019; Wu et al., 2019; Velickovic et al., 2019; Li et al., 2020; Xu et al., 2018;
Revised: GAT (Vaswani et al., 2017), GraphSAGE (Hamilton et al., 2017), GIN (Xu et al., 2019) and many others (Klicpera et al., 2019; Wu et al., 2019; Velickovic et al., 2019; Li et al., 2020; Xu et al., 2018;

Original: Chien et al., 2021b).
Revised: Chien et al., 2021b).

Original: There have also been some recent attempts to use Transformers in GNNs (Baek et al., 2021;
Revised: There have also been some recent attempts to use Transformers in GNNs (Baek et al., 2021;

Original: Yun et al., 2019;
Revised: Yun et al., 2019;

Original: Hu et al., 2020b).
Revised: Hu et al., 2020b).

Original: Nevertheless, all these methods only apply to graphs and not to hypergraphs.
Revised: Nevertheless, all these methods only apply to graphs and not to hypergraphs.

Original: Although CE can be used in all these case, this is clearly not an optimal strategy due to the previously mentioned distortion issues.
Revised: Although CE can be used in all these case, this is clearly not an optimal strategy due to the previously mentioned distortion issues.

Original: There are also many techniques that can improve GNNs in various directions.
Revised: There are also many techniques that can improve GNNs in various directions.

Original: For example, PairNorm (Zhao & Akoglu, 2019) and DropEdge (Rong et al., 2019) allow one to build deeper GNNs.
Revised: For example, PairNorm (Zhao & Akoglu, 2019) and DropEdge (Rong et al., 2019) allow one to build deeper GNNs.

Original: ClusterGCN (Chiang et al., 2019) and
Revised: ClusterGCN (Chiang et al., 2019) and

Original: GraphSAINT (Zeng et al., 2019), on the other hand, may be used to signiﬁcantly scale up GNNs.
Revised: GraphSAINT (Zeng et al., 2019), on the other hand, may be used to signiﬁcantly scale up GNNs.

Original: Residual correlation (Jia & Benson, 2020) and CopulaGNN (Ma et al., 2021) have been shown to improve GNNs on graph regression problems.
Revised: Residual correlation (Jia & Benson, 2020) and CopulaGNN (Ma et al., 2021) have been shown to improve GNNs on graph regression problems.

Original: These works highlight new directions and limitations of not only our method, but also hypergraph neural networks in general.
Revised: These works highlight new directions and limitations of not only our method, but also hypergraph neural networks in general.

Original: Nevertheless, we believe that AllSet layers can be adapted to resolve these issues using similar ideas as those proposed for graph layers.
Revised: Nevertheless, we believe that AllSet layers can be adapted to resolve these issues using similar ideas as those proposed for graph layers.

Original: Learning with hypergraphs.
Revised: Additional contributions to learning on hypergraphs.

Original: Some recent works also share a few similar ideas with AllSet.
Revised: After the submission of our manuscript, we were made aware of some loosely related lines of works.

Original: Ding et al.
Revised: Ding et al.

Original: (2020) and Wang et al.
Revised: (2020) and Wang et al.

Original: (2021) proposed hypergraph attention networks based on the V → E and E → V formulation,where they specialized to speciﬁc downstream tasks.
Revised: (2021) proposed hypergraph attention networks based on the V → E and E → V formulation, specialized for certain downstream tasks.

Original: However, they can again be modeled by our AllSet framework and have less expressiveness compare to AllSet.
Revised: However, these approaches fall under our AllSet framework and have smaller expressiveness compared to AllSet.

Original:  Jo et al.
Revised: The work Jo et al.

Original: (2021) proposed to learn edge representations in a graph via message passing on its dual hypergraph.
Revised: (2021) proposes to learn edge representations in a graph via message passing on its dual hypergraph.

Original: Although they used GMT (Baek et al., 2021) as their node pooling module, they do not explore the idea of treating f V→E and f E→V as multiset functions for hypergraph neural networks as in our AllSet.
Revised: Although the authors used GMT (Baek et al., 2021) as their node pooling module, they do not explore the idea of treating f V→E and f E→V as multiset functions for hypergraph neural networks as AllSet does.

Original: Finally, it is also possible to deﬁne propagation schemes based on other tensor eigenproblems, such as H eigenproblems (Qi & Luo, 2017).
Revised: It is also possible to deﬁne propagation schemes based on other tensor eigenproblems, such as H eigenproblems (Qi & Luo, 2017).

Original: Deﬁning propagation rules based on H eigenproblems can lead to imaginary features which is problematic:
Revised: Deﬁning propagation rules based on H eigenproblems can lead to imaginary features which is problematic:

Original: A more detailed discussion regarding this direction is available in the Appendix H.
Revised: A more detailed discussion regarding this direction is available in the Appendix H.

Original: B C ONCLUSIONS
Revised: B C ONCLUSIONS

Original: We proposed AllSet, a novel hypergraph neural network paradigm that represents a highly general framework for hypergraph neural networks.
Revised: We proposed AllSet, a novel hypergraph neural network paradigm that represents a highly general framework for hypergraph neural networks.

Original: We implemented hypergraph neural network layers as compositions of two multiset functions that can be efﬁciently learned for each task and each dataset.
Revised: We implemented hypergraph neural network layers as compositions of two multiset functions that can be efﬁciently learned for each task and each dataset.

Original: Furthermore, for the ﬁrst time, we integrated Deep Set and Set Transformer methods within hypergraph neural networks for the purpose of learning the described multiset functions.
Revised: Furthermore, for the ﬁrst time, we integrated Deep Set and Set Transformer methods within hypergraph neural networks for the purpose of learning the described multiset functions.

Original: Our theoretical analysis demonstrated that most of the previous hypergraph neural networks are strictly less expressive then our proposed AllSet framework.
Revised: Our theoretical analysis demonstrated that most of the previous hypergraph neural networks are strictly less expressive then our proposed AllSet framework.

Original: We conducted the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent significant challenges for hypergraph node classiﬁcation.
Revised: We conducted the most extensive experiments to date involving ten known benchmarking datasets and three newly curated datasets that represent significant challenges for hypergraph node classiﬁcation.

Original: The results showed that our method has the unique ability to either match or outperform all other baselines, including the state-of-the-art hypergraph neural networks UniGNN (Huang & Yang, 2021).
Revised: The results showed that our method has the unique ability to either match or outperform all other baselines, including the state-of-the-art hypergraph neural networks UniGNN (Huang & Yang, 2021).

Original: Remarkably, our baseline also included
Revised: Remarkably, our baseline also included

Original: HAN (Wang et al., 2019b), the heterogeneous hypergraph neural networks, via star expansion as the new comparative method in the hypergraph neural network literature for the ﬁrst time.
Revised: HAN (Wang et al., 2019b), the heterogeneous hypergraph neural networks, via star expansion as the new comparative method in the hypergraph neural network literature for the ﬁrst time.

Original: Our results show that leveraging powerful multiset function learners in our AllSet framework is indeed helpful for hypergraph learning.
Revised: Our results show that leveraging powerful multiset function learners in our AllSet framework is indeed helpful for hypergraph learning.

Original: As a promising future direction, more advanced multiset function learners such as Janossy pooling (Murphy et al., 2019) can be used in our AllSet framework and potentially further improve the performance.
Revised: As a promising future direction, more advanced multiset function learners such as Janossy pooling (Murphy et al., 2019) can be used in our AllSet framework and potentially further improve the performance.

Original: We left this topic in future works.
Revised: We left this topic in future works.

Original: C P ROOF OF T HEOREM 3. 
Revised: C P ROOF OF T HEOREM 3.

Original: First, we show that one can recover CEpropH (1) from AllSet (5).
Revised: First, we show that one can recover CEpropH (1) from AllSet (5).

Original: By choosing f V→E and f E→V to be the sums over appropriate input multisets and by ignoring the second input, we obtain
Revised: By choosing f V→E and f E→V to be the sums over appropriate input multisets and by ignoring the second input, we obtain

Original: [Equation]
Revised: [Equation]

Original: Combining these two equations gives
Revised: Combining these two equations gives

Original: [Equation]
Revised: [Equation]

Original: which matches the update equations of CEpropH (1)
Revised: which matches the update equations of CEpropH (1)

Original: [Equation]
Revised: [Equation]

Original: Next, we prove that one can recover CEpropA (1) from AllSet (6).
Revised: Next, we prove that one can recover CEpropA (1) from AllSet (6).

Original: We once again choose f V→E and f E→V to be the sums over appropriate input multisets and ignore the second input term.
Revised: We once again choose f V→E and f E→V to be the sums over appropriate input multisets and ignore the second input term.

Original: This results in the following update equations:
Revised: This results in the following update equations:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining the two equations we obtain
Revised: Combining the two equations we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which represents the update rule of CEpropA from (1).
Revised: which represents the update rule of CEpropA from (1).

Original: In the last step, we prove that one can recover Zprop (4) from AllSet (6).
Revised: In the last step, we prove that one can recover Zprop (4) from AllSet (6).

Original: By choosing f V→E and f E→V to be the product and sum over appropriate input multisets, respectively, and by ignore the second input, we have:
Revised: By choosing f V→E and f E→V to be the product and sum over appropriate input multisets, respectively, and by ignore the second input, we have:

Original: [Equation]
Revised: [Equation]

Original: Combining these two equations we arrive at
Revised: Combining these two equations we arrive at

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which matches Zprop from (4)
Revised: which matches Zprop from (4)

Original: This completes the proof.
Revised: This completes the proof.

Original: D P ROOF OF T HEOREM 3. 
Revised: D P ROOF OF T HEOREM 3.

Original: First we prove that the hypergraph neural network layer of HGNN (2) is a special instance of
Revised: First we prove that the hypergraph neural network layer of HGNN (2) is a special instance of

Original: AllSet (5).
Revised: AllSet (5).

Original: By choosing f V→E and f E→V to be the weighted sum over appropriate input multisets, where the weights are chosen according to the node degree, edge degree, and edge weights, and by ignore the second input, we have:
Revised: By choosing f V→E and f E→V to be the weighted sum over appropriate input multisets, where the weights are chosen according to the node degree, edge degree, and edge weights, and by ignore the second input, we have:

Original: [Equation]
Revised: [Equation]

Original: Since f V→E and f E→V both use the hypergraph G as their input, the functions can use the node degrees, edge degrees and edge weights as their arguments.
Revised: Since f V→E and f E→V both use the hypergraph G as their input, the functions can use the node degrees, edge degrees and edge weights as their arguments.

Original: Also, it is not hard to see that our choices for f V→E (18) and f E→V (19) are permutation invariant in V e, X ( t ) and E v, Z ( t +1) , respectively.
Revised: Also, it is not hard to see that our choices for f V→E (18) and f E→V (19) are permutation invariant in V e, X ( t ) and E v, Z ( t +1) , respectively.

Original: Plugging (18) into (19) leads to:
Revised: Plugging (18) into (19) leads to:

Original: [Equation]
Revised: [Equation]

Original: which represents the HGNN layer (2).
Revised: which represents the HGNN layer (2).

Original: For the HCHA layer, we have a node-wise formulation that reads as follows:
Revised: For the HCHA layer, we have a node-wise formulation that reads as follows:

Original: [Equation]
Revised: [Equation]

Original: where the attention weight α ( t ) ue depends on the node features X ( t ) and the feature of hyperedge e .
Revised: where the attention weight α ( t ) ue depends on the node features X ( t ) and the feature of hyperedge e .

Original: Here, σ ( · ) is a nonlinear activation function such as LeakyReLU and eLU.
Revised: Here, σ ( · ) is a nonlinear activation function such as LeakyReLU and eLU.

Original: An analysis similar to that described for HGNN can be used in this case as well.
Revised: An analysis similar to that described for HGNN can be used in this case as well.

Original: The only difference is in the choice of the attention weights α ue
Revised: The only difference is in the choice of the attention weights α ue

Original: and α ve .
Revised: and α ve .

Original: The deﬁnition of attention weights for HCHA is presented below.
Revised: The deﬁnition of attention weights for HCHA is presented below.

Original: [Equation]
Revised: [Equation]

Original: Clearly, the attention function in (22) has V e, X ( t ) and Z ( t ) e, : as its arguments.
Revised: Clearly, the attention function in (22) has V e, X ( t ) and Z ( t ) e, : as its arguments.

Original: Furthermore, we can choose f V→E and f E→V as:
Revised: Furthermore, we can choose f V→E and f E→V as:

Original: [Equation]
Revised: [Equation]

Original: Note that both (23) and (24) are permutation invariant, which means that they are valid choices for f V→E and f E→V .
Revised: Note that both (23) and (24) are permutation invariant, which means that they are valid choices for f V→E and f E→V .

Original: Combining these two equations reproduces the HCHA layer of (21).
Revised: Combining these two equations reproduces the HCHA layer of (21).

Original: For the HyperGCN layer, ignoring the degree normalization, one can use the following formulation:
Revised: For the HyperGCN layer, ignoring the degree normalization, one can use the following formulation:

Original: [Equation]
Revised: [Equation]

Original: Here, the weight w ( t ) uv,e depends on all node features within the hyperedge { X ( t ) u, : : u ∈ e } , and σ ( · ) is a nonlinear activation function (for example, ReLU).
Revised: Here, the weight w ( t ) uv,e depends on all node features within the hyperedge { X ( t ) u, : : u ∈ e } , and σ ( · ) is a nonlinear activation function (for example, ReLU).

Original: The same analysis as presented for the previous cases may be used in this case as well: The only difference lies in the choice of the weights w ( t ) uv,e .
Revised: The same analysis as presented for the previous cases may be used in this case as well: The only difference lies in the choice of the weights w ( t ) uv,e .

Original: According to the original HyperGCN paper (Yadati et al., 2019), these weights are deﬁned as
Revised: According to the original HyperGCN paper (Yadati et al., 2019), these weights are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ( i e , j e ) = argmax u,v ∈ e (cid:107) ( X ( t ) u, : − X ( t ) v, : ) Θ ( t ) (cid:107) .
Revised: where ( i e , j e ) = argmax u,v ∈ e (cid:107) ( X ( t ) u, : − X ( t ) v, : ) Θ ( t ) (cid:107) .

Original: [Equation]
Revised: [Equation]

Original: Again, it is straightforward to see that w ( t ) uv,e is a function of V e \ v, X ( t ) and X ( t ) v, : .
Revised: Again, it is straightforward to see that w ( t ) uv,e is a function of V e \ v, X ( t ) and X ( t ) v, : .

Original: Also, it is permutation invariant with respect to V e \ v, X ( t ) .
Revised: Also, it is permutation invariant with respect to V e \ v, X ( t ) .

Original: Hence, we can choose f V→E and f E→V as
Revised: Hence, we can choose f V→E and f E→V as

Original: [Equation]
Revised: [Equation]

Original: Combining the two equations leads to the HyperGCN layer from (25).
Revised: Combining the two equations leads to the HyperGCN layer from (25).

Original: Next, we show that HNHN layer introduced in Dong et al.
Revised: Next, we show that HNHN layer introduced in Dong et al.

Original: (2020) is a special case of AllSet (5).
Revised: (2020) is a special case of AllSet (5).

Original: The deﬁnition of HNHN layer is as follows
Revised: The deﬁnition of HNHN layer is as follows

Original: [Equation]
Revised: [Equation]

Original: where d e,l,β = (cid:88)
Revised: where d e,l,β = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Note that α and β are two hyperparameters that can be tuned in HNHN.
Revised: Note that α and β are two hyperparameters that can be tuned in HNHN.

Original: As before, σ ( · ) is a nonlinear activation function (e.g., ReLU).
Revised: As before, σ ( · ) is a nonlinear activation function (e.g., ReLU).

Original: It is obvious that we can choose f V→E and f E→V according to (30) and (31), respectively.
Revised: It is obvious that we can choose f V→E and f E→V according to (30) and (31), respectively.

Original: This is due to the fact that these expressions only involves degree normalizations and linear transformations.
Revised: This is due to the fact that these expressions only involves degree normalizations and linear transformations.

Original: Finally, we show that the HyperSAGE layer from Arya et al.
Revised: Finally, we show that the HyperSAGE layer from Arya et al.

Original: (2020) is a special case of AllSet (5).
Revised: (2020) is a special case of AllSet (5).

Original: The HyperSAGE layer update rules are as follows:
Revised: The HyperSAGE layer update rules are as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 35) where σ ( · ) is a nonlinear activation function.
Revised: 35) where σ ( · ) is a nonlinear activation function.

Original: The update (33) can be recovered by simply choosing f V→E to be the l p (power) mean.
Revised: The update (33) can be recovered by simply choosing f V→E to be the l p (power) mean.

Original: For the f E→V component, we ﬁrst model f E→V as a composition of another two multiset functions.
Revised: For the f E→V component, we ﬁrst model f E→V as a composition of another two multiset functions.

Original: The ﬁrst is the l p (power) mean with respect to its ﬁrst input, while the second is addition with respect to the second input.
Revised: The ﬁrst is the l p (power) mean with respect to its ﬁrst input, while the second is addition with respect to the second input.

Original: This recovers (34).
Revised: This recovers (34).

Original: The second of the two deﬁning functions can be chosen as (35), which is also a multiset function.
Revised: The second of the two deﬁning functions can be chosen as (35), which is also a multiset function.

Original: This procedure leads to f E→V .
Revised: This procedure leads to f E→V .

Original: This completes the proof of the ﬁrst claim pertaining to the universality of AllSet.
Revised: This completes the proof of the ﬁrst claim pertaining to the universality of AllSet.

Original: To address the claim that the above described hypergrpah neural network layers have strictly smaller expressive power then AllSet, one only has to observe that these hypergraph neural network layers cannot approximate arbitrary multiset functions in either the f V→E or f E→V component.
Revised: To address the claim that the above described hypergrpah neural network layers have strictly smaller expressive power then AllSet, one only has to observe that these hypergraph neural network layers cannot approximate arbitrary multiset functions in either the f V→E or f E→V component.

Original: We discuss both these functions separately.
Revised: We discuss both these functions separately.

Original: With regards to the HGNN layer, it is clear that two linear transformations cannot model arbitrary multiset functions such as the product in Zprop (4).
Revised: With regards to the HGNN layer, it is clear that two linear transformations cannot model arbitrary multiset functions such as the product in Zprop (4).

Original: For the HCHA layer, we note that if all node features are identical and the hyperedge features are all-zero vectors, all the attention weights α ue are the same.
Revised: For the HCHA layer, we note that if all node features are identical and the hyperedge features are all-zero vectors, all the attention weights α ue are the same.

Original: This setting is similar to that described for the HGNN layer and thus this rule cannot model arbitrary multiset functions.
Revised: This setting is similar to that described for the HGNN layer and thus this rule cannot model arbitrary multiset functions.

Original: For the HyperGCN layer (25), consider a 3 -uniform hypergraph as its input.
Revised: For the HyperGCN layer (25), consider a 3 -uniform hypergraph as its input.

Original: In this case, by deﬁnition, the weights w uv,e are all equal.
Revised: In this case, by deﬁnition, the weights w uv,e are all equal.

Original: Again, using only linear transformations one cannot model the product operation in Zprop (4); the same argument holds even when degree normalizations are included.
Revised: Again, using only linear transformations one cannot model the product operation in Zprop (4); the same argument holds even when degree normalizations are included.

Original: For the HNHN layer, note that the E → V component of (31) is just one MLP layer that follows the sum (cid:80) e : v ∈ e , which cannot model arbitrary multiset functions based on the results of Deep Sets (Zaheer et al., 2017).
Revised: For the HNHN layer, note that the E → V component of (31) is just one MLP layer that follows the sum (cid:80) e : v ∈ e , which cannot model arbitrary multiset functions based on the results of Deep Sets (Zaheer et al., 2017).

Original: As a ﬁnal note, we point out that the
Revised: As a ﬁnal note, we point out that the

Original: HyperSAGE layer involves only one learnable matrix Θ and is hence also a linear transformation, which cannot model arbitrary multiset functions.
Revised: HyperSAGE layer involves only one learnable matrix Θ and is hence also a linear transformation, which cannot model arbitrary multiset functions.

Original: This completes the proof.
Revised: This completes the proof.

Original: E P ROOF OF T HEOREM 3. 
Revised: E P ROOF OF T HEOREM 3.

Original: The propagation rule of MPNN reads as follows:
Revised: The propagation rule of MPNN reads as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, m denotes the message obtained by aggregating messages from the neighborhood of the node v , while M t and U t are certain functions selected at the t -th step of propagation.
Revised: Here, m denotes the message obtained by aggregating messages from the neighborhood of the node v , while M t and U t are certain functions selected at the t -th step of propagation.

Original: To recover MPNN from AllSet (6), we simply choose f V→E and f E→V as
Revised: To recover MPNN from AllSet (6), we simply choose f V→E and f E→V as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s.t
Revised: s.t

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: \ v
Revised: \ v

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, MPNN is a special case of AllSet for graph inputs.
Revised: Hence, MPNN is a special case of AllSet for graph inputs.

Original: F P ROOF OF P ROPOSITION 4. 
Revised: F P ROOF OF P ROPOSITION 4.

Original: The proof is largely based on the proof of Proposition 2 of the Set Trasnformer paper (Lee et al., 2019) but is nevertheless included for completeness.
Revised: The proof is largely based on the proof of Proposition 2 of the Set Trasnformer paper (Lee et al., 2019) but is nevertheless included for completeness.

Original: We ignore all layer normalizations as in the proof of (Lee et al., 2019) and start with the following theorem.
Revised: We ignore all layer normalizations as in the proof of (Lee et al., 2019) and start with the following theorem.

Original: Theorem F.1 (Theorem 1 in Lee et al.
Revised: Theorem F.1 (Theorem 1 in Lee et al.

Original: Functions of the form MLP (cid:80) ( MLP ) are universal approximators in the space of permutation invariant functions.
Revised: Functions of the form MLP (cid:80) ( MLP ) are universal approximators in the space of permutation invariant functions.

Original: This result is based on Zaheer et al.
Revised: This result is based on Zaheer et al.

Original: According to Wagstaff et al.
Revised: According to Wagstaff et al.

Original: (2019), we have the constraint that the input multiset has to be ﬁnite.
Revised: (2019), we have the constraint that the input multiset has to be ﬁnite.

Original: Next, we show that the simple mean operator is a special case of a multihead attention module.
Revised: Next, we show that the simple mean operator is a special case of a multihead attention module.

Original: For simplicity, we consider the case of a single head, which corresponds to h = 1 , as we can always set the other heads to zero by choosing MLP V,i ( S )
Revised: For simplicity, we consider the case of a single head, which corresponds to h = 1 , as we can always set the other heads to zero by choosing MLP V,i ( S )

Original: = 

Revised: =

Original: for all i ≥ 2 .
Revised: for all i ≥ 2 .

Original: Following the proof of Lemmain Lee et al.
Revised: Following the proof of Lemmain Lee et al.

Original: (2019), we set the learnable weight θ to be the all-zero vector and use ω ( · ) = 1 + g ( · ) (element-wise) as an activation function such that g (0) = 0 .
Revised: (2019), we set the learnable weight θ to be the all-zero vector and use ω ( · ) = 1 + g ( · ) (element-wise) as an activation function such that g (0) = 0 .

Original: The MH 1 ,ω module in (8) then becomes
Revised: The MH 1 ,ω module in (8) then becomes

Original: [Equation]
Revised: [Equation]

Original: | S | (cid:88)
Revised: | S | (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The AllSetTranformer layer (8) takes the form f V→E ( S ) =
Revised: The AllSetTranformer layer (8) takes the form f V→E ( S ) =

Original: Y + MLP ( Y ) , where Y = | S | (cid:88) i =
Revised: Y + MLP ( Y ) , where Y = | S | (cid:88) i =

Original: MLP V, 1 ( S ) .
Revised: MLP V, 1 ( S ) .

Original: Note that Y is a vector.
Revised: Note that Y is a vector.

Original: Clearly, we can choose an MLP such that MLP ( Y ) equals to another
Revised: Clearly, we can choose an MLP such that MLP ( Y ) equals to another

Original: MLP ( Y ) that results in subtracting Y .
Revised: MLP ( Y ) that results in subtracting Y .

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: By Theorem F.1, it is clear that f V→E ( S ) is a universal approximator for permutation invariant functions.
Revised: By Theorem F.1, it is clear that f V→E ( S ) is a universal approximator for permutation invariant functions.

Original: The same analysis applies to f E→V ( S ) .
Revised: The same analysis applies to f E→V ( S ) .

Original: G A D ISCUSSION OF LEGCN
Revised: G A D ISCUSSION OF LEGCN

Original: Line expansion (LE) is a procedure that transform a hypergraph or a graph into a homogeneous graph; a node in the LE graph represents a pair of node-hyperedge from the original hypergraph.
Revised: Line expansion (LE) is a procedure that transform a hypergraph or a graph into a homogeneous graph; a node in the LE graph represents a pair of node-hyperedge from the original hypergraph.

Original: Nodes in the LE graph are linked if and only if there is a nonempty intersection of their associated node-hyperedge pairs.
Revised: Nodes in the LE graph are linked if and only if there is a nonempty intersection of their associated node-hyperedge pairs.

Original: As stated by the authors of LEGCN (Yang et al., 2020), for a fully connected d -uniform hypergraph their resulting LE graph has Θ( d |E| ) nodes and Θ( d 2 |E| 2 ) edges.
Revised: As stated by the authors of LEGCN (Yang et al., 2020), for a fully connected d -uniform hypergraph their resulting LE graph has Θ( d |E| ) nodes and Θ( d 2 |E| 2 ) edges.

Original: This expansion hence leads to very large graphs which require large memory units and have high computational complexity when coupled with GNNs such as GCN.
Revised: This expansion hence leads to very large graphs which require large memory units and have high computational complexity when coupled with GNNs such as GCN.

Original: To alleviate this drawback, the authors use random sampling techniques which unfortunately lead to an undesired information loss.
Revised: To alleviate this drawback, the authors use random sampling techniques which unfortunately lead to an undesired information loss.

Original: In contrast, we deﬁne our AllSet layers directly on hypergraph which leads to a signiﬁcantly more efﬁcient approach.
Revised: In contrast, we deﬁne our AllSet layers directly on hypergraph which leads to a signiﬁcantly more efﬁcient approach.

Original: H H YPERGRAPH P ROPAGATION R ULES B ASED ON THE H E IGENPROBLEM
Revised: H H YPERGRAPH P ROPAGATION R ULES B ASED ON THE H E IGENPROBLEM

Original: As outlined in the main text, in this case we associate a d -uniform hypergraph G with an adjacency tensor A .
Revised: As outlined in the main text, in this case we associate a d -uniform hypergraph G with an adjacency tensor A .

Original: The H eigenproblem for A states that
Revised: The H eigenproblem for A states that

Original: [Equation]
Revised: [Equation]

Original: where x [ d − 1] i = ( x i ) d − 1 .
Revised: where x [ d − 1] i = ( x i ) d − 1 .

Original: Similar to Zprop, we can deﬁne Hprop (41) in a node-wise fashion as
Revised: Similar to Zprop, we can deﬁne Hprop (41) in a node-wise fashion as

Original: [Equation]
Revised: [Equation]

Original: Although taking the 1 / ( d − 1) -th root resolves the unit issue that exists in Zprop (4), it may lead to imaginary features during updates.
Revised: Although taking the 1 / ( d − 1) -th root resolves the unit issue that exists in Zprop (4), it may lead to imaginary features during updates.

Original: It remains an open question to formulate Hprop in a manner that ensures that features remain real-valued during propagation.
Revised: It remains an open question to formulate Hprop in a manner that ensures that features remain real-valued during propagation.

Original: I A DDITIONAL D ETAILS P ERTAINING TO THE T ESTED D ATASETS
Revised: I A DDITIONAL D ETAILS P ERTAINING TO THE T ESTED D ATASETS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5.18 med
Revised: 5.18 med

Original: [Table]
Revised: [Table]

Original: We use 10 available benchmark datasets from the existing hypergraph neural networks literature and introduce three newly created datasets as described in the exposition to follow.
Revised: We use 10 available benchmark datasets from the existing hypergraph neural networks literature and introduce three newly created datasets as described in the exposition to follow.

Original: The benchmark datasets include cocitation networks Cora, Citeseer and Pubmed 5 , obtained from Yadati et al.
Revised: The benchmark datasets include cocitation networks Cora, Citeseer and Pubmed 5 , obtained from Yadati et al.

Original: The coauthorship networks Cora-CA 6 and DBLP-CA 7 are also adapted from Yadati et al.
Revised: The coauthorship networks Cora-CA 6 and DBLP-CA 7 are also adapted from Yadati et al.

Original: In the cocitation and coauthorship networks datasets, the node features are the bag-of-words representations of the corresponding documents.
Revised: In the cocitation and coauthorship networks datasets, the node features are the bag-of-words representations of the corresponding documents.

Original: Datasets from the UCI Categorical Machine Learning
Revised: Datasets from the UCI Categorical Machine Learning

Original: Repository (Dua & Graff, 2017) include 20Newsgroups, Mushroom and Zoo.
Revised: Repository (Dua & Graff, 2017) include 20Newsgroups, Mushroom and Zoo.

Original: In 20Newsgroups, the node features are the TF-IDF representations of news messages.
Revised: In 20Newsgroups, the node features are the TF-IDF representations of news messages.

Original: In the Mushroom dataset, the node features represent categorical descriptions of 23 species of mushrooms.
Revised: In the Mushroom dataset, the node features represent categorical descriptions of 23 species of mushrooms.

Original: In Zoo, the node features are a mix of categorical and numerical measurements describing different animals.
Revised: In Zoo, the node features are a mix of categorical and numerical measurements describing different animals.

Original: The computer vision and graphics datasets include the Princeton CAD ModelNet40 (Wu et al., 2015) and the NTU 3D dataset (Chen et al., 2003).
Revised: The computer vision and graphics datasets include the Princeton CAD ModelNet40 (Wu et al., 2015) and the NTU 3D dataset (Chen et al., 2003).

Original: The visual objects contain features extracted using Group-View
Revised: The visual objects contain features extracted using Group-View

Original: Convolutional Neural Network(GVCNN)
Revised: Convolutional Neural Network(GVCNN)

Original: (Feng et al., 2018) and Multi-View Convolutional Neural
Revised: (Feng et al., 2018) and Multi-View Convolutional Neural

Original: Network(MVCNN)
Revised: Network(MVCNN)

Original: (Su et al., 2015).
Revised: (Su et al., 2015).

Original: The hypergraph construction follows the setting described in Feng et al.
Revised: The hypergraph construction follows the setting described in Feng et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: The three newly introduce datasets are adapted from Yelp (Yelp), House (Chodrow et al., 2021) and
Revised: The three newly introduce datasets are adapted from Yelp (Yelp), House (Chodrow et al., 2021) and

Original: Walmart (Amburg et al., 2020).
Revised: Walmart (Amburg et al., 2020).

Original: For Yelp, we selected all businesses in the “restaurant” catalog as our nodes, and formed hyperedges by selecting restaurants visited by the same user.
Revised: For Yelp, we selected all businesses in the “restaurant” catalog as our nodes, and formed hyperedges by selecting restaurants visited by the same user.

Original: We use the number of stars in the average review of a restaurant as the corresponding node label, starting from 1 and going up to 5 stars, with an interval of 0 .
Revised: We use the number of stars in the average review of a restaurant as the corresponding node label, starting from 1 and going up to 5 stars, with an interval of 0 .

Original: 5 stars.
Revised: 5 stars.

Original: We then form the node features from the latitude, longitude, one-hot encoding of city and state, and bag-of-word encoding of the topwords in the name of the corresponding restaurants.
Revised: We then form the node features from the latitude, longitude, one-hot encoding of city and state, and bag-of-word encoding of the topwords in the name of the corresponding restaurants.

Original: In the House dataset, each node is a member of the US House of Representatives and hyperedges are formed by grouping together members of the same committee.
Revised: In the House dataset, each node is a member of the US House of Representatives and hyperedges are formed by grouping together members of the same committee.

Original: Node labels indicate the political party of the representatives.
Revised: Node labels indicate the political party of the representatives.

Original: In Walmart, nodes represent products being purchased at Walmart, and hyperedges equal sets of products purchased together; the node labels are the product categories.
Revised: In Walmart, nodes represent products being purchased at Walmart, and hyperedges equal sets of products purchased together; the node labels are the product categories.

Original: Since there are no node features in the original
Revised: Since there are no node features in the original

Original: House and Walmart dataset, we impute the same using Gaussian random vectors, in a manner similar to what was done for the contextual stochastic block model (Deshpande et al., 2018).
Revised: House and Walmart dataset, we impute the same using Gaussian random vectors, in a manner similar to what was done for the contextual stochastic block model (Deshpande et al., 2018).

Original: In both datasets, we ﬁx the feature vector dimension to 100 and use one-hot encodings of the labels with added Gaussian noise N (0 , σ 2 I ) as the actual features.
Revised: In both datasets, we ﬁx the feature vector dimension to 100 and use one-hot encodings of the labels with added Gaussian noise N (0 , σ 2 I ) as the actual features.

Original: The noise standard deviation σ is chosen to be 1 and 0 . 6 .
Revised: The noise standard deviation σ is chosen to be 1 and 0 . 6 .

Original: J C OMPUTATIONAL E FFICIENCY AND E XPERIMENTAL S ETTINGS
Revised: J C OMPUTATIONAL E FFICIENCY AND E XPERIMENTAL S ETTINGS

Original: All our test were executed on a Linux machine with 48 cores, 376 GB of system memory, and two NVIDIA Tesla P 100 GPUs with 12 GB of GPU memory each.
Revised: All our test were executed on a Linux machine with 48 cores, 376 GB of system memory, and two NVIDIA Tesla P 100 GPUs with 12 GB of GPU memory each.

Original: In AllSetTransformer, we also used a single layer MLP at the end for node classiﬁcation.
Revised: In AllSetTransformer, we also used a single layer MLP at the end for node classiﬁcation.

Original: The average training times with their corresponding standard deviations in second per run, for all baseline methods at all tested datasets, are reported in Table 4.
Revised: The average training times with their corresponding standard deviations in second per run, for all baseline methods at all tested datasets, are reported in Table 4.

Original: Note that the reported times do not include any preprocessing times for the hypergraph datasets as these are only performed once before training.
Revised: Note that the reported times do not include any preprocessing times for the hypergraph datasets as these are only performed once before training.

Original: Table 4 lists the average training time per run with the optimal set of hyperparameters obtained after tunning; the average training times per run for different sets of hyperparameters used in the tuning process are reported in Table 5.
Revised: Table 4 lists the average training time per run with the optimal set of hyperparameters obtained after tunning; the average training times per run for different sets of hyperparameters used in the tuning process are reported in Table 5.

Original: Choices of hyperparameters.
Revised: Choices of hyperparameters.

Original: We tune the hidden dimension of all hypergraph neural networks over { 64 , 128 , 256 , 512 } , except for the case of HyperGCN and HAN, where the original implementations do not allow for changing the hidden dimension.
Revised: We tune the hidden dimension of all hypergraph neural networks over { 64 , 128 , 256 , 512 } , except for the case of HyperGCN and HAN, where the original implementations do not allow for changing the hidden dimension.

Original: For HNHN, AllSetTransformer and
Revised: For HNHN, AllSetTransformer and

Original: AllDeepSets we use one layer (a full V → E → V propagation rule layer), while for all the other methods we use two layers as recommended in the previous literature.
Revised: AllDeepSets we use one layer (a full V → E → V propagation rule layer), while for all the other methods we use two layers as recommended in the previous literature.

Original: We tune the learning rate over { 0 . 1 , 0 . 01 , 0 . 001 } , and the weight decays over { 0 , 0 . 00001 } , for all models under consideration.
Revised: We tune the learning rate over { 0 . 1 , 0 . 01 , 0 . 001 } , and the weight decays over { 0 , 0 . 00001 } , for all models under consideration.

Original: For models with multihead attentions, we also tune the number of heads over the set { 1 , 4 , 8 } .
Revised: For models with multihead attentions, we also tune the number of heads over the set { 1 , 4 , 8 } .

Original: The best hyperparameters for each model and dataset are listed in Table 6.
Revised: The best hyperparameters for each model and dataset are listed in Table 6.

Original: Note that we only use default setting for HAN due to facts that its DGL implementation has all parameters set as constants and its much higher time complexity per run.
Revised: Note that we only use default setting for HAN due to facts that its DGL implementation has all parameters set as constants and its much higher time complexity per run.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CEGAT
Revised: CEGAT

Original: HNHN
Revised: HNHN

Original: [Table]
Revised: [Table]

Original: wd
Revised: wd

Original: [Table]
Revised: [Table]

Original: Citeseer
Revised: Citeseer

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Cora
Revised: Cora

Original: [Table]
Revised: [Table]

Original: 00E-

Revised: 00E-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 1 20News
Revised: 1 20News

Original: OOM OOM OOM
Revised: OOM OOM OOM

Original: [Table]
Revised: [Table]

Original: 1 Mushroom 0.01 1.00E-
Revised: 1 Mushroom 0.01 1.00E-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 NTU 0.
Revised: 1 NTU 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 ModelNet
Revised: 1 ModelNet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 Yelp
Revised: 1 Yelp

Original: OOM
Revised: OOM

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 House
Revised: 1 House

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 1 Walmart
Revised: 1 Walmart

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1 Walmart(
Revised: 1 Walmart(

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 1 HyperGCN
Revised: 1 HyperGCN

Original: UniGCNII lr wd
Revised: UniGCNII lr wd

Original: [Table]
Revised: [Table]

Original: wd
Revised: wd

Original: [Table]
Revised: [Table]

Original: heads Cora
Revised: heads Cora

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 512 8 Citeseer
Revised: 512 8 Citeseer

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Pubmed
Revised: Pubmed

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Cora-
Revised: Cora-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]