Original: J ANOSSY P OOLING : L EARNING D EEP P ERMUTATIONINVARIANT F UNCTIONS FOR V ARIABLE - SIZE I NPUTS
Revised: J ANOSSY P OOLING : L EARNING D EEP P ERMUTATIONINVARIANT F UNCTIONS FOR V ARIABLE - SIZE I NPUTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider a simple and overarching representation for permutation-invariant functions of sequences (or set functions).
Revised: We consider a simple and overarching representation for permutation-invariant functions of sequences (or set functions).

Original: Our approach, which we call
Revised: Our approach, which we call

Original: Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence.
Revised: Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence.

Original: This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and ﬂexible permutation-invariant functions.
Revised: This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and ﬂexible permutation-invariant functions.

Original: If carried out naively, Janossy pooling can be computationally prohibitive.
Revised: If carried out naively, Janossy pooling can be computationally prohibitive.

Original: To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with k -order interactions, and stochastic optimization algorithms with random permutations.
Revised: To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with k -order interactions, and stochastic optimization algorithms with random permutations.

Original: Our framework uniﬁes a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions.
Revised: Our framework uniﬁes a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions.

Original: We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.
Revised: We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pooling is a fundamental operation in deep learning architectures (LeCun et al., 2015).
Revised: Pooling is a fundamental operation in deep learning architectures (LeCun et al., 2015).

Original: The role of pooling is to merge a collection of related features into a single, possibly vector-valued, summary feature.
Revised: The role of pooling is to merge a collection of related features into a single, possibly vector-valued, summary feature.

Original: A prototypical example is in convolutional neural networks (CNNs)(LeCun et al., 1995), where linear activations of features in neighborhoods of image locations are pooled together to construct more abstract features.
Revised: A prototypical example is in convolutional neural networks (CNNs)(LeCun et al., 1995), where linear activations of features in neighborhoods of image locations are pooled together to construct more abstract features.

Original: A more modern example is in neural networks for graphs, where each layer pools together embeddings of neighbors of a vertex to form a new embedding for that vertex, see for instance, (Kipf & Welling, 2016; Atwood & Towsley, 2016; Hamilton et al., 2017; Velickovic et al., 2017;
Revised: A more modern example is in neural networks for graphs, where each layer pools together embeddings of neighbors of a vertex to form a new embedding for that vertex, see for instance, (Kipf & Welling, 2016; Atwood & Towsley, 2016; Hamilton et al., 2017; Velickovic et al., 2017;

Original: Monti et al., 2017; Xu et al., 2018;
Revised: Monti et al., 2017; Xu et al., 2018;

Original: Liu et al., 2018; Liben-Nowell & Kleinberg, 2007; van den Berg et al., 2017; Duvenaud et al., 2015; Gilmer et al., 2017;
Revised: Liu et al., 2018; Liben-Nowell & Kleinberg, 2007; van den Berg et al., 2017; Duvenaud et al., 2015; Gilmer et al., 2017;

Original: Ying et al., 2018).
Revised: Ying et al., 2018).

Original: A common requirement of a pooling operator is invariance to the ordering of the input features.
Revised: A common requirement of a pooling operator is invariance to the ordering of the input features.

Original: In CNNs for images, this allows invariance to translations  of objects, while for graphs, this allows invariance to relabeling of graph vertices.
Revised: In CNNs for images, this allows invariance to translations and rotations of objects, while for graphs, this allows invariance to relabeling of graph vertices.

Original: Existing pooling operators are mostly limited to predeﬁned heuristics such as max-pool, min-pool, sum, or average.
Revised: Existing pooling operators are mostly limited to pre-deﬁned heuristics such as max-pool, min-pool, sum, or average.

Original: Another desirable characteristic of pooling layers is the ability to take variable-size inputs.
Revised: Another desirable characteristic of pooling layers is the ability to take variable-size inputs.

Original: This is less important in images, where neighborhoods are usually ﬁxed a priori .
Revised: This is less important in images, where neighborhoods are usually ﬁxed a priori .

Original: However in applications involving graphs, the number of neighbors of different vertices can vary widely.
Revised: However in applications involving graphs, the number of neighbors of different vertices can vary widely.

Original: Our goal is to design ﬂexible and learnable pooling operators satisfying these two desiderata.
Revised: Our goal is to design ﬂexible and learnable pooling operators satisfying these two desiderata.

Original: Abstractly, we will view pooling as a permutation-invariant (or symmetric) function acting on ﬁnite but arbitrary length sequences h .
Revised: Abstractly, we will view pooling as a permutation-invariant (or symmetric) function acting on ﬁnite but arbitrary length sequences h .

Original: All elements h
Revised: All elements h

Original: i of the sequences are features belonging to some space H (which itself could be a high-dimensional Euclidean space R n or some subset thereof, n > 0 ).
Revised: i of the sequences are features lying in some space H (which itself could be a high-dimensional Euclidean space R d or some subset thereof).

Original: The sequences h are  elements of the union of products of the H -space: h ∈ (cid:83) 
Revised: The sequences h are themselves elements of the union of products of the H -space: h ∈ (cid:83) ∞

Original: [Equation]{+j =
 H j ≡ H ∪ .+}
Revised: {+j =
 H j ≡ H ∪ .+}

Original: Throughout the paper, we will use Π n to represent the set of all permutations of the integers 1 to n , where n will often be clear from the context.
Revised: Throughout the paper, we will use Π n to represent the set of all permutations of the integers 1 to n , where n will often be clear from the context.

Original: In particular, h π , π ∈ Π | h | , will represent a reordering of the elements of a sequence h according to π , where |
Revised: In addition, h π , π ∈ Π | h | , will represent a reordering of the elements of a sequence h according to π , where |

Original: h | is the length of the sequence h .
Revised: h | is the length of the sequence h .

Original: We will use the double bar superscript f to indicate that a function is permutation invariant, returning the same value no matter the order of its arguments: f ( h ) =
Revised: We will use the double bar superscript f to indicate that a function is permutation-invariant, returning the same value no matter the order of its arguments: f ( h ) =

Original: f ( h π ) , ∀ π ∈ Π | h | .
Revised: f ( h π ) , ∀ π ∈ Π | h | .

Original: We will use the arrow superscript f (cid:42) to indicate general functions on sequences h which may or may not be permutationinvariant.
Revised: We will use the arrow superscript f (cid:42) to indicate general functions on sequences h which may or may not be permutationinvariant 1 .

Original: Functions f without any markers are ‘simple’ functions, acting on elements in H , scalars or any other argument that is not a sequence of elements in H .
Revised: Functions f without any markers are ‘simple’ functions, acting on elements in H , scalars or any other argument that is not a sequence of elements in H .

Original: Our goal in this paper is to model and learn permutation-sensitive functions f (cid:42) that can be used to construct ﬂexible and learnable permutation-invariant neural networks.
Revised: Our goal in this paper is to model and learn permutation-sensitive functions f (cid:42) that can be used to construct ﬂexible and learnable permutation-invariant neural networks.

Original: A recent step in this direction is work on deep sets by Zaheer et al.
Revised: A recent step in this direction is work on deep sets by Zaheer et al.

Original: (2017), who argued for learning permutation-invariant functions through the following composition:
Revised: (2017), who argued for learning permutation-invariant functions through the following composition:

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Here (a) x ∈ X is  the training data, h ∈ H is the embedding space of h , and h :X × R a → [Equation]
Revised: Here (a) x ∈ X is one observation in the training data, h ∈ H is the embedding (output) of the data given by the lower layers h :

Original:  a > 0is the embedding given by lower layers of the neural network with parameters θ 
Revised: X × R a → H ∪ , a > 0 with parameters θ ( h ) ∈

Original: ( h ) ∈ R a , and (b) f : H × R b → F is a middle-layer embedding function with parameters θ ( f ) ∈ R b , b > 0 , and F is the embedding(output) space of f ; and (c) ρ : F × R c
Revised: R a ; (b) f : H × R b → F is a middle-layer embedding function with parameters θ ( f ) ∈ R b , b > 0 , and F is the embedding space of f ; and (c) ρ : F × R c

Original: → Y is a neural network with parameters θ ( ρ ) ∈ R c , c > 0 , that maps to the ﬁnal output space Y .
Revised: → Y is a neural network with parameters θ ( ρ ) ∈ R c , c > 0 , that maps to the ﬁnal output space Y .

Original: Typically H and F are high-dimensional real-valued spaces; Y is often R d in d -dimensional regression problems or the simplex in classiﬁcation problems.
Revised: Typically H and F are high-dimensional real-valued spaces; Y is often R d in d -dimensional regression problems or the simplex in classiﬁcation problems.

Original: Effectively, the neural network f learns an embedding for each element in H , and given a sequence h , its component embeddings are added together before a second neural network transformation ρ is applied.
Revised: Effectively, the neural network f learns an embedding for each element in H , and given a sequence h , its component embeddings are added together before a second neural network transformation ρ is applied.

Original: Note that the function h may be the identity mapping h ( x ; · ) =
Revised: Note that the function h may be the identity mapping h ( x ; · ) =

Original: x that makes f act directly on the input data.
Revised: x that makes f act directly on the input data.

Original: Zaheer et al.
Revised: Zaheer et al.

Original: (2017)  argues that, under reasonable assumptions, the above architecturewith suitably ﬂexible functions f and ρ is capable of approximating any symmetric function on h -sequences, which justiﬁes the widespread use of average (sum) pooling to make neural networks permutation invariant in Duvenaud et al.  Hamilton et al.
Revised: (2017) essentially argues that if ρ is a universal function approximator, the above architecture is capable of approximating any symmetric function on h -sequences, which justiﬁes the widespread use of average (sum) pooling to make neural networks permutation-invariant in Duvenaud et al. (2015), Hamilton et al.

Original: (2017), Kipf & Welling (2016), Atwood & Towsley (2016),Niepert et al. (2016), among other works. 
Revised: (2017), Kipf & Welling (2016), Atwood & Towsley (2016), among other works. computationally

Original: In practice, however, there is a gap between ﬂexibility and learnability.
Revised: In practice, however, there is a gap between ﬂexibility and learnability.

Original: While the architecture of equations 1 and 2 is a universal approximator to all permutation-invariant functions, it does not allow structural knowledge about the function y to be encoded easily.
Revised: While the architecture of equations 1 and 2 is a universal approximator to all permutation-invariant functions, it does not allow structural knowledge about the function y to be encoded easily.

Original: Consider for instance trying to learn the permutation-invariant function y (cid:48)
Revised: Consider for instance trying to learn the permutationinvariant function y (cid:48)

Original:  ( x ) = max i,j ≤| x | | x i − x j | .
Revised: = ( x ) = max i,j ≤| x | | x i − x j | .

Original: With such higher-order interactions between the elements of h , the functions f(cid:42) of equation 2 cannot capture any useful intermediate representations towards the ﬁnal output, with the burden shifted entirely to the function ρ .
Revised: With such higher-order interactions between the elements of h , the functions f of equation 2 cannot capture any useful intermediate representations towards the ﬁnal output, with the burden shifted entirely to the function ρ .

Original: Learning ρ means learning to undo mixing performed by the summation layer
Revised: Learning ρ means learning to undo mixing performed by the summation layer

Original: f ( | h | , h ; θ ( f ) )
Revised: f ( | h | , h ; θ ( f ) )

Original: =1 | h | (cid:80) | h | j =1 f ( h j ; θ ( f ) ) .
Revised: = (cid:80) | h | j =1 f ( h j ; θ ( f ) ) .

Original: Unfortunately, as we show in our experiments, in many applications this is too much to ask of ρ .
Revised: Unfortunately, as we show in our experiments, in many applications this is too much to ask of ρ .

Original: [Table]

Original: … ⇢ ( f ; ✓ ( ⇢ ) )

Original: [Table]

Original: f

Original: [Table]

Original: ( n

Original: [Table]

Original: [Table]

Original: ( f )

Original: [Table]

Original: </latexit>

Original: The embedding h is permuted in all | h | !

Original: possible ways, and for each permutation h π , (cid:42) f

Original: ( | h | , h π ; θ ( f ) ) is computed.

Original: These results are added together, and the result is passed to a second function ρ parameterized by θ ( ρ ) , which gives the ﬁnal permutation-invariant output y ( x ; θ ( h ) , θ ( f ) , θ ( ρ ) ) .

Original: We discuss how this scheme can be made computationally tractable.

Original: Contributions.
Revised: Contributions.

Original: In this work we investigate a learnable permutation-invariant pooling layer for variable-size inputs inspired by the Janossy density framework (widely used in the theory of point processes (Daley & Vere-Jones, 2003,
Revised: We investigate a learnable permutation-invariant pooling layer for variable-size inputs inspired by the Janossy density framework, widely used in the theory of point processes (Daley & Vere-Jones, 2003,

Original: Chapter 7)).
Revised: Chapter 7).

Original: This approach, which we call Janossy pooling, directly allows the user to model what higher-order dependencies in h are relevant in the pooling.
Revised: This approach, which we call Janossy pooling, directly allows the user to model what higher-order dependencies in h are relevant in the pooling.

Original: Figure 1 summarizes a neural network with a single  pooling layer f : given an input embedding h , we apply a learnable (permutationsensitive) function f (cid:42) to every permutation h π of the input sequence h .
Revised: Figure 1 summarizes a neural network with a single Janossy pooling layer f (detailed in Deﬁnition 2.1 below): given an input embedding h , we apply a learnable (permutation-sensitive) function f (cid:42) to every permutation h π of the input sequence h .

Original: These outputs are added together, and fed to the second function ρ .
Revised: These outputs are added together, and fed to the second function ρ .

Original: Examples of function f (cid:42) include feedfoward and recurrent neural networks.
Revised: Examples of function f (cid:42) include feedforward and recurrent neural networks (RNNs).

Original: We call the operation used to construct f from f (cid:42) the Janossy pooling .
Revised: We call the operation used to construct f from f (cid:42) the Janossy pooling .

Original: A more detail description of Janossy pooling is given in Deﬁnition 2.1.
Revised: Deﬁnition 2.1 gives a more detailed description.

Original: Further, we contribute the following analysis: (a) We show deep sets (Zaheer et al., 2017) is a special case of Janossy pooling if the function f (cid:42) depends only on the ﬁrst element of the vector h π .
Revised: Further, we contribute the following analysis: (a) We show deep sets (Zaheer et al., 2017) is a special case of Janossy pooling where the function f (cid:42) depends only on the ﬁrst element of the vector h π .

Original: In the most general form of Janossy pooling (as described above), f (cid:42) depends on its entire input sequence h π .
Revised: In the most general form of Janossy pooling (as described above), f (cid:42) depends on its entire input sequence h π .

Original: This naturally raises the possibility of intermediate choices of f (cid:42) that allow practitioners to trade between ﬂexibility and tractability.
Revised: This naturally raises the possibility of intermediate choices of f (cid:42) that allow practitioners to trade between ﬂexibility and tractability.

Original: We will show that functions f (cid:42) that depend on their ﬁrst k arguments of h π allow the Janossy pooling layer to capture up to k -ary dependencies in h .
Revised: We will show that functions f (cid:42) that depend on their ﬁrst k arguments of h π allow the Janossy pooling layer to capture up to k -ary dependencies in h .

Original: (b) We show Janossy pooling can be used to learn permutation-invariant neural networks y ( x ) by sampling a random permutation  during training, and then modeling this permutation using a sequence model such as a recurrent neural network (LSTMs (Hochreiter & Schmidhuber, 1997), GRUs (Cho et al., 2014)) or a vector model such as a feedforward network.
Revised: (b) We show Janossy pooling can be used to learn permutation-invariant neural networks y ( x ) by sampling a random permutation of h during training, and then modeling this permuted sequence using a sequence model such as a recurrent neural network (LSTMs (Hochreiter & Schmidhuber, 1997), GRUs (Cho et al., 2014)) or a vector model such as a feedforward network.

Original: We call this permutation-sampling learning algorithm π -SGD ( π -Stochastic Gradient Descent).
Revised: We call this permutation-sampling learning algorithm π -SGD ( π -Stochastic Gradient Descent).

Original: Our analysis explains why this seemly unsound procedure is theoretically justiﬁed, which sheds light on the recent puzzling success of permutation sampling and LSTMs in relational models Moore & Neville (2017); Hamilton et al.
Revised: Our analysis explains why this seemingly unsound procedure is theoretically justiﬁed, which sheds light on the recent puzzling success of permutation sampling and LSTMs in relational models (Moore & Neville, 2017; Hamilton et al., 2017).

Original: We show that this property relates towhat we randomized model ensemble techniques.
Revised: We show that this property relates to randomized model ensemble techniques.

Original: (c)
Revised: (c)

Original: In Zaheer et al.
Revised: In Zaheer et al.

Original: (2017), the authors describe a connection between deep sets and inﬁnite de Finetti exchangeabilty.
Revised: (2017), the authors describe a connection between deep sets and inﬁnite de Finetti exchangeabilty.

Original: We provide a probabilistic connection between Janossy pooling and ﬁnite de Finetti exchangeabilty (Diaconis, 1977).
Revised: We provide a probabilistic connection between Janossy pooling and ﬁnite de Finetti exchangeabilty (Diaconis, 1977).

Original:  J ANOSSY P OOLING
Revised:  J ANOSSY P OOLING

Original: In this section we formalize the Janossy pooling function f .
Revised: We ﬁrst formalize the Janossy pooling function f .

Original: We start with a function f (cid:42) , parameterized by θ ( f ) , which can take any variable-size sequence as input: a sequence of matrices (such asa sequence of images), a sequence of vectors (such as a sequence of vector embeddings), or a variablesize sequence of features or embeddings representing the neighbors of a node in an attributed graph.
Revised: Start with a function f (cid:42) , parameterized by θ ( f ) , which can take any variable-size sequence as input: a sequence of matrices (such as images), a sequence of vectors (such as a sequence of vector embeddings), or a variable-size sequence of features or embeddings representing the neighbors of a node in an attributed graph.

Original: In practice, we implement f (cid:42) with a neural network.
Revised: In practice, we implement f (cid:42) with a neural network.

Original: Formalizing Figure 1 from Section 1, we use f (cid:42) to deﬁne f :
Revised: Formalizing Figure 1 from Section 1, we use f (cid:42) to deﬁne f :

Original: Deﬁnition 2.1:
Revised: Deﬁnition 2.1:

Original: [Janossy pooling] Consider a function f (cid:42) :
Revised: [Janossy pooling] Consider a function f (cid:42) :

Original: N × (cid:83)

Original: [Equation]

Original: H j

Original: [Equation] R b [Equation] Y on variablelength but ﬁnite sequences h , parameterized by θ ( f ) ∈ R b , b > 0 .
Revised: N × H ∪ × R b → F on variable-length but ﬁnite sequences h , parameterized by θ ( f ) ∈ R b , b > 0 .

Original: A permutation-invariant function f :

Original: [Equation]

Original: ∞ j = [Equation] H j → Y is the Janossy function associated with f (cid:42) if
Revised: A permutation-invariant function f : N × H ∪ → F is the Janossy function associated with f (cid:42) if

Original: [Equation]
Revised: [Equation]

Original: where Π | h | is the set of all permutations of the integers 1 to | h | , and h π represents a particular reordering of the elements of sequence h according to π ∈ Π | h | .
Revised: where Π | h | is the set of all permutations of the integers 1 to | h | , and h π represents a particular reordering of the elements of sequence h according to π ∈ Π | h | .

Original: We denote Janossy pooling the operation used to construct f from f (cid:42) .
Revised: We refer the operation used to construct f from f (cid:42) as Janossy pooling.

Original: ♦
Revised: ♦

Original: Deﬁnition 2.1 provides a conceptually simple approach to constructing permutation-invariant functions from arbitrary sequence-functions.
Revised: Deﬁnition 2.1 provides a conceptually simple approach for constructing permutation-invariant functions from arbitrary and powerful permutation-sensitive functions such as feedforward networks, recurrent neural networks, or convolutional neural networks.

Original: If f (cid:42) is a vector-valued function, then so is f , and in practice, one might pass this vector output of f through a second function ρ (e.g. a neural network parameterized by θ ( ρ ) ):
Revised: If f (cid:42) is a vector-valued function, then so is f , and in practice, one might pass this vector output of f through a second function ρ (e.g. a neural network parameterized by θ ( ρ ) ):

Original: [Equation]
Revised: [Equation]

Original: Theoretically, equation 3 can capture any permutation-invariant function for a ﬂexible-enough family of  functionsfor f (cid:42) , thus, at least theoretically, ρ in equation 4 provides no additional representational power.
Revised: Theoretically, equation 3 can capture any permutation-invariant function for a ﬂexible enough family of permutation-sensitive functions f (cid:42) (the proof is straightforward). Thus, at least theoretically, ρ in equation 4 provides no additional representational power.

Original: In practice, ρ can improve learnability by capturing common aspects across all terms in the summation.
Revised: In practice, ρ can improve learnability by capturing common aspects across all terms in the summation.

Original: Furthermore, when we look at approximations to equation 3, adding ρ can help recover some of the lost model capacity.
Revised: Furthermore, when we look at approximations to equation 3 or restrictions of f (cid:42) to more tractable families, adding ρ can help recover some of the lost model capacity.

Original: Overall then, equation 4 represents one layer of Janossy pooling, forming a constituent part of a bigger neural network.
Revised: Overall then, equation 4 represents one layer of Janossy pooling, forming a constituent part of a bigger neural network.

Original: Figure 1 summarizes this.
Revised: Figure 1 summarizes this.

Original: Janossy pooling, as deﬁned in equation 3 and 4 is intractable; the computational cost of summing over all permutations (for prediction), and backpropagating gradients (for learning) is likely prohibitive for most problems of interest.
Revised: Janossy pooling, as deﬁned in equation 3 and 4 is intractable; the computational cost of summing over all permutations (for prediction), and backpropagating gradients (for learning) is likely prohibitive for most problems of interest.

Original: Nevertheless, it provides an overarching framework to unify existing methods, and to extend them.
Revised: Nevertheless, it provides an overarching framework to unify existing methods, and to extend them.

Original: In what follows we present strategies for mitigating this computational cost.
Revised: In what follows we present strategies for mitigating this, allowing novel and effective trade-offs between learnability and computational cost.

Original: 2. T RACTABILITY THROUGH C ANONICAL I NPUT O RDERINGS
Revised: 2. T RACTABILITY THROUGH C ANONICAL I NPUT O RDERINGS

Original: A simple way to achieve permutation-invariance without the summation in equation 3 is to order the elements of h according to some canonical ordering based on their values, and then feed the reordered sequence to f (cid:42) .
Revised: A simple way to achieve permutation-invariance without the summation in equation 3 is to order the elements of h according to some canonical ordering based on its values, and then feed the reordered sequence to f (cid:42) .

Original: This way, all the permutations of h are mapped to the same canonical sequence.
Revised: This way, all the permutations of h are mapped to the same canonical sequence.

Original: Effectively, this imposes the following structural constraint on f (cid:42) : f (cid:42) ( h ) equals 0 for all sequences that do not follow the canonical permutation.
Revised: Effectively, this imposes the following structural constraint on f (cid:42) : f (cid:42) ( h ) equals 0 for all sequences that do not follow the canonical permutation.

Original: This  allowsus to use more complex f (cid:42) models, such as LSTMs and GRUs, that can capture arbitrary relationships in the canonical ordering of h .
Revised: This then allows complex f (cid:42) models, such as LSTMs and GRUs, that can capture arbitrary relationships in the canonical ordering of h .

Original: Examples of the canonical ordering approach already exist in the literature. We ﬁrst describe Niepert et al.
Revised: Examples of the canonical ordering approach already exist in the literature, for example, Niepert et al.

Original: (2016), which orders nodes in a graph according to a user-speciﬁed ranking such as betweenness centrality (say from high to low).
Revised: (2016) order nodes in a graph according to a user-speciﬁed ranking such as betweenness centrality (say from high to low).

Original: This approach is useful only if the canonical ordering is relevant to the task at hand.
Revised: This approach is useful only if the canonical ordering is relevant to the task at hand.

Original: Niepert et al. acknowledges this shortcoming by showing an example where a poor vertex traversal order signiﬁcantly degrades classiﬁcation accuracy.
Revised: Niepert et al. (2016)acknowledges this shortcoming and Moore & Neville (2017) demonstrates that an ordering by Personalized PageRank (Page et al., 1999; Jeh & Widom, 2003) achieves a lower classiﬁcation accuracy than a random ordering.

Original: As an idealized example, consider input sequences
Revised: As an idealized example, consider input sequences

Original: h
Revised: h

Original: = (cid:0) ( h i, 1 , h i, 2 ) (cid:1) n i =1 , with ( h i, 1 , h i, 2 ) ∈
Revised: = (cid:0) ( h i, 1 , h i, 2 ) (cid:1) n i =1 , with ( h i, 1 , h i, 2 ) ∈

Original: H =
Revised: H =

Original: R 2 , and compo- nents x i, 1 and x i, 2 sampled independently of each other.
Revised: R 2 , and components h i, 1 and h i,sampled independently of each other.

Original: Choosinga canonical ordering to sort h according to h · , 1 when the task at hand depends on a canonical ordering based on h · , 2 can lead to poor prediction accuracy.
Revised: Choosing to sort h according to h · , 1 when the task at hand depends on sorting according to h · , 2 can lead to poor prediction accuracy.

Original: Rather than pre-deﬁning a good canonical order, one can try to learn it from the data.
Revised: Rather than pre-deﬁning a good canonical order, one can try to learn it from the data.

Original: This requires searching over the discrete space of all | h | !
Revised: This requires searching over the discrete space of all | h | !

Original: permutations of the input vector h .
Revised: permutations of the input vector h .

Original: In practice, this discrete optimization relies on heuristics (Vinyals et al., 2016; Rezatoﬁghi et al., 2018).
Revised: In practice, this discrete optimization relies on heuristics (Vinyals et al., 2016; Rezatoﬁghi et al., 2018).

Original: Alternatively, instead of choosing a single canonical ordering, one can choose multiple orderings, resulting in ensemble methods that average across multiple permutations.
Revised: Alternatively, instead of choosing a single canonical ordering, one can choose multiple orderings, resulting in ensemble methods that average across multiple permutations.

Original: These can be viewed as more reﬁned (possibly data-driven) approximations to equation 3.
Revised: These can be viewed as more reﬁned (possibly data-driven) approximations to equation 3.

Original: 2. T RACTABILITY THROUGH k - ARY DEPENDENCIES
Revised: 2. T RACTABILITY THROUGH k - ARY DEPENDENCIES

Original: Here, we provide a different spectrum of options to trade-off ﬂexibility, complexity, and generalizability in Janossy pooling.
Revised: Here, we provide a different spectrum of options to trade-off ﬂexibility, complexity, and generalizability in Janossy pooling.

Original: Now, to simplify the sum over permutations in equation 3, we impose structural constraints where f (cid:42) ( h ) depends only on the ﬁrst k elements of its input sequence.
Revised: Now, to simplify the sum over permutations in equation 3, we impose structural constraints where f (cid:42) ( h ) depends only on the ﬁrst k elements of its input sequence.

Original: This amounts to the assumption that only k -ary dependencies in h are relevant to the task at hand.
Revised: This amounts to the assumption that only k -ary dependencies in h are relevant to the task at hand.

Original: Deﬁnition 2.2: [ k -ary Janossy pooling]
Revised: Deﬁnition 2.2: [ k -ary Janossy pooling]

Original: Fix k ∈ N .
Revised: Fix k ∈ N .

Original: For any sequence h , deﬁne ↓ k ( h ) as its projection to a length k sequence; in particular, if | h | ≥ k , we keep the ﬁrst k elements, while if | h | < k , we append ( k − | h | ) zeros at the end of h to make it at least of size k .
Revised: For any sequence h , deﬁne ↓ k ( h ) as its projection to a length k sequence; in particular, if | h | ≥ k , we keep the ﬁrst k elements.

Original: Then, a k -ary permutation-invariant Janossy function f is given by
Revised: Then, a k -ary permutation-invariant Janossy function f is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ♦
Revised: ♦

Revised: Note that if some of the embeddings have length | h | < k , then we can zero pad to form the length- k sequence ( ↓ k ( h π ) , 0 , . . . , 0) .

Original: Proposition 2.1 shows that if | h | > k , equation 5 only needs to sum over | h | ! /
Revised: Proposition 2.1 shows that if | h | > k , equation 5 only needs to sum over | h | ! /

Original: ( | h | − k )!
Revised: ( | h | − k )!

Original: terms.
Revised: terms, which can be tractable for small k .

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: The Janossy pooling in equation 5 requires summing over only 
Revised: The Janossy pooling in equation 5 requires summing over only | h | !

Original: [Equation] k )!
Revised: ( | h |− k )!

Original: terms, thus saving computation when k < | h | .
Revised: terms, thus saving computation when k < | h | .

Original: Proof.
Revised: Proof.

Original: Without loss of generality assume | h | ≥ k , we can pad h with zeros if needed.

Original: Deﬁne two permutations π, π (cid:48) ∈ Π | h | that agree on the ﬁrst k elements as k -equivalent .
Revised: Deﬁne two permutations π, π (cid:48) ∈ Π | h | that agree on the ﬁrst k elements as k -equivalent .

Original: Such permutations satisfy f (cid:42) ( k, ↓ k ( h π ); θ ( f ) )
Revised: Such permutations satisfy f (cid:42) [Equation] | , [Equation]

Revised: h π ); θ ( f ) )

Original: =
Revised: =

Original: f (cid:42) ( k, ↓ k ( h π(cid:48) )
Revised: f (cid:42) ( | h | , [Equation] k ( h π

Revised: [Equation]

Original: ; θ ( f ) ) .
Revised: ); θ ( f ) ) .

Original: These two permutations belong to the same equivalence class, containing a total of ( | h | − k )!
Revised: These two permutations belong to the same equivalence class, containing a total of ( | h | − k )!

Original: permutations (obtained by permuting the last ( | h | − k ) elements).
Revised: permutations (obtained by permuting the last ( | h | − k ) elements).

Original: Overall, we then have a total of | h | ! /
Revised: Overall, we then have a total of | h | ! /

Original: ( | h | − k )!
Revised: ( | h | − k )!

Original: equivalence classes.
Revised: equivalence classes.

Original: Write the set of equivalence classes as Π k | h | , and represent each by one of its elements.
Revised: Write the set of equivalence classes as Π k | h | , and represent each by one of its elements.

Original: Then, 
Revised: Then, f

Original: [Equation]
Revised: [Equation]

Original: is now a summation over only | h | ! /
Revised: is now a summation over only | h | ! /

Original: ( | h | − k )!
Revised: ( | h | − k )!

Original: terms.
Revised: terms.

Original: The sum in equation 6 can be writtenequivalently as 
Revised: Thus, the sum in equation 5 can be written as (cid:80)

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: f

Original:  I n is the set of all permutations of { 1 , 2 , . .. , n } taken k at a time.
Revised: [Equation] (cid:1) , where I | h | is the set of all permutations of { 1 , 2 , . .

Revised: | h |} taken k at a time, and h j is the j -th element of h .

Original: Note that the value of k balances computational savings and model ﬂexibility; it can be selected as a hyperparameter based on a-priori beliefs or through typical hyperparameter tuning strategies.
Revised: Note that the value of k balances computational savings and model ﬂexibility; it can be selected as a hyperparameter based on a-priori beliefs or through typical hyperparameter tuning strategies.

Original: Remark 2.1 (Deep sets (Zaheer et al., 2017) is a 1 -ary (unary) Janossy pooling) .
Revised: Remark 2.1 (Deep sets (Zaheer et al., 2017) is a 1 -ary (unary) Janossy pooling) .

Original: Equation 6 represented with k = 1  yields the model ρ [Equation] Noting that f (cid:42) only takes a single input, we can redeﬁne it and recover equation 2.
Revised: Equation 5 represented with k = 1 and composing with ρ as in equation 4 yields the model ρ (cid:0) 1 | h | (cid:80) | h | j =1 f (cid:42) ( | h | , h j ; θ ( f ) ); θ ( ρ ) (cid:1) and thus equation 2 for an appropriate choice of f (cid:42) .

Original: Not surprisingly, the computational savings obtained from k
Revised: Not surprisingly, the computational savings obtained from k

Original: -ary Janossy pooling come at the cost of reduced model ﬂexibility.
Revised: -ary Janossy pooling come at the cost of reduced model ﬂexibility.

Original: The next result formalizes this.
Revised: The next result formalizes this.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: For any k ∈ N , deﬁne F k as the set of all permutation invariant functions that can be represented by Janossy pooling with k -ary dependencies.
Revised: For any k ∈ N , deﬁne F k as the set of all permutation-invariant functions that can be represented by Janossy pooling with k -ary dependencies.

Original: Then, F k − 1 is a proper subset of F k if the space H is not trivial (i.e. if the cardinality of H is greater than 1 ).
Revised: Then, F k − 1 is a proper subset of F k if the space H is not trivial (i.e. if the cardinality of H is greater than 1 ).

Original: Thus, Janossy pooling with k -ary dependencies can express any Janossy pooling function with ( k − 1) -ary dependencies, but the converse does not hold.
Revised: Thus, Janossy pooling with k -ary dependencies can express any Janossy pooling function with ( k − 1) -ary dependencies, but the converse does not hold.

Original: Proof. ( F k − 1 ⊂ F k ): Consider any element f k − 1 ∈ F k − 1 . For any order-dependent function f (cid:42) , deﬁne f (cid:42) ↓ k ( h ) = f (cid:42) ( ↓ k ( h )) .
Revised: The proof is given in the Supplementary Material.

Original: Then, by deﬁnition, there exists a function f (cid:42) such that [Equation] ( F k (cid:54)⊂ F k − 1 ): Here, we need to construct a function f ∈ F k that is not present in F k − 1 . The construction is given in the Supplementary Material.
Revised: Theorem 2.1 has the following implication:

Original: Corollary 2.1.
Revised: Corollary 2.1.

Original: For k > 1 , the deep sets function in equation 1 (Zaheer et al., 2017) pushes the modeling of k -ary relationships to ρ .
Revised: For k > 1 , the deep sets function in equation 1 (Zaheer et al., 2017) pushes the modeling of k -ary relationships to ρ .

Original: Proof.
Revised: Proof.

Original: Deep sets functions can be expressed via Janossy pooling with k = 1 .
Revised: Deep Sets functions can be expressed via Janossy pooling with k = 1 .

Original: Thus, by Theorem 2.1, f in equation 2 cannot express all functions that can be expressed by higher-order (i.e. k > 1 )
Revised: Thus, by Theorem 2.1, f in equation 2 cannot express all functions that can be expressed by higher-order (i.e. k > 1 )

Original: Janossy pooling operations.
Revised: Janossy pooling operations.

Original: Consequently, if the deep sets function can express a premutationinvariant function, the expressive power must have been pushed to ρ .
Revised: Consequently, if the Deep Sets function can express any permutationinvariant function, the expressive power must have been pushed to ρ .

Original: 2. T RACTABILITY THROUGH P ERMUTATION S AMPLING
Revised: 2. T RACTABILITY THROUGH P ERMUTATION S AMPLING

Original: Another approach to tractable Janossy pooling uses random permutations of the input h during training.
Revised: Another approach to tractable Janossy pooling samples random permutations of the input h during training.

Original: Like the  approach in Section 2.1, this offers signiﬁcant computational savings, allowing more complex models for f (cid:42) such as LSTMs and GRUs.
Revised: Like the canonical ordering approach of Section 2.1, this offers signiﬁcant computational savings, allowing more complex models for f (cid:42) such as LSTMs and GRUs.

Original: Unlike thatearlier approach, this is considerable more ﬂexible, avoiding the need to learn a canonical-ordering or making assumptions about the dependencies between the elements of h and the objective function.
Revised: However, in contrast with that approach, this is considerably more ﬂexible, avoiding the need to learn a canonical ordering or to make assumptions about the dependencies between the elements of h and the objective function.

Original: The approach of sampling random permutations has been previously used in relational learning tasks (Moore & Neville, 2017; Hamilton et al., 2017) as a heuristic with an LSTM as f (cid:42) .
Revised: The approach of sampling random permutations has been previously used in relational learning tasks (Moore & Neville, 2017; Hamilton et al., 2017) as a heuristic with an LSTM as f (cid:42) .

Original: Both these papers report that permutation sampling outperforms or closely marches all other tested neural network models they tried.
Revised: Both these papers report that permutation sampling outperforms or closely matches other tested neural network models they tried.

Original: We provide a theoretical framework to understand and extend such approaches.
Revised: We provide a theoretical framework to understand and extend such approaches.

Original: For  simplicity, we will focus on a single Janossy pooling layer and a single sampled permutation, although generalizing these conditions is easy.
Revised: For the sake of simplicity, we analyze the optimization with a single sampled permutation. However, note that increasing the number of sampled permutations in the estimate of f decreases variance, and we recover the exact algorithm when all | h | ! permutations are sampled.

Original: We assume a supervised learning setting, though our analysis easily extends to unsupervised learning.
Revised: We assume a supervised learning setting, though our analysis easily extends to unsupervised learning.

Original: We are given training data D ≡ { ( x 1 , y 1 ) , . . .
Revised: We are given training data D ≡ { ( x (1) , y (1)) , . . .

Original: , ( x  N  , y  N ) } , where y  i  ∈ Y is the target output and x  i  its corresponding in- put.
Revised: , ( x ( N ) , y ( N )) } , where y ( i ) ∈ Y is the target output and x ( i ) its corresponding vector input.

Original: Our original goal was to minimize the empirical loss
Revised: Our original goal was to minimize the empirical loss

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: , where

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  and h ( i ) =
Revised: 7) and h ( i ) =

Original: h ( x  i ; θ ( h ) ) ∈∪ ∞ j =0 H j .
Revised: h ( x ( i ); θ ( h ) ) ∈ H ∪ with h ( i ) π ≡ ( h ( i ) )

Revised: π for permutation π .

Original: Computing the gradient of equation 9 is intractable for large inputs
Revised: Computing the gradient of equation 6 is intractable for large inputs

Original: h ( i ) , as the backpropagation computation graph branches out for every permutation in the sum.
Revised: h ( i ) , as the backpropagation computation graph branches out for every permutation in the sum.

Original: To address this computational challenge, we will turn out attention to stochastic optimization.
Revised: To address this computational challenge, we will turn our attention to stochastic optimization.

Original: Permutation sampling.
Revised: Permutation sampling.

Original: Consider replacing the Janossy sum in equation 10 with the estimate
Revised: Consider replacing the Janossy sum in equation 7 with the estimate

Original: [Equation]
Revised: [Equation]

Original: where s is a random permutation sampled uniformly, s ∼ Unif (Π | h | ) .
Revised: where s is a random permutation sampled uniformly, s ∼ Unif (Π | h | ) .

Original: The estimator in equation 2.3 is unbiased: E s [ˆ f (  h  ; θ ( f ) )]
Revised: The estimator in equation 8 is unbiased: E s [ ˆ f ( | h | , h s ; θ ( f ) )]

Original: =
Revised: =

Original: f
Revised: f

Original: (  h ; θ ( f ) ) .
Revised: ( | h | , h ; θ ( f ) ) .

Original: Note however that when f is chained with another nonlinear function ρ and/or nonlinear loss L , the composition is no longer unbiased: 
Revised: Note however that when f is chained with another nonlinear function ρ and/or nonlinear loss L , the composition is no longer unbiased: E s [ [Equation] ( y ,

Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: | ,

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ) ))] (cid:54) =

Revised: L

Revised: [Equation]

Revised: y ,

Revised: [Equation]

Revised: ( E s [ f (cid:42) ( | h s | , h s ; θ ( f ) )]

Revised: ; θ ( ρ ) )) .

Original: Using this estimate we propose the following stochastic approximation algorithm for gradient descent:
Revised: Nevertheless, we use this estimate to propose the following stochastic approximation algorithm for gradient descent:

Original: Deﬁnition 2.3: [ π -SGD] Let B = { ( x 1 , y 1 ) , . . .
Revised: Deﬁnition 2.3: [ π -SGD] Let B = { ( x (1) , y (1)) , . . .

Original: , ( x  B  , y  B ) } be a mini-batch i.i.d.
Revised: , ( x ( B ) , y ( B )) } be a mini-batch i.i.d.

Original: sampled uniformly from the training data D .
Revised: sampled uniformly from the training data D .

Original: At step t , consider the stochastic gradient descent update
Revised: At step t , consider the stochastic gradient descent update

Original: [Equation]
Revised: [Equation]

Original: where Z t is the random gradient
Revised: where Z t = 1 B (cid:80) Bi

Original: [Equation]
Revised: =1 ∇ θ L (cid:16) y ( i ) , ρ (cid:16) f (cid:42) ( | h ( i,t ) | , h ( i,t ) s i ; θ ( f ) t ); θ ( ρ ) t (cid:17)(cid:17) is the random gradient, where h ( i,t ) π ≡ ( h ( x ( i ); θ ( h ) t ))

Original: where h ( i ) = h ( x i ; θ ( h ) t ) , θ = ( θ ( ρ ) , θ ( f ) , θ ( h ) ) , with the random permutations { s i } Bi =1 ,  s i ∼ 
Revised: π for a permutation π , θ ≡ ( θ ( ρ ) , θ ( f ) , θ ( h ) ) , with the random permutations { s i } Bi =1 , sampled independently s i ∼ Uniform

Original: Uniform (Π | h ( i ) | ) ; the learning rate is η t ∈ (0 , 1) [Equation]
Revised: (Π | h ( i ) | ) ; the learning rate is η t ∈ (0 , 1) s.t.

Original: .t. [Equation] and [Equation]
Revised: lim t →∞ η n = 0 , (cid:80) ∞ t =1 η t = ∞ , and (cid:80) ∞ t =1 η 2 t < ∞ .

Original: ♦
Revised: ♦

Original: Effectively, this is a Robbins-Monro stochastic approximation algorithm of gradient descent (Robbins & Monro, 1951; Bottou & Cun, 2004) and optimizes the following modiﬁed objective function:
Revised: Effectively, this is a Robbins-Monro stochastic approximation algorithm of gradient descent (Robbins & Monro, 1951; Bottou & LeCun, 2004) and optimizes the following modiﬁed objective:

Original: [Equation]
Revised: [Equation]

Original: where h ( i ) π ≡ ( h ( x i ; θ ( h ) ))

Original: π for a permutation π .

Original: Observe that the expectation over permutations is now outside the L and ρ functions.
Revised: Observe that the expectation over permutations is now outside the L and ρ functions.

Original: Like equation 9, the loss in equation 12 is also permutation invariant.
Revised: Like equation 6, the loss in equation 10 is also permutation-invariant, though we note that π -SGD, after a ﬁnite number of iterations, returns a ρ ( f (cid:42) ( · · · , h ( i ) , · · · )) sensitive to the random input permutations of h ( i ) presented to the algorithm.

Original: In general though, the optima of J are different from those of the original objective function L .
Revised: Further, unless the function f (cid:42) itself is permutation-invariant ( f = f (cid:42) ), the optima of J are different from those of the original objective function L .

Original: In the setting where ρ is the identity function (equation 3), and L is convex, the function J is an upper bound to L (via Jensen’s inequality), and can be viewed as a tractable surrogate to the original Janossy objective. 
Revised: Instead, J is an upper bound to L via Jensen’s inequality if L is convex and ρ is the identity function (equation 3); minimizing this upper bound forms a tractable surrogate to the original Janossy objective. If the function-class used to model f (cid:42) is rich enough to include permutation-invariant functions, then the global minima of J will include those of L .

Original:  While this property no longer holds for general ρ , our results show that this holds for a composition f (cid:48) (cid:42) ≡ ρ ◦ f (cid:42) and still exhibits competitive results.In general, increasing the number of sampled permutations decreases bias and variance, and we recover the exact algorithm when all | h | ! permutations are sampled.
Revised: In general, minimizing the upper bound implicitly regularizes f (cid:42) to return functions that are as insensitive the permutations of the training data. While a general ρ no longer upper bounds the original objective, the implicit regularization of permutationsensitive functions still applies to the composition f (cid:48) (cid:42) ≡ ρ ◦ f (cid:42) and our results exhibit competitive results.

Original: The next result provides some insight into the covergence properties of ouralgorithm. Proposition 2.2 ( π -SGD Convergence) .
Revised: It is important to observe that the function ρ plays a very different role in our π -SGD formulation compared to k -ary

Original: Consider the π -SGD algorithm in Deﬁnition 2.3. If (a) there exists a constant G such that for all θ , G Tt θ ≤ G (cid:107) θ − θ (cid:63) (cid:107) 22 , where G t is the true gradient for the full batch over all permutations, G t = ∇ θ J ( D ; θ ( ρ ) t , θ ( f ) t , θ ( h ) t ) , where θ ≡ ( θ ( ρ ) , θ ( f ) , θ ( h ) ) .
Revised: Janossy pooling.

Original: (b) there exists a constant δ > 0 such that for all θ , E t [ (cid:107) Z t (cid:107) 22 ] ≤ δ 2 (1 + (cid:107) θ − θ (cid:63) (cid:107) 22 ) , where the expectation is taken with respect to all the data prior to step t . Then, the algorithm in equation 11 converges to θ (cid:63) with probability one.
Revised: Previously ρ was composed with an average over f (cid:42) to model dependencies not captured in the average– and was in some sense separate from f (cid:42) – whereas here it becomes absorbed directly into f (cid:42) (cid:48)

Original: Proof. First, we can show that E [ Z t ] =G t by equation 12, the linearity of the derivative operator, and the fact that the permutations are independently sampled for each training example in the minibatch and are assumed independent of θ .
Revised: =

Original: That equation 11 converges to θ (cid:63) is a consequence of our conditions and  thesupermartingale convergence theorem (Grimmett & Stirzaker, 2001, pp. 481). The following argument follows Yuille (2004).
Revised: ρ ◦ f (cid:42) . The next result, which we state and prove more formally in the Supplementary Material, provides some insight into the convergence properties of our algorithm.

Original: If A t = 12 (cid:107) θ t − θ (cid:63) (cid:107) 22 , B t = 12 δ 2 η 2 t , and C t = −(cid:107) θ t − θ (cid:63) (cid:107) 22 ( δ 2 η 2 t − G 2 η t ) , noting that C t is positive for a sufﬁciently large t , and (cid:80) ∞ t =1 B t ≤ ∞ by our deﬁnition of η t . The inequality E [ A t ] ≤ A t − 1 + B t − 1 − C t − 1 , for all t , is a consequence of E (cid:2) [Equation] θ
Revised: Proposition 2.2.

Original: [Equation] [Equation] E [Equation]
Revised: [ π -SGD Convergence]

Original: ( θ [Equation] ) T G t + η 2 t − 1 G 2 . Then, A t converges to zero with probability one and
Revised: The optimization of π -SGD enjoys properties of almost sure convergence to the optimal θ under similar conditions as SGD.

Original: [Equation] There are also ways to cope with the difference in objective functions between equation 9 and equation 12.
Revised: There are also ways to cope with the difference in objective functions between equation 6 and equation 10.

Original: An interesting approach adds an output regularization penalty for two distinct sampled permutations s i and s (cid:48)
Revised: An interesting approach adds an output regularization penalty for two distinct sampled permutations s i and s (cid:48)

Original: i , (cid:107) f (cid:42) (  h s i ( x ( i )  ; θ (h ) ); θ ( f ) ) − f (cid:42) (  h s(cid:48) i ( x ( i ) ; θ ( h) ); θ ( f ) ) 
Revised: i , (cid:107) f (cid:42) ( | h s i | , h ( i ) s i ; θ ( f ) ) − f (cid:42) ( | h s i | , h ( i ) s (cid:48)

Original:  (cid:107) 22 , so as to reduce the variance of the sampled Janossy pooling output.
Revised: i ; θ ( f ) ) (cid:107) 22 , so as to reduce the variance of the sampled Janossy pooling output.

Original: Such a variance-reduction procedure allows
Revised: Such a variance-reduction procedure allows

Original: E s [ L ( y  i  , f (cid:42) ( | h  s | , h  s ; θ ( f ) )]
Revised: E s [ L ( y ( i ) , f (cid:42) ( | h ( i ) s | , h ( i ) s ; θ ( f ) )]

Original: ≈ L ( y  i  , E s [ f (cid:42) ( | h  s | , h  s ; θ ( f ) )]) , inducing a near-equivalence between optimizing equation 9 and equation 12, assuming ρ is the identity or ρ is combined with f (cid:42) to form a single function f (cid:48) (cid:42) ≡ ρ ◦ f (cid:42) .
Revised: ≈ L ( y ( i ) , E s [ f (cid:42) ( | h ( i ) s | , h ( i ) s ; θ ( f ) )]) , inducing a near-equivalence between optimizing equation 6 and equation 10.

Original: This type of output-regularization-for-variance-reduction approach has been successfully used before to improve test performance of the averageDropout masks on
Revised: This type of output-regularization-for-variancereduction approach has been successfully used before to improve test performance of the average

Original:  LSTMs (Zolna et al., 2018).
Revised: Dropout masks on LSTMs (Zolna et al., 2018).

Original: Inference.
Revised: Inference.

Original: The use of π -SGD to optimize the Janossy pooling layer optimizes the objective J , and thus has the following implication on how outputs should be calculated at inference time:
Revised: The use of π -SGD to optimize the Janossy pooling layer optimizes the objective J , and thus has the following implication on how outputs should be calculated at inference time:

Original: Remark 2.2 (Inference) .
Revised: Remark 2.2 (Inference) .

Original: Assume L is convex (e.g., L is the L2, cross entropy, or negative loglikelihood losses), which via Jensen’s inequality makes π -SGD a proper surrogate to the original
Revised: Assume L is convex (e.g., L is the L 2 norm, cross-entropy, or negative loglikelihood losses), which via Jensen’s inequality makes π -SGD a proper surrogate to the original

Original: Janossy objective.
Revised: Janossy objective.

Original: At test time we estimate the output 
Revised: At test time we estimate the output y

Original: y i  of input x  i  by computing (or estimating)
Revised: ( i ) of input x ( i ) by computing (or estimating)

Original: [Equation]
Revised: [Equation]

Original: where f (cid:48) (cid:42) ≡ ρ ◦ f (cid:42) , θ ( f (cid:48) )
Revised: where f (cid:48) (cid:42) ≡ ρ ◦ f (cid:42) , θ ( f (cid:48) )

Original: (cid:63) ≡ ( θ ( f ) (cid:63) , θ ( ρ ) (cid:63) ) , h ( i )(cid:63) s i ≡ ( h ( x  i ; θ ( h ) (cid:63) ))
Revised: (cid:63) ≡ ( θ ( f ) (cid:63) , θ ( ρ ) (cid:63) ) , h ( i,(cid:63) ) s i ≡ ( h ( x ( i ); θ ( h ) (cid:63) ))

Original: s i and θ ( ρ ) (cid:63) , θ ( f ) (cid:63) , θ ( h ) (cid:63) are ﬁxed points of the π -SGD optimization.
Revised: s i and θ ( ρ ) (cid:63) , θ ( f ) (cid:63) , θ ( h ) (cid:63) are ﬁxed points of the π -SGD optimization.

Original: Equation 13 is a permutation-invariant function.
Revised: Equation 11 is a permutation-invariant function.

Original: Our work has a strong connection with ﬁnite exchangeability.
Revised: Combining π -SGD and Janossy with k -ary Dependencies

Original: Some researchers may be more familiar with the concept of inﬁnite exchangeability through de Finetti’s theorem (De Finetti, 1937;
Revised: In some cases one may consider k -ary

Original: Diaconis, 1977), which imposes strong structural requirements: the probability of any subsequence must equal the marginalized probability of  the original sequence.
Revised: Janossy pooling with a moderately large value of k in which case even the summation over | h | ! ( | h |− k )! terms (see proposition 2.1) becomes expensive.

Original: Finite exchangeability drops this projectivity requirement (Diaconis, 1977), which in general, cannot be simpliﬁed beyond ﬁrst sampling the number of observations m , and then sampling their locations from some exchangeable but non-i.i.d.
Revised: In these cases, one may sample s ∼ Unif (cid:0) Π | h | (cid:1) and compute ˆ f k = f (cid:42) ( k, ↓ k ( h s ); θ ( f ) ) in lieu of the sum in equation 5.

Original: distribution p m xchgx ( x 1 , . .
Revised: Note that equation 5 deﬁning k

Original: x m ) (Daley & Vere-Jones, 2003).
Revised: -ary

Original: Finite exchangeability also arises from the theory of spatial point processes; our framework of Janossy pooling is inspired by Janossy densities (Daley & Vere-Jones, 2003), which model the ﬁnite exchangeable distributions as mixtures of non-exchangeable distributions applied to permutations.
Revised: Janossy pooling constitutes exact inference of a simpliﬁed model whereas π -SGD with k -ary dependencies constitutes approximate inference.

Original: This literature also studies simplied exchangeable point processes such as ﬁnite Gibbs models (Vo et al., 2018; Moller & Waagepetersen, 2003) thatrestrict the structure ofp xchg to ﬁxed-order dependencies, and are related to k -ary Janossy.
Revised: We will return to this idea in our results section where we note that the GraphSAGE model of Hamilton et al. (2017) can be cast as a π -SGD approximation of k -ary Janossy pooling.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In what follows we empirically evaluate two tractable Janossy pooling approaches, k -ary dependencies (section 2.2) and sampling permutations for stochastic optimization (section 2.3), to learn permutation-invariant functions for tasks of different complexities.
Revised: In what follows we empirically evaluate two tractable Janossy pooling approaches, k -ary dependencies (section 2.2) and sampling permutations for stochastic optimization (section 2.3), to learn permutation-invariant functions for tasks of different complexities.

Original: One baseline we compare against is deep sets (Zaheer et al., 2017); recall that this corresponds to unary ( k = 1 )
Revised: One baseline we compare against is deep sets (Zaheer et al., 2017); recall that this corresponds to unary ( k = 1 )

Original: Janossy pooling (Remark 2.1).
Revised: Janossy pooling (Remark 2.1).

Original: Corollary 2.1 shows that explicitly modeling higher-order correlations simpliﬁes the task of the upper layers  of the neural network, and we evaluate the gains of increasing k = 1 , 2 , 3 , | h | over different  tasks.
Revised: Corollary 2.1 shows that explicitly modeling higher-order dependencies during pooling simpliﬁes the task of the upper layers ( ρ ) of the neural network, and we evaluate this experimentally by letting k = 1 , 2 , 3 , | h | over different arithmetic tasks.

Original: We also evaluate Janossy pooling in graph tasks, where it can be used as a permutation-invariant function to aggregate the features and embeddings of the neighbors of a node in the graph.
Revised: We also evaluate Janossy pooling in graph tasks, where it can be used as a permutation-invariant function to aggregate the features and embeddings of the neighbors of a vertex in the graph.

Original: Note that in graph tasks, permutation-invariance is required to ensure that the neural network is invariant to permutations in the adjacency matrix (graph isomorphism).
Revised: Note that in graph tasks, permutation-invariance is required to ensure that the neural network is invariant to permutations in the adjacency matrix (graph isomorphism).

Revised: The code used to generate the results in this section are available on GitHub 2 .

Original: 3. A RITHMETIC T ASKS ON S EQUENCES OF I NTEGERS
Revised: 3. A RITHMETIC T ASKS ON S EQUENCES OF I NTEGERS

Original: We ﬁrst consider the task of predicting the sum of a sequence of integers (Zaheer et al., 2017), and extend it to consider other harder tasks: range , unique sum ,and unique count  .
Revised: We ﬁrst consider the task of predicting the sum of a sequence of integers (Zaheer et al., 2017) and extend it to predicting other permutation-invariant functions: range , unique sum , unique count , and variance .

Original: In the sum task we predict the sum of a sequence of 5 integers drawn uniformly  with replacement from { 0 , 1 , . . .
Revised: In the sum task we predict the sum of a sequence of 5 integers drawn uniformly at random with replacement from { 0 , 1 , . . .

Original: , 99 } ; the range task also receives a sequence 5 integers distributed the same way, and tries to predict the range (the difference between the maximum and minimum values); the unique sum task receives a sequence of 10 integers, sampled uniformly with replacement from { 0 , 1 , . .
Revised: , 99 } ; the range task also receives a sequence 5 integers distributed the same way and tries to predict the range (the difference between the maximum and minimum values); the unique sum task receives a sequence of 10 integers, sampled uniformly with replacement from { 0 , 1 , . .

Original: , 9 } , and predicts the sum of all unique elements; the unique count task  receives a sequence of repeating elements, sampled uniformly with replacement from { 0 , 1 , . .
Revised: , 9 } , and predicts the sum of all unique elements; the unique count task also receives a sequence of repeating elements from { 0 , 1 , . . , 9 } , distributed in the same was as with the unique sum task, and predicts the number of unique elements; the variance task receives a sequence of 10 integers drawn uniformly with replacement from { 0 , 1 , . .

Original:  , 9 }, and predicts the number of unique elements.
Revised: . , 99 } and tries to predict the variance 1 | x | (cid:80) i ( x i − ¯ x ) 2 = 1 2 | x | 2 (cid:80) i,j ( x i − x j ) 2 , where ¯ x denotes the mean of x .

Original: Unlike Zaheer et al.
Revised: Unlike Zaheer et al.

Original: , we  work with the digits themselves, rather than with MNIST images. This allows a more direct assessment of the different Janossy pooling approximations, and it is a simple matter to pass digit images through another neural network layer before applying our algorithms.
Revised: (2017), we choose to work with the digits themselves, to allow a more direct assessment of the different Janossy pooling approximations. Note that the summation task of Zaheer et al.

Original: Note that sum task maps directly to the architecture of deeps sets (which computes embeddings of the individual digits and adds them together). However the other taskswe selected involved calculating permutation-invariant quantities that require reasoning about high-order relationships within the sequence. 
Revised: (2017) is naturally a unary task that lends itself to the approach of embedding individual digits then adding them together while the other tasks require exploiting high-order relationships within the sequence. Following Zaheer et al. , we report accuracy (0-1 loss) for all tasks with an integer target; we report root mean squared error (RMSE) for the variance task.

Original: We explore several Janossy pooling variants.
Revised: Here we explore two Janossy pooling tractable approximations: (a) ( k -ary dependencies)

Original: First are Deep Sets, or Janossy ( k = 1 )  and Janossy( k = 2 , 3) where f (cid:42) is a feedforward network with a single hidden layer comprised of 30 neurons. 
Revised: Janossy ( k = 1 ) (Deep Sets) , and Janossy k = 2 , 3 where f (cid:42) is a feedforward network with a single hidden layer comprised of 30 neurons. As detailed in the Supplementary

Original: Second, full ( k = | h | )
Revised: Material, the models are constructed to have the same number of parameters regarless of k by modifying the embedding (output) dimension of h . (b) ( π -SGD) Full k

Original:  Janossy pooling where f (cid:42) is an LSTM or a GRU (50 and 80 units in the hidden state, respectively) and the LSTM or GRU hidden state of the last temporal unitis the output of f (cid:42) (the c t of Cho et al.with t
Revised: = | h | Janossy pooling where f (cid:42) is an LSTM or a GRU that returns the short-term hidden state of the last temporal unit (the h t of Cho et al.

Original:  = | h | ).
Revised: with t = | h | ).

Original: The full Janossy pooling trained with  π -SGD is denoted JanossyM inf-LSTM andJanossy- M inf-GRU , where the value of M is the  number of  sampled permutations used to estimate ˆ y i ( x i ) of equation 13 at test time for a test sequence x i .
Revised: The LSTM has 50 hidden units and the GRU 80, trained with the π -SGD stochastic optimization. The number of hidden units was chosen so that the LSTM and the GRU have the same number of parameters (and the same number of parameters as the MLP (30) models described above). At test time, we experiment with approximating (estimating) equation 11 using 1 and 20 sampled permutations. We also explore two functions for ρ of equation 4 (upper-layer): (i)

Original: To deﬁne ρ , we followed the experimental setup of Zaheer et al.
Revised: [Linear] a single dense layer with identity activation as in the experiments of Zaheer et al.

Original: (2017), which uses a single dense layer with identity activation.
Revised: (2017), and (ii) [MLP (100)] a feedforward network with one hidden layer using tanh activations and 100 units.

Original: A consequence of this choice is that it provides direct insight into the  capacity of f to exploit kwise relationships.
Revised: Choosing a simple and complex form for ρ allows insight into the extent to which ρ supplements the capacity of the model by capturing relationships not exploited during pooling, and serves as an evaluation of the strategy of optimizing J as a tractable approximation of L .

Original:  We have also tested these tasks where ρ is a feedforward network with one hidden layer using tanh activations and 100 units; some preliminary results from this more powerful nonlinear ρ are also reported.
Revised: Much of our implementation, architectural, and experimental design are based on the deep sets codeof Zaheer et al. (2017), see the Supplementary Material for details. We tuned the Adam learning rate for each model and report the results using the rate yielding top performance on the validation set.

Original: Table 1 shows the accuracy (average 0-1 loss)  for  15 runs of the above Janossy pooling approximations under the sum , range , unique sum , and unique count tasks with ρ being a dense layer with identity activation.
Revised: Table 1 shows the accuracy (average 0-1 loss) of all tasks except variance, for which we report RMSE in the last column. We trained each model with 15 random intializations of the weights to quantify variability.

Original: Table 3 in the Supplementary Material shows the same results measured by mean absolute error.
Revised: Table 3 in the Supplementary Material shows the same results measured by mean absolute error.

Original: The data consists of 100,000 training examples and 10,000 test examples.
Revised: The data consists of 100,000 training examples and 10,000 test examples.

Original: ( Sum task) Note that the sum task (Zaheer et al., 2017) is easy for all Janossy pooling approximations. Also note that for the methods trained with π -SGD(LSTM and GRU), using 20 samples instead of one to estimate equation 13 yields signiﬁcant accuracy gains. ( Range , unique sum , and unique count tasks)
Revised: The results in Table 1 show that: (1) models trained with π -SGD using LSTMs and GRUs as f (cid:42) typically achieve top performance or are comparable to the top performer (within conﬁdence intervals) on all tasks, for any choice of ρ . We also observe for LSTMs and GRUs that adding complexity to ρ can yield small but meaningful performance gains or maintain similar performance, lending credence to the approach of optimizing J as a tractable approximation to L .

Original: These results corroborate both theorem 2.1 and corollary 2.1, showing that performance increases consistently withk on the tasks that require reasoning about relationships within elements of the sequence. This serves as evidence that optimizing using π -SGD yields strong accuracy gainsfor modeling the entire sequence, albeit over the modiﬁed loss
Revised: (2) Speciﬁcally, in the variance task, GRUs and LSTMs with π -SGD provide signiﬁcant accuracy gains over k ∈ { 1 , 2 , 3 } , showing that modeling full-dependencies can be advantageous even if model training with π -SGD is approximate.

Original: J of equation 12.
Revised: For a more complex ρ (MLP as opposed to Linear), lower-complexity Janossy pooling achieves consistently better results: k ∈ { 2 , 3 } gives good results when ρ is linear but poorer results when ρ is an MLP (as these models are more expressive, the only feasible explanation is an optimization issue as the performance is similarly not as good on the training data).

Original: For the methods trained with π -SGD (LSTM and GRU), we also see signiﬁcant gains of using 20 samples instead of one to estimate equation 13, whenever such gains are possible.
Revised: We also note that when ρ is an MLP, it takes a signiﬁcantly more epochs for k ∈ { 2 , 3 } to ﬁnd the best model (2000 epochs) while k = 1 ﬁnds good models much quicker (1000 epochs).

Original: When ρ is a feedforward network with one hidden layer (and 100 hidden units) with tanh activations, our preliminary results (Table 5 in the Supplementary Material) show that Janossy k = 1 performs [Table]
Revised: The results we report come from training with 1000 epochs on all models with a linear ρ and 2000 epochs for all models where ρ is an MLP. (4) We observe that for k = 1 (Deep Sets), a more complex ρ (MLP) is required as the pooling pushes the complexity of modeling high-order interactions over the input to ρ .

Original: [Table] similarly to Table 1 in the range task, while Janossy  k = 2 , 3 perform signiﬁcantly better. For the  unique sum and unique counttasks Janossy k = 1 , 2 , 3 have signiﬁcant improvements, while Janossy k = 2 , 3 either outperforms k = 1 or performs equally.
Revised: The converse is also true, if ρ is simple (Linear) then a Janossy pooling that models high-order interactions k ∈ { 2 , 3 , | h |} gives higher accuracy, speciﬁcally LSTMs and GRUs as shown in the range , unique sum , unique count , and variance tasks. 3. J ANOSSY POOLING AS AN AGGREGATOR FUNCTION FOR VERTEX CLASSIFICATION

Original: These preliminary results also show that Janossy LSTMs and GRUs perform worse when ρ is nonlinear, possibly due to  the larger divergence between the objective functions equation 9 and equation 12 when ρ is nonlinear.
Revised: Here we consider Janossy pooling in the context of graph neural networks to learn vertex representations enabling vertex classiﬁcation. The GraphSAGE algorithm (Hamilton et al., 2017) consists of sampling vertex attributes from the neighbor set of each vertex v before performing an aggregation operation which generates an embedding of v ; the authors consider permutation-invariant operations such as mean and max as well as the permutation-sensitive operation of feeding a randomly permuted neighborhood sequence to an LSTM.

Original: Investigating this further isa topic for in future work.
Revised: The sample and aggregate procedure is repeated twice to generate an embedding. Each step can be considered as Janossy pooling with π -SGD with k -ary subsequences, where k l , l ∈ { 1 , 2 } is the number of vertices sampled from each neighborhood and f (cid:42) is for instance a mean, max, or LSTM.

Original: 3. J ANOSSY POOLING AS AN AGGREGATOR FUNCTION FOR VERTEX CLASSIFICATION
Revised: However, at test time, GraphSAGE only samples one permutation s of each neighborhood to estimate equation 11.

Original: Here we consider the task of vertex classiﬁcation over two graph datasets: Cora and Pubmed (Sen et al., 2008).
Revised: In this section we investigate two conditions: (a) the impact of increasing k in the k -ary dependencies; and (b) the beneﬁts of increasing the number of sampled permutations at inference time. To implement the model and design our experiments, we modiﬁed the reference PyTorch code provided [Table] by the authors 4 .

Revised: We consider the three graph datasets considered in Hamilton et al.

Revised: (2017): Cora and Pubmed (Sen et al., 2008) and the larger Protein-Protein Interaction (PPI) (Zitnik & Leskovec, 2017).

Original: These datasets are citation networks where vertices represent papers, edges represent citations, and vertex features are bag-of-words representations of the document text.
Revised: The ﬁrst two are citation networks where vertices represent papers, edges represent citations [Table] and vertex features are bag-of-words representations of the document text.

Original: The task is to classify the paper topic.
Revised: The task is to classify the paper topic.

Revised: The PPI dataset is a collection of several graphs each representing human tissue; vertices represent proteins, edges represent protein interaction, features include genetic and immunological features, and we try to classify protein roles.

Original: More details of these datasets and our experimental test/train splits are shown in Table 4 in the Supplementary Material.
Revised: More details of these experiments are shown in Table 8 in the Supplementary Material.

Original: We use the graph convolutional model of Hamilton et al. (2017) for its simplicity and the use what the authors call Mean Pooling (Janossy pooling k = 1 ) and LSTM aggregator . The LSTM aggregator is effectively a Janossy pooling approximation using an LSTM for f (cid:42) and trained with π -SGD, described as Janossy-1inf-LSTM in our results. At test time, however, Hamilton et al. estimates the class label using equation 13 but with a single permutation.
Revised: (a) Table 2 shows the impact (on accuracy) of increasing the number of k -ary dependencies.

Original: Their experiments indicate that mean pooling (Janossy pooling k= 1 ) yields better accuracy than the LSTM approximation on citation network tasks.
Revised: We use k 1 , k 2 ∈ { 3 , 5 , 10 , 25 } for the two pooling layers of our graph neural network (GNN). The function f (cid:42) is an LSTM (except when we try mean-pooling).

Original: Here, we show that gains ofmean pooling over the LSTM aggregator reported in Hamilton et al.
Revised: Note that even for the LSTM, the number of parameters of the model is independent of k .

Original: vanish once we provide a better estimate of equation 13. We note in passing that we have closely followed experiment setup of Hamilton et al.
Revised: At inference time, we sample 20 random permutations of each sequence and average the predicted probabilities before making a ﬁnal prediction of the class label.

Original: detailed in Section C.1 of the Supplementary Material. The Micro-F1 scores (same metric used by Hamilton et al.) are averaged over 21 runs and shown in Table 2, along with the Janossy k= 1 approximation (the mean pooling aggregator of Hamilton et al.).
Revised: The results in Table 2 show that the choice of k 1 , k 2 ∈ { 3 , 5 , 10 , 25 } makes little difference on Cora and Pubmed due to the small neighborhood sizes: k 1 , k 2 ≥ 5 often amounts to sampling the entire neighborhood. In PPI, whose average degree is 28.8, increasing k yields consistent improvement.

Original: As in (Hamilton et al., 2017), Janossy k = 1 seems to have higher accuracy than Janossy-1infLSTM (one permutation sample for estimating the node label using equation 13). However, with a label estimate averaged over 20 permutation samples (Janossy-20inf-LSTM), these gains disappear and Janossy-20inf-LSTM accuracy is not signiﬁcantly different from mean pooling (unary Janossy).
Revised: The strong performance of mean-pooling points to both a relatively easy task 5 and the beneﬁts of utilizing the entire neighborhood of each vertex. (b) We now investigate if increasing the number of sampled permutation to better estimate equation 11 at test (inference) time impacts accuracy.

Original: The absence of difference between the unary Janossy and Janossy-LSTM pooling approximations points to an easy task. It would have been pointless to try k -ary
Revised: Figure 2 in the Supplementary Material shows that going from one to three sampled permutations lead to a reasonable increase in accuracy in the PPI task (Cora and Pubmed degrees are too small for this test).

Original: Janossy pooling approximations in these tasks.
Revised: The beneﬁts have diminishing returns with more samples.

Original: It is reasonable to suppose that the topic of a paper can be adequately predicted by computing the average bag-of-words representations of papers in the neighborhood without reasoning about relationships between neighboring papers (i.e. it may be a small- k task). For example, a high proportion of mentions of neural in neighboring papers is suggestive about the paper topic and can be computed independently for each paper.
Revised: Using paired tests – t and Wilcoxon signed rank – we see that test inference with seven sampled permutations versus one permutation is signiﬁcant with p < 10 − 3 over 12 replicates.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Under the Janossy pooling framework presented in this work, existing literature falls under one of three approaches to approximating to the intractable Janossy-pooling layer: Canonical orderings , k -ary dependencies , and permutation sampling .
Revised: Under the Janossy pooling framework presented in this work, existing literature falls under one of three approaches to approximating to the intractable Janossy-pooling layer: Canonical orderings , k -ary dependencies , and permutation sampling .

Original: Canonical Ordering Approaches.
Revised: Canonical Ordering Approaches.

Original: In section 2.1, we saw how permutation invariance can be achieved by mapping permutations to a canonical ordering.
Revised: In section 2.1, we saw how permutation invariance can be achieved by mapping permutations to a canonical ordering.

Original: Rather than trying to deﬁne a good canonical ordering, one can try to learn it from the data, however searching among all  h  !
Revised: Rather than trying to deﬁne a good canonical ordering, one can try to learn it from the data, however searching among all | h | !

Original: | permutations for one that correlates with the task of interest is a difﬁcult discrete optimization problem.
Revised: permutations for one that correlates with the task of interest is a difﬁcult discrete optimization problem.

Original: Recently, Rezatoﬁghi et al.
Revised: Recently, Rezatoﬁghi et al.

Original: (2018) proposed a method that computes the posterior distribution of  all permutations, conditioned on the model and the data.
Revised: (2018) proposed a method that computes the posterior distribution of [Table] all permutations, conditioned on the model and the data.

Original: This posterior-sampling approach is intractable for large inputs, unfortunately.
Revised: This posterior-sampling approach is intractable for large inputs, unfortunately.

Original: We note in passing that Rezatoﬁghi et al.
Revised: We note in passing that Rezatoﬁghi et al.

Original: (2018) is interested in permutation-invariant outputs, and that Janossy pooling is also trivially applicable to these tasks.
Revised: (2018) is interested in permutation-invariant outputs [Table] and that Janossy pooling is also trivially applicable to these tasks.

Original: Vinyals et al.
Revised: Vinyals et al.

Original: (2016) proposes a heuristic using ancestral sampling while learning the model.
Revised: (2016) proposes a heuristic using ancestral sampling while learning the model.

Original: k -ary Janossy Pooling Approaches.
Revised: k -ary Janossy Pooling Approaches.

Original: In section 2.2 we described k -ary
Revised: In section 2.2 we described k -ary

Original: Janossy pooling, which considers k -order relationships in the input vector h to simplify optimization.
Revised: Janossy pooling, which considers k -order relationships in the input vector h to simplify optimization.

Original: Deep Sets (Zaheer et al., 2017) can be characterized as unary Janossy pooling (i.e., k -ary for k = 1 ).
Revised: Deep Sets (Zaheer et al., 2017) can be characterized as unary Janossy pooling (i.e., k -ary for k = 1 ).

Original: Our experiments considered tasks that see signiﬁcant gains from higher-order k

Original: -ary

Original: Janossy pooling, k > 1 .

Original: Ravanbakhsh et al.
Revised: Ravanbakhsh et al.

Original: (2017) proposes a similar unary Janossy pooling to deep sets .
Revised: (2017) proposes a similar unary Janossy pooling to deep sets .

Original: Cotter et al.
Revised: Cotter et al.

Original: (2018) proposes to add inductive biases to the deep sets model in the form of monotonicity constraints with respect to the vector valued elements of the input sequence by modeling φ and ρ with Deep Lattice Networks (You et al., 2017); one can extend Cotter et al.
Revised: (2018) proposes to add inductive biases to the deep sets model in the form of monotonicity constraints with respect to the vector valued elements of the input sequence by modeling f and ρ with Deep Lattice Networks (You et al., 2017); one can extend Cotter et al.

Original: by using higher-order k -ary Janossy pooling for k > 1 .
Revised: by using higher-order ( k > 1 ) pooling.

Original: Exploiting dependencies within a sequence to learn a permutation invariant function has been discussed elsewhere.
Revised: Exploiting dependencies within a sequence to learn a permutation-invariant function has been discussed elsewhere.

Original: For instance Santoro et al.
Revised: For instance Santoro et al.

Original: (2017) exploits pairwise relationships to perform relational reasoning about pairs of objects in an image  Battaglia et al.
Revised: (2017) exploits pairwise relationships to perform relational reasoning about pairs of objects in an image and Battaglia et al.

Original: (2018) contemplates modeling the center of mass of a solar system by including the pairwise interactions among planets.
Revised: (2018) contemplates modeling the center of mass of a solar system by including the pairwise interactions among planets.

Original: However, Janossy pooling provides a general framework for capturing dependencies within a permutation invariant pooling layer.
Revised: However, Janossy pooling provides a general framework for capturing dependencies within a permutation-invariant pooling layer.

Original: Permutation Sampling Approaches.
Revised: Permutation Sampling Approaches.

Original: In section 2.3 we have seen a that permutation sampling can be used as a stochastic gradient procedure ( π -SGD) to learn a model with a Janossy pooling layer.
Revised: In section 2.3 we have seen a that permutation sampling can be used as a stochastic gradient procedure ( π -SGD) to learn a model with a Janossy pooling layer.

Original: The learned model provides only an approximate solution to original permutation-invariant function.
Revised: The learned model provides only an approximate solution to original permutation-invariant function.

Original: Permutation sampling has been used as a heuristic (without a theoretical justiﬁcation) in both Moore & Neville (2017) and Hamilton et al.
Revised: Permutation sampling has been used as a heuristic (without a theoretical justiﬁcation) in both Moore & Neville (2017) and Hamilton et al.

Original: (2017), which found that randomly permuting sequences and feeding them forward to an LSTM is effective in relational learning tasks that require permutation invariant pooling layers.
Revised: (2017), which found that randomly permuting sequences and feeding them forward to an LSTM is effective in relational learning tasks that require permutation-invariant pooling layers.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Our approach of permutation-invariance through Janossy pooling uniﬁes a number of existing approaches, and opens up avenues to develop both new methodological extensions, as well as better theory.
Revised: Our approach of permutation-invariance through Janossy pooling uniﬁes a number of existing approaches, and opens up avenues to develop both new methodological extensions, as well as better theory.

Original: Our paper focused on two main approaches: k -ary interactions and random permutations.
Revised: Our paper focused on two main approaches: k -ary interactions and random permutations.

Original: The former involves exact Janossy pooling for a restricted class of functions f (cid:42) .
Revised: The former involves exact Janossy pooling for a restricted class of functions f (cid:42) .

Original: Adding an additional neural network ρ can recover lost model capacity, but now hurts learnability.
Revised: Adding an additional neural network ρ can recover lost model capacity, but now hurts tractability and identiﬁability.

Original: Placing restrictions on ρ (convexity, Lipschitz continuity etc.) can allow a more reﬁned control of this trade-off, allowing theoretical and empirical work to shed light on the compromises involved.
Revised: Placing restrictions on ρ (convexity, Lipschitz continuity etc.) can allow a more reﬁned control of this trade-off, allowing theoretical and empirical work to shed light on the compromises involved.

Original: The  random permutation approach  involves no clear trade-offs between model capacity, and learnability, instead it approximates the  Janossy loss Lwith a more tractable approximation J .
Revised: The second was a random permutation approach which conversely involves no clear trade-offs between model capacity and computation when ρ is made more complex, instead it modiﬁes the relationship between the tractable approximate loss J and the original Janossy loss L .

Original: The difference between  the two reduces as more permutations are sampled, however, despite excellent empirical performance, it is not clear what problems this is best suited for.
Revised: While there is a difference between J and L , we saw the strongest empirical performance coming from this approach in our experiments (shown in the last row of Table 1); future work is required to identify which problems π -SGD is best suited for.

Original: Better understanding how the loss-functions L and J relate to each other, can shed light on the slightly black-box nature of this procedure.
Revised: Further, a better understanding how the loss-functions L and J relate to each other can shed light on the slightly black-box nature of this procedure.

Original: It is also important to understanding the relationship between the random-permutation approach to canonical ordering, and how one might be used to improve the other.
Revised: It is also important to understand the relationship between the random permutation optimization to canonical ordering and how one might be used to improve the other.

Original: Finally, it is important to apply our methodology to a wider range of applications.
Revised: Finally, it is important to apply our methodology to a wider range of applications.

Original: Two immediate domains are more challenging tasks involving graphs and tasks involving non-Poisson point processes.
Revised: Two immediate domains are more challenging tasks involving graphs and tasks involving non-Poisson point processes.

Original: A P ROOFS OF R ESULTS
Revised: A P ROOFS OF R ESULTS

Original: We restate and prove the remaining portion of theorem 2.1.
Revised: We restate and prove the remaining portion of Theorem 2.1.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: For any k ∈ N , deﬁne F k as the set of all permutation invariant functions that can be represented by Janossy pooling with k -ary dependencies.
Revised: For any k ∈ N , deﬁne F k as the set of all permutation-invariant functions that can be represented by Janossy pooling with k -ary dependencies.

Original: Then, F k − 1 is a proper subset of F k if the space H is not trivial (i.e. if the cardinality of H is greater than 1 ).
Revised: Then, F k − 1 is a proper subset of F k if the space H is not trivial (i.e. if the cardinality of H is greater than 1 ).

Original: Thus, Janossy pooling with k -ary dependencies can express any Janossy pooling function with ( k − 1) -ary dependencies, but the converse does not hold.
Revised: Thus, Janossy pooling with k -ary dependencies can express any Janossy pooling function with ( k − 1) -ary dependencies, but the converse does not hold.

Original: Proof.
Revised: Proof.

Revised: ( F k − 1 ⊂ F k ): Consider any element f k − 1 ∈ F k − 1 , and write f (cid:42) ( | h | , · ; θ ( f ) ) for its associated

Revised: Janossy function.

Revised: For any sequence h , f (cid:42) ( | h | ,

Revised: [Equation]

Revised: [Equation]

Original:  ( F k (cid:54)⊂ F k 1 ):
Revised: ); θ ( [Equation] = f (cid:42) ( | h | , ↓

Revised: [Equation]

Revised: −

Revised: [Equation]

Original: We will demonstrate the existence of f  k ∈ F k such that f  k − 1 (cid:54) =
Revised: ( h )); θ ( f ) ) : = f (cid:42) + ( | h | , ↓ k ( h ); θ ( f ) ) , where the function f (cid:42) + looks at its ﬁrst k elements. Thus,

Revised: [Equation]

Revised: (12) where f k is the Janossy function associated with f (cid:42) + and thus belongs to F k .

Revised: ( F k (cid:54)⊂ F k − 1 ): As noted previously, the case where k = 1 is trivial, so assume k > 1 .

Revised: We will demonstrate the existence of f k ∈ F k such that f k − 1 (cid:54) =

Original: f k for all f k − 1 ∈ F k − 1 .
Revised: f k for all f k − 1 ∈ F k − 1 .

Revised: Let f k and f k −be associated with f (cid:42) k and f (cid:42) k − 1 , respectively.

Original: It sufﬁces to consider | x | = k .
Revised: It sufﬁces to consider | h | = k .

Original: Let f (cid:42) k
Revised: Let f (cid:42) k

Original: ( | x | , x π ; θ  ) = (cid:81)
Revised: ( | h | , h π ; θ ( f ) k ) = (cid:81)

Original: | x | l =1 x π ( l ) whence f k ( x π ; θ  ) 
Revised: | h | l =1 h π ( l ) whence f k ( | h | , h ; θ ( f ) k ) =

Original: = (cid:81) | x | l =1 x l .
Revised: (cid:81) | h | l =1 h l .

Original: Thus, for any f k − 1  ,[Equation] =
Revised: Thus, for any f k − 1 and any θ ( f ) k − 1 ,

Original: [Equation]
Revised: [Equation]

Original: where Π { 1 ,..., | x |}\ j is the set of permutation functions of { 1 , 2 , . .
Revised: where Π { 1 ,..., | h |}\ j is the set of permutation functions of { 1 , 2 , . .

Original: , j − 1 , j +1 , . .
Revised: , j − 1 , j +1 , . .

Original: , | x |} and ( x − j )
Revised: , | h |} and ( h − j )

Original: ˜ π is a permutation of { x 1 , . . .
Revised: ˜ π is a permutation of { h 1 , . . .

Original: , x j − 1 , x j +1 , . . .
Revised: , h j − 1 , h j +1 , . . .

Original: , x | x |} .This can be written as 
Revised: h | h | } .This can be written as | h | !

Revised: | h | (cid:88)

Revised: j =  h j (cid:16) (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:123)(cid:122) (cid:125) denote by a j, | x | , whence
Revised: denote by a j

Original: [Equation]
Revised: [Equation]

Revised: | , therefore

Original: [Equation]
Revised: [Equation]

Original: Now, f k − 1 = f
Revised: Now, f k − 1 = f

Original: k if and only if their quotient in equation 14 is unity for all x .
Revised: k if and only if their quotient in equation 13 is unity for all h .

Original: But this is clearly not possible in general unless X is a singleton, which we have precluded in our assumptions.
Revised: But this is clearly not possible in general unless H is a singleton, which we have precluded in our assumptions.

Original: B DE F INETTI ’ S T HEOREMS
Revised: Proposition 2.2 is repeated below and is followed by a more rigorous restatement.

Original:  The inﬁnite form of de Finetti’s theorem arises when contemplating an inﬁnite sequence of exchangeable random variables or when contemplating a ﬁnite sequence whose distribution could arise as a marginalization of an inﬁnitely long exchangeable sequence (Diaconis, 1977; Orbanz & Roy, 2015;
Revised: Proposition 2.2. [ π -SGD Convergence] The optimization of π -SGD enjoys properties of almost sure convergence to the optimal θ under similar conditions as SGD. The following statement is similar to that in Yuille (2004), which also provides intuition behind the theoretical assumptions, which are indeed quite general.

Original: Gelman et al., 2014).
Revised: See also (Younes, 1999).

Original: When this holds, the data distribution p ( X 1 , X 2 , . . . ) can be represented as a mixture of some prior distribution p ( θ ) for parameter θ ∈ Θ and a conditionally iid likelihood for the data, given θ : in the case of an inﬁnite exchangeable sequence 1 ,
Revised: This is a familiar application of stochastic approximation algorithms already used in training neural networks. Proposition A.1 ( π -SGD Convergence) . Consider the π -SGD algorithm in Deﬁnition 2.3.

Original: [Equation] [Equation]
Revised: If

Original: [Table]
Revised: [Table]

Original: or if ( X 1 , . . .
Revised: (a) there exists a constant M > 0 such that for all θ , − G Tt θ ≤ M (cid:107) θ − θ (cid:63) (cid:107) 22 , where G t is the true gradient for the full batch over all permutations, G t = ∇ θ J ( D ; θ ( ρ ) t , θ ( f ) t , θ ( h ) t ) , where θ

Original:  , X n ) can be extended to an inﬁnitely exchangeable sequence, its distribution isarbitrarily close to the form in equation 15Diaconis (1977).
Revised: ≡ ( θ ( ρ ) , θ ( f ) , θ ( h ) ) , and θ (cid:63) is the optimium. (b) there exists a constant δ > 0 such that for all θ , E t

Original: However, Diaconis (1977) shows that not all ﬁnitely exchangeable sequences can be written as in equation 15, by demonstrating that one can write an exchangeable probability mass function of two binary random variables (over events { 0 , 0 } , { 0 , 1 } , { 1 , 0 } , { 1 , 1 } )
Revised: [ (cid:107) Z t (cid:107) 22 ] ≤ δ 2 (1 + (cid:107) θ t − θ (cid:63)t (cid:107) 22 ) , where the expectation is taken with respect to all the data prior to step t . Then, the algorithm in equation 9 converges to θ (cid:63) with probability one. Proof.

Original: where p ( { 0 , 1 } ) =
Revised: First, we can show that E t [ Z t ] =

Original: p ( { 1 , 0 } ) , which cannot be written in general as a mixture over conditionally iid Bernoulli random variables. Conversely, a representational form for exchangeable ﬁnite sequences has also been studiedDiaconis & Freedman (1980); Easton (1989).
Revised: G t by equation 10, the linearity of the derivative operator, and the fact that the permutations are independently sampled for each training example in the minibatch and are assumed independent of θ . That equation 9 converges to θ (cid:63) is a consequence of our conditions and the supermartingale convergence theorem (Grimmett & Stirzaker, 2001, pp. 481). The following argument follows Yuille (2004).

Original: For example, in the case of m binary random variables{+Let A t = (cid:107) θ t − θ (cid:63) (cid:107) 22 , B t = δ 2 η 2 t , and C t = −(cid:107) θ t − θ (cid:63) (cid:107) 
+}
Revised: {+Let A t = (cid:107) θ t − θ (cid:63) (cid:107) 22 , B t = δ 2 η 2 t , and C t = −(cid:107) θ t − θ (cid:63) (cid:107) 
+}

Original: X 1 , X 2 , . . .
Revised: ( δ 2 η 2 t − 2 Mη t ) .

Original:  , X m ,  the  probability distribution is represented by the following generating process: (1) randomly select the number of successes
Revised: Note that C t is positive for a sufﬁciently large t , and (cid:80) ∞ t =1 B t ≤ ∞ by our deﬁnition of η t (Deﬁnition 2.3). We will demonstrate that E t [ A t ] ≤ A t − 1 + B t − 1 − C t − 1 , for all t , in the Supplementary Material from which it follows that A t converges to zero with probability one and (cid:80) ∞ t =1 C t < ∞ . We write

Original: [Equation]
Revised: [Equation]

Original: an urn with m balls , then ) randomly draw balls from that urn. Observe that if an urn has s successes, and the ﬁrst s draws are all successes, the ( s + 1) th draw must be a failure, and thus the draws are not iid.
Revised: and the result follows.

Original: C E XPERIMENTS : F URTHER R ESULTS AND I MPLEMENTATION D ETAILS C. R ESULTS
Revised: B E XPERIMENTS : F URTHER R ESULTS AND I MPLEMENTATION D ETAILS B. R ESULTS

Original: Mean Absolute errors for arthimetic tasks on integer sequences are reported in Table 3.
Revised: Mean Absolute errors for arthimetic tasks on integer sequences are reported in Table 3 and Table 4.

Original: These largely corroborate the results from Table 1, with a drop in the mean absolute error as the value of k increases and when using more sampled permutations at test-time (e.g., Janossy-20inf-LSTM versus Janossy-1inf-LSTM).
Revised: These largely corroborate the results from Table 1, with a drop in the mean absolute error as the value of k increases and when using more sampled permutations at test-time (e.g., Janossy-20inf-LSTM versus Janossy-1inf-LSTM).

Original: Beyond the performance gains, we also observe a drop in variance when sampling more permutations at test time. 
Revised: Beyond the performance gains, we also observe a drop in variance when sampling more permutations at test time. [Table]

Original: C. I MPLEMENTATION  D ETAILS
Revised: B. I MPLEMENTATION AND E XPERIMENT D ETAILS

Original: Sequence tasks We extended the code from Zaheer et al.
Revised: Sequence tasks We extended the code from Zaheer et al.

Original: (2017), which was written in
Revised: (2017), which was written in

Original: Keras(Chollet et al., 2015).
Revised: Keras(Chollet et al., 2015), and subsequently ported to PyTorch.

Original: Regardless of the model, we always sort the sequence x beforehand.
Revised: Regardless of the model, we always sort the sequence x beforehand.

Original: In the notation of Figure 1, h is an Embedding with dimension of ﬂoor( 100 k ) , f (cid:42) is either an MLP with a single hidden layer or an RNN depending on the model ( k -ary Janossy or full-Janossy, respectively), and ρ is always a single dense layer.
Revised: In the notation of Figure 1, h is an Embedding with dimension of ﬂoor( 100 k ) , f (cid:42) is either an MLP with a single hidden layer or an RNN depending on the model ( k -ary Janossy or full-Janossy, respectively), and ρ is either a linear dense layer or one hidden layer followed by a linear dense layer.

Original: The MLPs  have 30 neurons, the LSTMs haveneurons,  and the GRUs have 80 hidden neurons.
Revised: The MLPs in f (cid:42) have 30 neurons whereas the MLPs in ρ haveneurons, the LSTMs have 50 neurons, and the GRUs have 80 hidden neurons.

Original: All activations are tanh except for the output layer which is linear.
Revised: All activations are tanh except for the output layer which is linear.

Revised: We chose 100 for the embedding dimension to be consistent with Zaheer et al.

Original: We constructed these models to be as comparable as possible; each has a similar number of total parameters and the same number of layers.
Revised: We constructed these models to be as comparable as possible; each has a similar number of total parameters and the same number of layers.

Original: We unify the number of parameters by adjusting the output dimension of the embedding (approximately 1 k ) for the different k -ary models and by using fewer neurons in the LSTM than the GRU since LSTMs are more complicated models with more parameters.
Revised: We unify the number of parameters by adjusting the output dimension of the embedding (approximately 1 k ) for the different k -ary models and by using fewer neurons in the LSTM than the GRU since LSTMs are more complicated models with more parameters.

Original: Optimization is done with Adam (Kingma & Ba, 2015) using a  learning rate of 10 − 4 . We did not do a tuning parameter sweep.
Revised: Optimization is done with Adam (Kingma & Ba, 2015). We performed a sweep over the learning rate, with learning rate in [0.01,0.001,0.0001,0.00001].

Original: Training was performed on GeForce GTX 1080 Ti GPUs.
Revised: Training was performed on GeForce GTX 1080 Ti GPUs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Janossy-MLP ( k = 1 )
Revised: Graph-based tasks

Original: [Table] Janossy-MLP ( k = 2 ) [Table]
Revised: The datasets used for this task are summarized in Table 8.

Original: Janossy-MLP ( k = 3 ) [Table] Janossy-1inf-LSTM [Table]
Revised: Our implementation is in PyTorch using Python 2.7, following the PyTorch code associated with Hamilton et al.

Original: Janossy-1inf-GRU [Table] Janossy-20inf-LSTM
Revised: That repo did not include an LSTM aggregator, so we implemented our own following the

Original: [Table] Janossy-20inf-GRU [Table] Graph-based tasks
Revised: TensorFlow implementation of GraphSAGE, and describe it here.

Original: The characteristics of the datasets used are tabulated in Table 4.
Revised: At the beginning of every forward pass, each vertex v is associated with a p -dimensional vertex attribute h (see Table8).

Original: This task is implemented in PyTorch and trained on the same GeForce GTX 1080 Ti GPUs.
Revised: For every vertex in a batch, k 1 neighbors of v are sampled, their order is shufﬂed, and their features are fed through an LSTM. From the LSTM, we take the short-term hidden state associated with the last element in the input sequence (denoted h ( T ) in the LSTM literature, but this h is not to be confused with a vertex attribute).

Original: As this experiment is using Janossy pooling as  a GraphSAGE (Hamilton et al., 2017) aggregator, we forked the GraphSAGE datasets and code for experimental set-up and the mean-pool aggregator; we implemented our own LSTM as it is not yet available in the GraphSAGE PyTorch repo.
Revised: This short-term hidden state is passed through a fully connected layer to yield a vector of dimension q 2 , where q is a user-speciﬁed positive even integer referred to as the embedding dimension . The vertex’s own attribute h is also fed forward through a fully connected layer with q 2 output neurons. At this point, for each vertex, we have two representation vectors of size q 2 representing the vertex v and its neighbor set, which we concatenate to form an embedding of size q . This describes one convolution layer, and it is repeated a second time with a distinct set of learnable weights for the fully connected and LSTM layers, sampling k 2 vertices from each neighborhood and using the embeddings of the ﬁrst layer as features.

Revised: After each convolution, we may optionally apply a ReLU activation and/or embedding normalization, and we follow the decisions shown in the GraphSAGE code Hamilton et al.

Revised: After both convolution operations, we apply a ﬁnal fully connected layer to obtain the score, followed by a softmax (Cora, Pubmed) or sigmoid (PPI).

Revised: The loss function is cross entropy for Cora and Pubmed, and binary cross entropy for PPI.

Revised: The number of trainable parameters in each model is independent of k 1 and k 2 by the design of

Revised: LSTMs (the same is true for the mean-pooling aggregator).

Revised: The only variation in the number of weights is in the dimensions of the input and output features, which differ by dataset.

Original: Following
Revised: Please see

Revised: Table 9 for details.

Revised: Optimization is performed with the Adam optimizer (Kingma & Ba, 2015).

Revised: The training routine for the smaller graphs (Cora, Pubmed) is not guaranteed to see the entire training data, in contrast with the scheme applied to the larger PPI graph.

Revised: For Cora and Pubmed, we form 100 minibatches by randomly sampling subsets of 256 vertices from the training dataset (with replacement).

Revised: With PPI, we perform 10 full epochs: at each epoch, the training data is shufﬂed, partitioned into minibatches

Original: Hamilton et al.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: (2017) a depth (i.e. number of aggregator steps) of 2, an Adam optimizer and a hidden LSTM dimension of 256.
Revised: [Table] [Table] of size 512, and we pass over each. In either case, the weights are updated after computing the gradient of the loss on each minibatch.

Original: Theremaining hyperparameters are dataset-speciﬁc:
Revised: The hyperparameters were set by following Hamilton et al.

Revised: (2017); no hyperparameter optimization was performed.

Original: • On the Cora dataset,for all models, the learning rate is 0.005; the number of random samples atdepth one and two are both ﬁve. • On the Pubmed dataset, for all models the learning rate is 0.01; the number of random samples at depth one and two are 10 and 25, respectively.
Revised: For every dataset, the embedding dimension was set to q = 256 at both conv layers. For Pubmed and PPI, the learning rate is set at 0.01 while for Cora it is set at 0.005. At test time, we load the weights obtained from training, perform 20 forward passes – which shufﬂes the input sequence by design – average the predicted probabilities (i.e. softmax output) from each forward pass, and choose the class that maximizes the averaged probabilities.

Original: Note that the embedding function h , in terms of Figure 1 is an identity in this task.
Revised: The implementation for Mean Pooling is similar in spirit but replaces f (cid:42) with a permutation invariant function. The details can be found by viewing our repo on GitHub.

Original: D S UPPLEMENTARY RESULTS ON S EQUENCE A RITHMETIC T ASKS
Revised: C L A TEX FOR J ANOSSY FUNCTION MARKERS

Original: Here we present our preliminary results on the sequence task where ρ is modeled as a feedforward network with one hidden layer containing 100 hidden neurons and tanh activations.
Revised: The commands below can be directly copied and pasted into L A T E Xsource to create the Janossy function markers. Please use the amsmath package. To typeset f , we deﬁne in L A TEXas \newcommand

Original: Other than the model for ρ , the implementation details are as in section C.1.
Revised: [Equation] and type [Table] Similarly, for f (cid:42)

Original: No hyperparameter tuning was performed.
Revised: [Table]

Original: We are not considering results for the sum task since the experimental interest is to investigate whether a richer model for ρ can carry the burden of exploiting dependencies in the sequence when k = 1 . The results are shown in Table 5.
Revised: we deﬁne in L A TEXas [Table] Last, the deﬁnition above for f (cid:42) caused difﬁculties in environments such as figure , so we deﬁned and occasionally used in L A TEX

Original: We can see that deep sets does not perform well across tasks and that increasing k helps to boost performance.
Revised: [Table]