A BELIAN N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In several domains such as natural language processing, it has been empirically reported that simple addition and subtraction in a somehow learned embedding space capture analogical relations.
However, there is no guarantee that such relation holds for a new embedding space acquired by some training strategies.
To tackle this issue, we propose to explicitly model analogical structure with an
Abelian group.
We construct an Abelian group network using invertible neural networks and show its universal approximation property.
In experiments, our model successfully learns to capture word analogies from word2vec representations and shows better performance than other learning-based strategies.
As a byproduct of modeling Abelian group operations, we furthermore obtain its natural extension to permutation invariant models with theoretical size-generalization capability.
I NTRODUCTION
The vector representations of words called word2vec (Mikolov et al., 2013a;b) trained only on large unlabeled text data are known to capture linear regularities between words.
For example, vec ( “king” ) − vec ( “man” )
+ vec ( “woman” ) results in the most similar vector to vec ( “queen” ) .
Similar results have been observed not only in other word embeddings (Mnih & Kavukcuoglu, 2013;
Pennington et al., 2014) but also in various embedding spaces, such as combined embeddings of text and image (Kiros et al., 2014), emoji embeddings (Eisner et al., 2016), latent representation of deep convolutional generative adversarial networks (Radford et al., 2016), and feature space of pretrained image models (Upchurch et al., 2017).
Although the reports are interesting and attractive, such approaches have shortcomings.
Since those methods of learning from unlabelled data usually do not explicitly incorporate learning analogical structure, there is no guarantee that the acquired embedding space has linear relations between instance pairs even if training itself works well.
Even when an embedding space captures some kinds of analogies, it might not work for other kinds of analogies.
Indeed, word2vec representation works well for inﬂectional analogies ( 68 .
22% accuracy) but poorly for encyclopedic analogies ( 7 . 11% accuracy) in our preliminary experiments (see Table 3 in Section 4.2).
In such a case, it is quite difﬁcult to tune the training algorithm for certain kinds of analogies you want to use.
To alleviate these issues, we propose to directly learn analogical relations on the embedding space from labeled data.
One challenge in learning analogy in a supervised manner is how to model analogical functions.
A naive way to do this is to train two separate models corresponding to addition and subtraction, respectively; however, it does not reﬂect the analogical structure and might be inefﬁcient.
In this work, we propose an Abelian group network to incorporate an analogical inductive bias into a neural network.
The proposed network is designed to satisfy the Abelian group condition by using an invertible neural network.
We also show that the Abelian group network is a universal approximator of smooth Abelian group operations.
Since the inverse element in the Abelian group network and its gradient are analytically computable, we can train it for analogy tasks by common techniques for deep learning, such as stochastic gradient descent.
As a side effect of the algebraic structure, we can construct a permutation invariant function, i.e., a function for multisets, by repeatedly composing an Abelian group operation for multiple inputs.
Multiset models can handle inputs of different sizes, and it is important for the models to automatically generalize between different size inputs, especially from small to large ones.
However, existing multiset models (Zaheer et al., 2017; Qi et al., 2017) have no such theoretical guarantee.
On the other hand, our multiset models naturally induce the size-generalization capability because the output for larger inputs can be written as the composition of small elements.
Further, we show that a necessary and sufﬁcient condition for the composed function being permutation invariant is that the binary operation forms an Abelian semigroup, and we propose an Abelian semigroup network , by using the characterization of associative symmetric polynomials.
P RELIMINARIES 2. D EFINITIONS
In this section, let us introduce some basic notations and important deﬁnitions that will play a key role in this work.
2.1. B ASIC N OTATIONS
By N , we represent the set of the natural numbers including 0 .
We denote a vector by a bold symbol, e.g., x .
Let x ∈ R
d be a d -dimensional vector.
We represent the i -th element ( 1 ≤ i ≤ d ) of x by x i .
For 1 ≤ k ≤ d , x ≤ k ∈ R k is the k -dimensional vector ( x 1 , . . .
x k ) and x <k ∈ R k − 1 is the ( k − 1)
-dimensional vector ( x 1 , . . .
x k − 1 ) .
We denote the elementwise product of two vectors x , y ∈ R
d by x ⊗ y , such that ( x ⊗ y )
i
= x
i y i and the elementwise division of two vectors x ∈ R d , y ∈
( R \ { 0 } )
d by x (cid:11) y , such that ( x (cid:11) y ) i =
x i /y
i .
By (cid:107) · (cid:107) , we represent the L(Euclidean) norm
. 2.1. M ULTISET AND P ERMUTATION I NVARIANCE
Here, we use X and Y to describe some domains, which are typically Euclidean spaces, i.e., X = R dand Y = R d 2 .
We denote the set of multisets over X by N X .
We use { x 1 , . . .
x n } ∈ N X to describe a multiset composed of x 1 , . . .
, x n ∈ X (any confusion with sets is not problematic in this paper).
The cardinality of a multiset is the number of elements with multiplicity and is expressed by | · | , e.g., |{ 1 , 2 , 2 , 3 }| = 4 .
For n ∈ N , a symmetric group S n is the set of all n !
bijective functions σ : { 1 , 2 , . .
. , n } → { 1 , 2 , . .
. , n } .
For a permutation σ ∈ S n and X ∈ X , σ · X is deﬁned such that ( σ · X ) i =
X σ ( i ) for all i ∈ { 1 , . . . , n } .
A function f : X n → Y is said to be permutation invariant if for any X ∈ X n and for any permutation σ ∈ S n , f ( σ · X ) =
f ( X ) holds.
This concept can be extended to functions that take vectors of different dimensions.
Namely, a function f : (cid:83) k ∈ N X k
→ Y is called permutation invariant if for any k ∈ N , for any X ∈ X k and for any permutation σ ∈ S k , f ( σ · X ) =
f ( X ) holds.
When f : (cid:83) k ∈ N X k
→ Y is permutation invariant, it can be also viewed as a function that takes multisets as input.
For notation simplicity, we sometimes use the same symbol to express the multiset function: f : N X → Y .
2.1. U NIVERSALITY
Universality is an important theoretical property of neural networks’ expressive power.
Let X and
Y be an input domain and an output domain, respectively.
We consider a model M and a class of target functions F , both of which are sets of functions X
→ Y .
The model M is a sup-universal approximator of F if for any target function f ∗ ∈ F , for any (cid:15) > 0 , and for any compact subset
K ⊂ X , there exists a function f ∈ M such that
[Equation]
If not noted otherwise, universality refers to the sup-universal property.
2.1. B ASIC A LGEBRA
Here, we introduce the basic deﬁnition of important algebraic structures in this study.
Let G be a set and ◦ : G × G → G be a binary operation.
Below, we review four properties to deﬁne Abelian semigroups and groups.
Associativity For any x, y, z ∈ G , ( x ◦ y ) ◦
z = x ◦ ( y ◦ z ) .
Identity Element There exists an element e ∈ G , called the identity element, such that for any x ∈ G , x ◦ e =
e ◦ x = x .
Inverse Element For any x ∈ G , there exists an element x − 1 ∈ G , called the inverse element of x , such that x ◦ x − 1 = x − 1 ◦ x = e .
Commutativity For any x, y ∈ G , x ◦ y =
y ◦ x .
Table 1 shows which properties are required in each algebraic structure.
A semigroup only requires associativity to the binary operation.
A group is a semigroup with an identity element and inverse elements.
An Abelian (semi)group is a (semi)group with commutativity.
[Table]
2. I NVERTIBLE N EURAL N ETWORKS
Invertible neural networks are neural networks that approximate invertible functions R d → R d .
Here, we review some existing studies for multi-dimensional case, i.e., d ≥ 2 , and singledimensional case, i.e., d =
. 2.2. N ORMALIZING F LOWS
Multi-dimensional invertible neural networks have been studied mainly in the context of normalizing ﬂows (Tabak & Vanden-Eijnden, 2010), which iteratively apply invertible functions to a simple original probability distribution to express complex probability distributions (Kobyzev et al., 2020;
Papamakarios et al., 2019).
There have been many variants proposed including residual ﬂows (Behrmann et al., 2019), neural ODEs (Chen et al., 2018), and autoregressive ﬂows (Kingma et al., 2017).
Here we review afﬁne coupling ﬂows (Dinh et al., 2015), one of the most popular models with parallelizable efﬁcient inverse computation.
Each layer of the afﬁne coupling ﬂows maps x = ( x 1 , . . .
, x d ) ∈ R d to y =
( y 1 , . .
. , y d ) ∈ R d such that
[Equation]
where exp is applied elementwise and α, β : R k
→ R d − k are trainable functions.
The inverse is computed as follows:
[Equation]
They are used in many successful applications such as NICE (Dinh et al., 2015), Real NVP (Dinh et al., 2017), and Glow (Kingma & Dhariwal, 2018).
Although the normalizing ﬂows have a limited form of transform, they still admit universalities on certain classes of functions (Teshima et al., 2020).
The afﬁne coupling ﬂows are L p -universal (weaker condition of universality) for C 2 -diffeomorphisms.
Some more complex models, including deep sigmoidal ﬂows (Huang et al., 2018) and sum-of-squares polynomial ﬂows (Jaini et al., 2019), are universal for C 2 -diffeomorphisms.
2.2. S INGLE - DIMENSIONAL I NVERTIBLE N EURAL N ETWORKS
For single-dimensional functions, invertibility is equivalent to strict monotonicity.
Monotonic networks (Sill, 1997) model strictly monotonic functions.
The monotonically increasing version with
[Equation]
K groups and J k units for k -th group is as follows:
[Equation]
where ˜ w ( k,j ) , b ( k,j ) ∈
R are trainable parameters.
The monotonic networks are a universal approximator for strictly monotonic differentiable functions.
Monotonic rational-quadratic transforms (Durkan et al., 2019) are another universal model for the single-dimensional case.
P ROPOSED M ETHODS
Here, we introduce the proposed methods.
First, we propose a model for Abelian group operations and show its universality.
Next, we explain how to model analogical relations by our model.
Finally, we present architectures for multiset input and show the size-generalization ability.
3. A BELIAN G ROUP N ETWORK
Let X be a Euclidean space, i.e., X =
R d ( d ∈ N ).
We present the Abelian group network that models Abelian group operations as follows:
[Equation]
where φ : X → X is a trainable invertible function, typically modeled by an invertible neural network.
First, we check that this binary operation satisﬁes the four conditions of the Abelian group described in Section 2.1.4.
Proposition 1 (Semigroup Conservation) .
Let ρ : X → X be a bijective function.
When ∗ : X × X → X is associative, x ◦ y =
ρ − 1 ( ρ ( x ) ∗ ρ ( y )) is also associative.
Similarly, when ∗ is commutative, ◦ is also commutative.
Proof.
Associativity:
[Equation]
Commutativity:
[Equation]
By this proposition, since + is associative and commutative, the Abelian group network is also associative and commutative.
The identity element is
[Equation] [Equation]
which satisﬁes
[Equation]
for all x ∈ X .
The inverse element of x ∈ X is
[Equation] [Equation]
which satisﬁes
[Equation]
It is worth noting that we can analytically compute the inverse function (Equation 10).
Next, we check the expressive power of the Abelian group network.
An Abelian Lie group is an
Abelian group over a manifold in which the group operation ( x, y )
(cid:55)→ x ◦ y and the inverse function x (cid:55)→ x − 1 are both differentiable.
The following theorem states that the Abelian group network can approximate any Abelian Lie group operation with arbitrary precision.
Theorem 1 (Universality of Abelian group networks) .
Let X be a Euclidean space.
Abelian group networks are a universal approximator of Abelian Lie group operations over X .
In other words, for any Abelian Lie group operation ∗ : X × X → X , for any (cid:15) > 0 , and for any compact subset
K ⊂ X , there exists a binary operation function ◦ :
X × X →
X represented by an Abelian group network such that
[Equation]
We provide the proof in Appendix C.1.
It is based on the theory of the Abelian Lie group and the universality of invertible neural networks.
3. M ODELING A NALOGY BY A BELIAN G ROUP N ETWORK
Now we consider an analogical task of predicting an element d in a relation a :
b = c : d .
Let a , b , c , d ∈ X be embedded vectors of corresponding elements.
We need to deﬁne a model f : X ×
X × X → X that predicts d from a , b , and c .
Then, if the set of candidate elements are ﬁnite as in word embedding, we can search the nearest vector to f ( a , b , c ) .
We explained in Section 1 that several studies heuristically use f ( a , b , c ) =
b − a + c .
Analogical relations have two natural laws: when a = b , then c
= d holds; a : b
= c : d is equivalent to a : c = b : d . when a = c , then b = d holds.
Therefore, we would like to design f to satisfy the two condition that f ( a , b , c ) =
c when a = b , and f ( a , b , c ) =
f ( a , c , b ) .
These conditions hold in f ( a , b , c ) =
b − a + c .
However, when we try to parametrize f by neural networks, it is not trivial to reﬂect these inductive biases using existing architectures.
By using the Abelian group network, we can model f so that it reﬂects the conditions as follows:
[Equation]
Also, this is strictly more expressive than the simple arithmetic operations because when φ is the identity, φ ( a , b , c )
=
b − a + c .
We can train this model by minimizing the loss: l ( f ( a , b , c ) , d ) , where l : X → R can typically be an L 2 loss or negative cosine similarity.
3. M ULTISET A RCHITECTURE AND S IZE G ENERALIZATION
Here, we construct a multiset architecture by combining the Abelian group networks and show it naturally generalizes in sizes.
It is easy to check that if ◦ : X × X
→ X is associative and commutative, f : X k → X deﬁned by
[Equation]
(14) is permutation invariant.
When this condition holds, we represent the multiset version of f : N X
→
X as follows by denoting a composition of ◦ by (cid:13) : f ( X ) =
(cid:13) x ∈ X x , where X ∈ N X is a multiset of X .
By modeling ◦ by the Abelian group network, we can construct a multiset architecture of the
Abelian group network:
[Equation]
where the invertible function φ : X → X is typically modeled by an invertible neural network.
Now, we consider the size-generalization ability of the multiset architectures.
An intuitive explanation is as follows.
As an extreme case, even if trained only on multisets of two elements, our models can learn the correct binary operation.
Therefore, they generalize to multisets of larger sizes.
In general, we can derive the following theorem.
Appendix C.2 provides the proof.
Theorem 2 (Size Generalization of Abelian Group Networks) .
Let f ∗ : N X → X be a target function expressed by a composition of Abelian semigroup ( X , ◦ ) :
f ∗ ( X ) = (cid:13) x ∈ X x .
Let f : N X → X be a multiset architecture of the Abelian group network: f ( X ) =
φ − 1 (cid:0)(cid:80) x ∈ X φ
[Equation]
x ) (cid:1) .
When
[Equation]
holds for any X ( ∈ N X ) whose size is smaller than a ( ≥ 2 ), then
[Equation]
holds for any X ( ∈ N X ) whose size is b ( ≥ a ), under the condition that the Lipschitz constants of φ and φ − 1 are K 1 and K 2 , respectively.
A necessary and sufﬁcient condition of f in Equation 14 being permutation invariant is that ◦ is an Abelian semigroup operation.
In Appendix A, we provide the proof and propose an Abelian semigroup network by using the characterization of associative symmetric polynomials.
E XPERIMENTS
Here, we summarize the main experiment of learning word analogies.
In addition to this section, we provide the experiments of size generalization on learning multiset functions in Appendix B.
4. C OMMON S ETTINGS
We implemented the neural networks in the PyTorch framework (Paszke et al., 2019) and optimized them using the Adam algorithm (Kingma & Ba, 2015).
The hyperparameters for each model in each problem were tuned with validation datasets using the Bayesian optimization of the Optuna framework (Akiba et al., 2019).
The experiments were run on Intel Xeon E5-2695 v4 with NVIDIA
Tesla P100 GPU.
See Appendix D for the detailed settings, such as the model architecture and the range of hyperparameters.
4. W ORD A NALOGIES
In Section 1, We discussed the general motivation of training analogical functions over an embedding space.
In addition, here, we explain the speciﬁc issue in word2vec (Mikolov et al., 2013a;b) and justify the motivation of using the Abelian group network.
In word2vec, for predicting a word d in a relation a : b = c : d , the word with the most similar vector to b − a
+
c
(we denote the corresponding vector for each word by using a bold symbol) is selected in terms of the cosine similarity:
[Equation]
Usually, the words a, b, c are excluded from the candidate vocabulary under the assumption that a common word does not appear in one analogy example.
Although this assumption is reasonable in many cases, it prevents us from solving certain problems such as a past tense verb analogy “do”:“did” = “split”:“split” or a plural noun analogy “apple”:“apples” = “deer”:“deer”.
On the other hand, if we do not exclude the words a, b, c from the candidates, word2vec suffers from severe performance degradation e.g., falling from 73 .
59% to 20 .
64% in our preliminary experiment on the
Google analogy test set.
This is due to the nature of the word2vec algorithm: the result of the simple arithmetic calculation b − a
+ c has a high probability of being close to b or c in the cosine similarity, especially in a high dimensional space.
We mitigate this issue by learning richer functions than addition and subtraction.
In this experiment, we trained the Abelian group network from a labeled dataset and compared it with the original word2vec and other learning-based approaches.
Word Embedding We used a 300 -dimensional word2vec model for 3 billion words trained on
Google News corpus of about 100 billion words 1 .
We normalized each word embedding by Lnorm, following the implementation of the Gensim framework ( ˇReh˚uˇrek & Sojka, 2010).
Word Analogy Models We compared different models for a word analogy function f : R 300 × R× R 300 → R 300 that takes the vectors of words a, b, c and predicts the vector of a word d .
In the proposed method (WV + AGN), we modeled f by Equation 13.
For the invertible neural network φ : R 300 → R 300 , we implemented a model based on the afﬁne coupling ﬂows described in Section 2.2.1.
In addition to the original word2vec (WV): f ( a , b , c ) =
b − a + c , we prepared two trainable baselines based on a multilayer perceptron (WV + MLP and WV + MLP C, respectively):
[Equation]
where we trained MLP : R 300 → R 300 or MLP C : R 900 → R 300 .
Setup Except for WV, we trained f by minimizing the loss function: loss f ( a , b , c , d ) = − cos( f ( a , b , c ) , d ) on the training set.
We measured the accuracy on the test set by calculating the most similar vector to the model output for each word:
[Equation]
[Equation]
where V is the set of all word embeddings in the word2vec model.
For reference, we also tested the setting where we removed the words a, b, c from the candidates:
[Equation]
[Equation] Datasets We trained our models on the bigger analogy test set (Rogers et al., 2016), which consists of 4 categories, each of which has 10 smaller subcategories of 50 unique relations.
First, we extracted the pairs included in the word2vec vocabularies.
Then for each subcategory, randomly split them into a training set ( 60% ), validation set ( 20% ), and test set ( 20% ).
Finally, for each set, we generated all the combinations of the pairs for each subcategory and concatenated them among all subcategories.
For some relations that contain multiple acceptable candidates, such as mammal and canine for hypernyms of dog , we used the ﬁrst candidate for training and accepted any for the test.
To check the transferability to another dataset, we also tested our models by the Google analogy test set (Mikolov et al., 2013a).
It includes 19 , 544 question pairs ( 8 , 869 semantic and 10 , 675 syntactic), and all the words were included in the word2vec vocabulary.
Tables 9 and 10 in Appendix
D.2 summarize the explanation and the number of extracted pairs for all subcategories.
[Table]
Results In Table 2, we summarized the results on the bigger analogy test set when we selected from the whole vocabulary.
In this setting, WV performed extremely poorly.
WV + AGN achieved the best accuracy in overall, while the three trainable models all outperformed WV.
Table 3 shows the accuracy comparison when we excluded a, b, c from the candidates.
While the proposed method outperformed WV again, interestingly, the other learning-based models degraded their performance compared to the original WV.
We summarized the results on the Google analogy test set in Tables 4 and 5 for each test setting.
Here, MLP-based models signiﬁcantly dropped performance while the proposed model did not.
[Table] [Table]
Only in the setting where we excluded a
[Table]
b, c from the candidates in the Google analogy test set
[Table]
the proposed method did not outperform WV.
This is possibly because the word2vec model was highly tuned for the Google analogy test set for this evaluation method.
Indeed, it has been pointed out that word embedding algorithms are quite dependent on system design choices and hyperparameter tuning Levy et al. (2015).
We show the full results of all subcategories on each dataset in each evaluation setting in Tables 11, 12, 13, and 14 in Appendix D.2.
Finally, we enumerated the answers of each model for some toy example relations in Table 6.
The ﬁrst four relations are the cases where a = b or a = c .
Thanks to the inductive bias incorporated in the Abelian group network, WV + AGN answered all the questions correctly as well as WV.
WV
+
AGN also predicted the correct words for the last ﬁve relations, where we need to predict the words that appeared as c .
On the other hand, the MLP-based models failed in most cases, which indicates that they are not good at the type of examples different from the training dataset.
Overall, while the naive learning approach overﬁtted to the certain dataset and evaluation criteria, the inductive biases incorporated in the Abelian group network successfully prevented the model from overﬁtting.
R ELATED W ORK 5. A LGEBRAIC S TRUCTURES IN N EURAL N ETWORKS
In the literature of deep learning, algebraic structures mainly appear in the context of group invariant/equivariant neural networks.
For image input, some studies tried to incorporate reﬂection and rotation invariance into convolutional neural networks (Cohen & Welling, 2016; Worrall et al., 2017).
Neural networks for (multi)sets (Zaheer et al., 2017; Qi et al., 2017) adopted invariance/equivariance to symmetric group actions.
Recent studies have investigated symmetries invariant/equivariant to more general group actions, such as a subgroup of the symmetric group (Maron et al., 2019b) and sets of symmetric elements (Maron et al., 2020).
Our work differs from the above studies since we try to model an Abelian group/semigroup operation itself.
5. I NDUCTIVE B IAS AND E XPRESSIVE P OWER OF N EURAL N ETWORKS
Inductive biases are assumptions on the nature of the data-generating process or the space of solutions in machine learning (Battaglia et al., 2018).
Many studies have constructed special neural networks that reﬂect the inductive biases of a given problem setting.
At the same time, since those networks are often composed of limited forms of neural operations, expressive power including universal approximation properties has been studied.
[Table]
[Table]
Convolutional layers of convolutional neural networks (CNN) (LeCun et al., 1989) are designed to reﬂect spacial structures in images.
CNN without fully connected layers has been shown to be universal (Zhou, 2018).
Invertible neural networks incorporate the inductive bias of being bijective, which we summarized in Section 2.2.
Message passing graph neural networks (Gilmer et al., 2017) such as graph convolutional networks (Kipf & Welling, 2017) and graph attention networks (Vaswani et al., 2017) are designed under the assumption that neighboring nodes have similar properties.
They have been shown to have limited expressive power in terms of graph isomorphism (Xu et al., 2019; Morris et al., 2019)
More expressive models have also been studied (Sato et al., 2019;
Maron et al., 2019a; Keriven & Peyr ´ e, 2019; Maehara & Hoang, 2019).
For a (multi)set learning problem, DeepSets (Zaheer et al., 2017) are one of the most popular models with universal approximation property.
5. S IZE G ENERALIZATION
Graph neural networks and neural networks for (multi)sets can handle graphs and sets of different sizes, and their size-generalization ability has been empirically shown in some applications such as physical systems (Battaglia et al., 2016) and combinatorial optimization (Khalil et al., 2017; Abe et al., 2019; Veli ˇ ckovi ´ c et al., 2020).
However, from a theoretical perspective, there exist simple tasks on which graph neural networks do not naturally generalize to larger graphs (Yehudai et al., 2020).
Recent work has analyzed the extrapolation of graph neural networks trained by gradient descent (Xu et al., 2021).
There have been few studies on size generalization of (multi)sets probably because of difﬁculty in analyzing DeepSets for inputs of different sizes.
C ONCLUSION AND F UTURE W ORK
In this work, we presented a novel neural network architecture to model an Abelian group with universality.
In the experiment of learning word analogies, we conﬁrmed that the inductive bias of our model that reﬂects analogical structure successfully enhanced the performance.
This brings up the possibility of solving other analogical tasks such as image generation by image analogies (Hertzmann et al., 2001), which is left as future work.
Moreover, as an application other than analogies, we constructed a permutation invariant architecture (i.e., multiset model) by combining the Abelian group models, which has the theoretical capability of size generalization.
We hope that our attempt to model algebraic structure by neural networks gives a new insight into the ﬁeld of machine learning.
A A BELIAN S EMIGROUP N ETWORK A. M OTIVATION
Below, we present a necessary and sufﬁcient condition for multiset functions that are represented by the composition of binary operations to be well-deﬁned.
Proposition 2 (Permutation Invariant Conditions for Binary Operation) .
Let f : (cid:83) k ∈ N X k
→ X be a function represented as
[Equation]
where X = ( x 1 , . . .
, x n ) ∈ X n and ◦ :
X × X
→ X is a binary operation (left-associative).
The function f is invariant if and only if ◦ forms an Abelian semigroup, namely, ◦ is commutative and associative.
Proof.
It is obvious that when ◦ is commutative and associative, f is permutation invariant.
Let us consider the case when f is permutation invariant.
f (( x 1 , x 2 ))
=
f
(( x 2 , x 1 )) leads to x 1 ◦ x 2 = x 2 ◦ x 1 (commutativity).
From f (( x 1 , x 2 , x 3 ))
=
f
(( x 2 , x 3 , x 1 )) , we have ( x 1 ◦ x 2 ) ◦ x 3 = ( x 2 ◦ x 3 ) ◦ x 1 and commutativity leads to ( x 1 ◦ x 2 ) ◦ x 3 = x 1 ◦ ( x 2 ◦ x 3 ) (associativity).
On the basis of this proposition, our goal decomposes into learning Abelian semigroup operations over X .
In Section A.2, we propose neural network architectures for Abelian semigroups.
A. A RCHITECTURE
Although the Abelian group network proposed in Section 3.1 is universal for smooth group operations, it is not sufﬁcient for approximating an Abelian semigroup operation such as the product over R , i.e., x ◦ y
=
xy .
Now we extend the Abelian group network and propose the Abelian semigroup network.
Our idea is to extend + of Equation 5 to a polynomial.
From Proposition 1, Equation 5 is still a semigroup after we replace + by a polynomial of x and y as long as the polynomial is associative and symmetric as a binary operation.
We call the polynomials with this property associative symmetric polynomials , which are characterized by the following theorem.
Since the original paper only gives a brief explanation, we give detailed proof in Appendix C.3.
Theorem 3 (Characterization of Associative Symmetric Polynomials, Commutative Case of (Yoshida, 1963)) .
An associative symmetric polynomial of x and y is one of the following three forms:
[Equation]
where α, β, γ are coefﬁcients.
By applying this theorem to Proposition 1 for X with elementwise product and division, we obtain the following three kinds of Abelian semigroup operations:
[Equation]
where ρ : X → X is an invertible function and α , β , γ ∈ X are parameters ( γ is nonzero for all elements in Equation 27).
Equation 25 is a constant case, on which we do not put a focus due to its trivialness.
Equation 26 forms a group where e = ρ − 1 ( − α ) , x − 1 =
ρ − 1 ( − ρ ( x ) − 2 α ) .
It can be expressed by the Abelian group network with φ ( x ) =
ρ ( x ) + α and φ − 1 ( x ) =
ρ − 1 ( x − α ) in Equation 5.
Equation 27 is a semigroup but not a group.
Just using this equation is ﬁne, but we propose a simpler form, as the Abelian semigroup network:
[Equation]
28) where φ : X → X is a trainable invertible function typically modeled by an invertible neural network.
This is a special case of Equation 27 when β = 0 , γ =
and therefore is a semigroup.
Conversely, the Abelian semigroup network can express Equation 27 by
[Equation]
Moreover, the Abelian semigroup network can approximate the Abelian group network: φ (cid:48)− 1 ( φ (cid:48)
( x ) + φ (cid:48)
( y )) .
Let X > 0 = R d> 0 , where X = R d .
One construction is approximating a bijective function π : X → X > 0 ,
[Equation]
) by φ , where exp and log act elementwise.
This is possible in any compact subset of X .
From the previous discussions so far, the Abelian semigroup network can approximate any binary operation which is homeomorphic to an associative symmetric polynomial.
We conﬁrm this fact in the experiments.
By calculating Equation 14, we can write the model for multiset input X = { x 1 , . . .
x n } ∈ X in a simple form:
[Equation]
31) where the invertible function φ : X → X is typically modeled by an invertible neural network.
For the Abelian semigroup network, the error bound for small multiset propagates in the form of the product with the values φ ( x i ) , which prevents us from inducing the bound like Theorem 2.
However, it still has the size-generalization ability in most real applications where the values are not too large.
We conﬁrm this by an experiment in Section B.
B E XPERIMENTS OF S IZE G ENERALIZATION
To check the size generalization over semigroup and group operations on multisets, we trained the models on synthetic data.
The binary operation forms of the examined functions are x ◦ y
=
x
+ y, x + y + 1 , 3 (cid:112) x 3 + y 3 (group cases) and x ◦ y
= xy, x + y + xy 2 (semigroup cases).
Setup For the single-dimensional invertible neural network of the Abelian group network and
Abelian semigroup network, we used monotonic networks (Sill, 1997).
We tuned the hyperparameters, the number of groups and the number of units for each group.
As a baseline, we used DeepSets (Zaheer et al., 2017), one of the most popular models for (multi)set learning.
It incorporates two multilayer perceptrons (MLP).
We used the same number of hidden layers for the two MLPs and tuned the hyperparameters, the number of layers in each MLP, the middle dimension, and the hidden dimension.
Each model was trained to minimize the mean squared error on a training set.
Data Generation As training data, we generated 500 multisets of size { 2 , 3 , 4 } (chosen uniformly random).
All the elements were single-dimensional and selected uniformly at random from [ − 5 . 0 , 5 . 0] .
A validation data of 100 multisets were generated from the same distribution.
We prepared two kinds of test data.
One consisted of 100 multisets drawn from the same distribution as the training and validation data, which we refer to by small .
To see the size-generalization ability, the other consisted of 100 multisets of size { 10 , 11 , 12 } (chosen uniformly at random) with the same element distribution, which we refer to by large .
Results Table 7 summarizes the results.
For the group functions, all models including the Abelian semigroup network performed well.
This is consistent with the fact that group operations can be approximated by the Abelian semigroup network, as discussed in Section A. While the Abelian group network was better on the other two cases, DeepSets outperformed the Abelian group network on 3 (cid:112) x
3 + y 3 .
This is possibly due to the optimization of MLPs in DeepSets being easier than monotonic networks in the Abelian group network and Abelian semigroup network.
Invertible neural networks for the single-dimensional case that are easy to optimize are important for future work.
For the semigroup operations, as well as DeepSets, the Abelian group network did not work well.
This is reasonable because these semigroup operations can not be expressed by the Abelian group network.
On the size generalization, although DeepSets worked fairly well, our models worked better.
For example, the Abelian semigroup network was better than DeepSets on large of x + y despite being worse on small ; The Abelian group network had similar results on xy and x + y + xy 2 .
[Table]
C P ROOFS C. P ROOF OF T HEOREM
First, we review a theorem of the Abelian Lie group
[Table]
which is important in our proof.
The real numbers R with the addition + forms a Lie group, which we denote by ( R , +) .
Also, the torus T = R / 2 π Z with the addition + modulo 2 π Z forms a Lie group
[Table]
which we denote by ( T , +) .
It is known that any connected Abelian Lie group is isomorphic to ( R , +) k
×
( T , +) h for some k, h ∈ N (Section 4.4.2 of (Procesi, 2007)).
Now
[Table]
we give the proof of Theorem 1.
Proof.
We use the fact that any connected Abelian Lie group is isomorphic to ( R , +) k × ( T , +) h for some k, h ∈ N .
Since there is no homeomorphic function R (which is not compact) to T (which is compact), Abelian Lie groups over a Euclidean space has the form of only ( R , +) k .
Therefore, any ∗ can be represented as
[Equation]
32) where π : X → X is a homeomorphic function in terms of Lie groups, i.e., π ( · ) and π − 1 ( · ) are analytic.
Take any (cid:15) > 0 and a compact subset K ⊂ X .
We denote the image of K × K through the function ( x , y )
(cid:55)→ π ( x ) + π ( y ) by S (cid:48)
= { π ( x ) + π ( y ) | x , y ∈ K} .
Let
[Equation]
33) and
[Equation]
Then, we have a Lipschitz constant L > 0 of π − 1 over S since π − 1 is continuous and S is compact.
Also, from the universality of invertible neural networks (Teshima et al., 2020) for the compact set
K (cid:48) , there exists an invertible neural network φ : X → X such that for any x ∈ K (cid:48) (cid:107)
[Equation]
x
[Equation]
x
[Equation]
(cid:107)
[Equation]
(cid:15)
[Equation]
L +(35) and for any x (cid:48) ∈
π ( K (cid:48) )
(cid:107) π − 1 ( x (cid:48) )
− φ − 1 ( x (cid:48) )
(cid:107) <
(cid:15) 2 L + 1 .
Then, for any x , y ∈ K , φ ( x ) + φ ( y ) ∈ S ( ⊂ π ( K (cid:48) ))
because
[Equation]
x
[Equation]
x
[Equation]
x )
[Equation]
π
[Equation] [Equation]
<
[Equation] [Equation]
(cid:15).
Therefore, for any x , y ∈ X , we have from the Lipshitz continuity of π −
[Equation]
) and from Equation
[Equation]
From Equation 38 and
[Equation]
for any x
[Equation]
y ∈ K
[Equation]
we obtain
[Equation]
This concludes that Abelian group networks are universal.
C. P ROOF OF T HEOREM
Proof.
We prove that for any X ∈ N X of size smaller than b ≥ a ,
[Equation]
(41) by induction on size b .
Note that K 1 K 2 > 1 because they are the Lipschitz constants of inverse functions.
Base Case When b = a , Inequality 41 holds.
Inductive Step We assume Inequality 41 holds for size b (cid:48)
= 1 , . . .
, b − 1 .
We divide X of size b into balanced a subsets X 1 , . . .
, X a so that X
=
X 1 + · · · X a and each | X i | ≤ (cid:100) ba (cid:101)
(Addition over multisets is deﬁned as follows: { x 1 , . . .
, x n } + { x n +1 , . .
, x N } = { x 1 , . .
, x n , x n +1 , . .
, x N } ).
Then
[Table]
[Equation]
i
[Equation]
X
[Equation] [Equation]
X
[Equation]
(cid:107) f
( X i )
[Equation]
f ∗ ( X i ) (cid:107)
[Equation]
From the assumption on size (cid:100) ba (cid:101) , we obtain
[Equation]
< aK
[Equation]
K
[Equation]
(cid:15) (( aK
[Equation]
K
[Equation] [Equation]
log a (cid:100) ba (cid:101)(cid:101) − 1) aK 1 K 2 −
[Equation]
which establishes the inductive step.
C. P ROOF OF T HEOREM
Proof.
First, we prove that associative polynomials are at most ﬁrst-order for each variable.
Assume that we have a n -order ( n ≥ 2 ) associative polynomial
[Equation]
(cid:88)
[Equation]
(44) where α i,j ∈ R for 0 ≤ i, j ≤ n .
Then, we have ( x ∗ y )
∗ z = n (cid:88) i = n (cid:88)
j = α i,j ( n (cid:88) k
= n (cid:88) l = α k,l x k y l ) i
z j (45) and x ∗ ( y ∗ z ) =
n (cid:88) i = n (cid:88)
j = α i,j x i ( n (cid:88) k
= n (cid:88) l = α k,l y k z l ) i .
Since ∗ is associative, these two must form an identity.
By comparing a coefﬁcient of x n 2 , we obtain
[Equation]
If we have 0 ≤ j (cid:48) ≤ n such that α n,j
[Equation]
= 0 , from the coefﬁcient of z j (cid:48) ,
[Equation]
Recursively, we get α n, 0 = α n, 1 = · · · = α n,n = 0 , which leads to contradiction.
Therefore, we now have
[Equation]
In the same way, we can also prove
[Equation]
From Equation 49 and 50 for n ≥ 2 , now we know that associative polynomials are at most ﬁrstorder for each variable.
Therefore, symmetric associative polynomials have the form:
[Equation]
Then we have
[Equation]
and
[Equation]
By solving this identity, we obtain
[Equation]
This condition is equivalent to the associativity of ∗ .
It decomposes into three cases: ( γ = 0 , β = 0) , ( γ = 0 , β = 1) , and γ (cid:54) = 0 .
For each case, we obtain
[Equation]
D E XPERIMENTAL D ETAILS
Here, we explain the detailed setting and further discussion of the experiments that we did not cover in the main part.
D. E XPERIMENTS OF S IZE G ENERALIZATION
Model Architecture For the implementation of the monotonic networks, we followed the Equation 4, except that we added a coefﬁcient term s ∈ R which automatically learn the sign of the weights:
[Equation]
[Equation]
Hyperparameters All networks were trained by the Adam algorithm of lr = 10 − 3 , beta = (0 . 9 , 0 . 999) for 1000 epochs with the batch size of 32 .
Hyperparameters of each model were tuned with the validation dataset using the Optuna framework for each function.
For DeepSets, the number of layers for each MLP was selected from [2 , 8] and the middle dimension and hidden dimension were selected from [2 , 32] .
For the Abelian group network and Abelian semigroup network, the number of groups and the number of units in each group were selected from [2 , 32] .
D. W ORD A NALOGIES
Model architecture For the invertible neural network for the Abelian group network and Abelian semigroup network, we implemented a model based on the afﬁne coupling ﬂows using the FrEIA framework 2 .
We stacked Glow coupling layers and random permutation layers of the dimensions in turn.
For each Glow coupling layer, we used three layer feedforward neural networks with a hyperparameter of hidden dim as a sub network.
For MLP and MLP C, we implemented multilayer perceptrons with the ReLU activation function.
Hyperparameters All networks were trained by the Adam algorithm of lr = 10 − 3 , beta = (0 . 9 , 0 . 999) for 100 epochs with the batch size of 32 .
The hyperparameters of each model were tuned with the validation dataset using the Optuna framework.
For MLP, the number of layers was selected from [2 , 6] and the hidden dimension was selected from [8 , 256] .
For the Abelian group network, the number of layers was selected from [2 , 6] and the hidden dimension was selected from [8 , 256] .
Weight decay was selected from [0 , 10 − 3 ] for all models.
Table 8 summarizes the selected hyperparameters for each model.
[Table]
Detailed Results In Table 9 and 10, we explained all the subcategories of the bigger analogy test set and the Google analogy test set.
We summarized the full results for each subcategory in Table 11, 12, 13, and 14.
[Table]
Derivational
D09 verb+tion irreg accuse:accusation
Derivational
D10 verb+ment irreg accomplish:accomplis...
Encyclopedic
E01 country - capital abuja:nigeria
Encyclopedic
E02 country - language andorra:catalan
Encyclopedic
E03 UK city - county aberdeen:aberdeenshi...
Encyclopedic
E04 name - nationality aristotle:greek
Encyclopedic
E05 name - occupation andersen:writer/poet...
Encyclopedic
E06 animal - young ape:baby/infant
Encyclopedic
E07 animal - sound alpaca:bray
Encyclopedic
E08 animal - shelter ant:anthill/insectar...
Encyclopedic
E09 things - color ant:black/brown/red
Encyclopedic
E10 male - female actor:actress
Lexicographic
L01 hypernyms - animals allosaurus:dinosaur/...
Lexicographic
L02 hypernyms - misc armchair:chair/seat/...
Lexicographic
L03 hyponyms - misc backpack:daypack/kit...
Lexicographic
L04 meronyms - substance atmosphere:gas/oxyge...
Lexicographic
L05 meronyms - member acrobat:troupe
Lexicographic
L06 meronyms - part academia:college/uni...
Lexicographic
L07 synonyms - intensity afraid:terriﬁed/hor...
Lexicographic
L08 synonyms - exact airplane:aeroplane/p...
Lexicographic
L09 antonyms - gradable able:unable/incapabl...
Lexicographic
L10 antonyms - binary after:before/earlier...
[Table]
[Table]
D  0 ( 0 . 
 D  0(0 .
D  0(0 .
D  0 ( 0 . 
 D  0(0 .
D  3(3 .
D  0(0 .
E  0(0 .
E  4(7 .
E  6 ( 30 . 00 %) 0(0 .
E  2(10 . 00%) 3(15 . 00%) 0(0 .
E  5(16 .
E  4(4 .
E  3(3 .
E  12(13 .
E  10(11 .
E  6(6 .
L  0(0 .
L  1(1 .
L  0 ( 0 . 00 %) 0 ( 0 .
L  0(0 . L  0(0 .
L  9 ( 10 . 00 %) 0(0 .
L  11 ( 12 . 22 %) 5(5 .
L  0 ( 0 . 00 %) 0 ( 0 .
L  0(0 .
L
[Table] [Table]
D  11 ( 12 . 22 %) 0(0 .
D  21 ( 29 .
D  13(14 .
D  1(1 . D  1(1 .
D  21(23 .
D  6(6 .
E  18 ( 32 .
E  0(0 .
E  0 ( 0 . 00 %) 0 ( 0 . 00 %) 0 ( 0 . 00 %) 0 ( 0 .
E  0(0 .
E  0(0 .
E  5(5 .
E  3(3 .
E  2(2 .
E  12(13 .
E  43 ( 47 .
L  7(7 .
L  3(3 . L  3 ( 3 .
L  1(1 . L  1(1 .
L  0(0 .
L  12 ( 13 .
L  27 ( 30 . 00 %) 0(0 .
L  2(2 .
L
[Table] [Table]
[Table] [Table]
superlati.
[Table]
present
[Table]