E NCODING WORD ORDER IN COMPLEX EMBEDDINGS
Grue
A BSTRACT
Sequential word order is important when processing text.
Currently, neural networks (NNs) address this by modeling word position using position embeddings.
The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions.
We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships.
Our solution generalizes word embeddings, previously deﬁned as independent vectors, to continuous word functions over a variable (position).
The beneﬁt of continuous functions over variable positions is that word representations shift smoothly with increasing positions.
Hence, word representations in different positions can correlate with each other in a continuous function.
The general solution of these functions is extended to complex-valued domain due to richer representations.
We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available).
Experimentson text classiﬁcation, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings.
To our knowledge, this is the ﬁrst work in NLP to link imaginary numbers in complexvalued representations to concrete meanings (i.e., word order).
I NTRODUCTION
When processing text, the sequential structure of language is important, but can be computationally costly to model with neural networks (NNs)
(Socher et al., 2011) due to the difﬁculty in parallelization.
This has been alleviated by modeling word sequence not on the NN architecture level, but by adding position embeddings on the feature level.
This has been done by the convolutional sequence model (ConvSeq) (Gehring et al., 2017) and the Transformer model (Vaswani et al., 2017) that replaces recurrent and convolution operations with purely attention mechanisms.
More generally, vanilla position embeddings (Gehring et al., 2017) assume that individual word positions are independent and do not consider relations between neighbouring word positions.
We posit that both the global absolute positions of words and their inner sequential and adjacent relationships are crucial in language.
This is supported by recent empirical ﬁndings by Shaw et al.
and Dai et al.
(2019) who show the importance of modeling distance between sequential elements, and explicitly use extra relative position encodings to capture the relative-distance relationship of words.
We present a novel and principled approach to model both the global absolute positions of words and their inner sequential and adjacent relationships as follows: we extend each word embedding, previously deﬁned as an independent vector, as a continuous function over an independent variable i.e., position.
The beneﬁt of continuous functions over variable positions is that word representations shift smoothly with increasing positions.
Hence, word representations in different positions [Figure] can correlate with each other in a continuous function.
Fig.
1 illustrates this type of word representation with a three-dimensional complex-valued embedding, where the amplitudes { r 1 , r 2 , r 3 } denote semantic aspects corresponding to classical word vectors, and periods { p 1 , p 2 , p 3 } denote how sensitive the word is to positional information.
We further discuss the necessary properties of these functions to model sequential information and obtain a general solution in the form of a complexvalued embedding.
Interestingly, there is a direct connection between a speciﬁc case of our general embedding and the well-known positional encoding in Vaswani et al.
(2017) (see App.
A).
We contribute (i) a novel paradigm that extends word vectors as continuous functions over changing variables like word position, and (ii) a general word embedding that models word order in a mathematically-sound manner.
We integrate our complex word embeddings in state-of-the-art (SOTA) NN architectures (CNN, RNN, Transformer and experimentally ﬁnd that it yields gains over both classical word embeddings and position-enriched word embeddings in text classiﬁcation, machine translation and language modeling.
Note that this is the ﬁrst work in NLP to link imaginary numbers in complex-valued representation to concrete meanings (i.e., word order). [Equation]
ODELLING W ORD O RDER IN E MBEDDING S PACE
A Word Embedding (WE) generally deﬁnes a map f we : N → R D from a discrete word index to a D -dimensional real-valued vector and N = { 0 , 1 , 2 , . . . } .
Similarly, a Position Embedding (PE) (Gehring et al., 2017; Vaswani et al., 2017) deﬁnes another map f pe : N → R D from a discrete position index to a vector.
The ﬁnal embedding for word w j ( w j ∈ W with index j in a given vocabulary W ) in the pos -th position in a sentence is usually constructed by the sum f
( j, pos )
=
f we ( j ) + f pe ( pos ) , (1) and f ( j, pos ) ∈ R D .
Since both the word embedding map f w and the position embedding map f p only take integer values as word indexes or position indexes, embedding vectors for individual words or positions are trained independently.
The independent training for each word vector is reasonable, since a word index is based on the order of a given arbitrary vocabulary and does not capture any speciﬁc sequential relationship with its neighboring words.
However, the position index captures an ordered relationship, for instance adjacency or precedence, leading to the problem that position embeddings in individual positions (Gehring et al., 2017) are independent of each other; the ordered relationship between positions is not modelled.
We refer to this as the position independence problem .
This problem becomes more crucial when position embeddings are used in position-insensitive NNs, e.g., FastText (Mikolov et al., 2013b), ConvSeq (Gehring et al., 2017) and
Transformer (Vaswani et al., 2017), because it is hard for such position-insensitive NNs with vanilla position embeddings (Gehring et al., 2017) to infer that w j 1 in the pos -th position is close to w jin the pos + 1 -th position, or that w j 1 precedes w j 2 ; instead, it is only inferred that w j 1 and w jare in different positions, while the relative distance between them is almost unknown.
Thus vanilla position embeddings (Gehring et al., 2017) cannot fully capture the sequential aspect of language.
Next, we ﬁrst introduce the necessary properties to model word order in embeddings, and then give a unique solution to meet such properties.
2. E XTENDING VECTORS TO FUNCTIONS
In the general deﬁnition in Eq.
1, each dimension of the position embedding is obtained based on the discrete position indexes { 0 , 1 , 2 , ..., pos , ... } .
This makes it difﬁcult to model the ordered relationship between the positions.
One solution to this problem is to build continuous functions over a variable (i.e., position index) to represent a speciﬁc word in an individual dimension.
Formally, we deﬁne a general embedding as
[Equation]
where g j is short for g we ( j ) ∈ ( F ) D , indicating D functions over position index pos, and g we ( · ) :
N → ( F ) D is a mapping from a word index to D functions.
By expanding the D dimension of g j , a word w j in the pos -th position can be represented as a D -dimensional vector as shown in [ g [Equation] ( pos [Equation]
[Equation] in which ∀ g j,d ( · ) ∈ F : N → R , d ∈ { 1 , 2 , ..., D } is a function over the position index pos .
To move the word w j from the current position pos to another one pos (cid:48) , it needs only replace the variable pos to pos (cid:48) without changing g j .
Functions for words, especially continuous functions, are expected to capture smooth transformation from a position to its adjacent position therefore modeling word order.
The position-independent position embedding (Gehring et al., 2017) can be considered as a special case of our deﬁnition when it only takes independent values for individual positions in the embedding function.
2. P ROPERTIES FOR THE F UNCTIONS TO CAPTURE WORD ORDER
Relative distance is hard to compute because position indices are not visible in NNs after vector embedding (discrete position indices are necessarily embedded as vectors like words to be backpropagated with the gradient).
Hence, we claim that the modeling of relative distance in NNs should be position-free : absolute position indices cannot be directly accessed in intermediate layers.
Instead of processing position-free operations in NNs to capture relative distance between words, prior work (Shaw et al.,
; Dai et al., 2019) ﬁrst calculates the relative distance between words, and then feeds the relative distance as an additional feature or as embeddings/weights to NNs, instead of directly feeding with the raw position indices.
Assume that words are embedded into R D , and let, for 1 ≤ d ≤ D , the function g j,d : N → R be the embedding function giving the d -th coordinate of the representation of word w j (i.e., g j,d ( pos ) is the d -th coordinate of the embedding of w j if it occurs at position pos.
In the following, we simply write g instead of g j,d when there is no risk of confusion.
Ideally, one would like there to exist a function
Transform n : R → R that transforms the embedding of any word at some position pos to the embedding of a word at position pos + n such that Transform n is only dependent on the embedded value itself, but independent of the position pos, that is ∀ pos : g ( pos + n ) =
Transform n ( g ( pos )) .
Prior work in NLP (Li et al., 2019), Information Retrieval (Van Rijsbergen, 2004) and Machine
Learning (Trabelsi et al., 2017) has shown the usefulness of complex numbers as richer representations.
Complex word embeddings (Wang et al., 2019; Li et al., 2019; Li et al., 2018) have been used to model language.
To investigate the potential of complex-valued representation, we extend the target domains of g ( · ) from R D to C D without losing generality, since real-valued numbers are speciﬁc complex numbers with their imaginary part being zero.
This property regarding “position-free offset transformation” in complex-valued domains is formally deﬁned in Property 1 below.
Property 1.
Position-free offset transformation : An embedding function g : N → C is said to be a position-free offset transformation if there exists a function Transform : N × C → C (called the witness ) such that for all n ≥ 1 , the function Transform n ( · ) =
Transform ( n, · ) satisﬁes ∀ pos ∈ N : g ( pos + n ) =
Transform n ( g ( pos )) .
A position-free offset transformation g is said to be linearly witnessed if there is a function w : N → C such that g has a witness Transform satisfying, for all n ,
Transform ( n, pos ) =
Transform n ( pos ) =
w ( n ) (i.e., each Transform n is a linear function).
Additionally, a boundedness property is necessary to ensure that the position embedding can deal with text of any length ( pos could be large in a long document).
Property 2.
Boundedness : The function over the variable position should be bounded, i.e. ∃ δ ∈
R + , ∀ pos ∈ N , | g ( pos ) | ≤ δ .
Formally, we prove the following claim that there is a unique solution that meets Properties 1 and 2 under the condition that the embedding function is linearly witnessed.
We use linear functions because they are well-understood and simple with a single ﬂoating-point operation in NNs.
Claim 1.
A function g : N → C is a bounded and linearly witnessed position-free offset transformation
iff it is on the form g ( pos ) =
z 2 z pos 1 for z 1 , z 2 ∈ C with | z 1 | ≤ 1 .
Proof.
Assume that g is a bounded and linearly witnessed position-free offset transformation.
Then, by linear witnessing, we have for all pos , n 1 , n 2 ∈ N :
[Equation]
whence w ( n 1 + n 2 ) =
w ( n 1 ) w ( n 2 ) .
Write w (1) =
z 1 and g (0) =
z 2 .
As n 1 , n 2 ∈ N were arbitrary, we have w ( n ) =
( w (1))
n = z n 1 for all n ∈ N .
But then g ( pos + n ) = w ( n ) g ( pos ) =
z n 1 g ( pos ) .
Furthermore, observe that for pos ≥ 1 , we have g ( pos ) =
g (1+ pos − 1) =
w ( pos ) g (0) =
z pos 1 z 2 =
z 2 z pos 1 .
For pos = 0 , g (0) =
z 2 =
z 2 z 01 , whence g ( pos ) =
z 2 z pos 1 , as desired.
Observe that if | z 1 | > 1 , then g ( pos ) is unbounded, whence we have | z 1 | ≤ 1 .
Conversely, assume that g is on the form g ( pos ) =
z 2 z pos 1 with | z 1 | ≤ 1 .
Then, | g ( pos )
[Equation]
≤
[Equation] z pos 1 | ≤ | z {+
+} || z pos 1 | ≤ | z 2 | , whence g is bounded.
Deﬁne, for each n ∈ N , w ( n ) =
z n 1 and Transform n ( pos ) = w ( n ) pos.
Then, for all pos , n ∈ N ,
[Equation]
showing that g is a linearly witnessed position-free offset transformation.
For any z ∈ C , we may write z = re iθ = r
(cos θ + i sin θ ) .
Thus, for the general form of the embedding g from Theorem 1, we have:
[Equation]
In implementations, the above deﬁnition of g will lead to an optimization problem due to the constraint | r 1 | ≤ 1 .
A natural and simple way to avoid this is to ﬁx r 1 = 1 ; note that | e ix | ≡ 1 , thus automatically satisfying the constraint, in contrast to a real-valued embedding where one would need to explicitly devise functions satisfying the constraint.
Finally, Eq. 4 can be written in the simpliﬁed form: g ( pos ) =
re i ( ω pos + θ ) .
Thus, one can think of g as embedding positions counterclockwise on a complex circle of radius r with a ﬁxed period ( r is the amplitude term, θ is the initial phase term, ω
2 π is the frequency, and 2 πω is the period term).
2. C OMPLEX - VALUED WORD E MBEDDING
We now deﬁne our complex-valued word embedding g as a map taking a word index j and position word index pos to C D .
For a word w j in position pos, our general complex-valued embedding is deﬁned as f ( j, pos ) =
g [Equation] pos
[Equation]
=
[Equation]
e
[Equation] j pos [Equation] θ j ) .
Therefore, f ( j, pos ) is deﬁned as:
[Equation]
Note that each coordinate d ( 1 ≤ d ≤ D ) has a separate amplitude r j,d , period p j,d = 2 π ω j,d , and initial phase θ j,d .
In Fig.
1 each dimension is represented as a wave which is parameterized by an amplitude, a period/frequency, and an initial phase.
The trainable parameters of the embedding are the amplitudes vector r j =
[ r j, 1 , ..., r j,D ] , the period/frequency related weights ω j = [ ω j, 1 , ..., ω j,D ] , and the initial phase vector θ j = [ θ j, 1 , ..., θ j,D ] .
Note that the mean values of f ( j, · ) over all positions are linearly dependent on the amplitude.
Observe that the period/frequency determines to what degree the word is sensitive to the position.
With an extremely long period (i.e., ω j very small), the complex-valued embedding is approximately constant for all possible values of pos, and hence approximates a standard word embedding.
Conversely, if the period is short, the embedding will be highly sensitive to the position argument.
In our embedding, the mean vectors of f ( j, · ) taken over all positions are linearly correlated to the amplitude embedding r j = [ r j, 1 , ..., r j,K ] with a coefﬁcient 2 π .
The amplitude r j,d of our embedding depends only on the word w j (and coordinate d ), not on the position of the word, whence one can [Table] think of the vector g pe ( j, pos ) =
[ e i ( ω j, 1 pos + θ j, 1 ) , · · · , e i ( ω j,D pos
+ θ j,D ) ] as a “purely” positional embedding.
Consequently, our complex embedding can be considered an element-wise multiplication between the word embedding g we ( j ) =
[ r j, 1 , ..., r j,K ] and position embedding g pe .
[Equation]
Prior work (Gehring et al., 2017; Vaswani et al., 2017) uses mean-weight addition between word embeddings
f we and position embeddings f pe (all words share the weights).
In our work, word embeddings and position embeddings are decoupled to some extent by element-wise multiplication and therefore the frequency/period terms (related to ω j,d ) can adaptively adjust the importance between semantic and position information for each word and each dimension.
In particular, with higher frequency (i.e., large ω j,d ), the ﬁnal embedding will change dramatically with the changing positions, while it can be ﬁxed for any positions with an extremely-small frequency (i.e., small ω j,d ).
Interestingly, the well-known position embedding in Transformer (Vaswani et al., 2017) can be seen as a degraded version of one of our speciﬁc complex word embeddings (see the Appendix A).
E XPERIMENTAL E VALUATION
We evaluate our embeddings in text classiﬁcation, machine translation and language modeling.
3. T EXT C LASSIFICATION
Experimental Setup.
We use six popular text classiﬁcation datasets: CR, MPQA, SUBJ, MR,
SST, and TREC (see Tab. 1).
We use accuracy as evaluation measure based on ﬁxed train/dev/test splits or cross validation, as per prior work.
We use Fasttext (Joulin et al., 2016), CNN (Kim, 2014),
LSTM and Transformer (Vaswani et al., 2017) as NN baselines 2 .
We use each of them: (1) without positional information; (2) with Vanilla Position Embeddings (PE) (randomly initialized and updated during training using the sum between word and position vectors (Gehring et al., 2017); (3) with Trigonometric Position Embeddings (TPE) (deﬁning position embeddings as trigonometric functions as per Eq. 7); (4) with Complex-vanilla word embeddings (where the amplitude embedding is initialized by the pre-trained word vectors, and the phrase embedding is randomly initialized in a range from − π to π without considering word order (Wang et al., 2019)); and (5) with our order-aware complex-valued word embeddings, Complex-order (which encode position in the phase parts, train the periods, and where the amplitude embedding is also initialized by pretrained word vectors).
For more details on the complex-valued extensions of NNs, see App.
B and App.
C.
Our embedding generally has 3 × D × | W | parameters with D-dimensional word vectors and | W | words, while previous work (Mikolov et al., 2013b; Pennington et al., 2014) usually employs only
D × | W | parameters for embedding lookup tables.
To increase efﬁciency and facilitate fair comparison with previous work we set initial phases θ j
=
[ θ j, 1 , ..., θ j,D ] to a shared constant value (such as zero).
Furthermore, the period vectors ω
j,d depend on word index j with length | W | and the coordinate index d with length D .
To decrease the number of parameters, one can either use a word-sharing scheme (i.e., ω j,d = ω · ,d ), or a dimension-sharing scheme ( ω j,d = ω j, · ), leading to | W | ∗ D + | W | and | W | ∗ D + D parameters in total for the embedding layer. [Table] [Table]
We search the hyper parameters from a parameter pool, with batch size in { 32 , 64 , 128 } , learning rate in { 0 . 001 , 0 . 0001 , 0 . 00001 } , L2-regularization rate in { 0 , 0 .
001 , 0 . 0001 } , and number of hidden layer units in { 120 , 128 } .
We use pre-trained 300-dimensional vectors from word2vec (Mikolov et al., 2013a) in all models except for Transformers.
The models with trainable trigonometric position embedding produce nearly identical results compared to the non-trainable version, therefore we report the result of ﬁxed position embeddings as per Vaswani et al.
We adopt narrow convolution and max pooling in CNN, with number of ﬁlters in { 64 , 128 } , and size of ﬁlters in { 3 , 4 , 5 } .
In all Transformer models, we only use the encoder layer to extract feature information, where the layer is 1, dimension of word and inner hidden are 256 and 512 respectively, and head number is 8.
Results.
The results are shown in Tab. 2.
Our complex-order embeddings outperform all other variations at all times.
This gain in effectiveness comes at a negligible (or non-existent) cost in efﬁciency (it varies per NN architecture – see Fig. 2).
CNNs are the best performing NN as expected following Bai et al.
Tranformer NNs beneﬁt the most from our complex-order embeddings, most likely because they are our weakest baseline.
To contextualise these results, Tab. 3 shows classiﬁcation accuracy of ﬁve typical approaches on the same datasets (as reported in the original papers).
Our complex-order embeddings outperform all methods, except for the CR dataset, where
InferSent is marginally better.
Overall, our approach is on a par with the SOTA in embeddings.
We perform an ablation test (Tab. 4) on Transformer because it is the most common NN to be used with position embeddings. The two period-sharing schemas (dimension-sharing and word-sharing) w/t PE
PE TPE Complex-vanilla Complex-order FasttextBiLSTMCNNTransformer
[Table]
vanilla [Table]
[Table] et al., 2017) [Table] slightly drop performance, because fewer parameters limit the representative power.
Adding initial phases also hurts performance, although we observed that the loss could decrease faster in early epochs compared to the setting without offset.
The negative effect of initial phases may be due to periodicity, and ω cannot be directly regularized with L2-norm penalties.
The sharing schemes slightly decrease the performance with less parameters.
More details of the learned periods/frequencies (e.g. the distributions of periods/frequencies and case studies) are shown in App.
D.
Note that the word-sharing schema outperform the Vanilla Transformer, (both have a comparable number of parameters).
If we choose (cid:60) ( W Q/K/V ) = (cid:61) ( W Q/K/V ) , the additional parameters in the embedding layers will affect much less the whole parameter scale in the multiple-layer Transformer, since a embedding layer is only used in the ﬁrst layer instead of the following Transformer layers.
3. M ACHINE T RANSLATION
Experimental Setup.
We use the standard WMT 2016 English-German dataset (Sennrich et al., 2016), whose training set consists of 29,000 sentence pairs.
We use four baselines: basic Attentional encoder-decoder (AED) (Bahdanau et al., 2014); AED with Byte-pair encoding (BPE) subword segmentation for open-vocabulary translation (Sennrich et al., 2016); AED with extra linguistic features (morphological, part-of-speech, and syntactic dependency labels) (Sennrich & Haddow, 2016); and a 6-layer Transformer.
Our approach (Transformer Complex-order) uses a batch size of 64, a head of 8, 6 layers, the rate of dropout is 0.1, and the dimension of the word embedding is 512.
The embedding layer does not use initial phases, i.e., following f ( j, pos ) =
r j e i ( ω j pos ) .
We evaluate MT performance with the Bilingual Evaluation Understudy (BLEU) measure.
[Table]
[Table]
[Table]
Results Tab. 5 shows the MT results.
Our approach outperforms all baselines.
Two things are worth noting: (1) Both the vanilla Transformer and our Transformer Complex-order outperform the three Attentional encoder-decoder baselines which are based on an LSTM encoder and decoder, even when AED uses additional features.
(2) Our Transformer Complex-Order outperforms the Vanilla
Transformer and complex-vanilla Transformer by 1 . 3 and 1 . 1 in absolute BLEU score respectively.
3. L ANGUAGE M ODELING
Experimental Setup.
We use the text8 (Mahoney, 2011) dataset, consisting of English Wikipedia articles.
The text is lowercased from a to z, and space.
The dataset contains 100M characters (90M for training, 5M for dev, and 5M for testing, as per Mikolov et al.
We use as baselines
BN-LSTM, LN HM-LSTM RHN and Large mLSTM, which are typical recurrent NNs for language modeling in this dataset.
We evaluate performance with the Bits Per Character (BPC) measure, (the lower, the better).
We run the coder in Dai et al.
with 6 layers for Transformer XL 6L; our model, named Transformer XL complex-order, directly replaces the word embedding with our proposed embedding under the same setting.
We choose 6 layers due to limitations in computing resources.
For Transformer XL Complex-order, all other parameter settings are as for Transformer
XL (Dai et al., 2019).
Our complex-order model does not use initial phases.
Results.
We see in Tab. 6 that our method outperforms all baselines.
The ﬁrst four baselines (BNLSTM, LN HM-LSTM, RHN and Large mLSTM) are based on recurrent NNs and rely on different regulation methods to become applicable in multiple-layer recurrent architectures.
Transformerbased architectures can easily be stacked with multiple layers due to their advantages in parallelization, however the vanilla Transformer does not outperform the multiple-layer recurrent baselines, most likely due to its limitation of 6 layers.
Our Transformer XL Complex-order outperforms its vanilla counterpart under the 6-layer setting (and also strong recurrent network baselines), demonstrating that our embedding also generalizes well in tasks with long-term dependency.
With limited resources, slightly increasing the parameters in the feature layer like our proposed embedding could be more beneﬁcial than stacking more layers with linearly increasing parameters.
R ELATED W ORK
Complex-valued NNs are not new (Georgiou & Koutsougeras, 1992; Kim & Adalı, 2003; Hirose, 2003).
Complex-valued weights have been used in NNs, motivated by biology (Reichert & Serre, 2013), and also as signal processing in speech recognition (Shi et al., 2006).
More recently, Arjovsky et al.
(2016) shifted RNNs into the complex domain and Wolter & Yao (2018) proposed a novel complex gated recurrent cell.
Trabelsi et al.
(2017) also developped a complex-valued NN for computer vision and audio processing.
Complex numbers have also been applied to text processing like (Van Rijsbergen, 2004; Melucci, 2015; Blacoe et al., 2013).
Trouillon et al.
(2016) adopt complex embedding for entities in Knowledge Graph Completion to represent antisymmetric relations with Hermitian dot product.
Li et al.
; Wang et al.
(2019) extend word embeddings to complex-valued fashion in quantum probability driven NNs, seeing the overview in Wang et al.
However, the physical meaning of both the complex-valued entity and word embeddings is unknown, since a complex number was considered as two real numbers in a black-box learning paradigm.
Our work ﬁrst links the phase in complex numbers to word position to deﬁne concrete physical meaning in document representations.
C ONCLUSIONS
We extended word vectors to word functions with a variable i e. position, to model the smooth shift among sequential word positions and therefore implicitly capture relative distances between words.
These functions are well-deﬁned to model the relative distances, therefore we derive a general solution in complex-valued fashion.
Interestingly, the position embedding in Vaswani et al.
(2017) can be considered a simpliﬁed version of our approach.
We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding.
Experiments on text classiﬁcation, machine translation and language modeling show that our embeddings are more effective than vanilla position embeddings (Vaswani et al., 2017).
A CKNOWLEDGMENTS
We thank Massimo Melucci and Emanuele Di Buccio for their helpful comments, Xindian Ma for his detailed experimental suggestions.
This work is supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union‘s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.
A L INKING TO THE POSITION EMBEDDINGS IN (V ASWANI ET AL ., 2017)
Vaswani et al.
(2017) proposed a new initialization for position embedding, resulting in comparable performance with previous one (Gehring et al., 2017) even without ﬁne-tuning.
The position embedding is empirically selected as
[Equation]
Where pos is the position index, 2 k and 2 k + 1 is the dimension index and d model is the dimension size of embedding.
The reason for choosing this position embedding was not well-explained and its general extension is unknown, leading to some difﬁculties to improve it.
We claim that the proposed position embedding in (Vaswani et al., 2017) is a degraded version of one of our speciﬁc complex word embedding in word-sharing schema (i.e., ω j,d = ω · ,d ), in which p j,k = 2 π × 10000 2 k/d model
and the initial phases are set as zero.
In our complex-valued position embedding, let f pe,k ( · , pos ) =
e i × 10000 2 k/dmodel = cos(10000 2 k/d model pos )
+ i sin(10000 2 k/d model pos ) .
Note that there exists a bi-jection between PE ( · , pos ) and f pe,k ( · , pos ) :
[Equation]
where (cid:60) and (cid:61) are the operations to take the real and imaginary part of a complex-valued number.
Its inverse transformation is
[Equation]
[Equation]
In our overall embedding, each dimension f k ( j, pos ) =
f we,k ( j ) (cid:12) f pe,k
( · , pos ) in our approaches, while it is E k ( j, pos )
=
WE k
( j ) +
PE k
( · , pos ) in (Gehring et al., 2017; Vaswani et al., 2017).
Hence the position embedding in (Vaswani et al., 2017) is equivalent, albeit not identical, to our complex-valued position embedding with p j,k = 2 π × 10000 2 k/d model .
It is therefore a particular case of our complex-valued position embedding, the word-sharing schema in which all words share the same period at a certain dimension, i.e, p j,k = p · ,k is irrelevant to the choice of j .
B I NTEGRATING COMPLEX - VALUED EMBEDDING TO GENERAL NEURAL NETWORKS
Neural networks are typically given real numbers as inputs and return real numbers as outputs.
To accommodate complex numbers as in- and output, we devise a complex-valued version of various neural network layers i.e. complex-valued FastText with dense layer, CNN, and RNN.
Unlike existing complex-valued neural networks (Trabelsi et al., 2017; Wolter & Yao, 2018), our feature layers are also converted into complex-valued layers.
Complex-valued FastText FastTest (Joulin et al., 2016) is a simple and efﬁcient neural network architecture using a dense layer over the sum of all word embeddings for general text classiﬁcation.
For a linear dense layer, i.e., z = dense ( x + i y ) , where x + i y and z denote the complex-valued in- and output, respectively.
Let W
=
A + i B and b =
c + i d be complex-valued linear weights and bias, respectively.
Then, the complex-valued dense layer is given by:
[Equation]
10) where σ is a real-valued activation function such as the sigmoid function.
By rewriting (10) in matrix form (Trabelsi et al., 2017), we obtain:
[Equation]
where, for z = x + iy , (cid:60) ( z ) = x and (cid:61) ( z ) =
y .
To save parameters and fairly compare with our real-valued baselines, the weights for real-part and imaginary-part input can be shared, i.e.,
A = B , c = d .
Complex-valued CNN For the complex-valued version of the convolution operation Trabelsi et al.
(2017), we similarly deﬁne a complex-valued convolution with separate real and imaginary kernels
A and B , to compute convolutions on the real and imaginary parts of the input in Eq.
A complex-valued CNN network is constructed by stacking the operations based on a complex-valued convolution kernel, adding a complex dense layer in (10), and taking their norm as the ﬁnal prediction in the last layer.
Complex -valued RNN The basic complex RNN formulation is:
[Equation]
(12) where z t and h t represent the complex-valued input and complex-value hidden state vectors at time t , b is a complex-valued bias, W h and W z are complex-valued weight transitions for hidden state and input state, and f ( z )
=
σ ( (cid:60) ( z )) + iσ ( (cid:61) ( z )) is the activation function.
The multiplication
W h
h t − 1 and W z z t is computed as deﬁned in (10) above.
Similarly, the complex-valued gates are used in LSTM via operations as in (12).
In the ﬁnal layer, a l 2 -norm operation is adopted to obtain a real-valued loss for backpropagation.
Complex-valued Transformer The main components in the Transformer are self-attention sublayers and position-wise feed-forward (FFN) sublayers.
A self-attention sublayer employs h attention heads and the concatenation of all heads is used as the output followed by a parameterized linear transformation.
For a sequence embedded as complex-valued vector input = { w 1 , w 2 , ..., w n } , the output of each head is computed as a weighted sum of a linear transformation of the input sequence itself, namely
[Equation]
where w j is a complex-valued vector and W V is a complex linear transformation; therefore output i is also complex.
Hence, output is a sequence of complex-valued vectors with the same shape as input .
The weight coefﬁcient, a i,j , which is deﬁned as in real-valued domain, is calculated as the softmax of the product between complex-valued query vectors and key vectors: a i,j = softmax [Equation] i [Equation]
and
[Equation]
z is a complex number since w i and w j are complex-valued vectors and W Q , W K are complexvalued transformation.
To extend another variant of Transformer called Transformer XL, we keep its original relative position embedding and additionally replace its word embedding with our proposed embedding.
Correspondingly, the FFN sublayer can easily be extended to a complex-valued version by replacing the real-valued layers with complex-valued ones.
We use batch-normalization separately for the real and imaginary parts.
C I MPLEMENTATION OF THE PROPOSED EMBEDDING
Words functions are implemented in neural networks by storing the function parameters { r , ω , θ } and then construct the values based on the arguments.
Based on the deﬁnition, the implementation of the proposed embedding can easily be implemented with only modifying the embedding layer.
We list the basic code to construct our general embedding as below:
import
torch import math
[Equation]
[Table]
Note that both the frequency vectors ω and initial-phase vectors θ can be shared between words or dimensions, to save parameters.
The proposed embedding can be also used in real-valued neural networks if one directly concatenates the real-part numbers and imaginary-part numbers as a doublesize real-valued vector; therefore it could easily be extended in any existing networks without any complex-valued components.
For instance, it could be a good extension for Transformer based pretrained models like (Devlin et al., 2018) by enriching the feature layer.
D V ISUALIZATION OF FREQUENCIES / PERIODS
After training, we obtain the frequency vector ω for each word.
For each word, the mean value of the absolute frequency values, i.e., δ j = 1 | D | (cid:80) Dd =1 | ω j,d | is considered as a metric to test the positional sensitivities of the word, since a period value could be negative during training.
The density of the δ j is shown in Fig.
Words with the 50 greatest, and the 50 smallest, frequencies in the SST dataset are shown in Tab.
,“stupid” and “powerful”; a reason for this may be that such words appear in many positions in many documents during training, and thus they are more sensitive to the positions.
Conversely, there are fewer words expressing strong sentiment among words with smaller frequencies, as shown in the second row.
[Table]