R EFLECTION - BASED W ORD A TTRIBUTE T RANSFER
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a word attribute transfer framework based on reﬂection to obtain a word vector with an inverted target attribute for a given word in a word embedding space.
Word embeddings based on Pointwise Mutual Information (PMI) represent such analogic relations as −−→ king − −−→ man + −−−−−→ woman ≈ −−−→ queen .
These relations can be used for changing a word’s attribute from king to queen by changing its gender.
This attribute transfer can be performed by subtracting a difference vector −−→ man − −−−−−→ woman from −−→ king when we have explicit knowledge of the gender of given word king .
However, this knowledge cannot be developed for various words and attributes in practice.
For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it.
In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice.
We introduce a framework based on reﬂection mapping that satisﬁes this property; queen should be transferred back to king with the same mapping as the transfer from king to queen .
Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.
I NTRODUCTION
Distributed representation (Hinton et al., 1984) is a kind of data representation that can capture data similarities in a vector space.
In natural language processing, various studies have been conducted on word embeddings (Mikolov et al., 2013a;b; Pennington et al., 2014; Peters et al., 2018;
Bojanowski et al., 2017).
Word embedding models, such as skip-gram with negative sampling (SGNS) (Mikolov et al., 2013b) or GloVe (Pennington et al., 2014), capture some analogic relations, such as −−→ king − −−→
[Equation]
Previous work offer theoretical explanation based on Pointwise Mutual
Information (PMI; Church & Hanks (1990)) for maintaining the analogic relations in word vectors (Levy & Goldberg, 2014b; Arora et al., 2016; Gittens et al., 2017; Ethayarajh et al., 2019; Allen &
Hospedales, 2019).
These relations can be used for transferring a certain attribute of a word, such as changing king into queen by transferring the gender.
This task, which is called word attribute transfer , enables us to rewrite He is a boy as She is a girl .
Word attribute transfer is expected to be applicable for natural language inference and data augmentation in natural language processing.
The above analogic relations can be used, including adding difference vector −−−−−→ woman −−−→ man to −−→ king to transfer −−→ king to −−−→ queen .
This operation requires the explicit knowledge whether an input word is male or female; we have to add a difference vector to a male word and subtract it from a female word for a gender transfer.
We also have to avoid changing words without any gender attributes, such as is and a in the example above.
Thus, analogy-based word attribute transfer requires explicit knowledge of word attributes, such as king is male, queen is female, and is has no gender attribute.
Developing such knowledge is very difﬁcult for various words and attributes in practice.
In this study, we propose a novel framework based on reﬂection , which enables word attribute transfer by a single reﬂection-based mapping for a certain attribute.
Reﬂection in geometry is a mapping that exchanges the locations of two vectors in a Euclidean space by a hyperplane called a mirror , which satisﬁes the above desired property: working as identity mapping when it is applied twice and when it is applied to vectors on the mirror.
We apply this reﬂection mapping to the problem of word attribute transfer by estimating an appropriate mirror that maps word pairs with a binary target attribute (e.g., male and female) and keeps the other words without that attribute using training data.
We also extend this approach by introducing parameterized mirrors , which work as different mirrors based on the given input words, to overcome a limitation using a single ﬁxed mirror to represent complex transfer mappings for different words.
Experimental results show that the reﬂection-based method enables such transfers, achieves comparable performance to analogy-based methods with explicit attribute knowledge, even though our proposed method does not use such knowledge.
The following are the contributions of this paper:
• We propose a novel representation learning framework that obtains a vector with an inverted attribute in embedding space without explicit attribute knowledge of the given word.
• Our proposed reﬂection-based word attribute transfer enables us to transfer word attributes in up to 76% for words with target attributes and to avoid changing words without target attributes in over 99% in our experiments.
[Equation]
[Figure]
[Equation]
Let x denote a word and let v x denote its vector representation.
Here we assume that v x is learned in advance with an embedding model such as SGNS.
In this task, we have two inputs, word x and onehot vector z , representing a certain target attribute, and one output, return word t with the inverted attribute of x for z .
Let A denote a set of a triplet ( x, t, z ) , e.g., ( man, woman, z gender ) ∈ A .
Let N denote a set of words without attribute z , e.g., apple ∈ N , when z represents gender.
The purpose of this task is to transfer v x to v t by transfer function f z that inverts attribute z of v x .
In other words, output v y should be close to the vector of corresponding target word v t , which is typically the nearest neighbor of v t in the word embedding space.
[Equation]
Note that mapping f z transfers word x if it has target attribute z ; otherwise f z works as identity mapping.
For instance with z gender , given input word man , gender attribute transfer f z gender ( v man ) should result in a vector close to v woman .
Given input word apple as x , the results should be v apple .
A NALOGY - BASED W ORD A TTRIBUTE T RANSFER
Analogy is a general idea for realizing attribute transfer.
Several PMI-based word embedding methods (Mikolov et al., 2013c; Linzen, 2016) tackled to embed words into word embedding space to capture the analogic relations.
An embedded vector with SGNS or GloVe captures analogic relations (Levy & Goldberg, 2014a; Mikolov et al., 2013c; Linzen, 2016).
For instance, v queen is near the vector obtained on the right side of Eq.
By rearranging Eq. 2, Eq. 3 is obtained:
[Equation]
We can transfer the gender attribute by subtracting difference vector v man − v woman from input word vectors, e.g., v king .
The analogy-based transfer function is
[Equation]
where d is a difference vector of the given word pair such as man and woman , M is a set of words having a target attribute, and F is a set of words having an inverse attribute, for example, man ∈ M and woman ∈ F for gender attributes.
Eq.
4 indicates that the operation changes depending on whether input word x belongs to M or F .
For gender words, we subtract difference vector
d if x is male, and add it if x is female.
Therefore, we need such explicit knowledge.
However, this knowledge cannot be developed for various words and attributes in practice.
R EFLECTION - BASED W ORD A TTRIBUTE T RANSFER 4. I DEALIZED TRANSFER WITHOUT EXPLICIT KNOWLEDGE
What is an idealized transfer function φ z for the word attribute transfer?
The following are the idealized natures of such a transfer function:
[Equation]
where m ∈ M and w ∈ F
This function φ z enables to transfer a word without explicit knowledge.
Function φ z transfers v m to v w and v w to v m without such explicit knowledge as m ∈ M and w ∈ F .
By combining Eqs.
5 and 6, we obtain the following formula:
[Equation]
and
[Equation]
Hence, the idealized transfer function is a mapping that becomes an identity mapping when we apply it twice for any v .
Such a mapping is called involution in geometry.
For example, φ : v (cid:55)→ − v is one example of an involution.
Note that the identity map itself, such as φ : v (cid:55)→ v , is excluded from the involution.
4. R EFLECTION
A reﬂection is an involution that reverses the location between two vectors in a Euclidean space through an afﬁne hyperplane (mirror).
Reﬂection is an idealized function because every point returns to its original location when reﬂection is applied twice:
[Equation]
Given vector v in Euclidean space R n , the formula for the reﬂection in the mirror is given:
[Equation]
where a ∈ R n is a vector orthogonal to the mirror (normal vector) and c ∈ R n is a point through which the mirror passes.
a and c are parameters that determine the mirror.
4. R EFLECTION - BASED W ORD A TTRIBUTE T RANSFER
We apply reﬂection to the word attribute transfer to invert a speciﬁc attribute of an input word without its explicit attribute knowledge.
We learn a mirror (hyperplane) in a pre-trained embedding space using training word pairs with a common (binary) attribute z (Fig. 2).
Here since the mirror is uniquely determined by two parameter vectors, a and c , we estimate a and c from target attribute z using fully connected multi-layer perceptrons:
[Equation]
Transferred vector v y is obtained by inverting attribute z of v x by reﬂection:
[Equation]
[Figure]
apple apples orange oranges bananas banana
Mirror
[Figure] [Figure]
4. P ARAMETERIZED M IRRORS
Reﬂection with a mirror by Eqs.
11 and 12 assumes a single mirror depending only on z .
Previous discussion assumed that there will be pairs sharing a stable attribute such king and queen .
However, often gendered words don’t come in pairs, and gender is far from a stable attribute.
For example, actress may be feminine, but actor is clearly neutral in many cases (Fig. 3).
Thus, actor isn’t as obvious a masculine counterpart as king .
In fact, it is known that there are biases in gender words in the embedding space (Zhao et al., 2018; Kaneko & Bollegala, 2019).
This phenomenon can occur not only with the gender attribute, but also with other attributes.
With this assumption of a single mirror, the mirror must be a hyperplane that goes through the midpoints for all word vector pairs.
However, the vector pairs shown on the left of Fig.
3 cannot be transferred well since the single mirror does not satisfy this constraint due to the bias of the embedding space.
To solve this problem, we introduce different mirrors for different words.
We propose parameterized mirrors determined by input vector v x in addition to attribute z .
The following are the deﬁnitions of the mirror parameters:
[Equation]
where [ · ; · ] indicates the vector concatenation in the column.
The parameterized mirrors are expected to work ﬂexibly on different words.
For instance, as shown in Fig. 3, suppose we learned the mirror (the blue line) that transfers v hero to v heroine in advance.
If input word vector v actor resembles v hero , a mirror that is similar to the one for v hero should be derived and used for the attribute transfer.
4. L OSS F UNCTION
Loss function L is deﬁned:
[Equation]
where 1 |A| (cid:80) ( x i ,t i , z i ) ∈A ( v y i − v t i ) 2 is a term that draws target word vector v t i closer to corresponding transferred vector v y i and 1 |N| (cid:80) x
j ∈N ( v y j − v x j ) 2 is a term that prevents words without a target attribute from being moved by transfer function f z .
Θ represents the set of all the trainable parameters.
The parameters in the proposed model are the MLP weights used to determine mirror hyperplanes via a and c .
We iteratively update Θ to minimize L :
[Equation] [Equation]
where t is the number of parameter updates at that time.
E XPERIMENT
We evaluated the performance of the proposed reﬂection-based word attribute transfer using data with some different attributes.
5. E XPERIMENTAL S ETUP
We used three different datasets of word pairs with three binary attributes: Male-Female 1 , SingularPlural and Capital-Country, shown in Table 1.These word pairs were collected from analogy test sets (Mikolov et al., 2013a; Gladkova et al., 2016), the Internet and Nguyen et al.
(2017) for antonyms of noun 2 .
Since these datasets are very small, we added Gaussian noise ( σ = 0 . 1 ) to every input vector v x during training to avoid overﬁt.
Random noise was applied independently to every sample in every iteration.
For a non-attribute dataset N , we sampled words from the three-million-word vocabulary of the word embedding model.
We sampled from 4 to 50 words for training ( 0 ≤ |N train | ≤ 50 ) and 1000 words for the test ( |N test | = 1000 ).
We used a mixed dataset that included both |N train | and |A train | .
Note that N test was sampled and excluded words from N train and A train .
We had no N val because the tuning was conducted with only |A val | .
We used word2vec (Mikolov et al., 2013b) 3 and GloVe (Pennington et al., 2014) 4 as the pre-trained embedding model.
The embedded vector dimension is n = 300 .
[Table]
5. E VALUATION M ETRICS
We measured the accuracy and stability performances of the word attribute transfer.
The accuracy measures how many input words in A test were transferred correctly to the corresponding target words.
The stability score measures how many words in N test are not mapped to other words.
For example, in a gender transfer, given man , the transfer is regarded as correct if woman is the closest word to the transferred vector; otherwise it is incorrect.
Given apple , the transfer is regarded as correct if apple is the closest word to the transferred vector; otherwise its stability is incorrect.
Here we used cosine similarity to measure the similarity of output vector v y and target vector v t .
The accuracy and stability scores are calculated by the following formula:
[Equation]
Accuracy = |A test | (cid:88) ( x i ,t i , z i ) ∈A test δ ( v y i , t i ) , (19)
Stability =
|N test | (cid:88)
x i ∈N test δ ( v y i , x i ) , (20) where V is the vocabulary of the word embedding model
5. M ETHODS AND C ONFIGURATIONS
In our experiment we compared our proposed method with the following baseline methods:
R EF Reﬂection-based word attribute transfer with a single mirror.
We used a fully connected 2-layer MLP with 300 hidden units and ReLU activations to
estimate
a and c .
R EF + PM Reﬂection-based word attribute transfer with parameterized mirrors .
We used the same MLP as the R EF .
MLP Fully connected MLP: v y =
MLP ([ v x ; z ]) .
The highest accuracy models are a 2-layer MLP for Capital-Country and 3-layer MLP for the other datasets.
D IFF Analogy-based word attribute transfer with a difference vector:
d = v m − v w , where m and w are in the training data of A .
We chose d because it achieved the best accuracy in the validation data of A .
We determined whether to add or subtract d to v x based on attribute knowledge (Eq. 4).
D IFF + Analogy-based word attribute transfer with a difference vector regardless of the attribute knowledge.
d was obtained in the same way as the D IFF .
We added d to v x for any input x : f z ( v x ) = v x + d , ∀ v x ∈ R n .
D IFF − Analogy-based word attribute transfer with a difference vector regardless of the attribute knowledge.
d was obtained in the same way as the D IFF .
We subtracted d from v x for any input x : f z ( v x ) = v x − d ,
[Equation]
R n .
M EAN D IFF Analogy-based word attribute transfer with a mean difference vector ¯ d : ¯ d =|A train | (cid:80)
( m i ,w i ) ∈A train ( v m i − v w i ) .
We determined whether to add or subtract ¯ d to v x based on the attribute knowledge (Eq.4).
M EAN D IFF + Analogy-based word attribute transfer with a mean difference vector regardless of the attribute knowledge: f z ( v x ) =
v x + ¯ d , ∀ v x ∈ R n .
¯ d was obtained in the same way as the M EAN D IFF .
M EAN D IFF − Analogy-based word attribute transfer with a mean difference vector regardless of the attribute knowledge: f z ( v x ) = v x − ¯ d , ∀ v x ∈ R n .
¯ d was obtained in the same way as the M EAN D IFF .
Based on the tuning, we used the Adam optimizer (Kingma & Ba, 2015) with a learning rate of α = 10 − 4 (the other hyperparameters were the same as the original one (Kingma & Ba, 2015)), and a batchsize of 62 for male-female, and 32 for the others.
These hyperparameters were identical for the learning-based methods: R EF , R EF + PM, or MLP.
We did not use such regularization methods as dropout (Srivastava et al., 2014) or batch normalization (Ioffe & Szegedy, 2015) because they did not show any improvement in our pilot test.
5. A CCURACY AND S TABILITY
Table 2 and 3 shows the transfer accuracy and stability score results.
Both experiments using GloVe or word2vec obtained similar results.
R EF + PM achieved the best accuracy among the methods that did not use explicit attribute knowledge.
This means that reﬂection can be used for word attribute transfers even without attribute knowledge.
In the stability evaluation, reﬂection-based methods (R EF , R EF + PM) and the analogy-based methods with a mean difference vector (M EAN D IFF − ,
M EAN D IFF + ) achieved high stability.
In particular, reﬂection-based transfers achieved outstanding stability scores exceeding 99%.
The stability of D IFF + and D IFF − was much lower than the other methods.
Although M EAN D IFF − and M EAN D IFF + achieved high stability, their accuracy results
[Table]
[Table]
were very low.
Interestingly, reﬂection-based transfer with parameterized mirrors (R EF + PM) achieved high performance in both accuracy and stability.
For example, the accuracy of R EF + PM was 41.67%, and the stability was 99.9% in Male-Female (MF), and the accuracy was 58 % and the stability was 99.40 % in Capital-Country (CC).
These results show that the proposed method transfers an input word if it has a target attribute and does not transfer an input word even though it does not use explicit attribute knowledge on the input words.
MLP worked poorly both in accuracy and stability.
In the antonym (AN), while the transfer accuracy by the proposed method was a bit lower than that by MLP, the stability of the proposed method was 100% and that of MLP was really poor (almost 0%).
We investigated the relation between the size of |N train | and the stability of learning-based methods by conducting an additional experiment by varying |N train | from 0 to 50.
The stability scores by
MLP did not improve (Table 4).
On the other hand, R EF and R EF + PM achieved high stability scores with just |N train | = 4 and maintained the accuracy.
5. T RANSFER E XAMPLE
Table 5 shows examples of a gender transfer at the sentence level, where the attribute transfer was applied to words in sentence X
= { x 1 , x 2 , ... } .
Here since such words as a and .
are not in the vocabulary of the original word embedding model, we omitted them from the inputs.
MLP made many wrong transfers on words without gender attributes, e.g., the became By Katie Klingsporn , was became she , when became Doughty Evening Chronicle , and woman became girlfriend .
D IFF + can transfer if x is female, e.g., it transferred from woman to man , but it could not transfer grandfather and boy .
Similarly, D IFF − failed to transfer from female to male.
In addition, since the stability of these methods was low, they erroneously transferred.
For example, in D IFF − , the and when became she .
R EF + PM can selectively transfer words with a gender attribute without using explicit gender information.
For example, when woman was given, Ref ( v woman ) became man without knowledge that woman is a female word, and when man was given, it became woman .
When non-attribute word
[Table]
married was given, Ref ( v married ) became married without knowledge that married has no gender attribute.
When we applied the reﬂection-based transfer twice, the transferred word returned to its original word, e.g., Ref ( Ref ( v woman )) gives woman .
[Table]
We can transfer some different attributes of words with reﬂection-based transfer one-by-one.
Table 6 shows that the words having different target attributes were transferred by each reﬂection-based transfer in the order of Male-Female, Singular-Plural, and Country-Capital.
Given actress for a
Male-Female transfer, it was transferred to actor and to actors for Singular-Plural.
Given Japan for
Male-Female and Singular-Plural, it was not transferred, but it was transferred to Tokyo for CountryCapital.
R ELATED W ORK
The embedded vectors obtained by SGNS (Mikolov et al., 2013a;b) and GloVe (Pennington et al., 2014) have analogic relations.
The theory of analogic relations in word embeddings has been widely discussed: Levy & Goldberg (2014b); Arora et al. (2016);
Gittens et al.
; Ethayarajh et al.
; Allen & Hospedales (2019); Linzen (2016).
Levy & Goldberg (2014b) offer the explanation that SGNS factorizes a shifted PMI matrix.
Allen & Hospedales (2019) and Ethayarajh et al.
(2019) argued that they proved the existence of such analogic relations without strong assumptions.
In our work, we focus on the analogic relations in a word embedding space and propose a novel framework to obtain a word vector with inverted attributes.
Style transfers (Niu et al., 2018; Prabhumoye et al., 2018; Jain et al., 2019; Logeswaran et al., 2018; Dai et al., 2019; Zhang et al., 2018) resemble our task.
In a style transfer, the text style of the input sentences is changed.
For instance, Jain et al.
[Table]
beautiful
[Table]
in
[Table]
in
[Table]
+
[Table]
the poor actors
[Table]
and
[Table]
rich actresses
[Table]
stay the beautiful towns in Japan or see animal in Canberra (.)
(2019) transferred from formal to informal sentences.
Logeswaran et al.
(2018) transferred sentences by controlling such attributes as mood and tense.
These style transfer tasks use sentence pairs; our word attribute transfer task uses word pairs.
Style transfer changes sentence styles, but our task changes the word attributes (contents).
Soricut & Och (2015) studied the problem of morphological transformation based on character information.
Our work aims more general attribute transfer such as gender transfer and country-capital and is not limited to the morphological transformation.
C ONCLUSION AND F UTURE W ORK
We proposed a novel representation learning framework based on reﬂection to invert a certain attribute of a word vector.
We proposed a reﬂection-based method for word attribute transfers without relying on the explicit attribute knowledge of an input word, which is necessary for a simple analogy-based transfer.
Experimental results showed that our proposed method can transfer the word attributes if the input word has a target attribute.
If not, reﬂection does not transfer the word.
Future work includes applications to other transfer tasks: sentence by sentence transfer, such Niu et al.
; Prabhumoye et al.
; Jain et al.
(2019), and entity prediction on an analogic graph embedding space (Liu et al., 2017), in the ﬁeld of computer vision,
visual analogy
Reed et al., 2015), or style transfer (Zhu et al., 2017;
Liao et al., 2017) with GANs (Radford et al., 2016;
Goodfellow et al., 2014) because their latent space holds analogic relations (Radford et al., 2016).
A T OP T HREE A CCURACY AND S TABILITY
[Table]
[Table] [Table]
+ 36.
M EAN D IFF − 34.
D IFF (cid:88) 80.
- - - - M EAN D IFF (cid:88) 70.
[Table]
B V ISUALIZATION OF PARAMETERIZED MIRRORS
Figures below visualize PCA results of a obtained for the test words.
We normalized the L2 norm of a to 1 ( a (cid:107) a (cid:107) ).
Corresponding word pairs are connected by solid lines.
Figs.
4 and 5 suggest not only the mirror parameters of paired words are similar to each other but also the parameters with the attribute form a cluster — words with the same attribute has similar mirror parameter a .
The mirrors of paired words are close to each other in the same attribute (Fig. 4 and 5).
In the ﬁgure 5, there are some MF pairs lying far away from MF cluster.
This is presumed to be the result of ignoring principal components that have a high inﬂuence on MF due to the small number of data.
Figs. 6, 7, 8, 9, 10, 11, 12, and 13 are the detailed PCA results for four different attributes: MF, SP, CC, and
AN.
These results show that a reﬂection transfers a paired word each other by using a similar mirror.
For example, rich and poor use almost the same mirror (Fig. 9).
On the other hand, different mirrors are used for different word pairs since the mirrors are parameterized.
These results shows the effect of the mirror as described in section 4.4.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Islamabad
[Figure]
[Figure]
nearside foregroundbackground thinness thickness rightnesswrongness maximizationminimization departurearrival nationalisminternationalism immoralitymorality
[Figure]
convergencedivergence richpoor reputationdisrepute
[Figure]
contrastanalogy frown
[Figure]
purityimpurity loyaltydisloyalty
AN
[Figure]
grandfather
[Figure]
fatheractor
[Figure]
[Figure]
[Figure]
baku azerbaijan tashkent uzbekistan honduras tegucigalpa taiwan taipei angola luanda hungary budapest bangladesh dhaka islamabad pakistan iran tehran croatia zagreb caracas venezuela bangkok thailand rwanda kigali antananarivo madagascar niamey
[Figure]
bujumbura burundimanagua nicaragua ukraine kiev stockholm sweden afghanistan kabul
[Figure]
[Figure]