Reflection-based Word Attribute Transfer (2020)

Abstract: Word embeddings, which often represent such analogic relations as king - man + woman queen, can be used to change a word’s attribute, including its gender. For transferring king into queen in this analogy-based manner, we subtract a difference vector man - woman based on the knowledge that king is male. However, developing such knowledge is very costly for words and attributes. In this work, we propose a novel method for word attribute transfer based on reflection mappings without such an analogy operation. Experimental results show that our proposed method can transfer the word attributes of the given words without changing the words that do not have the target attributes.

## introduction word-embedding methods handle word semantics in natural language processing @xcite @xcite . such wordembedding models as skip-gram with negative sampling (sgns; @xcite or glove @xcite capture such analogic relations as --→ king---→ man+ -----→ woman ≈ ---→ queen. previous work @xcite @xcite @xcite offers theoretical explanation based on pointwise mutual information (pmi; @xcite for maintaining analogic relations in word vectors. these relations can be used to transfer a certain attribute of a word, such as changing king into queen by transferring its gender. this transfer can be applied to perform data augmentation; for example, rewriting he is a boy to she is a girl. it can be used to generate negative examples for natural language inference, for example. we tackle a novel a naive way for word attribute transfer is to use a difference vector based on analogic relations, such as adding -----→ woman ---→ man to --→ king to obtain ---→ queen. this requires explicit knowledge whether an input word is male or female; we have to add a difference vector to a male word and subtract it from a female word for the gender transfer. we also have to avoid changing words without gender attributes, such as is and a in the example above, since they are non-attribute words. developing such knowledge is very costly for words and attributes in practice. in this work, we propose a novel framework for a word attribute transfer based on reflection that does not require explicit knowledge of the given words in its prediction. the contribution of this work is two-fold: (1) we propose a word attribute transfer method that obtains a vector with an inverted binary attribute without explicit knowledge. (2) the proposed method demonstrates more accurate word attribute transfer for words that have target attributes than other baselines without changing the words that do not have the target attributes. ## word attribute transfer task in this task, we focus on modeling the binary attributes (e.g. male and female 1 ). let x denote a word and let v x denote its vector representation. we assume that v x is learned in advance with an embedding model, such as skip-gram. in this task, we have two inputs, word x and vector z, which represent a certain target attribute, and output word t with the inverted attribute of x for z. in this paper, z is a 300-dimensional vector embedded from a target attribute id using an embedding function of a deep learning framework. for example, given a set of attributes @xmath0, antonym}, we assign different random vectors z gender for gender and z antonym for antonym, respectively. let a denote a set of triplets (x, t, z), e.g., (man, woman, z gender ) ∈ a, and n denote a set of words without attribute z, e.g., (person, z gender ) ∈ n . this task transfers input word vector v x to target word vector v t by transfer function f z that inverts attribute z of v x : the following property must be satisfied: (1) attribute words {x|(x, t, z) ∈ a} are transferred to their counterparts and (2) non-attribute words {x|(x, z) ∈ n } are not changed (transferred back into themselves). for instance with z gender , given input word man, gender attribute transfer f z gender (v man ) should result in a vector close to v woman . given another input word person as x, the results should be v person . ## analogy-based word attribute transfer analogy is a general idea that can be used for word attribute transfer. pmi-based word embedding, such as sgns and glove, captures analogic relations, including eq. 2 @xcite @xcite . by rearranging eq. 2, eq. 3 is obtained: ## experiment we evaluated the performance of the word attribute transfer using data with four different attributes. we used 300-dimensional word2vec and glove as the pre-trained word embedding. we used four different datasets of word pairs with four binary attributes: male-female, singular-plural, capital-country, and antonym ( ## related work the theory of analogic relations in word embeddings has been widely discussed @xcite @xcite @xcite . in our work, we focus on the analogic relations in a word embedding space and propose a novel framework to obtain a word vector with inverted attributes. the style transfer task @xcite @xcite @xcite resembles ours. in style transfer, the text style of the input sentences is changed. for instance, @xcite transferred from formal to informal sentences. these style transfer tasks use sentence pairs; our word attribute transfer task uses word pairs. style transfer changes sentence styles, but our task changes the word attributes. @xcite studied morphological transformation based on character information. our work aims for more general attribute transfer, such as gender transfer and antonym, and is not limited to morphological transformation. ## conclusion this research aims to transfer word binary attributes (e.g., gender) for applications such as data augmentation of a sentence. we can transfer the word attribute with analogy of word vectors, but it requires explicit knowledge whether the input word has the attribute or not (e.g., @xmath1, @xmath2, person / ∈ gender). the proposed method transfers binary word attributes using reflectionbased mappings and keeps non-attribute words unchanged, without attribute knowledge in inference time. the experimental results showed that the proposed method outperforms analogy-based and mlp baselines in transfer accuracy for attribute words and stability for non-attribute words.