A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders (2021)

Abstract: Powerful sentence encoders trained for multiple languages are on the rise. These systems are capable of embedding a wide range of linguistic properties into vector representations. While explicit probing tasks can be used to verify the presence of specific linguistic properties, it is unclear whether the vector representations can be manipulated to indirectly steer such properties. For efficient learning, we investigate the use of a geometric mapping in embedding space to transform linguistic properties, without any tuning of the pre-trained sentence encoder or decoder. We validate our approach on three linguistic properties using a pre-trained multilingual autoencoder and analyze the results in both monolingual and cross-lingual settings.

## introduction recently, the design of sentence encoders, monolingual @xcite and multilingual @xcite has enjoyed a lot of attention. many works have used probing tasks to investigate the presence of specific linguistic properties in sentence representations @xcite @xcite @xcite . however, it remains unclear to what extent these linguistic properties can be actually steered by manipulating the representations. by analogy to the definition of style-transfer from @xcite , we refer to modifying a particular linguistic property in a given text (e.g., a sentence's tense) while preserving all of the property-independent content as linguistic property transfer. training dedicated models to transfer linguistic properties requires substantial computational effort and a lot of training data. adding the ability to transform a new property may require an entire retraining of the text encoder and decoder. this is especially challenging for low-resource languages or when reusing or building transfer models for more than one language. assuming that pre-trained autoencoders capture the linguistic properties of interest, we investigate (i) whether they can be used without further tuning to efficiently transfer the properties, and (ii) whether this extends to the cross-lingual setting, when based on a multilingual pre-trained autoencoder. our starting point is a pre-trained sentence encoder, with a corresponding decoder trained on an autoencoder objective. we show how a geometric transformation of pre-trained multilingual sentence embeddings can be efficiently learned on cpu for transferring specific linguistic properties. we also experiment with cross-lingual linguistic property transfer, using a language-agnostic pretrained encoder. in summary, this paper presents a set of preliminary experiments on linguistic property transfer, and shows that there may be value in further research on manipulating distributed representations to efficiently tackle language generation tasks. ## related work linguistic properties usually denote the grammatical behavior of linguistic units in sentences. this contrasts with styles which concern semantic aspects of sentences such as sentiment and gender. nevertheless, transferring linguistic properties can be situated in the broader style transfer setting. style transfer systems can be categorized into (i) methods that learn disentangled representations, in which the content is explicitly separated from the style, making the style aspect controllable and interpretable @xcite @xcite @xcite and (ii) methods that learn entangled representations in which the content and style are not explicitly separated @xcite @xcite 2019; @xcite . our approach falls under the entangled methods because encoder-decoder systems trained on an autoencoding objective yield representations in which there is no explicit separation between content and style. conceptually, our method is most similar to @xcite , but differs in (i) that it can use any existing pre-trained autoencoder as opposed to training an autoencoder from scratch on a variational objective, (ii) that a simple geometric transformation is applied on the representations instead of training a computational heavy neural transformation network, and (iii) that it generalizes to the cross-lingual setting. ## linguistic property transfer our system consists of three components: (1) a pretrained multilingual autoencoder, (2) linear classifiers for the targeted linguistic properties and (3) a component that geometrically transforms sentence embeddings to transfer the selected properties in the dense sentence representation space. these components are presented schematically in fig. 1 . we start from a pre-trained autoencoder (fig. 1a ) that consists of an encoder (enc : @xmath0, and a decoder (dec : r @xmath1. the second component (fig. 1b ) is a linear classifier c : r @xmath2. we will limit our experiments to binary properties, i.e., @xmath3, 1}. finally, the last component (fig. 1c ), performs a geometric transformation. it allows flipping the value of the selected linguistic property by projecting the original encoding z into the opposite half-space with respect to the property classifier, over an estimated distance λ. this leads to the transferred encoding z , designed to be decoded into a sentence x close to the original sentence, but with the transformed target property. the three components shown in fig. 1 are further described below. ## experiments to investigate whether linguistic properties embedded in representations of pre-trained encoders can be transferred without finetuning, we first apply the senteval tool from @xcite to laser-embeddings (section 3.1) and identify three properties that have a strong presence. we then investigate how well our approach performs on these properties in the monolingual setting (ml), in which our cmab model is both trained and evaluated on english sentences (q1). finally, we investigate the performance of our approach in the crosslingual setting (cl), in which the model is trained on english but evaluated on dutch sentences. in particular, after training on english, dutch sentences are passed into the laser encoder to obtain the transformed encodings z which in turn are decoded by the dutch decoder (q2). ## conclusion and future work we have introduced a simple and efficient geometric method to transfer linguistic properties which has been evaluated on three properties in both monolingual and cross-lingual settings. while there is room for improvement, our preliminary results indicate that it can allow pre-trained autoencoders to transfer linguistic properties without additional tuning, such that there is no need to train dedicated transfer systems. this potentially makes learning faster and better scalable than with