On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation (2022)

Abstract: We present a methodology that explores how sentence structure is reflected in neural representations of machine translation systems. We demonstrate our model-agnostic approach with the Transformer English-German translation model. We analyze neuron-level correlation of activations between paraphrases while discussing the methodology challenges and the need for confound analysis to isolate the effects of shallow cues. We find that similarity between activation patterns can be mostly accounted for by similarity in word choice and sentence length. Following that, we manipulate neuron activations to control the syntactic form of the output. We show this intervention to be somewhat successful, indicating that deep models capture sentence-structure distinctions, despite finding no such indication at the neuron level. To conduct our experiments, we develop a semi-automatic method to generate meaning-preserving minimal pair paraphrases (active-passive voice and adverbial clause-noun phrase) and compile a corpus of such pairs.

## introduction understanding the roles neurons play is important for the interpretability of neural machine translation (nmt) models. finding neurons that are either invariant or sensitive to particular structural distinctions may explain how such structures are encoded, and validates the robustness of translation systems, which is a challenging but important problem @xcite . furthermore, understating how these encodings are used by the network may potentially enable controlling the output by direct manipulation of neurons. previous work analyzing what aspects of sentence structure are encoded in network representations mostly took a probing approach or focused on syntactic agreements. works that compared activations did so either across models with identical input @xcite @xcite or by representation words, not sentences @xcite . our novelty lies in using the same model with paraphrased input pairs, to analyze sentence structure encoding, without probing (c.f. §7). we derive inspiration from computer vision works that analyze model behavior under non-semantic changes to the input @xcite . for our proposed approach, we provide a dataset of minimal paraphrases (along with a code to extend it). we take two phenomena as test cases: active to passive voice and an adverbial clause to a noun phrase (see to compare the activation patterns of sentences that may be comprised of a different number of tokens, we aggregate tokens representations. we then measure the correlation of neuron activations between paraphrases and provide a confound analysis. we find that the main contributors to strong correlation are similar positional encodings and bag-of-words overlap, suggesting strong correlation is derived from similar input encoding and not high-level abstractions learned by the model. the identification of these confounds may be beneficial to future work on network analysis (see §4). our findings suggest that these paraphrase distinctions are nonetheless encoded and used, as evident by our experiments of manipulating neurons ( §5). we control the structure of the translation output (e.g., active/passive) by translating neuron activations in a fixed direction. we show that this manipulation generates outputs that are more similar to the desired form with an in-depth evaluation, using bleu, dependency parsing, and manual analysis. we provide ablation studies that show that the results are not simply random artifacts of manipulation, with a non-local effect. lastly, we compare different methods for selecting subsets of neurons source paraphrased active @xmath0: examples produced by our paraphrasing engine to be manipulated ( §6) with counter-intuitive results, attributed to neurons of general importance and multiple roles per neuron. overall, the similarity between neuron activation over paraphrases is mostly explained by shallow input features: the positional and token embeddings. therefore, some neurons represent input features, but sentence-level information is not localized, even in higher layers. moreover, we show that sentence phrasing can be naïvely controlled, with a manipulation of a large number of neurons. this suggests that the distinction between different sentence structures is encoded in the model, probably in a distributed manner. lastly, the neurons most effective for such manipulations are the ones most correlated across paraphrases, not necessarily those that vary the most. ## dataset: minimal paraphrase pairs we curate datasets that isolate specific sentence structure distinctions. to achieve that, we require sentence pairs with the following attributes: existing paraphrasing tools and datasets fail to satisfy these criteria (see §7). therefore, we develop our own paraphrasing method, with which we compile two parallel sets: active voice to passive voice and an adverbial clause to a noun phrase. sentence examples can be found in the proposed process is automatic, following predefined syntactic rules while utilizing several nlp models. first, we identify sentences that match some source patterns (active voice, adverbial clause) according to a dependency parsing and pos tags model @xcite and a semantic role labeling model @xcite . then, we rephrase the sentence to the desired structure. we complement missing prepositions by choosing the one with the highest probability as predicted by bert @xcite . for example, the adverbial clause sentence "she felt accomplished when she met the investor" requires the preposition "with" in the noun phrase form "she felt accomplished during her meeting with the investor", and the temporal preposition when is replaced with during. in ambiguous instances, we choose whether or not to insert a preposition by opting for the sentence with the higher probability according to gpt2 language model @xcite . when replacing a verb with a noun (e.g., arrival is replaced with arrive), we look for the most suitable conversion in existing lexicons, including nomlex @xcite , amr's and verb forms. see appendix a for details and examples. foot_1 paraphrased valid adverbial clause to noun phrase 376 114 active voice to passive voice 3107 1169 we apply our paraphrasing engine to the wmt19 english-german development set @xcite . some results are disfluent. for example, the sentence "he took his time" is converted to "his time was taken by him", which is syntactically well-formed, but anomalous. therefore, we manually filter the data. for more details and other failed filtering approaches, see appendix a.3. the number of pairs is given in ## technical setup model. we demonstrate our model-agnostic methodology with the transformer model for machine translation @xcite . we use the fairseq implementation @xcite , which was trained on the wmt19 english-german train set @xcite . the embedding dimension is 1024 with sinusoidal positional encoding. dataset. minimal paraphrases (see §2). due to space considerations, we present results on the active/passive set in the main paper. clause/noun phrase results are in appendices §b.3 and §c. notations. we refer to trained models with a different random seed as m 1 , m 2 . we denote the set of source sentences @xmath1, ..., s n }, and its corresponding paraphrased set with @xmath2, ..., p n } (e.g., s i is an active voice sentence and p i is its passive counterpart). we follow @xcite ; @xcite and take into account only the last sub-word token for each word (results with all subword tokens were similar). we consider as neurons the 1024 activation values in the output embedding of the 6 encoder layer blocks (following @xcite foot_2 . this leads to the following definition: the activation of some neuron l in model m, on sentence s i is x m,l s [i], while x m,l s is a vector of size n (with n being the number of sentences in set s, i.e. one activation value per sentence, see §4.1). ## detecting correlation patterns to detect activation patterns under experimental conditions, we measure pearson correlation foot_3 between neural activations. while correlation analysis has been previously used to analyze neuronlevel behavior @xcite @xcite , our novelty lies with the independent variable being a property of the input (i.e., paraphrases) and not the model (e.g., architecture, initialization). ## manipulation of neurons manipulation of neurons allows us to control the output translation (without additional training) and adds a causative explanation to the role neurons play. we look into changing the activation values to force the output to have a desired syntactic structural feature (e.g., active or passive voice). although we did not observe individual neurons that have a strong positive/negative correlation across paraphrases in §4, these sentence-structure distinctions could still be encoded in a decentralized manner in the model, and therefore susceptible to manipulation. we address three main questions: ## specific neuron set selection although finding a subset of neurons that carries a specific functionality is a difficult problem (sajjad manipulating the output translation to be in active voice when feeding passive voice as input. comparing the choice of neurons to manipulate when starting from the top or bottom according to the rank given by paracorr. figure 4 : top ranked neurons have a stronger impact on the translation quality of a test set, measured in bleu. erasure of neurons from the top or bottom of the rank given by the value of correlation between paraphrases. model performance. top-ranked neurons are important for overall performance. we follow @xcite who identified important neurons by deleting them (i.e., setting activations to zero) and examining the impact on the model performance. we delete an increasing amount of neurons, according to paracorr rank. we measure bleu on a held-out set of 552 active voice sentences, and their references, extracted from the wmt19 test set. results (figure 4 ) show that top-ranked neurons have a stronger impact on the translation quality than bottom-ranked do, suggesting that paracorr partially ranks neurons by their general importance. role overlap. some of the top paracorr neurons account for lexical identity and positional information. this fact explains why they have the most impact when manipulating sentence structure. word order is essential for active-passive paraphrasing, where the subject and direct object exchange places. notably, when we tested non-paired active-passive sentences the phenomena did not repeat itself, see appendix c.6. word tokens are the building blocks for the semantic meaning of a sentence (which is the same across paraphrases), even when a bagof-word is not exclusive to a single meaning. the first evidence to support this claim is seen in §4, where most of the strong correlations in paracorr are explained by similarity in the tokens and the positional embeddings between the inputs (i.e., to-kencorr and poscorr, respectively). in an additional test, we check how many of the top paracorr neurons are also in the top poscorr and tokencorr neurons. figure 5 shows that for any count x, the set of top x paracorr neurons have an intersection with the sets of top x poscorr or tokencorr neurons. 7 related work understanding nlp neural networks. various approaches were previously proposed @xcite , each with a methodology that differs from ours. probing tasks investigate whether linguistic properties of the input text can be effectively predicted from model representations @xcite @xcite . they shed light on what information is kept within a model, but not necessarily what is used, or how (antverg and belinkov, 2021). others employ mediation analysis theory: vig et al. (2020); finlayson et al. (2021) study semantic behavior and syntactic agreement, respectively. some works analyze attention heads (voita et al., 2019) or follow attention flow (abnar and zuidema, 2020). visualization tools interpret activations, but with some exceptions (e.g., lenc and vedaldi, 2015), they are mostly limited to qualitative examples. durrani et al. (2020) interpreted individual neurons with probing-like methods for fine-grained analysis. challenge sets @xcite and adversarial examples @xcite , expose challenging cases by analyzing the nmt system's behavior, rather than representation. @xcite analyzes semantically equivalent inputs by clustering their embeddings. they improve prediction by continual training, while we manipulate translation posttraining. interpretation in other domains. some computer vision work inspired our approach. @xcite study the interaction between input transformation and its representation through the layers, while @xcite examine invariant neurons, those that are selective to high-level features but robust given semantically identical transformations. their methodologies do not fit the nlp domain since they rely on a mathematically well-defined input transformation (e.g., rotation). we propose an alternative with our paraphrases in §2. linguistic encoding in the human brain has been studied in neuroscience works. @xcite analyzed the correlation of neuroimaging where subjects are presented with sentences with subtle syntactic variations or violations, and found that well-correlated regions are considered to process syntax. fedorenko et al. (2016) presented human subjects with various inputs, which are analogous to our correlation experiments: word lists (tokencorr), meaningless grammatical sentences (poscorr), non-words lists (combination of tokencorr and poscorr), and regular sentences (paracorr). individual neurons analysis with correlation. bau et al. (2019) detected neurons that correlate across lstm models while showing these are the most important for performance. they manipulated individual neurons to control single words in the output (e.g., gender, tense), with their linguistic role identified by probing with a gmm classifier. the technique to identify neurons that behave similarly in different models was previously suggested by dalvi et al. (2019), who found neurons in lstm models to have role polysemy, aligning with our discussion in §6. later, wu et al. (2020) employed correlation to examine similarities of different transformer architectures. meftah et al. (2021) adapted correlations to quantify the impact of fine-tuning by measuring activations of neurons before and after domain adaptation. controlling active-passive voice in translation. yamagishi et al. (2016) controlled voice (active/passive) in rnn-based machine translation, from japanese to english, when an indicator was given as input. their method required additional model training, unlike ours. paraphrases. existing paraphrasing tools vary by how localized their edits are. some alter the lexical level @xcite , other alter whole phrases @xcite , some are sentence-level paraphrases @xcite , while some split source sentences into subsentences (e.g., @xcite . other than paraphrasing tools, existing datasets include the ppdb database @xcite that contains sentence paraphrases that are lexical, phrasal, or syntactic. @xcite ; @xcite include paraphrase and non-paraphrase pairs, the former with high lexical overlap, while @xcite contains multiple paraphrases of lexical diversity. none of these match our criteria for paraphrases ( §2). ## conclusion with our curated dataset, we introduced a modelagnostic methodology to detect activation patterns across paraphrases. by a meticulous confound analysis, we found that activation similarity is likely due to shallow features of sequence length or word identity, which are not exclusive to meaningpreserving variations. we emphasize how these confounds must be taken into account when attempting to detect local correlation under any ex-perimental setup. we controlled syntactic structures of generated output, which provides evidence of the ability of models to capture them. while we found the modification technique to be important for manipulation success, the selection of a subset of neurons was more challenging. future work should test additional architectures and language pairs or examine the representation significance of our paraphrase pairs in other nlp tasks.