Understanding Pure Character-Based Neural Machine Translation: The Case of Translating F innish into E nglish (2020)

Abstract: Recent work has shown that deeper character-based neural machine translation (NMT) models can outperform subword-based models. However, it is still unclear what makes deeper character-based models successful. In this paper, we conduct an investigation into pure character-based models in the case of translating Finnish into English, including exploring the ability to learn word senses and morphological inflections and the attention mechanism. We demonstrate that word-level information is distributed over the entire character sequence rather than over a single character, and characters at different positions play different roles in learning linguistic knowledge. In addition, character-based models need more layers to encode word senses which explains why only deeper models outperform subword-based models. The attention distribution pattern shows that separators attract a lot of attention and we explore a sparse word-level attention to enforce character hidden states to capture the full word-level information. Experimental results show that the word-level attention with a single head results in 1.2 BLEU points drop.

## introduction neural machine translation (nmt) has boosted machine translation significantly in recent years @xcite @xcite @xcite @xcite . however, it is still unclear how nmt models work due to the black-box nature of neural networks. better understandings of nmt models could guide us in improving nmt systems. currently most of the studies towards understanding nmt models only take into account subword-based (e.g. bpe-based) models. deeper character-based (char) models have been shown to perform better than bpe-based models @xcite . in this paper, we try to investigate the working mechanism of char models. we explore the ability of char models to learn word senses and morphological inflections and the attention mechanism. previous studies have tried to interpret and understand nmt models by interpreting attention weights @xcite @xcite , using gradients @xcite , applying layer-wise relevance propagation @xcite , probing classification tasks @xcite @xcite @xcite , and more intrinsic analysis @xcite . however, only @xcite have probed character-based representations. @xcite have only explored character-aware word-level representations, while we investigate fully character-level representations, which are also studied in @xcite . we apply more composition methods to explore how char models learn linguistic knowledge and how attention extracts features directly from characters. probing classification tasks @xcite have emerged as a popular method to interpret the internal representations from neural networks. given a probing classifier, the input is usually the representation of a word and the output is the corresponding linguistic tag. char models pose new challenges for interpretability, and we investigate whether we can probe char models in a way similar to (sub)word-based models. in addition, can we extract word sense and morphological information about the full word from individual hidden states, or is this information distributed across multiple states? this has implications for interpreting neural char models, but can also inform novel architectures, such as sparse attention mechanisms. thus we first investigate the ability of char models to learn word senses and morphology in section 3. we apply different methods to compose information from characters and demonstrate that the word-level information is distributed over all the characters but characters at different positions play different roles in learning linguistic knowledge. we also explore the effect of encoder depth to answer why char models outperform bpe-based models only when they have the settings with deeper encoder. the probing results show that char models need more layers to learn word senses. then in section 4, we move on to explore the attention mechanism. the distribution pattern shows that separators attract much more attention compared to other characters. to study the effect of enforcing characters to capture the full word-level information, we investigate a sparse attention mechanism, i.e. a model that only attends to separators, which can be viewed as a word-level attention. the bleu score drops 1.2 points when we apply the word-level sparse attention. this implies that only attending to separators by a single attention head is workable but not enough to extract all the necessary information. the main findings are summarized as follows: ## experiments as rnn-based/char models can in principle achieve state-of-the-art performance in nmt @xcite and most of analysis of nmt models are based on bpe-based models, we are interested in analyzing the working mechanisms of pure rnn-based char models. 1 we follow @xcite in using rnn-based models, and we focus on @xmath0, because training char models requires huge computational resources. we first train char models with different encoder depths and a bpe-based model for comparison. then we explore how char models learn word senses and morphological inflections via probing classification tasks, using representations generated by the trained models. for the morphological probing tasks, the classifiers predict the morphological tag given the representation of a token. for the word sense disambiguation (wsd) probing task where learning word senses is needed, the input to the classifier and the output are different from the classifiers in the morphological probing tasks. instead, the representations of an ambiguous word and its candidate translation are both fed into the classifier and then the classifier predicts whether the candidate translation is correct or not. ## learning linguistics in this section, we interpret the ability of char models to encode word senses and morphological inflections, by exploring the effect of separators, composition methods, the evolution over layers, and the robustness to domain-mismatch. we only analyze the encoder hidden states in the context of probing tasks. as the attention extracts features from encoder hidden states in a different way from our composition methods -pooling and randlstm, the findings may not apply to the entire char model. ## attention mechanisms in the encoder-decoder attention @xcite , a higher attention weight means that the source token contributes more to the prediction at current step. thus, we could utilize the attention distributions to explore how char models pay attention to the source characters during translation. char models encode information into a longer sequence which essentially increases the representational capacity of the encoder. we are interested in exploring the effect of restricting the capacity of the encoder states that are passed to the decoder. thus, we apply the word-level attention which attends to a character of each word and potentially enforces the character captures the full word information. ## conclusion char models have been shown to perform better than bpe-based models in nmt yet they pose new challenges for interpretability. in this paper, we investigate char models via the wsd and six morphological probing tasks to learn how char models learn word senses and morphology, in the case of translating finnish into english. we also explore the attention distribution pattern and a sparse word-level attention to learn the working mechanism of attention. in the probing tasks, we find that separators also have captured some linguistic knowledge. we apply different composition methods to the characters of a word, and we demonstrate that the word sense and morphological information is distributed over all the characters rather than some specific characters. moreover, characters at different positions play different roles in learning linguistic knowledge. char models are better at learning morphology but we need a more complicated composition method, such as a randomly initialized lstm, to extract all the encoded information. these results on probing tasks show that we can extract word sense information and morphological features from character-level hidden states and that these features are encoded in different ways. in addition, we explore the effect of encoder depth and show that char models require more layers to encode word senses, which explains why only deeper char models outperform bpe-based models. the attention distribution shows that separators attract a lot of attention, and we show that the sparse word-level attention only attending to separators is workable but not enough for translation. as we have shown that characters at different positions specialize in learning word senses and morphology, it will be interesting to explore sparse attention with multiple heads in the future which could learn to extract features from different aspects.