Hierarchical Transformer for Multilingual Machine Translation (2021)

Abstract: The choice of parameter sharing strategy in multilingual machine translation models determines how optimally parameter space is used and hence, directly influences ultimate translation quality. Inspired by linguistic trees that show the degree of relatedness between different languages, the new general approach to parameter sharing in multilingual machine translation was suggested recently. The main idea is to use these expert language hierarchies as a basis for multilingual architecture: the closer two languages are, the more parameters they share. In this work, we test this idea using the Transformer architecture and show that despite the success in previous work there are problems inherent to training such hierarchical models. We demonstrate that in case of carefully chosen training strategy the hierarchical architecture can outperform bilingual models and multilingual models with full parameter sharing.

## introduction machine translation (mt) today is gradually approaching near-human quality; however, this holds true only when massive parallel corpus is available. as for the low-resource machine translation, the main way to improve it is to use additional data such as monolingual texts or parallel data in other languages. one of the ways to use the latter is to build a multilingual model: instead of training separate models for each translation direction, multiple parallel corpora can be combined to train a single model where languages can share some parameters to help each other learn. given that languages have a lot in common, a properly organized parameter sharing strategy could compensate for the lack of training examples in low-resource pairs. exploiting language relatedness can substantially improve translation quality @xcite . the question is which architectural setup allows getting the most benefit from between-language commonalities. while there are different approaches to this problem, such as full parameter sharing @xcite or shared encoder side with language-specific decoders @xcite , we find the recent approach by azpiazu and pera (2020) promising because it accounts for the degree of relatedness between languages in a multilingual model in a systematic way. linguistic trees organize languages in hierarchies by the degree of kinship, and the same approach can be applied to multilingual machine translation models. the idea is to organize both encoder and decoder in a hierarchical fashion, reflecting the degree of relatedness between languages, such that the most related languages share the largest number of parameters. figure 1 shows the outline of such model (will be explained in more detail in section 3.1). unlike azpiazu and pera (2020), who used long short term memory (lstm) recurrent neural network as a basic architecture, we implemented this idea using the state-of-the-art transformer @xcite architecture. our experiments demonstrated some persistent training problems associated with the hierarchical architecture-namely, the model is prone to early overfitting in low-resource directions leading to low translation scores. given that one of the main goals of introducing multilingual models in general and the hierarchical model in particular is improving translation accuracy for low-resource directions, this problem is critical and needs a solution. these are the main contributions of this work: 1. testing the hierarchical multilingual translation model using the transformer architecture; 2. identifying and analyzing problems related to the hierarchical nature of the model-namely, overfitting in low-resource directions; ## related work in this section, we review the existing approaches to parameter sharing in multilingual mt models to contrast them with the hierarchical approach. probably, the simplest form of parameter sharing was introduced by @xcite , where all parameters are shared and identifier tokens are used to distinguish between languages. while being easy to implement, such model may require more resources to capture relationships between languages, as no architectural hints are given. @xcite views the translation problem as a multi-task one, with a shared encoder and separate decoders for each target language. here the potential knowledge sharing between target languages is not considered. @xcite and @xcite introduce shared parameters between the encoder and decoder. this is an interesting idea that can be considered in the future. @xcite propose a one-tomany model, where parameters are partially shared between the multiple decoders. this is similar in spirit to the hierarchical approach; however, instead of having shared parts between individual decoders, the idea is to build a hierarchy of decoders. many other recent works, such as @xcite and @xcite add language-specific components/layers to the decoder side of a model. we observe that in these works languages on the encoder/decoder side of multilingual models are treated the same, regardless of the degree of their relatedness. in fact, @xcite do group languages by vocabulary overlap, nevertheless, in their model close languages may end up being in different groups. in the hierarchical model, however, the number of shared parameters directly depends on the degree of kinship between languages. ## approach using trees to model genealogical relationships between different languages has historically been a common approach in linguistics @xcite . although this is not the only model, and there exist alternatives such as wave model @xcite , tree structure naturally suits for depicting origination of languages one from another over time. this expert knowledge of relationships between languages can be utilized when building a multilingual mt model. the current trend in natural language processing is to make models learn linguistic rules and patterns on their own, without explicitly guiding them. however, in this case, when supervision does not really cost anything and language relatedness information is readily available, this expert knowledge can let models train faster and manage parameter space better. @xcite found out that different encoder levels specialize in different language as-pects: lower-level representations better capture part-of-speech tags, and higher-level ones are better at grasping semantics. another recent observation by @xcite showed that in multilingual mt models with a single shared encoder, representations of different languages cluster by language families, and as they move up the encoder, similarity between source languages increases. these facts taken together suggest that the encoder tries to find a common representation of different languages that initially cluster together by language families. as they move up the encoder, a model finds intermediate representations that smooth out the dissimilarities on different levels -morphological, syntactic, semantic. this closely resembles the structure of phylogenetic trees, where connections on the bottom level mean the greatest linguistic similarity between languages, and connections on the highest level of a hierarchy indicate that languages are far from each other. see the example of such tree in figure 2 . if we organize the architecture of a multilingual mt model according to language relationships in a phylogenetic tree, this will allow sharing parameters between languages on appropriate levels. for example, we can take two very close languages, whose vocabularies overlap significantly. since these languages are highly related, we assume that it will not take long to reduce them to the same representation. so, we "combine" them early on, i.e., introduce shared parameters for these languages in the first layers of the encoder. the third language, let us suppose, comes from the same family, but many base words are different. still, sentence structure remains the same as in the first two languages. therefore, we combine this language with the first two in the later stages (parameter sharing starts deeper in encoder layers), and so on. this parameter sharing strategy allows the economic utilization of parameter space and has the potential to lead to better, i.e., source-languageindependent representations of different languages on the output level of the encoder. ## experiments to test the hierarchical approach to parameter sharing we experimented with several hierarchical setups in multilingual models. unlike azpiazu and pera (2020) who only test complex multi-source multi-target models, we decided to start with simple setups when the hierarchy is just on the one side of the model (encoder or decoder) and then increase the complexity to the general case. we tested the hierarchical model's performance in three different cases: for each of these cases we trained two hierarchical models using different language sets. further, when we identified specific training problems of hierarchical models (described in section 4.3.2), we suggested and applied two variations to each hierarchical model. each hierarchical model is compared to two baselines: bilingual models and multilingual model with full parameter sharing (will be also called full sharing model later). comparing to bilingual models trained on the same parallel corpora helps to understand whether languages can learn from each other in the hierarchical model. whereas contrasting hierarchical models to full sharing models, with single shared encoder and decoder for all languages, as in @xcite , answers the question about the use of explicitly introducing the hierarchy into the model. in every multilingual model we trained there are low-resource and high-resource directions, and we explored whether low-resource ones can learn from high-resource ones, and which model structure better fits for this purpose. ## conclusion this work tested a new hierarchical approach in multilingual mt. we implemented the hierarchical model based on the transformer architecture and contrasted it to bilingual and full parameter sharing baselines. the straightforward approach to training turned out to be hampering the hierarchical model's performance. we found out that there is a problem of overfitting which is specific to training hierarchical models. regularizing low-resource directions solved the problem, substantially improving the model's performance. we showed that the hierarchical model greatly improves low-resource pairs' scores, however, at the expense of high-resource pairs. the comparison with the full sharing model provided positive evidence supporting the assumption about the usefulness of explicitly defining parameter sharing strategy in a multilingual model. overall, the hierarchical approach looks promising and hence should be further explored and developed. in the future, we would like to analyze how exactly learning happens in hierarchical models and whether hierarchical architecture is indeed capable of capturing different language aspects (morphological, syntactic, semantic) on different levels in a hierarchical manner.