A L ATENT M ORPHOLOGY M ODEL FOR O PENV OCABULARY N EURAL M ACHINE T RANSLATION
A
BSTRACT
Translation into morphologically-rich languages challenges neural machine
translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic.
This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters.
The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task.
The latter learns directly from translation data but requires rather deep architectures.
In this paper we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inﬂection.
Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms.
Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings
I NTRODUCTION
Neural machine translation (NMT) models are conventionally trained by maximizing the likelihood of generating the target side of a bilingual parallel corpus of observations one word at a time conditioned of their full observed context.
NMT models must therefore learn distributed representations that accurately predict word forms in very diverse contexts, a process that is highly demanding in terms of training data as well as the network capacity.
Under conditions of lexical sparsity, which includes both the case of unknown words and the case of known words occurring in surprising contexts, the model is likely to struggle.
Such adverse conditions are typical of translation involving morphologically-rich languages, where any single root may lead to exponentially many different surface realizations depending on its syntactic context.
Such highly productive processes of word formation lead to many word forms being rarely observed or ever observed with a particular set of morphosyntactic attributes.
The standard approach to overcome this limitation is to pre-process words into subword units that are shared among words, which are, in principle, more reliable as they are observed more frequently in varying context (Sennrich et al., 2016; Wu et al., 2016).
One drawback related to this approach, however, is that the estimation of the subword vocabulary relies on word segmentation methods optimized using corpus-dependent statistics, disregarding any linguistic notion of morphology and the translation objective.
This often produces subword units that are semantically ambiguous as they might be used in far too many lexical and syntactic contexts (Ataman et al., 2017).
Moreover, in this approach, a word form is then generated by prediction of multiple subword units, which makes generalizing to unseen word forms more difﬁcult—consider the case of a subword unit necessary to reconstruct a given word form being unlikely in a given context.
To alleviate the sub-optimal effects of using explicit segmentation and generalize better to new morphological forms, recent studies explored the idea of extending NMT to model translation directly at the level of characters (Kreutzer & Sokolov, 2018; Cherry et al., 2018), which, in turn, have demonstrated the requirement of using comparably deeper networks, as the network would then need to learn longer distance grammatical dependencies (Sennrich, 2017).
In this paper, we explore the beneﬁts of explicitly modeling variation in surface form of words using techniques from deep latent variable modeling in order to improve translation accuracy for lowresource and morphologically-rich languages.
Latent variable models allow us to inject inductive biases relevant to the task, in our case, word formation in translation.
We design a hierarchical latent variable model to mimick the process of morphological inﬂection.
Our model translates words one character at a time based on word representations learned compositionally from sub-lexical components.
In particular, for each word, it generates two latent representations: i) a continuousspace dense vector aimed at capturing the lexical semantics of the word in a given context, and ii) a set of (approximately) discrete features aimed at capturing that word’s morphosyntatic role in the sentence.
We then see inﬂection as decoding a word form, one character at a time, from a learned composition of these two representations.
These latent morphological features are more likely shared across contexts and word forms, and thus facilitate generalization under sparse settings.
We evaluate our method in translating English into three morphologically-rich languages each with a distinct morphological typology: Arabic, Czech and Turkish, and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT.
N EURAL M ACHINE T RANSLATION
In this paper, we use recurrent NMT architectures based on the model developed by Bahdanau et al.
The model essentially estimates the conditional probability of translating a source sequence x = (cid:104) x 1 , x 2 , . . .
x m (cid:105) into a target sequence y =
(cid:104)
y 1 , y 2 , . .
y l (cid:105) via an exact factorization:
[Equation]
where y <
i stands for the sequence preceding the i th target word.
At each step of the sequence, a ﬁxed neural network architecture maps its inputs, the source sentence and the target preﬁx, to the probability of the i th target word observation in context.
In order to condition on the source sentence fully, this network employs an embedding layer and a bi-directional recurrent neural network (bi-RNN) based encoder.
Conditioning on the target preﬁx y <
i is implemented using a recurrent neural network (RNN) based decoder, and an attention mechanism which summarises the source sentence into a context vector c
i as a function of a given preﬁx (Luong et al., 2015).
Given a parallel training set D , the parameters θ of the network are estimated to attain a local minimum of the negative log-likelihood function L ( θ |D )
=
− (cid:80) x,y ∼D log p ( y | x, θ ) via stochastic gradient-based optimization (Bottou & Cun, 2004).
Atomic parameterization estimates the probability of generating each target word y i in a single shot:
[Equation]
where E ∈ R v × d is the target embedding matrix and the decoder output h i ∈
R d represents x and y <i .
Clearly, the size v of the target vocabulary plays an important role in determining the complexity of the model, which creates an important bottleneck when translating into low-resource and morphologically-rich languages due to the sparsity in the lexical distribution.
Recent studies approached this problem by performing NMT with subword units, a popular one of which is based on the Byte-Pair Encoding algorithm (BPE; Sennrich et al., 2016), which ﬁnds the optimal description of a corpus vocabulary by iteratively merging the most frequent character sequences.
Atomic parameterization could also be used to model translation at the level of characters, which is found to be advantageous in generalizing to morphological variations (Cherry et al., 2018).
Hierarchical paramaterization further factorizes the probability of a target word in context:
[Equation]
where the i th word y i = (cid:104) y i, 1 , . . .
, y i,l i (cid:105) is seen as a sequence of l i characters.
Generation follows one character at a time, each with probability computed by a ﬁxed neural network architecture with varying inputs, namely, the source sentence x , the target preﬁx y <
i , and the preﬁx y i,<j of characters already generated for that word.
In this case there are two recurrent cells, one updated at the boundary of each token, much like in the standard case, and another updated at the character level.
Luong & Manning (2016) propose hierarchical parameterization to compute the probability p ( y i | x, y <
i , θ ) for unknown words, while for known words they use the atomic parameterization.
In this paper, we use the hierarchical parameterization method for generating all target words, where we also augment the input embedding layer with a character-level bi-RNN, which computes each word representation
y i as a composition of the embeddings of their characters (Ling et al., 2015).
A L ATENT M ORPHOLOGY M ODEL FOR L EARNING W ORD R EPRESENTATIONS
Studies analyzing the advantage of using a hierarchical structure in related tasks such as language modeling (Vania & Lopez, 2017) or semantic role labeling (Sahin & Steedman, 2018) have shown that representations of words learned compositionally via bi-RNNs from character units contain many cues about their morphological features, since word representations are encoded through a function which can establish a mapping between combinations of orthographic units and lexical context.
On the other hand, the quality of learned representations depends on the amount of observations (Sahin & Steedman, 2018), as the training data is essential in properly modeling the lexical distribution.
On the other hand, the process of word formation, particularly morphological inﬂection, has many properties that remain universal across languages, where a word is typically composed of a lemma , representing its lexical semantics, and a distinct combination of categorical inﬂectional features expressing the word’s syntactic role in the phrase or sentence.
In this paper, we propose to manipulate this universal structure in order to enforce an inductive bias on the prior distribution of words and allow the hierarchical parameterization model in properly learning lexical representations.
3. G ENERATIVE MODEL
Our generative latent morphology model for NMT formulates word formation in terms of a stochastic process, where each word is generated one character at a time by composing two latent representations: a continuous vector aimed at capturing the lemma, and a set of sparse features aimed at capturing the inﬂectional features.
The motivation for using a stochastic model is twofold.
First, deterministic models are by deﬁnition unimodal : when presented with the same input (the same context) they always produce the same output.
When we model the word formation process, it is reasonable to expect a larger degree of ambiguity, that is, for the same context ( e.g. a noun preﬁx), we may continue by inﬂecting the word differently depending on the (latent) mode of operation we are at ( e.g. generating nominative, accusative or dative noun).
Second, in stochastic models, the choice of distribution gives us a mechanism to favour a particular type of representation.
In our case, we use sparse distributions for inﬂectional features to accommodate the fact that morphosyntactic features are discrete in nature.
Our latent variable model is an instance of a variational auto-encoder (VAE; Kingma & Welling, 2013) inspired by the model of Zhou & Neubig (2017) for morphological reinﬂection.
Generation of the i th word starts by sampling a Gaussian-distributed representation in context.
This requires predicting the Gaussian location u i and scale s i vectors,
[Equation]
where prediction of the location (in R d ) and scale (in R d> 0 ) from the word-level decoder hidden state h
i (which represents x and y <i ) is performed by two dense layers, and the scale values are ensured to be positive with the softplus ( ζ ) activation.
Generation proceeds by then sampling a K -dimensional vector
f
i of sparse scalar features (see §3.2) conditioned on the source x , the target preﬁx y <
i , and the sampled lemma z i .
We model sampling of f i conditioned on z i in order to capture the insight that inﬂectional transformations typically depend on the category of a lemma.
Having sampled f i and z i , the representation of the i th target word is computed by a transformation of z i and
f i , i.e. t i =
dense
([ z i , f i ]; θ comp ) .
As shown in Figure 1, our model generates each word character by character auto-regressively by conditioning on the word representation t i predicted by the latent morphology model,
the current context c i , and the previously generated characters following the hierarchical parameterization.
See Algorithm 1 for details on generation.
Input: model parameters θ , latent lemma z i , latent morphological attributes f i , observed character sequence (cid:104)
y i, 1 , . . .
, y i,l i (cid:105) if training or placeholders if test, decoder state h i , and context vector c
i Result: updated decoder hidden state, prediction (a word), probability of prediction (for loss) initialization; t i =
dense
([ z i , f i ]; θ comp ) ; initialize char-rnn with a projection of [ t i , c i ] ; for j < l i and j
< max do compute
; if then set else end prediction; endAlgorithm 1: Word generation: in training the word is observed, thus we only update the decoder and assess the probability of the observation, in test, we use mean values of the distributions to represent most likely values for z and f and populate predictions with beam-search.
3. S PARSE FEATURES
Since each target word y i may have multiple inﬂectional features, ideally, we would like f i to be
K feature indicators, which could be achieved by sampling from K independent Bernoulli distributions parameterized in context.
The problem with this approach is that sampling Bernoulli outcomes is non-differentiable, thus, their training requires gradient estimation via REINFORCE (Williams, 1992) and sophisticated variance reduction techniques.
An alternative approach that has recently become popular is to use relaxations such as the Concrete distribution or Gumbel-Softmax (Maddison et al., 2017; Jang et al., 2017) in combination with the straight-through estimator (ST; Bengio et al., 2013).
This is based on the idea of relaxing the discrete variable from taking on samples in the discrete set { 0 , 1 } to taking on samples in the continuous set (0 , 1) using a distribution for which a reparameterization exists ( e.g. Gumbel).
Then, a non-differentiable activation ( e.g. a threshold function) maps continuous outcomes to discrete ones.
ST simply ignores the discontinuous activation in the backward pass, i.e. it assumes the Jacobian is the identity matrix.
This does lead to biased estimates of the gradient of the loss, which is in conﬂict with the requirements behind stochastic optimization (Robbins & Monro, 1951).
An alternative presented by Louizos et al.
(2018) achieves a different compromise, it gets rid of bias at the cost of mixing both sparse and dense outcomes.
The idea is to obtain a continuous sample c ∈ (0 , 1) from a distribution for which a reparameterization exists and stretch it to a continuous support ( l, r ) ⊃ (0 , 1) using a simple linear transformation s = l + ( r − l ) c .
A rectiﬁer is then employed to map the negative outcomes to 0 and the positive outcomes larger than one to 1 , i.e. f = min(1 , max(0 , s )) .
The rectiﬁer is only non-differentiable at s = 0 and at s = 1 , however, because the stretched variable s is sampled from a continuous distribution, the chance of sampling s = 0 and s = 1 is essentially 0 .
This stretched-and-rectiﬁed distribution allows: i) the sampling procedure to become differentiable with respect to the parameters of the distribution, ii) to sample sparse outcomes with an unbiased estimator, and iii) to calculate the probability of sampling f =and f = 1 in closed form as a function of the parameters of the underlying distribution, which corresponds to the probability of sampling s < 0 and s > 1 , respectively.
In their paper, Louizos et al.
(2018) used the BinaryConcrete (or Gumbel-Sigmoid) as the underlying continuous distribution, the sparsity of which is controlled via a temperature parameter.
However, in our study, we found this parameter difﬁcult to predict, since it is very hard to allow a neural network to control its value without unstable gradient updates.
Instead, we opt for a slight variant by Bastings et al.
based on the Kumaraswamy distribution (Kumaraswamy, 1980), a twoparameters distribution that closely resembles a Beta distribution and is sparse whenever its (strictly positive) parameters are between 0 and 1 .
In the context of text classiﬁcation, Bastings et al.
(2019) shows this stretch-and-rectify technique to work better than methods based on REINFORCE.
For each token y i , we sample K independent Kumaraswamy variables in context,
[Equation] [Equation]
which makes a continuous random vector c i in the support (0 , 1) K .
We then stretch-and-rectify the samples via f i,k = min(1 , max(0 , l − ( r − l ) c
i,k )) making f i a random vector in the support [0 , 1] K .
The probability that f i,k is exactly 0 is
[Equation]
and the probability that f i,k is exactly 1 is
[Equation]
and therefore the complement
[Equation]
We use l = − 0 .
1 and r = 1 . 1 .
Figure 2 in the appendix illustrates different instances of this distribution.
is the probability that f i,k be any continuous value in the open set (0 , 1) .
In §3.4, we will derive regularizers based on π (0 , 1) i,k to promote sparse outcomes to be sampled with large probability.
3. P ARAMETER ESTIMATION
Parameter estimation of neural network models is typically done via maximum-likelihood estimation (MLE), where we approach a local minimum of the negative log-likelihood function via stochastic gradient descent with gradient computation automated by the back-propagation algorithm.
Using the following shorthand notation:
[Equation]
The log-likelihood for a single data point can be formulated as:
[Equation]
the computation of which is intractable.
Instead, we resort to variational inference (VI; Jordan et al., 1999), where we optimize a lower-bound on the log-likelihood
[Equation]
expressed with respect to an independently parameterized posterior approximation q ( z, f | x, y, λ ) .
For as long as sampling from the posterior is tractable and can be performed via a reparameterization, we can rely on stochastic gradient-based optimization.
In order to have a compact parameterization, we choose
[Equation]
This simpliﬁes the lowerbound, which then takes the form of l nested expectations, the i th of which is E α
( z i ) β ( f i ) [log γ ( y i )] .
This is similar to the stochastic decoder of Schulz et al.
(2018), though our approximate posterior is in fact, also our parameterized prior.
6 Although this objective does not particularly promote sparsity, we employ sparsity-inducing regularization techniques that will be discussed in the next section.
Concretely, for a given source sentence x , target preﬁx y <
i , and a latent sample z ≤ i , f ≤ i , we obtain a single-sample estimate of the loss by computing L i ( θ )
= − log γ ( y i ) .
3. R EGULARIZATION
In order to promote sparse distributions for the inﬂectional features, we apply a regularizer inspired by expected L 0 regularization (Louizos et al., 2018).
Whereas L 0 is a penalty based on the number of non-zero outcomes, we design a penalty based on the expected number of continuous outcomes , which corresponds to π (0 , 1) i,k as shown in Equation (6).
For a given source sentence x , target preﬁx y <
i , and a latent sample z <i , f <i , we aggregate this penalty for each feature
[Equation]
and add it to the cost function with a positive weight ρ .
The ﬁnal loss of the NMT model is
[Equation]
3. P REDICTIONS
In our model, obtaining the conditional likelihood for predicting the most likely hypothesis requires marginalisation of the latent variables, which is intractable.
An alternative approach is to heuristically search through the joint distribution,
[Equation]
rather than the marginal, an approximation that has been referred to as Viterbi decoding (Smith, 2011).
During beam search, we populate the beam with alternative target words, and for each preﬁx y <i in the beam, we resort to deterministically choosing the latent variables based on a single sample which we deem representative of their distributions, which is a common heuristic in VAEs for translation (Zhang et al., 2016; Schulz et al., 2018).
For unimodal distributions, such as the
Gaussian p ( z i | x, y <
i , z <i , f <i ) , we use the analytic mean, whereas for multimodal distributions, such as the Hard Kumaraswamy p ( f i | x, y <i , z ≤ i , f <i ) , we use the argmax.
E VALUATION 4. M ODELS
We evaluate our model by comparing it in machine translation against three baselines which constitute the conventional open-vocabulary NMT methods, including architectures using atomic parameterization either with subword units segmented with BPE (Sennrich et al., 2016) or characters, and the hierarchical parameterization method employed for generating all words in the output.
We implement all architectures using Pytorch (Paszke et al., 2017) within the OpenNMT-py framework (Klein et al., 2017) 8 .
4. D ATA AND L ANGUAGES
In order to evaluate our model we design two sets of experiments.
The experiments in §4.4.1 aim to evaluate different methods under low-resource settings, for languages with different morphological typology.
We model the machine translation task from English into three languages with distinct morphological characteristics:
Arabic ( templatic ), Czech ( fusional ), and Turkish ( agglutinative ).
We use the TED Talks corpora (Cettolo, 2012) for training the NMT models for these experiments.
In §4.4.3, we conduct more experiments in Turkish to demonstrate the case of increased data sparsity using multi-domain training corpora, where we extend the training set using corpora from EU Bookshop (Skadin¸ˇs et al., 2014), Global Voices, Gnome, Tatoeba, Ubuntu (Tiedemann, 2012), KDE (Tiedemann, 2009), Open Subtitles (Lison & Tiedemann, 2016) and SETIMES (Tyers & Alperen, 2010) 9 .
The statistical characteristics of the training sets are given in Tables 4 and 5.
We use the ofﬁcial evaluation sets of the IWSLT 10 for validating and testing the accuracy of the models.
In order to increase the number of unknown and rare words in the evaluation sets we measure accuracy on large test sets combining evaluation sets from many years (Table 6 presents the evaluation sets used for development and testing).
The accuracy of each model output is measured using BLEU (Papineni et al., 2002) and chrF3 (Popovi´c, 2015) metrics, whereas the signiﬁcance of the improvements are computed using bootstrap hypothesis testing (Clark et al., 2011).
In order to measure the accuracy in predicting the correct syntactic description of the references, we also compute BLEU scores over the output sentences segmented with morphological analyzer.
We use the AlKhalil Morphosys (Boudchiche et al., 2017) for segmenting Arabic, Morphidata (Strakov´a et al., 2014) for segmenting
Czech and the morphological lexicon model of Oﬂazer (Oﬂazer & Kuru¨oz, 1994) and disambiguation tool of Sak (Sak et al., 2007) for segmenting Turkish sentences into sequences of lemmas and morphological features.
4. T RAINING S ETTINGS
All models are implemented using gated recurrent units (GRU) (Cho et al., 2014), and have a singlelayer bi-RNN encoder.
The source sides of the data used for training all NMT models, and the target sides of the data used in training the subword-level NMT models are segmented using BPE with 16,000 merge rules.
We implement all decoders using a comparable number of GRU parameters, including 3-layer stacked-GRU subword and character-level decoders, where the attention is computed after the 1 st layer (Barone et al., 2017) and a 3-layer hierarchical decoder which implements the attention mechanism after the 2 nd layer.
All models use an embedding dimension and GRU size of 512.
The latent morphology model (LMM) uses the same hierarchical GRU architecture, where the middle layer is augmented using 4 multi-layer perceptrons with 256 hidden units.
We use a lemma vector dimension of 150, 10 inﬂectional features (See §A.3 for experiments conducted to tune the feature dimensions) and set the regularization constant to ρ = 0 .
All models are trained using the Adam optimizer (Kinga & Ba, 2014) with a batch size of 100, dropout rate of 0.2, learning rate of 0.0004 and learning rate decay of 0.8, applied when the perplexity does not decrease at a given epoch.
11 Translations are generated with beam search with a beam size of 5, where the hierarchical models implement the hierarchical beam search algorithm (Ataman et al., 2019).
4. R ESULTS 4.4. T HE E FFECT OF M ORPHOLOGICAL T YPOLOGY
The experiment results given in Table 1 show the performance of each model in translating English into Arabic, Czech and Turkish.
In Turkish, the most sparse target language in our benchmark with rich agglutinative morphology, using character-based decoding shows to be more advantageous compared to the subword-level and hierarchical models, suggesting that reduced granularity in the vocabulary units might aid in better predicting words under conditions of high data sparsity.
In Arabic, on the other hand, using a hierarchical decoding model shows to be advantageous compared to the subword and character-level models, as it might be useful in better learning syntactic dependencies.
Using the latent morphology model provides improvements of 0.51 and 0.30 BLEU points in Arabic and Turkish over the best performing baselines, respectively.
The fact that our model can efﬁciently work in both Arabic and Turkish conﬁrms that it can handle the generation of both concatenative and non-concatenative morphological transformations.
The results in the English-to-Czech translation direction do not indicate a speciﬁc advantage of using either method for generating fusional morphology, where morphemes are already optimized at the surface level, although our model is still able to achieve translation accuracy comparable to the character and subword-level models.
4.4. P REDICTING U NSEEN W ORDS
In addition to the general machine translation evaluation using automatic metrics, we perform a more focused statistical analysis to illustrate the performance of different methods in predicting unseen words by computing the average perplexity per character on the input sentences which contain outof-vocabulary words as suggested by Cotterell et al.
We also analyze the outputs generated by each decoder in terms of the frequency of unknown words in each model output and the KullbackLeibler (KL) divergence between the character trigram distributions of the references and outputs, which represents the coherence between the statistical distribution learned by each model and the reference translations.
Our analysis results generally conﬁrm the advantage of increased granularity during the generation of unseen words, where the character-level decoder can generate a higher rate of unseen word forms and higher KL-divergence with the reference, suggesting superior ability in generalizing to new
[Table]
output and not necessarily copying previous observations as the subword-level model, however, this advantage is more visible in Turkish and less in Czech or Arabic.
The hierarchical decoder which performs the search at the level of words, on the other hand, behaves with less uncertainty in terms of the perplexity values although it cannot demonstrate the ability to generalize to new forms and neither can closely capture the actual distribution in the target language.
Due to its stochastic nature
[Table]
our model yields higher perplexity values compared to the hierarchical model, whereas the values range between subword and character-based models, possibly ﬁnding an optimal level of granularity between the two solutions.
The KL-divergence and out-of-vocabulary word rates conﬁrm that our model has the potential in better generalize to new word forms as well as different morphological typology
[Table]
4.4. T HE E FFECT OF D ATA S IZE
Repeating the experiments in the English-to-Turkish translation direction by increasing the amount of training data with multi-domain corpora demonstrates a more challenging case, where there is a greater possibility of observing rare words, either in the form of morphological inﬂections due to previously unobserved syntactic conditions, or a larger vocabulary extended with terminology from different domains.
In this experiment, the character-level model experiences a drop in performance and its accuracy is much lower than the subword-level one, suggesting that its capacity cannot cope with the increased amount of training data.
Empirical results suggest that with increased capacity, character-level models carry the potential to reach comparable performance to subword-level models (Cherry et al., 2018).
On the other hand, our model reaches a much larger improvement of 0.
BLEU points over the subword-level and 2.54 BLEU points over the character-level decoders, suggesting that it could make use of the increased amount of observations for improving the translation performance, which possibly aid the morphology model in becoming more accurate.
4.4. T HE I MPACT OF I NFLECTIONAL F EATURES
In order to understand whether the latent inﬂectional features in fact capture information about variations related to morphological transformations
[Table]
we ﬁrst try generating different surface forms of the same lemma by sampling a lemma and generating outputs using the ﬁxed lemma vector and assigning different values to the inﬂectional features.
In the second experiment
[Table]
we evaluate the impact of the inﬂectional features by setting all features to 0 and translating a set of English sentences with varying inﬂected forms in Turkish.
Table 3 presents different sets of feature values and the corresponding outputs generated by the decoder and the outputs generated with or without the inﬂection component
[Table]
The model generates different surface forms for different sets of features, conﬁrming that the latent variables represent morphological features related to the inﬁnitive form of the verb, as well as its formality conditions, prepositions, person, number and tense.
Decoding the set of sentences given in the second experiment LMM always generates the correct inﬂectional form, although when the feature values are set to 0 the model omits some inﬂectional features in the output, suggesting that despite partially relying on the source-side context, it still encodes important information for generating correct surface forms in the inﬂectional features.
C ONCLUSION
In this paper we presented a novel decoding architecture for NMT employing a hierarchical latent variable model to promote sparsity in lexical representations, which demonstrated promising application for morphologically-rich and low-resource languages.
Our model generates words one character at a time by composing two latent features representing their lemmas and inﬂectional features.
We evaluate our model against conventional open-vocabulary NMT solutions such as subword and character-level decoding methods in translationg English into three morphologically-rich languages with different morphological typologies under low to mid-resource settings.
Our results show that our model can signiﬁcantly outperform subword-level NMT models, whereas demonstrates better capacity than character-level models in coping with increased amounts of data sparsity.
We also conduct ablation studies on the effect of feature variations to the predictions, which prove that despite being completely unsupervised, our model can in fact capture morphosyntactic information and generalize to different surface forms of words.
A CKNOWLEDGMENTS
The authors would like to thank Marcello Federico, Orhan Fırat, Adam Lopez, Graham Neubig,
Akash Srivastava and Clara Vania for their feedback and suggestions.
This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements 825299 (GoURMET) and 688139 (SUMMA).
A A PPENDIX A. T HE STATISTICAL CHARACTERISTICS OF EXPERIMENTAL DATA
[Table] [Table]
[Table]
A. T HE K UMARASWAMY DISTRIBUTION
[Figure]
A. T HE E FFECT OF F EATURE D IMENSIONS
We investigate the optimal lemma and inﬂectional feature sizes by measuring the accuracy in
English-to-Turkish translation using different feature vector dimensions.
The results given in Figure 3 show that gradually compressing the word representations computed by recurrent hidden states, with an original dimension of 512, from 500 to 100, leads to increased output accuracy, suggesting that encoding more compact representations might provide the model with a better generalization capability.
Our results also show that using a feature dimension of 10 is sufﬁcient in reaching the best accuracy.