A L ATENT M ORPHOLOGY M ODEL FOR O PENV OCABULARY N EURAL M ACHINE T RANSLATION
A BSTRACT
Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic.
This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters.
The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task.
The latter learns directly from translation data but requires rather deep architectures.
In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inﬂection.
Our model generates words one character at a time by composing two latent representations a continuous one aimed at capturing the lexical semantics and a set of (approximately discrete features aimed at capturing the morphosyn - tactic function which are shared among different surface forms
Our model achieves better accuracy in translation into three morphologically rich languages than conventional open-vocabulary NMT methods while also demonstrating a better generalization capacity under low to mid -resource settings.
I NTRODUCTION
Neural machine translation (NMT) systems are conventionally trained based on the approach of maximizing the log-likelihood on a training corpus in order to learn distributed representations of words according to their sentence context, which is highly demanding in terms of training data as well as the network capacity.
Under conditions of lexical sparsity, which may include the cases when the amount of training examples is insufﬁcient to observe words in different context, and particularly in translation of morphologically-rich languages, where the same word can have exponentially many different surface realizations due to syntactic conditions, which are often rarely or ever observed in any set of collected examples, the model may suffer in learning accurate representations of words.
The standard approach to overcome this limitation is to replace the word representations in the model with subword units that are shared among words, which are, in principle, more reliable as they are observed more frequently in varying context (Sennrich et al., 2016; Wu et al., 2016).
One drawback related to this approach, however, is that the estimation of the subword vocabulary relies on word segmentation methods optimized using corpus-dependent statistics, disregarding any linguistic notion and the translation objective, which may result in morphological errors during splitting, resulting in subword units that are semantically ambiguous as they might be used in far too many lexical contexts (Ataman et al., 2017).
Moreover, the words are generated predicting multiple subword units, which makes generalizing to unseen word forms more difﬁcult, where some of the subword units that could be used to reconstruct a given word may be unlikely in the given context.
To alleviate the sub-optimal effects of using explicit segmentation and generalize better to new morphological forms, recent studies explored the idea of extending the same approach to model translation directly at the level of characters (Kreutzer & Sokolov, 2018; Cherry et al., 2018), which, in turn, have demonstrated the requirement of using comparably deeper networks, as the network would then need to learn longer distance grammatical dependencies (Sennrich, 2017).
In this paper, we explore the beneﬁt of explicitly modeling variations in the surface forms of words using methods from deep latent variable modeling in order to improve the translation accuracy in low-resource and morphologically-rich languages.
Latent variable models allow us to inject inductive biases relevant to the task, which, in our case, is word formation, and we believe that follows a certain hierarchical procedure.
Our model translates words one character at a time based on word representations learned compositionally from sub-lexical components, which are parameterized by a hierarchical latent variable model mimicking the process of morphological inﬂection, consisting of a continuous-space dense vector capturing the lexical semantics, and a set of (approximately) discrete features, representing the morphosyntactic role of the word in a given sentence.
Each word representation during decoding is reformulated based on the shared latent morphological features, aiding in learning more reliable representations of words under sparse settings by generalizing across their different surface forms.
We evaluate our method in translating English into three morphologicallyrich languages each with a distinct morphological typology: Arabic, Czech and Turkish, and show that our model is able to obtain better translation accuracy and generalization capacity than conventional approaches to open-vocabulary NMT.
N EURAL M ACHINE T RANSLATION
In this paper, we use recurrent NMT architectures based on the model developed by Bahdanau et al.
The model essentially estimates the conditional probability of translating a source sequence x = (cid:104) x 1 , x 2 , . . .
x m (cid:105) into a target sequence y =
(cid:104)
y 1 , y 2 , . .
y l (cid:105) via an exact factorization: p ( y | x, θ )
= l (cid:89) i = p
( y j | x, y <
i , θ ) (1) where y <i stands for the sequence preceding the i th target word.
At each step of the sequence, a ﬁxed neural network architecture maps its inputs, the source sentence and the target preﬁx, to the probability of the i th target word observation in context.
In order to condition on the source sentence fully, this network employs an embedding layer and a bi-directional recurrent neural network (bi-RNN) based encoder.
Conditioning on the target preﬁx y <
i , is implemented using a recurrent neural network (RNN) based decoder, and an attention mechanism which summarises the source sentence into a context vector c
i as a function of a given preﬁx (Luong et al., 2015).
Given a parallel training set D , the parameters θ of the network are estimated to attain a local minimum of the negative log-likelihood function L ( θ |D )
=
− (cid:80) x,y ∼D log p ( y | x, θ ) via stochastic gradient-based optimization (Bottou & Cun, 2004).
Atomic Paramaterization estimates the probability of generating each target word y i in a single shot:
[Equation]
where E ∈ R v × d is the target embedding matrix and the decoder output h i ∈
R d represents x and y <i .
Clearly, the size v of the target vocabulary plays an important role in deﬁning the complexity of the model, which creates an important bottleneck when translating into low-resource and morphologically-rich languages due to the sparsity in the lexical distribution.
Recent studies approached this problem by performing NMT with subword units, a popular one of which is based on the Byte-Pair Encoding algorithm (BPE; Sennrich et al., 2016), which ﬁnds the optimal description of a corpus vocabulary by iteratively merging the most frequent character sequences.
Atomic parameterization could also be be used to model translation at the level of characters, which is found to be advantageous in generalizing to morphological variations (Cherry et al., 2018).
Hierarchical Paramaterization further factorizes the probability of a target word in context:
[Equation]
where y i =
(cid:104)
y i, 1 , . . .
, y i,l i (cid:105) is generated one character at a time, each with probability computed by a ﬁxed neural network architecture with varying inputs, namely, the source sentence x , the target preﬁx y <
i , and the current word’s preﬁx y i,<j .
In this case there are two recurrent cells, one updated at the boundary of each token, much like in the standard case, and another updated at the character level.
Luong & Manning (2016) propose hierarchical parameterization to compute the probability p ( y i | x, y <
i , θ ) for unknown words, while for known words they use the atomic parameterization.
In this paper, we use the hierarchical parameterization method for generating all target words, where we also augment the input embedding layer with a character-level bi-RNN, which computes each word representation
y i as a composition of the embeddings of their characters (Ling et al., 2015).
A L ATENT M ORPHOLOGY M ODEL FOR L EARNING W ORD R EPRESENTATIONS
Studies analyzing the advantage of using a hierarchical structure in related tasks such as language modeling (Vania & Lopez, 2017) or semantic role labeling (Sahin & Steedman, 2018) have shown that representations of words learned compositionally via bi-RNNs from character units contain many cues about their morphological features, since word representations are encoded through a function which can establish a mapping between combinations of orthographic units and lexical context.
On the other hand, the quality of learned representations depends on the amount of observations (Sahin & Steedman, 2018), as the training data is essential in properly modeling the lexical distribution.
In this paper, we propose to extend the hierarchical parameterization model with a stochastic morphology model used to predict a set of shared latent features based on the information encoded in the compositional word representations, and reconstruct the lexical representations as a combination of the latent features, aiding in learning more reliable representations of words under sparse settings by generalizing across their different surface forms.
3. G ENERATIVE MODEL
Our generative latent morphology model for NMT formulates word formation in terms of a stochastic process, where each word is generated one character at a time by composing two latent representations: a continuous vector aimed at representing the lexical semantics of the word, or its lemma , and a set of sparse features aimed at capturing the word’s inﬂectional features .
The motivation for using a stochastic model is twofold.
First, deterministic models are by deﬁnition unimodal : when presented with the same input (the same context) they always produce the same output.
When we model the word formation process, it is reasonable to expect a larger degree of ambiguity, that is, for the same context ( e.g. a noun preﬁx), we may continue by inﬂecting the word differently depending on the (latent) mode of operation we are at ( e.g. generating nominative, accusative or dative noun).
Second, in stochastic models, the choice of distribution gives us a mechanism to favour a particular type of representation.
In our case, we use sparse distributions for inﬂectional features to accommodate the fact that morphosyntactic features are discrete in nature.
Our latent variable model is an instance of a variational auto-encoder (VAE; Kingma & Welling, 2013) inspired by the model of
Zhou & Neubig (2017) for morphological reinﬂection.
Generation of each word starts by sampling a Gaussian-distributed representation in context represented by a location vector u i and a scale vector s i ,
[Equation]
where prediction of the location (in R d ) and scale vectors (in R d> 0 ) from the word-level decoder hidden state h
i (which represents x and y <i ) is performed by two dense layers, and the scale values are ensured to be positive with the softplus ( ζ ) activation.
Generation proceeds by then sampling a vector
f
i of K sparse scalar features (see §3.2) conditioned on the source x , the target preﬁx y <
i , and the sampled lemma z i .
We model sampling of f i conditioned on z i in order to capture the insight that inﬂectional transformations typically depend on the category of a lemma.
Having sampled f i and z i , the representation of the i th target word is computed by
As shown in Figure 1, our model generates each word character by character auto-regressively by conditioning on the word representation t i predicted by the latent morphology model,
the current context c i , and the previously generated characters following the hierarchical parameterization.
3. S PARSE FEATURES
Since each target word y i may have multiple inﬂectional features, ideally, we would like f i to be
K feature indicators, which could be achieved by sampling from K independent Bernoulli distributions parameterized in context.
The problem with this approach is that sampling Bernoulli outcomes is non-differentiable, thus, their training requires gradient estimation via REINFORCE (Williams, 1992) and sophisticated variance reduction techniques.
An alternative approach that has recently become popular is to use relaxations such as the Concrete distribution or Gumbel-Softmax (Maddison et al., 2017; Jang et al., 2017) in combination with the straight-through estimator (ST; Bengio et al., 2013).
This is based on the idea of relaxing the discrete variable from taking on samples in the discrete set { 0 , 1 } to taking on samples in the continuous set (0 , 1) using a distribution for which a reparameterization exists ( e.g. Gumbel).
Then, a non-differentiable activation ( e.g. a threshold function) maps continuous outcomes to discrete ones.
ST simply ignores the discontinuous activation in the backward pass, i.e. it assumes the Jacobian is the identity matrix.
This does lead to biased estimates of the gradient of the loss, which is in conﬂict with the requirements behind stochastic optimization (Robbins & Monro, 1951).
An alternative presented by Louizos et al.
(2018) achieves a different compromise, it gets rid of bias at the cost of mixing both sparse and dense outcomes.
The idea is to obtain a continuous sample c ∈ (0 , 1) from a distribution for which a reparameterization exists and stretch it to a continuous support ( l, r ) ⊃ (0 , 1) using a simple linear transformation s = l + ( r − l ) c .
A rectiﬁer is then employed to map the negative outcomes to 0 and the positive outcomes larger than one to 1 , i.e. f = min(1 , max(0 , s )) .
The rectiﬁer is only non-differentiable at s = 0 and at s = 1 , however, because the stretched variable s is sampled from a continuous distribution, the chance of sampling s = 0 and s = 1 is essentially 0 .
This stretched-and-rectiﬁed distribution allows: i) the sampling procedure to become differentiable with respect to the parameters of the distribution, ii) to sample sparse outcomes with an unbiased estimator, and iii) to calculate the probability of sampling f = and f = 1 in closed form as a function of the parameters of the underlying distribution, which corresponds to the probability of sampling s < 0 and s > 1 , respectively.
In their paper, Louizos et al.
(2018) used the BinaryConcrete (or Gumbel-Sigmoid) as the underlying continuous distribution, the sparsity of which is controlled via a temperature parameter.
However, in our study, we found this parameter difﬁcult to predict, since it is very hard to allow a neural network to control its value without unstable gradient updates.
Instead, we opt for a slight variant by Bastings et al.
based on the Kumaraswamy distribution (Kumaraswamy, 1980), a twoparameters distribution that closely resembles a Beta distribution and is sparse whenever its (strictly positive) parameters are between 0 and 1 .
In the context of text classiﬁcation, Bastings et al.
(2019) shows this stretch-and-rectify technique to work better than methods based on REINFORCE.
For each token y i , we sample K independent Kumaraswamy variables in context,
[Equation]
which makes a continuous random vector c i in the support (0 , 1) K .
We then stretch-and-rectify the samples via f i,k = min(1 , max(0 , l − ( r − l ) c
i,k )) making f i a random vector in the support [0 , 1] K .
The probability that f i,k is exactly 0 is
[Equation]
and the probability that f i,k is exactly 1 is
[Equation]
and therefore the complement
[Equation]
is the probability that f i,k be any continuous value in the open set (0 , 1) .
In §3.4, we will derive regularizers based on π (0 , 1) i,k to promote sparse outcomes to be sampled with large probability.
3. P ARAMETER ESTIMATION
Parameter estimation of neural network models is typically done via maximum-likelihood estimation (MLE), where we approach a local minimum of the negative log-likelihood function via stochastic gradient descent with gradient computation automated by the back-propagation algorithm.
Using the following shorthand notation:
[Equation]
The log-likelihood for a single data point can be formulated as:
[Equation]
We use l = − 0 . 1 and r = 1 . 1 . Figure 2 in the appendix illustrates different instances of the computation of which is intractable.
Instead, we resort to variational inference (VI; Jordan et al., 1999), where we optimize a lower-bound on the log-likelihood
[Equation]
expressed with respect to an independently parameterized posterior approximation q ( z, f | x, y, λ ) .
For as long as sampling from the posterior is tractable and can be performed via a reparameterization, we can rely on stochastic gradient-based optimization.
In order to have a compact parameterization, we choose
[Equation]
This simpliﬁes the lowerbound, which then takes the form of l nested expectations, the i th of which is E α
( z i ) β ( f i ) [log γ ( y i )] .
This is similar to the stochastic decoder of Schulz et al.
(2018), though our approximate posterior is in fact, also our parameterized prior.
Although this objective does not particularly promote sparsity, we employ sparsity-inducing regularization techniques that will be discussed in the next section.
Concretely, for a given source sentence x , target preﬁx y <
i , and a latent sample z ≤ i , f ≤ i , we obtain a single-sample estimate of the loss by computing L i ( θ )
= − log γ ( y i ) .
3. R EGULARIZATION
In order to promote sparse distributions for the inﬂectional features, we apply a regularizer inspired by expected L 0 regularization (Louizos et al., 2018).
Whereas L 0 is a penalty based on the number of non-zero outcomes, we design a penalty based on the expected number of continuous outcomes , which corresponds to π (0 , 1) i,k as shown in Equation (6).
For a given source sentence x , target preﬁx y <
i , and a latent sample z <i , f <i , we aggregate this penalty for each feature
[Equation] [Equation]
and add it to the cost function with a positive weight ρ .
The ﬁnal loss of the NMT model is
[Equation]
3. P REDICTIONS
In our model, obtaining the conditional likelihood for predicting the most likely hypothesis requires marginalisation of the latent variables, which is intractable.
An alternative approach is to heuristically search through the joint distribution,
[Equation]
rather than the marginal, an approximation that has been referred to as Viterbi decoding (Smith, 2011).
During beam search, we populate the beam with alternative target words, and for each preﬁx y <i in the beam, we resort to deterministically choosing the latent variables based on a single sample which we deem representative of their distributions, which is a common heuristic in VAEs for translation (Zhang et al., 2016; Schulz et al., 2018).
For unimodal distributions, such as the
Gaussian p ( z i | x, y <
i , z <i , f <i ) , we use the analytic mean, whereas for multimodal distributions, such as the Hard Kumaraswamy p ( f i | x, y <i , z ≤ i , f <i ) , we use the argmax.
E VALUATION 4. M ODELS
We evaluate our model by comparing it in machine translation against three baselines which constitute the conventional open-vocabulary NMT methods, including architectures using atomic parameterization either with subword units segmented with BPE (Sennrich et al., 2016) or characters, and the hierarchical parameterization method employed for generating all words in the output.
We implement all architectures using Pytorch (Paszke et al., 2017) within the OpenNMT-py framework (Klein et al., 2017) 7 .
4. D ATA AND L ANGUAGES
In order to evaluate our model we design two sets of experiments.
The experiments in §4.4.1 aim to evaluate different methods under low-resource settings, for languages with different morphological typology.
We model the machine translation task from English into three languages with distinct morphological characteristics:
Arabic ( templatic ), Czech ( fusional ), and Turkish ( agglutinative ).
We use the TED Talks corpora (Cettolo, 2012) for training the NMT models for these experiments.
In §4.4.2, we conduct more experiments in Turkish to demonstrate the case of increased data sparsity using multi-domain training corpora, where we extend the training set using corpora from EU Bookshop (Skadin¸ ˇ s et al., 2014), Global Voices, Gnome, Tatoeba, Ubuntu (Tiedemann, 2012), KDE (Tiedemann, 2009), Open Subtitles (Lison & Tiedemann, 2016) and SETIMES (Tyers & Alperen, 2010) 8 .
The statistical characteristics of the training sets are given in Tables 4 and 5.
We use the ofﬁcial evaluation sets of the IWSLT 9 for validating and testing the accuracy of the models.
In order to increase the number of unknown and rare words in the evaluation sets we measure accuracy on large test sets combining evaluation sets from many years (Table 6 presents the evaluation sets used for development and testing).
The accuracy of each model output is measured using BLEU (Papineni et al., 2002) and chrF (Popovi ´ c, 2015) metrics, whereas the signiﬁcance of the improvements are computed using bootstrap hypothesis testing (Clark et al., 2011).
4. T RAINING S ETTINGS
All models are implemented using gated recurrent units (GRU) (Cho et al., 2014), and have a singlelayer bi-RNN encoder.
The source sides of the data used for training all NMT models, and the target sides of the data used in training the subword-level NMT models are segmented using BPE with 32,000 merge rules.
We implement all decoders using a comparable number of GRU parameters, including 3-layer stacked-GRU subword and character-level decoders, where the attention is computed after the 1 st layer (Barone et al., 2017) and a 3-layer hierarchical decoder which implements the attention mechanism after the 2 nd layer.
All models use an embedding dimension and GRU size of 512.
The latent morphology model uses the same hierarchical GRU architecture, where the middle layer is augmented using 4 multi-layer perceptrons with 256 hidden units.
We use a lemma vector dimension of 150, 10 inﬂectional features (See §A.3 for experiments conducted to tune the feature dimensions) and set the regularization constant to ρ = 0 .
All models are trained using the Adam optimizer (Kinga & Ba, 2014) with a batch size of 100, dropout rate of 0.2, learning rate of 0.0004 and learning rate decay of 0.8, applied when the perplexity does not decrease at a given epoch.
Translations are generated with beam search with a beam size of 5, where the hierarchical models implement the hierarchical beam search (Ataman et al., 2019).
4. R ESULTS 4.4. T HE E FFECT OF M ORPHOLOGICAL T YPOLOGY
The experiment results given in Table 1 shows the performance of each model in translating English into Arabic, Czech and Turkish.
In Turkish, the most sparse target language in our benchmark, [Table] using character-based decoding shows to be more advantageous compared to the subword-level and hierarchical models, due to the fact that reduced granularity in the vocabulary units might aid in better predicting words under conditions of high data sparsity.
In Arabic, on the other hand, using a hierarchical decoding model shows to be advantageous compared to the character-level decoder, as it might be useful in better learning syntactic dependencies, whereas it also outperforms the subword-level decoder.
Using the latent morphology model provides improvements of 0.51 and 0.30 BLEU points in Arabic and Turkish over the best performing baselines, respectively.
The fact that our model can efﬁciently work in both Arabic and Turkish suggests that it can handle the generation of both concatenative and non-concatenative morphological transformations.
The results in the English-to-Czech translation direction do not indicate a speciﬁc advantage of using either method for generating fusional morphology, where morphemes are already optimized at the surface level, although our model is still able to achieve translation accuracy comparable to the characterlevel model.
4.4. T HE E FFECT OF D ATA S IZE
The experiment conducted in the English-to-Turkish translation direction by increasing the amount of training data with multi-domain corpora demonstrates a more challenging case, where there is a greater possibility of observing rare words, either in the form of morphological inﬂections due to the complex agglutinative morphology of Turkish, or ambiguous terminology raising from the multi-domain characteristics.
In this experiment, the character-level model experiences a drop in performance and its accuracy is much lower than the subword-level one, suggesting that its capacity cannot cope with the increased amount of sparsity.
Empirical results suggest that with increased capacity, character-level models carry the potential to reach comparable performance to subwordlevel models (Cherry et al., 2018).
Our model reaches a much larger improvement of 0.82 BLEU points over the subword-level and 2.54 BLEU points over the character-level decoders, suggesting that it could make use of the increased sparsity in learning more accurate representations.
4.4. P REDICTING U NSEEN W ORDS
In addition to general evaluation using automatic metrics, we perform a more focused analysis to illustrate the performance of different methods in predicting unseen words.
We sample the sentences from the development sets which contain out-of-vocabulary words, and compute the average perplexity per character on these sentences using different NMT models, as suggested by Cotterell et al.
In general, the highest perplexities are obtained using the subword-based model, suggesting that generating unseen words using subword units is indeed increasing the difﬁculty of prediction, compared to the character-level which obtains the lowest perplexity.
This result indicates that increased granularity aids in reducing the uncertainty during prediction.
Similar to the results in §4.4.1, in Czech the values are almost comparable.
Due to its stochastic nature, our model yields higher perplexity values compared to the hierarchical model, whereas the values range between subword and character-based models, possibly ﬁnding an optimal level of granularity between the two solutions.
[Table]
4.4. F EATURE V ARIATIONS
In order to understand whether the latent inﬂectional features in fact capture information about variations related to morphological transformations
[Table]
we try generating different surface forms of the same lemma by assigning different values to the inﬂectional features.
We use the latent morphology model based decoder to translate the English word ‘go’
[Table]
and after sampling the lemma, we ﬁx its value and vary the values of the inﬂectional features at random positions for generating different outputs.
Table 3 presents different sets of feature values and the corresponding outputs generated by the decoder [Table] Features
[Table]
morphology model based on the lemma ‘git’ ( ‘go’ ) and different sets of inﬂectional features.
The model generates different surface forms for different sets of features, conﬁrming that latent variables encode information related to the inﬁnitive form of the verb, as well as its formality conditions, prepositions, person, number and tense.
We also observe that many trials based on different feature combinations may result in the same outputs, although some feature values may not be set in a single-word context.
Varying the features individually does not necessarily yield distinct changes in the output, suggesting that some features may act jointly in determining the word form.
C ONCLUSION
In this paper we presented a novel decoding architecture for NMT employing a hierarchical latent variable model to promote sparsity in lexical representations, which demonstrated promising application for morphologically-rich and low-resource languages.
Our model generates words one character at a time by composing two latent features representing their lemmas and inﬂectional features.
We evaluate our model against conventional open-vocabulary NMT solutions such as subword and character-level decoding methods in translationg English into three morphologically-rich languages with different morphological typologies under low to mid-resource settings.
Our results show that our model can signiﬁcantly outperform subword-level NMT models, whereas demonstrates better capacity than character-level models in coping with increased amounts of data sparsity.
We also conduct ablation studies on the effect of feature variations to the predictions, which prove that despite being completely unsupervised, our model can in fact capture morphosyntactic information and generalize to different surface forms of words.  A CKNOWLEDGMENTS
This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements 825299 (GoURMET) and 688139 (SUMMA).
A A PPENDIX A. T HE STATISTICAL CHARACTERISTICS OF TRAINING CORPORA
[Table] [Table]
[Table]
A. T HE K UMARASWAMY DISTRIBUTION
[Figure]
A. T HE E FFECT OF F EATURE D IMENSIONS
We investigate the optimal lemma and inﬂectional feature sizes by measuring the accuracy in
English-to-Turkish translation using different feature vector dimensions.
The results given in Figure 3 show that gradually compressing the word representations computed by recurrent hidden states, with an original dimension of 512, from 500 to 100, leads to increased output accuracy, suggesting that encoding more compact representations might provide the model with a better generalization capability.
Our results also show that using a feature dimension of 10 is sufﬁcient in reaching the best accuracy.