Paper:
Morphological Inflection Generation with Hard Monotonic Attention
We present a neural model for morphological inflection generation which employs a hard attention mechanism, inspired by the nearly-monotonic alignment commonly found between the characters in a word and the characters in its inflection. We evaluate the model on three previously studied morphological inflection generation datasets and show that it provides state of the art results in various setups compared to previous neural and nonneural approaches. Finally we present an analysis of the continuous representations learned by both the hard and soft attention (Bahdanau et al., 2014) models for the task, shedding some light on the features such models extract.
We present a neural model for morphological inflection generation which employs a hard attention mechanism, inspired by the nearly-monotonic alignment commonly found between the characters in a word and the characters in its inflection. We evaluate the model on three previously studied morphological inflection generation datasets and show that it provides state of the art results in various setups compared to previous neural and nonneural approaches. Finally we present an analysis of the continuous representations learned by both the hard and soft attention (Bahdanau et al., 2014) models for the task, shedding some light on the features such models extract.
1 Introduction
Morphological inflection generation involves generating a target word (e.g. “härtestem”, the German word for “hardest”), given a source word (e.g. “hart”, the German word for “hard”) and the morpho-syntactic attributes of the target (POS=adjective, gender=masculine, type=superlative, etc.).
The task is important for many down-stream NLP tasks such as machine translation, especially for dealing with data sparsity in morphologically rich languages where a lemma can be inflected into many different word forms. Several studies have shown that translating into lemmas in the target language and then applying inflection generation as a post-processing step is beneficial for phrase-based machine translation (Minkov et al., 2007; Toutanova et al., 2008; Clifton and Sarkar, 2011; Fraser et al., 2012; Chahuneau et al., 2013)
and more recently for neural machine translation (Garcı́a-Martı́nez et al., 2016).
The task was traditionally tackled with hand engineered finite state transducers (FST) (Koskenniemi, 1983; Kaplan and Kay, 1994) which rely on expert knowledge, or using trainable weighted finite state transducers (Mohri et al., 1997; Eisner, 2002) which combine expert knowledge with datadriven parameter tuning. Many other machinelearning based methods (Yarowsky and Wicentowski, 2000; Dreyer and Eisner, 2011; Durrett and DeNero, 2013; Hulden et al., 2014; Ahlberg et al., 2015; Nicolai et al., 2015) were proposed for the task, although with specific assumptions about the set of possible processes that are needed to create the output sequence.
More recently, the task was tackled using neural sequence to sequence learning over character sequences with impressive results (Faruqui et al., 2016). The vanilla encoder-decoder models as used by Faruqui et al. compress the input sequence to a single, fixed-sized continuous representation. Instead, soft-attention based sequence to sequence learning paradigm (Bahdanau et al., 2014) allows directly conditioning on the entire input sequence, and was utilized for morphological inflection generation with great success (Kann and Schütze, 2016b,a).
However, the sequence-to-sequence models require large training sets in order to perform well: their performance on the relatively small CELEX dataset is inferior to the latent variable WFST model of Dreyer et al. (2008). Interestingly, the neural WFST model by Rastogi et al. (2016) also suffered from the same issue on the CELEX dataset, and surpassed the latent variable model only when given twice as much data to train on.
We propose a model which handles the above issues by directly modeling an almost monotonic alignment between the input and output character
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
sequences, which is commonly found in the morphological inflection generation task (e.g. in languages with suffixing morphology). The model consists of an encoder-decoder neural network with a dedicated control mechanism: in each step, the model attends to a single input state and either writes a symbol to the output sequence or advances the attention pointer to the next input state from the bi-directionally encoded sequence, as described visually in Figure 1.
This modeling suits the natural monotonic alignment between the input and output, as the network learns to attend to the relevant inputs before writing the output which they are aligned to. The encoder is a bi-directional RNN, where each character in the input word is represented using a concatenation of a forward RNN and a backward RNN states over the word’s characters. The combination of the bi-directional encoder and the controllable hard attention mechanism enables to condition the output on the entire input sequence. Moreover, since each character representation is aware of the neighboring characters, non-monotone relations are also captured, which is important in cases where segments in the output word are a result of long range dependencies in the input word. The recurrent nature of the decoder, together with a dedicated feedback connection that passes the last prediction to the next decoder step explicitly, enables the model to also condition the current output on all the previous outputs at each prediction step. The hard attention mechanism allows the network to jointly align and transduce while using a focused representation at each step, rather then the weighted sum of representations used in the soft attention model. In contrast to previous sequence-to-sequence work, we do not require the training procedure to also learn the alignment. Instead, we use a simple training procedure which relies on independently learned character-level alignments, from which we derive gold transduction+control sequences. The network can then be trained using straightforward cross-entropy loss.
To evaluate our model, we perform extensive experiments on three previously studied morphological inflection generation datasets: the CELEX dataset (Baayen et al., 1993), the Wiktionary dataset (Durrett and DeNero, 2013) and the SIGMORPHON2016 dataset (Cotterell et al., 2016). We show that while our model is on par with
or better than the previous neural and non-neural state-of-the-art approaches, it also performs significantly better with very small training sets, being the first neural model to surpass the performance of the weighted FST model with latent variables specifically tailored for the task by Dreyer et al. (2008). Finally, we analyze and compare our model and the soft attention model, showing how they function very similarly with respect to the alignments and representations they learn, in spite of our model being much simpler. This analysis also sheds light on the representations such models learn for the morphological inflection generation task, showing how they encode specific features like a symbol’s type and the symbol’s location in a sequence.
To summarize, our contributions in this paper are three-fold:
1. We present a hard attention model for nearlymonotonic sequence to sequence learning, as common in the morphological inflection setting.
2. We evaluate the model on the task of morphological inflection generation, establishing a new state of the art on three previouslystudied datasets for the task.
3. We perform an analysis and comparison of our model and the soft-attention model, shedding light on the features such models extract for the inflection generation task.
2 The Hard Attention Model
We would like to transduce an input sequence, x1:n ∈ Σ∗x into an output sequence, y1:m ∈ Σ∗y, where Σx and Σy are the input and output vocabularies, respectively. Imagine a machine with read-only random access to the encoding of the input sequence, and a single pointer that determines the current read location. We can then model sequence transduction as a series of pointer movement and write operations. If we assume the alignment is monotone, the machine can be simplified: the memory can be read in sequential order, where the pointer movement is controlled by a single “move forward” operation (step) which we add to the output vocabulary. We implement this behavior using an encoder-decoder neural network, with a control mechanism which determines in each step of the decoder whether to write an
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
output symbol or promote the attention pointer the next element of the encoded input.
2.1 Model Definition
In prediction time, we seek the output sequence y1:m ∈ Σ∗y, for which:
y1:m = arg max y′∈Σ∗y
p(y′|x1:n, f) (1)
Where x ∈ Σ∗x is the input sequence and f = {f1, ..., fl} is a set of features influencing the transduction task (in the inflection generation task these would be the desired morpho-syntactic features of the output sequence). Given a nearlymonotonic alignment between the input and the output, we replace the search for a sequence of letters with a sequence of write and step actions s1:q ∈ Σ∗s, where Σs = Σy ∪ {step}. This sequence is a series of step and write actions required to go from x1:n to y1:m according to the monotonic alignment between them. In this case we define: 1
s1:q = arg max s′∈Σ∗s
p(s′|x1:n, f)
= arg max s′∈Σ∗s ∏ s′i∈s′ p(s′i|s′1...s′i−1, x1:n, f) (2)
we can then estimate this using a neural network:
s1:q = arg max s′∈Σ∗s NN(x1:n, f,Θ) (3)
Where the network’s parameters Θ are learned using a set of training examples. We will now describe the network architecture.
2.2 Network Architecture
Notation We use bold letters for vectors and matrices. We treat LSTM as a parameterized function LSTMθ(x1...xn) mapping a sequence of input vectors x1...xn to a an output vector hn. The equations for the LSTM variant we use are detailed in the supplementary material of this paper.
Encoder For every element in the input sequence: x1:n = x1...xn, we take the corresponding embedding: ex1 ...exn , where: exi ∈ RE .
1We note that our model (Eq 2) solves a different objective than (Eq 1), as it searches for the best derivation and not the best sequence. In order to accurately solve (1) we would need to marginalize over the different derivations leading to the same sequence, which is computationally challenging. However, as we see in the experiments section, the best-derivation approximation is effective in practice.
These embeddings are parameters of the model which will be learned during training. We then feed the embeddings into a bi-directional LSTM encoder (Graves and Schmidhuber, 2005) which results in a sequence of vectors: x1:n = x1...xn, where each vector xi ∈ R2H is a concatenation of: LSTMforward(ex1 , ex2 , ...exi) and LSTMbackward(exn , exn−1 ...exi), the forward LSTM and the backward LSTM outputs when fed with exi .
Decoder Once the input sequence is encoded, we feed the decoder RNN, LSTMdec, with three inputs at each step:
1. The current attended input, xa ∈ R2H , initialized with the first element of the encoded sequence, x1.
2. A set of feature embeddings that influence the generation process, concatenated to a single vector: f = [f1...fl] ∈ RF ·l.
3. si−1 ∈ RE , which is an embedding for the predicted output symbol in the previous decoder step.
Those three inputs are concatenated into a single vector zi = [xa, f , si−1] ∈ R2H+F ·l+E , which is fed into the decoder, providing the decoder output vector: LSTMdec(z1...zi) ∈ RH . Finally, to model the distribution over the possible actions, we project the decoder output to a vector of |Σs| elements, followed by a softmax layer:
p(si = c)
= softmax c(W · LSTMdec(z1...zi) + b) (4)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Control Mechanism When the most probable action is step, the attention is promoted so xa contains the next encoded input representation to be used in the next step of the decoder. The process is demonstrated visually in Figure 1.
2.3 Training the Model
For every example: (x1:n, y1:m, f) in the training data, we should produce a sequence of step and write actions s1:q to be predicted by the decoder. The sequence is dependent on the alignment between the input and the output: ideally, the network will attend to all the input characters aligned to an output character before writing it. While recent work in sequence transduction advocate jointly training the alignment and the decoding mechanisms (Bahdanau et al., 2014; Yu et al., 2016), we instead show that in our case it is worthwhile to decouple these stages and learn a hard alignment beforehand, using it to guide the training of the encoder-decoder network and enabling the use of correct alignments for the attention mechanism from the beginning of the network training phase. For this purpose, we first run a character level alignment process on the training data. We use the character alignment model of Sudoh et al. (2013) which is based on a Chinese Restaurant Process which weights single alignments (character-to-character) in proportion to how many times such an alignment has been seen elsewhere out of all possible alignments. Once we have the character level alignment per input-output sequence pair in the training set, we deterministically infer the sequence of actions s1:q from it by requiring to read all the input elements aligned to an output element (using the step action) before writing it. We then train the network to predict this sequence of actions by using a conventional cross entropy loss function per example: L(x1:n, y1:m, f,Θ) = − ∑
sj∈s1:q
log softmax sj (d),
d = W · LSTMdec(z1...zi) + b (5)
An alternative view of our encoder-decoder architecture is that of a transition system with ADVANCE and WRITE(CH) actions, where the oracle is derived from a given character-level alignment, the input is encoded using a biRNN, and the next action is determined by an RNN over the previous state/action combinations.
3 Experiments
We perform extensive experiments with three previously studied morphological inflection generation datasets to evaluate our hard attention model in various settings. In all experiments we report the results of the best performing neural and nonneural baselines which were previously published on those datasets. The implementation details for our models are described in the supplementary material section of this paper. The source code for our models is available on github.2
CELEX Our first evaluation is on a very small dataset, to see if our model indeed avoids the tendency to overfit with small training sets. We report exact match accuracy on the German inflection generation dataset compiled by Dreyer et al. (2008) from the CELEX database (Baayen et al., 1993). The dataset includes only 500 training examples for each of the four inflection types: 13SIA→13SKE, 2PIE→13PKE, 2PKE→z, and rP→pA which we refer to as 13SIA, 2PIE, 2PKE and rP, respectively.3 We compare our model to three competitive models that reported results on this dataset: the Morphological Encoder-Decoder (MED) of Kann and Schütze (2016a) which is based on the soft-attention model of Bahdanau et al. (2014), the neural-weighted FST of Rastogi et al. (2016) which uses stacked bi-directional LSTM’s to weigh its arcs (NWFST), and the model of Dreyer et al. (2008) which uses a weighted FST with latent-variables structured particularly for morphological string transduction tasks (LAT). Following previous reports on this dataset, we use the same data splits as Dreyer et al. (2008), dividing the data for each inflection type into five folds, each consisting of 500 training, 1000 development and 1000 test examples. We train a separate model for each fold and report exact match accuracy, averaged over the five folds.
Wiktionary To neutralize the negative effect of very small training sets on the performance of the different learning approaches, we also evaluate our model on the dataset created by Durrett and DeN-
2the link is removed in this version to keep the submission anonymized
3The acronyms stand for: 13SIA=1st/3rd person, singular, indefinite, past;13SKE=1st/3rd person, subjunctive, present; 2PIE=2nd person, plural, indefinite, present;13PKE=1st/3rd person, plural, subjunctive, present; 2PKE=2nd person, plural, subjunctive, present; z=infinitive; rP=imperative, plural; pA=past participle.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
13SIA 2PIE 2PKE rP Avg. MED 83.9 95 87.6 84 87.62 NWFST 86.8 94.8 87.9 81.1 87.65 LAT 87.5 93.4 87.4 84.9 88.3 Hard 85.8 95.1 89.5 87.2 89.44
Table 1: Results on the CELEX dataset
DE-N DE-V ES-V FI-NA FI-V FR-V NL-V Avg. DDN13 88.31 94.76 99.61 92.14 97.23 98.80 90.50 94.47 NCK15 88.6 97.50 99.80 93.00 98.10 99.20 96.10 96.04 FTND16 88.12 97.72 99.81 95.44 97.81 98.82 96.71 96.34 YBB16 87.5 92.11 99.52 95.48 98.10 98.65 95.90 95.32 Hard 88.87 97.35 99.79 95.75 98.07 99.04 97.03 96.55
Table 2: Results on the Wiktionary datasets
ero (2013), which contains up to 360k training examples per language. It was built by extracting Finnish, German and Spanish inflection tables from Wiktionary, used in order to evaluate their system based on string alignments and a semiCRF sequence classifier with linguistically inspired features. We also used the expansion made by Nicolai et al. (2015) to include French and Dutch inflections as well. Their system also performs an align-and-transduce approach, extracting rules from the aligned training set and applying them in inference time with a proprietary character sequence classifier. In addition to those systems we also compare to the results of the recent neural approaches of Faruqui et al. (2016), which did not use an attention mechanism, and Yu et al. (2016), which coupled the alignment and transduction tasks.
SIGMORPHON As different languages show different morphological phenomena, we also experiment with how our model copes with this various phenomena using the morphological inflection dataset from the SIGMORPHON2016 shared task (Cotterell et al., 2016). Here the training data consists of ten languages, with five morphological system types (detailed in Table 3): Russian (RU), German (DE), Spanish (ES), Georgian (GE), Finnish (FI), Turkish (TU), Arabic (AR), Navajo (NA), Hungarian (HU) and Maltese (MA) with roughly 12,800 training and 1600 development examples per language. We compare our model to two soft attention baselines on this dataset: MED (Kann and Schütze, 2016b), which was the best participating system in the shared
task, and our implementation of the global (soft) attention model presented by Luong et al. (2015).
4 Results
On the low resource setting (CELEX), our model significantly outperforms both the recent neural models of Kann and Schütze (2016a) and Rastogi et al. (2016) and the morphologically aware latent variable model of Dreyer et al. (2008), as detailed in Table 1. It is also, to our knowledge, the first model that surpassed in overall accuracy the latent variable model on this dataset. We attribute our advantage over the soft attention model to the ability of the hard attention control mechanism to harness the monotonic alignments found in the data, while also conditioning on the entire output history which wasn’t available in the FST models. Figure 2 plots the train-set and dev-set accuracies of the soft and hard attention models as a function of the training epoch. While both models perform similarly on the train-set (with the soft attention model fitting it slightly faster), the hard attention model performs significantly better on the dev-set. This shows the soft attention model’s tendency to overfit on the small dataset, as it is not enforcing the monotonic assumption of the hard attention model.
On the large training set experiments (Wiktionary), our model is the best performing model on German verbs, Finnish nouns/adjectives and Dutch verbs, resulting in the highest reported average accuracy across all the inflection types when compared to the four previous neural and nonneural state of the art baselines, as detailed in Ta-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
595
596
597
598
599
ble 2. This shows the robustness of our model also with large amounts of training examples, and the advantage the hard attention mechanism provides over the encoder-decoder approach of Faruqui et al. (2016) which does not employ an attention mechanism. Our model is also significantly more accurate than the model of Yu et al. (2016), which shows the advantage of using independently learned alignments to guide the network’s attention from the beginning of the training process. As can be seen in Table 3, on the SIGMORPHON 2016 dataset our model performs better than both soft-attention baselines for the suffixing+stem-change languages (Russian, German and Spanish) and is slightly less accurate than our implementation of the soft attention model on the rest of the languages, which is now the best performing model on this dataset to our knowledge. We explain this by looking at the languages from a linguistic typology point of view, as detailed in Cotterell et al. (2016). Since Russian, German and Spanish employ a suffixing morphol-
ogy with internal stem changes, they are more suitable for monotonic alignment as the transformations they need to model are the addition of suffixes and changing characters in the stem. The rest of the languages in the dataset employ more context sensitive morphological phenomena like vowel harmony and consonant harmony, which require to model long range dependencies in the input sequence which better suits the soft attention mechanism. While our implementation of the soft attention model and MED are very similar modelwise, we hypothesize that our soft attention model results are better due to the fact that we trained the model for 100 epochs and picked the best performing model on the development set, while the MED system was trained for a fixed amount of 20 epochs (although trained on more data – both train and development sets).
5 Analysis
The Learned Alignments In order to see if the alignments predicted by our model fit the mono-
suffixing+stem changes circ. suffixing+agg.+v.h. c.h. templatic RU DE ES GE FI TU HU NA AR MA Avg.
MED 91.46 95.8 98.84 98.5 95.47 98.93 96.8 91.48 99.3 88.99 95.56 Soft 92.18 96.51 98.88 98.88 96.99 99.37 97.01 95.41 99.3 88.86 96.34 Hard 92.21 96.58 98.92 98.12 95.91 97.99 96.25 93.01 98.77 88.32 95.61
Table 3: Results on the SIGMORPHON 2016 morphological inflection dataset. The text above each language lists the morphological phenomena it includes: circ.=circumfixing, agg.=agglutinative, v.h.=vowel harmony, c.h.=consonant harmony
0 10 20 30 40
0
0.5
1
epoch
ac cu
ra cy
soft-train hard-train soft-dev hard-dev
Figure 2: Learning curves for the soft and hard attention models on the first fold of the CELEX dataset
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
tonic alignment structure found in the data, and whether are they more suitable for the task when compared to the alignments found by the soft attention model, we examined alignment predictions of the two models on examples from the development portion of the CELEX dataset, as depicted in Figure 3. First, we notice the alignments found by the soft attention model are also monotonic, supporting our modeling approach for the task. Figure 3 (bottom-right) also shows how the hardattention model performs deletion (legte→lege) by predicting a sequence of two step operations. Another notable morphological transformation is the one-to-many alignment, found in the top example: flog→fliege, where the model needs to transform a character in the input, o, to two characters in the output, ie. This is performed by two consecutive write operations after the step operation of the relevant character to be replaced. Notice that in this case, the soft attention model performs a different alignment by aligning the character i to o and the character g to the sequence eg, which
is not the expected alignment in this case from a linguistic point of view.
The Learned Representations How does the soft-attention model manage to learn monotonic alignments? Perhaps the the network learns to encode the sequential position as part of its encoding of an input element? More generally, what information is encoded by the soft and hard alignment encoders? We selected 500 random encoded characters-in-context from input words in the CELEX development set, where every encoded representation is a vector in R200. Since those vectors are outputs from the bi-LSTM encoders of the models, every vector of this form carries information of the specific character with its entire context. We project these encodings into 2-D using SVD and plot them twice, each time using a different coloring scheme. We first color each point according to the character it represents (Figures 4a, 4b). In the second coloring scheme (Figures 4c, 4d), each point is colored according
(a) Colors indicate which character is encoded.
(b) Colors indicate which character is encoded.
(c) Colors indicate the character’s position.
(d) Colors indicate the character’s position.
Figure 4: SVD dimension reduction to 2D of 500 character representations in context from the encoder, for both the soft attention (top) and hard attention (bottom) models.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
to the character’s sequential position in the word it came from, blue indicating positions near the beginning of the word, and red positions near its end.
While both models tend to cluster representations for similar characters together (Figures 4a, 4b), the hard attention model tends to have much more isolated character clusters. Figures 4c, 4d show that both models also tend to learn representations which are sensitive to the position of the character, although it seems that here the soft attention model is more sensitive to this information as its coloring forms a nearly-perfect red-to-blue transition on the X axis. This may be explained by the soft-attention mechanism encouraging the encoder to encode positional information in the input representations, which may help it to predict better attention scores, and to avoid collisions when computing the weighted sum of representations for the context vector. In contrast, our hardattention model has other means of obtaining the position information in the decoder using the step actions, and for that reason it does not encode it as strongly in the representations of the inputs. This behavior may allow it to perform well even with fewer examples, as the location information is represented more explicitly in the model using the step actions.
6 Related Work
Many previous works on inflection generation used machine learning methods (Yarowsky and Wicentowski, 2000; Dreyer and Eisner, 2011; Durrett and DeNero, 2013; Hulden et al., 2014; Ahlberg et al., 2015; Nicolai et al., 2015) with assumptions about the set of possible processes needed to create the output word. Our work was mainly inspired by Faruqui et al. (2016) which trained an independent encoder-decoder neural network for every inflection type in the training data, alleviating the need for feature engineering. Kann and Schütze (2016b,a) tackled the task with a single soft attention model (Bahdanau et al., 2014) for all inflection types, which resulted in the best submission at the SIGMORPHON 2016 shared task (Cotterell et al., 2016). Another closely related work (Rastogi et al., 2016) modeled the task with a WFST in which the arcs are learned by optimizing a global loss function over all the possible paths in the state graph while modeling contextual features with bi-directional LSTM’s. Another recent work (Kann et al., 2016)
proposed performing neural multi-source morphological reinflection, generating an inflection from several source forms of a word.
Previous works on neural sequence transduction include the RNN Transducer (Graves, 2012) which uses two independent RNN’s over monotonically aligned sequences to predict a distribution over the possible output symbols in each step, including a null symbol to model the alignment. Yu et al. (2016) improved this by replacing the null symbol with a dedicated learned transition probability. Both models are trained using a forwardbackward approach, marginalizing over all possible alignments. Our model differs from the above by learning the alignments independently, thus enabling a dependency between the encoder and decoder. While providing better results than Yu et al. (2016), this also simplifies the model training using a simple cross-entropy loss. Jaitly et al. (2015) proposed the Neural Transducer model, which is also trained on external alignments. They divide the input into blocks of a constant size and perform soft attention separately on each block. Lu et al. (2016) used a combination of an RNN encoder with a CRF layer to model the dependencies in the output sequence. A line of work on attention-based speech recognition (Chorowski et al., 2015; Bahdanau et al., 2016) proposed improvements to the attention mechanism: adding location awareness by using the previous attention weights when computing the next ones, and preventing the model from attending on too many or too few inputs using “sharpening” and “smoothing” techniques on the attention weight distributions.
7 Conclusion
We presented a hard attention model for morphological inflection generation. The model employs an explicit alignment which is used to train a neural network to perform transduction by decoding with a hard attention mechanism. Our model performs on par or better than previous neural and non-neural approaches on various morphological inflection generation datasets, while staying competitive with dedicated models even with very few training examples. Future work may include applying the model to other nearly-monotonic alignand-transduce tasks like abstractive summarization, transliteration or machine translation.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Review:
- Strengths: A new encoder-decoder model is proposed that explicitly takes  into account monotonicity.  - Weaknesses: Maybe the model is just an ordinary BiRNN with alignments de-coupled. Only evaluated on morphology, no other monotone Seq2Seq tasks.  - General Discussion:  The authors propose a novel encoder-decoder neural network architecture with  hard monotonic attention . They evaluate it on three morphology datasets.  This paper is a tough one. One the one hand it is well-written, mostly very clear and also presents a novel idea, namely including monotonicity in morphology tasks.   The reason for including such monotonicity is pretty obvious: Unlike machine translation, many seq2seq tasks are monotone, and therefore general encoder-decoder models should not be used in the first place. That they still perform reasonably well should be considered a strong argument for neural techniques, in general. The idea of this paper is now to explicity enforce a monotonic output character generation. They do this by decoupling alignment and transduction and first aligning input-output sequences monotonically and then training to generate outputs in agreement with the monotone alignments. However, the authors are unclear on this point. I have a few questions:  1) How do your alignments look like? On the one hand, the alignments seem to be of the kind 1-to-many (as in the running example, Fig.1), that is, 1 input character can be aligned with zero, 1, or several output characters. However, this seems to contrast with the description given in lines 311-312 where the authors speak of several input characters aligned to 1 output character. That is, do you use 1-to-many, many-to-1 or many-to-many alignments?  2) Actually, there is a quite simple approach to monotone Seq2Seq. In a first stage, align input and output characters monotonically with a 1-to-many constraint (one can use any monotone aligner, such as the toolkit of Jiampojamarn and Kondrak). Then one trains a standard sequence tagger(!) to predict exactly these 1-to-many alignments. For example, flogtofliege (your example on l.613): First align as in  f-l-o-g   f-l-ie-ge . Now use any tagger (could use an LSTM, if you like) to predict  f-l-ie-ge  (sequence of length 4) from  f-l-o-g  (sequence of length 4). Such an approach may have been suggested in multiple papers, one reference could be [*, Section 4.2] below.  My two questions here are:   2a) How does your approach differ from this rather simple idea?  2b) Why did you not include it as a baseline?  Further issues:  3) It's really a pitty that you only tested on morphology, because there are many other interesting monotonic seq2seq tasks, and you could have shown your system's superiority by evaluating on these, given that you explicitly model monotonicity (cf. also [*]).  4) You perform  on par or better  (l.791). There seems to be a general cognitive bias among NLP researchers to map instances where they perform worse to  on par  and all the rest to  better . I think this wording should be corrected, but otherwise I'm fine with the experimental results.  5) You say little about your linguistic features: From Fig. 1, I infer that they include POS, etc.   5a) Where did you take these features from?  5b) Is it possible that these are responsible for your better performance in some cases, rather than the monotonicity constraints?  Minor points:  6) Equation (3): please re-write $NN$ as $  text{NN}$ or similar  7) l.231  Where  should be lower case  8) l.237 and many more: $x_1  ldots x_n$. As far as I know, the math community recommends to write $x_1,  ldots,x_n$ but $x_1  cdots x_n$. That is, dots should be on the same level as surrounding symbols.  9) Figure 1: is it really necessary to use cyrillic font? I can't even address your example here, because I don't have your fonts.  10) l.437: should be  these   [*]   @InProceedings{schnober-EtAl:2016:COLING,     author    = {Schnober, Carsten  and  Eger, Steffen  and  Do Dinh, Erik-L  ^{a}n  and  Gurevych, Iryna},   title     = {Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks},   booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},   month     = {December},   year                                                      = {2016},   address   = {Osaka, Japan},   publisher = {The COLING 2016 Organizing Committee},   pages     = {1703--1714},   url                                               = {http:  aclweb.org anthology C16-1160}  }  AFTER AUTHOR RESPONSE  Thanks for the clarifications. I think your alignments got mixed up in the response somehow (maybe a coding issue), but I think you're aligning 1-0, 0-1, 1-1, and later make many-to-many alignments from these.  I know that you compare to  Nicolai, Cherry and Kondrak (2015) but my question would have rather been: why not use 1-x (x in 0,1,2) alignments as in  Schnober et al. and then train a neural tagger on these (e.g. BiLSTM). I wonder how much your results would have differed from such a rather simple baseline. (A tagger is a monotone model to start with and given the monotone alignments, everything stays monotone. In contrast, you start out with a more general model and then put hard monotonicity constraints on this ...)  NOTES FROM AC  Also quite relevant is Cohn et al. (2016), http:  www.aclweb.org anthology N16-1102 .  Isn't your architecture also related to methods like the Stack LSTM, which similarly predicts a sequence of actions that modify or annotate an input?    Do you think you lose anything by using a greedy alignment, in contrast to Rastogi et al. (2016), which also has hard monotonic attention but sums over all alignments?