Tü- CL at SIGMORPHON 2023: Straight-Through Gradient Estimation for Hard Attention (2023)

Abstract: This paper describes our systems participating in the 2023 SIGMORPHON Shared Task on Morphological Inflection and in the 2023 SIGMORPHON Shared Task on Interlinear Glossing. We propose methods to enrich predictions from neural models with discrete, i.e. interpretable, information. For morphological inflection, our models learn deterministic mappings from subsets of source lemma characters and morphological tags to individual target characters, which introduces interpretability. For interlinear glossing, our models learn a shallow morpheme segmentation in an unsupervised way jointly with predicting glossing lines. Estimated segmentation may be useful when no ground-truth segmentation is available. As both methods introduce discreteness into neural models, our technical contribution is to show that straight-through gradient estimators are effective to train hard attention models.

## introduction this paper describes our systems participating in the sigmorphon-unimorph shared task on typologically diverse and acquisition-inspired morphological inflection generation @xcite and the sigmorphon 2023 shared task on interlinear glossing. for morphological inflection, we participate in part 1, and for interlinear glossing we mainly target the closed track. morphological inflection is the task of predicting the correct inflected form given a lemma and set of morphological tags. an example from the italian dataset in the shared task is votare ("to vote") v;ind;fut;nom(1,pl) ## st optimization of hard attention we discuss hard attention as mappings of the following form: let @xmath0.g. the number of decoder positions in a encoder-decoder sequence-to-sequence model), and @xmath1.g. learned embedding vectors). each target element y i , @xmath2, . . . , k} is calculated as a sum of source element encodings, formally y @xmath3, . . . , n} is the set of source elements aligned to target position i. note that a source element may be aligned to multiple target elements, i.e. appear @xmath4. this mapping can be calculated by a matrix multiplication ξ • @xmath5, where columns of ξ ∈ {0, 1} k×n are the multi-hot encodings of index sets (→ i ) @xmath6,...,k} . formally, this means we assume ξ is a sample from a underlying categorical distribution where we can compute the marginals ξi,j that specify the probability the main problem is how to side-step sampling during gradient-based optimization, because sampling is not differentiable. one solution is the so-called straight-through estimator @xcite @xcite which means using ξ for the forward pass, i.e. when computing model outputs, but using ξ for backpropagation, i.e. when computing gradients of model parameters w.r.t. the loss. however, gradients of x are affected by the discreteness of ξ as well, because ξ i,@xmath7. therefore, when using straight-through gradient estimation, we should use ξ when computing gradients of x. formally, for some differentiable function f that is applied to y, we set ## applications in this section, we describe how to apply the method from section 2 to sequence transduction (section 3.1) and sequence segmentation (section 3.2). we keep formulations more general than necessary for the shared tasks, because we want to highlight that the methods apply to similar problems as well. ## evaluation here, we evaluate the methods presented in section 3 by participating in the shared task on morphological inflection and in the shared task on interlinear glossing. technical details of the experimental setup and hyperparameters are in appendix a. ## related work much recent work in character-level transduction aims at making models both more interpretable and stronger by using sparse @xcite or hard attention @xcite @xcite . modifying the attention mechanism is necessary, because for soft attention, which does not realise hard alignment decisions, the relation of model outputs and attention weights is not fully understood @xcite . hard attention does not suffer from this problem, because it realises hard alignment decisions, so that we exactly know which information influences the output of attention. especially if the main purpose of models is to gain insights into the data and not mainly to achieve better performance on modelling it, interpretability of models is crucial. however, working with hard attention is notoriously difficult, because it introduces nondifferentiability into models. this means that the sophisticated machinery developed for gradientbased optimisation of deep neural models fails in this case. the most popular but also most expensive approach to train hard attention mechanisms is marginalising all alignments @xcite @xcite or approximating the marginalisation @xcite . different approaches to approximate gradients instead of having exact gradients by marginalisation are using reinforcement learning @xcite , reparametrising discrete distributions or working with continuous relaxations @xcite , and perturbation based gradient approximators @xcite . while all methods to use hard attention with gradient-based optimisation come with problems, we find straight-through gradient estimators @xcite very effective to compute informative discrete intermediate representations even for complicated attention scenarios. similar results have been reported for other fields as well, both practically @xcite and in theoretical analysis @xcite . therefore, we propose two models, one for interpretable sequence transduction and one for joint segmentation and segment classification, based on straight-through gradient estimators. ## conclusion in this work, we propose to optimise hard attention as building block in deep neural networks, which in principle is non-differentiable, by straight-through gradient estimation. in particular, we describe applications to interpretable sequence-to-sequence models and sequence segmentation models. we evaluate our approaches on two important tasks in computational morphology, namely morphological inflection and interlinear glossing which are shared tasks of the acl sigmorphon 2023 workshop. our approaches achieve good results in morphological inflection despite of constrained expressivity compared to fully differentiable models, and strong results in interlinear glossing. these results provide encouraging evidence that learning interpretable intermediate representations by deep neural network does not necessarily lead to intolerable sacrifices in performance. we hope that future work can benefit from these insights by combining interpretable representations and the generalisation abilities of neural models for scientific discovery. an exponential decay learning rate scheduler which multiplies the learning rate by factor γ after each epoch. we tune γ for each combination of model and language. furthermore, we clip gradients with absolute value greater than 1. models are trained exclusively on the training splits provided by the shared task organisers. after each training epoch, generalisation performance is estimated by performance on the validation split. in the case of morphological inflection, our main metric is normalised edit distance (lower is better). in the case of interlinear glossing, our main metric is accuracy of correctly predicted glossing lines (higher is better). if performance does not improve for 3 epochs, training is stopped. only the best model checkpoint, i.e. the checkpoint 3 epochs before ending training, is retained.