Impact of Sequence Length and Copying on Clause-Level Inflection (2022)

Abstract: We present the University of British Columbia’s submission to the MRL shared task on multilingual clause-level morphology. Our submission extends word-level inflectional models to the clause-level in two ways: first, by evaluating the role that BPE has on the learning of inflectional morphology, and second, by evaluating the importance of a copy bias obtained through data hallucination. Experiments demonstrate a strong preference for language-tuned BPE and a copy bias over a vanilla transformer. The methods are complementary for inflection and analysis tasks – combined models see error reductions of 38% for inflection and 15.6% for analysis; However, this synergy does not hold for reinflection, which performs best under a BPE-only setting. A deeper analysis of the errors generated by our models illustrates that the copy bias may be too strong - the combined model produces predictions more similar to the copy-influenced system, despite the success of the BPE-model.

## introduction morphology is often described as the "study of the shape of words", but such a description is not entirely accurate. without considering the somewhat nebulous definition of a "word", there are clearly inflectional processes that operate on a periphrastic level. for example, in english, the future tense is regularly inflected through the use of an auxiliary: will and an infinitive, such as in the case "i will go". previous tasks in inflectional morphology @xcite @xcite @xcite have largely been restricted to generating isolated inflected word forms, which could be viewed as a rather artificial task. while some have included periphrastic constructions @xcite , * *the first three authors contributed equally. we augment traditional transformer-based character models with two simple data modifications: we first apply a small bpe-vocabulary to learn common repeated sequences like function words and affixes, hoping to increase performance by reducing the known bias of long character sequences @xcite . secondly, we adopt a common data augmentation technique from wordlevel inflection: adding data that has an identical source and target to bias the model towards the copying of characters @xcite . we find that a combination of these simple techniques improves upon a vanilla transformer for inflection and analysis, while a bpe-only model has the best results for reinflection. we also contribute a significant error analysis. we investigate the types of errors that inflectional systems are prone to, and how our contributions alleviate them at the clause level; furthermore, we provide a thorough ablation study that compares errors across inflectional tasks, and how these errors are influenced by sequence length and copy biasing foot_1 . ## methods studies in neural machine translation have regularly shown character-and subword-level representations outperform word-level ones for morphologically-rich languages @xcite , and that optimizing the number of bpe operations can lead to substantial gains in model quality @xcite . although the sequences in inflectional models are typically shorter, there is evidence that inflection models, like machine translation models, can benefit from grouping common sequences @xcite . similarly, inflectional research has demonstrated that models can be significantly improved by establishing a heavy bias towards copying data directly from input to output @xcite . many variations of this theme exist, but some of the most successful have included establishing a hard attentional model @xcite , learning an explicit copy bias @xcite , and augmenting the model with hallucinated data @xcite . for our submission to the shared task, we investigate to what extent these methods are extensible to clausal morphology. previous work has largely occurred at the word-level, and while it is intuitive that word-level inflection should extend to the clause-level, it is unclear to what extent. as one of the first investigations into clause-level morphology, we investigate the influence of byte pair encoding and copy bias on the production of accurate morphological structures. ## data / experiments the shared task consists of three sub-tasks: (1) inflection, where a lemma and msd input are converted to an inflected output; (2) reinflection, where an initial clause, input-msd, and target-msd are used to generate a target clause, and (3) analysis, where an input clause generates a target lemma and msd. each task is evaluated across 9 languages. each language has a train/dev/test split of 10,000, 1000, and 1000 instances, respectively. 4 although extra data was allowed for the task, we instead concentrated on optimizing the models without additional data. each model is evaluated on a single training run; the seed is stabilized to lessen the effect of noise across experiments. we focus our experiments on an ablation of our proposed data augmentation techniques. the vanilla experiments train models using the vanilla transformer described above. +bpe tunes a byte pair encoding vocabulary on each respective language and task; we investigate bpe merges from 10-200, and choose the model that maximizes the results on the development set+copy augments the data with copied target-side data; the size of the bpe vocabulary is tuned individually for each language and amount of additional data. the transformer was trained with 4 attentional heads over 4 encoder and decoder layers. the adam{0.9, 0.98} optimizer was used, with an initial learning rate of 0.0001, and an inverse squareroot learning schedule and a label-smoothed cross entropy criterion. dropout and attentional dropout of 0.3 were applied to limit over-fitting, and a batch size of 400 was also used. models were trained for 20,000 updates, with the best model chosen via loss on the development set. ## results we break the discussion of our results down based on the three sub-tasks of the competition: inflection, reinflection, and analysis. all reported results and analysis are on the development set, and are cumulative: "+bpe" applies bpe to the vanilla transformer, and "+copy" further supplements the model with data hallucination. for official results on the test set, please see the task description paper @xcite . all systems submitted to the official task were the systems with both bpe and data hallucination. the results report the exact match accuracy of the systems. we first report the results for the inflection subtask in ## analysis / discussion in order to better understand the differences in model quality, we perform error analysis along several axes. we first consider the types of inflection errors produced by the bpe, copy, and bpe+copy models in figure 3 , while figure 4 shows error reduction compared to the vanilla transformer when using bpe, copy, and their combination bpe+copy. first, we notice that both bpe and copy individually reduce overall errors (the error type total in figure 4 ). the impact of the methods seems roughly equal, although copy is slightly more effective on its own. nevertheless, the combination bpe+copy clearly outperforms both individual methods. second, we observe somewhat different influence from the bpe and copy methods when they are used in isolation -the former significantly improves upon punctuation errors, while the latter removes a number of insertion errors from the vanilla model. moreover, the most prevalent error type in the vanilla model -deletion -is only moderately reduced by the bpe model, while a far greater error reduction can be seen when copy is employed. furthermore, we observe a largely complementary effect -the combined model improves over either individual model for all error categories. we next run an ablation to investigate the role that each of our contributions has on the quality of the models for each task. the results are plotted in figure 5 . in this graph, we investigate which errors are corrected or introduced by a particular method. bpe and copy "correct" an error if it was produced by the vanilla model, and "break" an example if it was correctly predicted by the vanilla model, but not the enhanced model. for the model with both bpe and copying, an instance is only considered "corrected" if both the bpe and copy models produced an incorrect solution. likewise, it "breaks" a prediction only if both the bpe and copy models produce the correct solution. we observe that both bpe and copying lead to large improvements in the model, regardless of the task -far more errors corrected than introduced. for inflection and analysis, both methods appear to contribute roughly equally to the quality of the model. furthermore, we observe a complementary effect, where the combination of both methods corrects notably more examples than either method on its own. contrarily, the combined model introduces fewer inflectional errors than either bpe or copying alone. interestingly, the trends observed in inflection and analysis do not hold for reinflection. although bpe and copying alone improve the model, their combination introduces a large number of errorssuch that they overwhelm the corrected instances obtained through the combination of methods. a closer inspection reveals that this outlier is largely attributable to a single language -swahili. when swahili is excluded, the results trend similar to the other tasks, although bpe still has a stronger influence. there are several areas where swahili could be contributing to this interesting finding, but lacking experts in the language on our team, we hesitate to make concrete hypotheses. figure 5 suggests that the biggest benefit of the combined model is its ability to correctly discern when one of the separate data augmentations correctly produces an inflection, but it isn't quite that clear. looking at examples where the bpe and copy models disagree, we observe that the combined inflection model correctly chooses the right solution 72.8% of the time. 5 however, for reinflection and analysis, the correct solution is only chosen slightly more than 50% of the time. considering only those instances where the original bpe and copy models disagree, we investigate the influence of the individual contributions. for inflection, we observe that the combined model produces output identical to the bpe model in 61% of cases, as opposed to only 40% for reinflection and 43% for analysis. it appears that the copied data has an unduly large influence on the combined model for the latter two tasks. doubling drop insertion deletion punct spell total 0 0.25 0.5 0.75 1 bpe copy bpe+copy doubling drop insertion deletion punct spell total avg 0 0.25 0.5 0.75 1 bpe copy bpe+copy doubling drop insertion deletion punct spell total avg (b) error reduction compared to baseline baseline error frequencies 0 0.25 0.5 0.75 1 0 0.25 0.5 0.75 1 doubling drop insertion deletion punct spell total 0 0.25 0.5 0.75 1 bpe copy bpe+copy doubling drop insertion deletion punct spell total avg 0 0.25 0.5 0.75 1 bpe copy bpe+copy doubling drop insertion deletion punct spell total avg (b) error reduction compared to baseline (a) baseline error frequencies ducing the size of input and output segments, we investigate the role that the length of a sequence plays on the quality of a model. since reinflection and analysis lengths vary based not only on the length of the verb being inflected, but on other factors such as the number of words in the input, etc, we limit this investigation to the inflection task. figure 6 demonstrates the number of errors produced by our best system, given the length of the input sequence (ie, the lemma). german, russian, and turkish show a strong preference for shorter input sequences. hebrew (both unvocalized and standard) and spanish instead demonstrate a somewhat surprising preference in the other directionproducing more errors for short sequences. in an attempt to further explain these conflicting results, we next investigate the relationship between lemmas in the training and development sets. figure 7 reports the number of errors made by our best system with respect to the distance between the development lemma and the closest analogue in the training data. now, unsurprisingly, we see that most languages perform better when there is a closely-related lemma in the training data. finally, we investigate the efficiency of our copying method by comparing it with two alternatives. rather than simply taking the training output data and copying it as extra data, random generates random sequences of characters to copy from source to target. similarly, lm creates new copy sequences, but first learns a neural language model from the training data, before generating the sequences. the results for inflection are shown in figure 8 . we observe that changing the hallucination method from copied training data to randomlygenerated sequences greatly improves the quality of the inflector, with an error reduction of more than 35%, on average. we hypothesize that while the copy method simply reinforces an existing signal, the random method introduces new contexts for copying, which allows the model to better generalize the copy operation overall. ## conclusion we have described the submission of the ubc team to the mrl shared task on multilingual clauselevel morphology. experiments on a series of morphologically-diverse languages have demonstrated that bpe and copy-biasing, two methods that have proven successful at the word-level, are largely extensible to clause-level morphology. we observe that the methods are largely complementary, with one exception -the task of reinflection. although we observe notable gains over a vanilla transformer when either performing bpe or copy hallucination, combining the two methods leads to a degradation in reinflection quality.