Exploring Looping Effects in RNN -based Architectures (2020)

Abstract: The paper investigates repetitive loops, a common problem in contemporary text generation (such as machine translation, language modelling, morphological inflection) systems. More specifically, we conduct a study on neural models with recurrent units by explicitly altering their decoder internal state. We use a task of morphological reinflection task as a proxy to study the effects of the changes. Our results show that the probability of the occurrence of repetitive loops is significantly reduced by introduction of an extra neural decoder output. The output should be specifically trained to produce gradually increasing value upon generation of each character of a given sequence. We also explored variations of the technique and found that feeding the extra output back to the decoder amplifies the positive effects.

## introduction over the last few years we witnessed a significant progress in the field of natural language processing (nlp). many state-of-the-art models are based on neural architectures with recurrent units. for instance, @xcite proposed one of the first neural machine translation models that achieved results comparable with statistical models. similarly, @xcite introduced a neural pos tagging model as a new state-of-the-art on the task. recently, neural architectures almost superseded non-neural (finite-state or rule-based) approaches in morphology modelling tasks such as morphological reinflection @xcite with average accuracy being over 90% on high-resource languages. error analysis conducted by @xcite demonstrated that among general misprediction errors such as syncretism, the models also produce certain "silly" errors that human learners do not make. one case of such errors, a looping error, is particularly notable. this type of error is not specific to the task and several other papers reported a similar problem @xcite . still, the causes and the nature of the error remains under-studied. here we provide some insights on the causes of the issues and possible remedy to it. we consider morphological reinflection task for our experiments since it has low time and space requirements and, therefore, allows us to reproduce cases of looping in sufficient quantities and analyse them relatively easy. ## morphological reinflection task morphological inflection is the task of generating a target word form (e.g., "runs") from its lemma ("to run") and a set of target morphosyntactic features (tags, "verb;present tense;singular;3rd person"). the task is called morphological reinflection when the lemma form is replaced with any other form and, optionally, its morphosyntactic features. this is a type of string-to-string transduction problem that in many cases pre-supposes nearly monotonic alignment between the strings. traditionally, researchers either hand-engineered @xcite or used trainable @xcite finite state transducers to solve the task. most recently, neural models were shown to outperform most non-neural systems, especially in the case of high-resource languages @xcite . in terms of the study we focus on two typologically diverse languages, nen @xcite @xcite and russian. nen is a papuan language of the morehead-maro (or yam) family, spoken in the western province of papua new guinea by approximately 400 people. the language is highly under-resourced, and @xcite is the only computational work on it we are aware of, and in current study we use the data derived from their corpus. russian, a slavic language from indo-european family, on the other hand, is considered as high-resource. we use the splits from the sigmorphon-conll 2017 shared task on morphological reinflection @xcite . we used medium sized training sets which occurred to yield highest rates of looped sequences in predicted word forms. the number of samples in the datasets are presented in ## experiments we reused the hard attention model specifically designed for the morphological reinflection task (aharoni and goldberg, 2017) for our explorations. the model uses an external aligner @xcite to extract input-to-output character sequence transformation steps for a given morphological sample. instances of a special character (step) are inserted into transformed words to represent alignment step advances. the resulting seq2seq model is trained to perform transformation from a given lemma into a target inflected form which contains step characters. the model consists of two modules; (1) an array of lstm @xcite encoders and (2) an lstm decoder. when a step character occurs in a target sequence (either learnt or predicted), the encoder array index advances to the next position. it corresponds to advancing current pointer in the lemma by one character. in such a way, a hard monotonic attention schema is implemented. in our experiments we computed counts of looped sequences in generated word forms during model evaluation rounds that were carried out upon each epoch of model training. we distinguished a generated character sequence as looped if it satisfies both of the following conditions: (1) the sequence contains at least 3 repeated instances of some character subsequence at its very end, and (2) the total length of those repeated subsequences reaches at least 8 characters. while applying such a criterion, we considered predicted sequences in their alphabetical form, with all step characters stripped off. 1 we hypothesized that the looping is primarily caused by merging of decoder states relevant to different word positions. therefore, introduction of variables that are guaranteed to be different at distinct stages of output word form production should reduce looped prediction rate. presence of such a variable would facilitate distinguishing states that correspond to different parts of generated word, if even closely surrounding character sequences are similar. to implement this idea, we introduced an extra decoder output that is trained to always be increasing while new output characters are produced. more specifically, we added an extra output r and an extra input r to the decoder. to ensure that r increases gradually while target word characters are generated, we modified calculation of total loss in the model training, allowing an extra (hinge-like) term as follows: here ∆r is the difference between current and previous r values. initially, for every predicted word form r is set to zero. having observed the dynamics of r value in preliminary training experiments, we chose γ = 50; @xmath0.05. for better exploration of different factors, we tested combinations of the following setting variations: we repeated experiments 15 times for each distinct setting. the result figures presented are normalized to single experiment. denotation goal for r r value n ("none") none zero i ("increment") r is ablated incrementing f ("feedback") none previous r u ("unused") increase zero s ("all set") increase previous r note: if r is a vector, its size is added before a mode symbol: '3f', '3u', '3s'. ## results the plots given in fig. 1 present counts of looped predictions at different epochs for the two datasets used (nen and russian). 2 it can be observed that 2 the curves shown at fig. 1 , 2 are generated by a polynomial smoothing procedure from a dataset with high variance. they may expose some irrelevant artifacts, for example, they fall to negative count values at some points. training a model with increasing r (modes 's', '3s') demonstrates significantly lower rates of looped word generation compared to the baseline mode ('n'). this is true for almost all considered epochs. one may also note that the 'u' mode yields results comparable to ones obtained with the 's' mode. this fact means that the presence of gradually incrementing decoder output is helpful for fighting looping even when the output isn't used. however, if the output is free of constraints and is fed back to the decoder (mode 'f'), the effect is mostly negative. fig. 1 demonstrates the results of the same kind for the modes that occur to be less looping-prone than the baseline mode. when its components weren't trained to gradually increase (mode '3f'), a vector of 3 feedback values drastically increased looping rate at all epochs. if a vector of 3 increasing components was produced but wasn't fed back as input, the results were still negative. this is surprising because the result for a respective scalar mode ('u') is positive. increasing the dimensionality of extra decoder output sometimes yields an improvement ('3s' mode) but generally the results suggest that vector size is a factor causing looping rate increase. finally, scalar seems to be more preferable than vector. ## discussion we have found a strong evidence that the presence of a decoder output which is trained to progressively increment reduces the average rate of looping sequences in multiple times. in most cases the positive effect is more significant if this output is fed back to the decoder, although there are exceptions of minor magnitude. attempts to scale the effect further by increasing dimensionality of progressively incrementing variables are sometimes successful. still, if we consider an average explored case, the mode 's' seems to be the most effective and consistent in fighting looping. we also observed that presence of an auto-incremented decoder input (mode 'i') leads to looping rate reduction, but the effect is superior if the decoder itself serves to produce a gradually increasing value. thus, the practical recommendation arising from our research should be (1) adding an extra scalar output to the decoder, (2) endorsing it to increase by inclusion a respective term into a training loss formula, and (3) feeding it back as an encoder input. conceptually, it isn't surprising that the presence of an increasing variable helps the decoder to distinguish states rated to different phases of output word production and such a way reduces probability of falling into a loop. still, the details of this mechanism yet need exploration. in our current work we made no attempt to enforce the usage of the new variable in any way; we only made such a usage potentially possible. a detailed exploration of its effect on the learning process is yet a subject of further research. and, what is even more practically important, we yet need to find how the system design may be changed to incorporate progressive variables in a more explicit, controllable and efficient way. the introduction of feedback variables adds elements of rnn architecture to the decoder. we observed highly negative results when such variable values weren't constrained (modes 'f' and, especially, '3f'). this indirectly suggests that rnn schema may not be a good solution for a decoder in terms of looping prevention. ## conclusion we proposed and explored a simple technique that reduces rate of repetitive loops occurrence in a neural decoder output. our work was inspired by a hypothesis that looping effects in a neural decoder are caused by its inability to distinguish states related to different positions in a generated word. we both provided a simple and universal practical solution and outlined a promising direction for further research.