Recurrent Inference in Text Editing (2020)

Abstract: In neural text editing, prevalent sequence-to-sequence based approaches directly map the unedited text either to the edited text or the editing operations, in which the performance is degraded by the limited source text encoding and long, varying decoding steps. To address this problem, we propose a new inference method, Recurrence, that iteratively performs editing actions, significantly narrowing the problem space. In each iteration, encoding the partially edited text, Recurrence decodes the latent representation, generates an action of short, fixed-length, and applies the action to complete a single edit. For a comprehensive comparison, we introduce three types of text editing tasks: Arithmetic Operators Restoration (AOR), Arithmetic Equation Simplification (AES), Arithmetic Equation Correction (AEC). Extensive experiments on these tasks with varying difficulties demonstrate that Recurrence achieves improvements over conventional inference methods.

## introduction for text editing, the sequence-to-sequence (seq2seq) framework has been applied to text simplification @xcite , punctuation restoration @xcite , grammatical error correction @xcite @xcite , machine translation post-editing @xcite , and etc. we observe that current inference methods can be roughly grouped into two categories: end-to-end (end2end) @xcite @xcite and tagging @xcite @xcite @xcite . for models from both categories, the encoders extract and encode information from the source text sequence. yet, the goal of the decoders is different for end2end and tagging. upon receiving the encoder's hidden states that comprise the source text information, the decoder of end2end directly decodes the hidden states and generates the completely edited target text sequence. but, the decoder of tagging produces a sequence of editing operations, such as deletion and insertion, that is later applied to the source text to yield the edited text via a realization step @xcite . the mechanisms of end2end and tagging are illustrated in figure 1 . however, both end2end and tagging are problematic because as decoding progresses, the divergence between the partially edited text and the original text grows, rendering the encoder hidden states less and less helpful for decoding the edited text or editing operations toward the end of the editing process; and as the number of decoding steps increases with edited text length, decoding the completely edited text or the full editing operation sequence becomes more and more demanding. to tackle the aforementioned issues, we propose a recurrent inference method, recurrence, for textediting with the encoder-decoder framework. recurrence consists of two components as illustrated in figure 1 : (i) an encoder-decoder model, namely the programmer; (ii) an interpreter. for a given source sequence, the programmer determines an editing action that consists of an editing operation with the tokens it needs and the position in the source sequence to apply the operation. after the interpreter executes the editing action, the partially edited text is again fed to the programmer to determine the next appropriate editing action. this process repeats until the programmer decides that no further editing is needed. intuitively, recurrence is advantageous because (i) as a novel recurrent inference process, it is not constrained by model structures and generally applicable; (ii) the programmer only produces one single editing step, easing the learning difficulty; (iii) the encoder hidden states are updated for each decoding step, providing faithful latent representations; (iv) the decoder outputs an editing action of fixed sequence length, alleviating the problem caused by long decoding steps. empirically, through three text editing tasks, namely arithmetic operators restoration (aor), arithmetic equation simplification (aes) and arithmetic equation correction (aec), we show that recurrence is dataefficient and more resilient to the text sequence length and the vocabulary size. our contributions are the followings: (1) we demonstrate that many text editing tasks can be solved by multiple inference steps recurrently; (2) we propose a novel recurrent inference method, recurrence, for text editing that tears an editing task down into iterations of editing actions; (3) we design three easily reproducible, proof-of-concept text editing tasks, aor, aes and aec; (4) we exhibit that recurrence outperforms end2end and tagging in all three text editing tasks and is (i) less sensitive to longer sequences; (ii) less sensitive to larger vocab sizes; (iii) less data-hungry to achieve superior or competitive performances. the code for three inference methods, text editing tasks, data generation, and experiments in this work is available at: https://github.com/ shininglab/recurrent-text-editing . ## related work text editing is an natural language processing (nlp) task in that systems change texts by inserting, deleting and rephrasing the words to meet certain needs. according to the length relationship between input and output texts, we summarize text editing tasks into three types: short-to-long, longto-short, and mixed. end-to-end is one of the early methods to perform text editing by casting the job as seq2seq @xcite text generation. without complicated preparation and subsequent processing, end2end has been proven to accomplish text editing well, in all three types @xcite @xcite @xcite . yet, conventional seq2seq-based approaches are well-known for their drawbacks, including dependency on large amounts of data, unexplainable processes, and uncontrollable outcomes @xcite . when texts do not need a complete modification, there are more appropriate methods than learning a direct mapping from unedited texts to edited texts. tagging solves text editing in two steps instead. it firstly employs a seq2seq framework to produce tag sequences, and secondly, edits input texts according to the tag sequences (the "realization" step) @xcite . tagging assigns the tag keep for words that do not need to be changed so that it does not need to learn a copy mechanism. some have reported that tagging is better than end2end in short-to-long @xcite , longto-short @xcite @xcite , and mixed editing @xcite @xcite . one notable member of the tagging family is neural programmer-interpreter (npi), a recurrent and compositional neural network @xcite . npi is adopted in text editing to predict tags, such as keep, delete, and insert, and execute operations during decoding simultaneously. npi-based methods have achieved state-of-the-art results in long-to-short @xcite , and mixed editing @xcite . nevertheless, like other tagging methods, npi's encoder hidden states are not updated during editing. its decoder considers operations and executions from previous time steps to predict the current operation while putting massive pressure on the decoder @xcite @xcite . also, tagging in general suffers from a performance decline caused by a large vocabulary that combines tags and words or too many decoding steps to assign tags. to resolve the aforementioned problems with tagging, in recurrence, we update the encoder hidden states iteratively and free the interpreter from the decoder to complete text editing in several program-interpret iterations (recurrent inference). npi belongs to neural program induction @xcite , but recurrence is part of neural program synthesis @xcite . consequently, recurrence always follows the latest hidden representation of its input text rather than a static context matrix and only needs to decode an editing action of a fixed length in each iteration. ## tasks an increasing number of studies takes synthetic benchmark tasks to examine ideas before extending to open-domain natural language data @xcite @xcite . following the fruitful results of previous work, we aim to evaluate three inference methods in the domain of arithmetic problems @xcite @xcite that can be treated as the test-beds for text editing. we introduce three tasks, namely aor, aes, and aec, corresponding to the three types of text editing tasks: short-to-long, long-to-short, and mixed. being able to control the aspects of the datasets allows us to compare the characteristics of the three inference methods more thoroughly and analyze the appropriate situations to apply each method. ## experiments we test recurrence in comparison with end2end and tagging across aor, aes, and aec. we describe the results conditioned on specific n , l, and d. later, we analyze the impact of each of them in section 6. data. in all tasks, the dataset is divided into three subsets: 70% for training, 15% for validation, and 15% for testing. for aes (many-to-one learning) and aec (many-to-many learning), we feed the training set to a data generator in every epoch to expose all the variants of targets as input sequences (see section 4). for the sake of fairness, we examine three methods in both online and offline training modes. to train end2end online and tagging online , in each epoch, we keep the targets, but uniformly pick a partially edited y (i) to alternate the original input x as the source sequence. the target equations can be used to train end2end directly. by contrast, further pre-processing is necessary for tagging and recurrence. training targets for tagging are tag sequences, while those for recurrence are editing actions. models. after testing transformer @xcite and a range of modern rnns @xcite @xcite , we focus on the overall best-performed architecture -bidirectional lstm @xcite with an attention mechanism @xcite . throughout all the experiments, three inference methods share the same model structure with d @xmath0, d @xmath1, n @xmath2, r @xmath3, r teacher @xmath4.5, and r @xmath5.5 @xcite . parameters are uniformly initialized to prevent uncontrolled interference, we train all models from scratch instead of pre-training. we use adam optimizer @xcite with an l2 gradient clipping of 5.0 @xcite . evaluation. we evaluate methods by three metrics: token accuracy, sequence accuracy, and equation accuracy. token accuracy marks the correct predictions at the token-level divided by the target sequence length and then averaged by the test size. sequence accuracy stands for the correct predictions at the sequence-level divided by the test size. equation accuracy is the number of true predicted equations divided by the test size; it emphasizes on whether an equation holds rather than whether an equation is the same as the target. we evaluate the performance via equation accuracy for aor (one-to-many), sequence accuracy for aes (manyto-one), and both equation accuracy and sequence accuracy for aec (many-to-many). sequence accuracy is accompanied by token accuracy for additional reference. training. we train on a single geforce rtx titan with a batch size of 256. the last batch is dropped if it does not contain 256 samples. to ensure convergence, we adopt early stopping @xcite with a patience of 512 epochs. ## analysis as shown in section 5, recurrence outperforms end2end and tagging in all three tasks in our experiment settings. in this section, we explore the limits of recurrence by running experiments with varying values of n , l and d, so as to determine in what scenario recurrence performs well (see figure 4 ). the impact of n . we conduct experiments with @xmath6, @xmath7, and n increasing from 10 to 50 with an interval of 10 for aor; @xmath8, @xmath9, and n increasing from 100 to 300 with an interval of 50 for aes; and @xmath10, @xmath11, and n increasing from 10 to 50 with an interval of 10 for aec. for aor, recurrence online and tagging show similar resilience, however, tagging offline performs better when @xmath12. for aes, recurrence online performs much better than tagging and end2end (by at least 20%) when @xmath13. note that end2end performs bad when @xmath14. we also observe that end2end offline can achieve a near-perfect performance when @xmath15. these results indicate that the end2end offline 's performance declines rapidly as n increases and re-quires a much larger d-to-n ratio to perform well. finally, for aec, recurrence online displays the most resilience and performs the best. the impact of l. we conduct experiments with @xmath16, @xmath17, and l increasing from 5 to 9 with an interval of 1 for aor; @xmath18, @xmath19, and l increasing from 3 to 7 with an interval of 1 for aes; and settings identical to aor for aec. for aor and aes, both recurrence online and tagging show similar trend, however, recurrence online performs the best. for aec, while recurrence online still outperforms tagging, end2end offline performs the best for @xmath20. we conduct experiments with @xmath21, @xmath22, and d increasing from 10k to 50k with an interval of 10k for aor; @xmath23, @xmath24, and d increasing from 10k to 50k with an interval of 10k for aes; and settings identical to aor for aec. all models benefit from the increasing of d as expected. however, it is clear that recurrence online is the best performing model when d is small. the only exception is that for aec, end2end has similar performance trend as recurrence. as discussed before, this is likely because end2end performs well with small n . the impact of online training. when comparing the performance between online and offline training, the online training, as expected, generally has better performances than offline training for end2end and tagging with only a few exceptions. note that online training is not part of the standard training procedure for end2end and tagging, however, we use online training with end2end and tagging for the sake of a fair comparison. therefore, for end2end and tagging, the online training acts like a data augmentation technique, providing more data points for training. surprisingly, offline training also allows recurrence to gain some editing ability, at times better than end2end and tagging. we believe for text editing tasks with very localized editing actions, such as aes, showing the immediate editing actions are enough for the model to generalize proper editing actions. in other words, when the editing actions are less sequentially dependent, even offline training enables recurrence to achieve performance better than end2end and tagging. this supports our intuition that letting the programmer produce one single editing step reduces the learning difficulty. the impact of ordering. in early experiments, we find that the programmer cannot converge if the data guide it to edit a sequence in a random order (a mixture of both left-to-right and right-to-left). hence, we think ordering matters for not only text generation @xcite but also recurrence in text editing. one of our assumptions is that random ordering may assign various actions to the same text state, and thus causes confusion in the list of actions used to edit the input text x to the output text y. when there are conflicting sample pairs in the training data set, the model cannot easily converge. we leave this problem for future work. to summarize our findings, under settings with moderate or large n and l, end2end performs much worse than tagging and recurrence with limited data. tagging performs slightly better than recurrence when n gets larger with fixed d and l in aor (short-to-long). however, tagging performs worse than recurrence in all other cases. therefore, we conclude that recurrence is more data-efficient and overall better performs than end2end and tagging in most situations, especially in aes (long-to-short). ## conclusions and future work we propose a recurrent inference method, recurrence, that edits a given text sequence iteratively such that in each iteration the programmer determines a single step of editing action and the interpreter executes the action. our method outperforms the other two inference methods, end2end and tagging, in three arithmetic equation editing tasks we introduced. for future work, we plan to apply recurrence to open-domain natural language data and investigate on how to relax its need for intermediate editing steps as extra supervision signals. we also wish to experiment with applying pointer attention @xcite to replace the position component in actions.