Text Editing as Imitation Game (2022)

Abstract: Text editing, such as grammatical error correction, arises naturally from imperfect textual data. Recent works frame text editing as a multi-round sequence tagging task, where operations – such as insertion and substitution – are represented as a sequence of tags. While achieving good results, this encoding is limited in flexibility as all actions are bound to token-level tags. In this work, we reformulate text editing as an imitation game using behavioral cloning. Specifically, we convert conventional sequence-to-sequence data into state-to-action demonstrations, where the action space can be as flexible as needed. Instead of generating the actions one at a time, we introduce a dual decoders structure to parallel the decoding while retaining the dependencies between action tokens, coupled with trajectory augmentation to alleviate the distribution shift that imitation learning often suffers. In experiments on a suite of Arithmetic Equation benchmarks, our model consistently outperforms the autoregressive baselines in terms of performance, efficiency, and robustness. We hope our findings will shed light on future studies in reinforcement learning applying sequence-level action generation to natural language processing.

## introduction text editing @xcite is an important domain of processing tasks to edit the text in a localized fashion, applying to text simplification @xcite , grammatical error correction @xcite , punctuation restoration @xcite , to name a few. neural sequence-tosequence (seq2seq) framework @xcite establishes itself as the primary approach to text editing tasks, by framing the problem as machine translation @xcite . applying a seq2seq modeling has the advantage of simplic- figure 1 : three approaches -sequence tagging (left), end-to-end (middle), sequence generation (right) -to turn an invalid arithmetic expression "1 1 2" into a valid one "1 + 1 = 2". in end-to-end, the entire string "1 1 2" is encoded into a latent state, which the string "1 + 1 = 2" is generated directly. in sequence tagging, a localized action (such as "insert_+", meaning insert a "+" symbol after this token) is applied/tagged to each token; these token-level actions are then executed, modifying the input string. in contrast, sequence generation output an entire action sequence, generating the location (rather than tagging it), and the action sequence is executed, modifying the input string. both token-level actions and sequence-level actions can be applied multiple times to polish the text further (up to a fixed point). in contrast, alternative approaches @xcite train the agent to explicitly generate freeform edit actions and iteratively reconstructs the text during the interaction with an environment capable of altering the text based on these actions. this sequence-level action generation @xcite @xcite allows higher flexibility of action design not limited to token-level actions, and is more advantageous given the narrowed problem space and dynamic context in the edit @xcite . the mechanisms of sequence tagging and sequence generation against end-to-end are exemplified in figure 1 . both methods allow multiple rounds of sequence refinement @xcite and imitation learning (il) @xcite . essentially an agent learns from the demonstrations of an expert policy and later imitates the memorized behavior to act independently @xcite . on the one hand, il in sequence tagging functions as a standard supervised learning in its nature and thus has attracted significant interest and been widely used recently @xcite @xcite , achieving good results in the token-level action generation setting @xcite . on the other hand, il in sequence-level action generation is less well defined even though its principle has been followed in text editing @xcite and many others @xcite . as a major obstacle, the training is on state-action demonstrations, where the encoding of the states and actions can be very different @xcite . for instance, the mismatch of the lengths dimension between the state and action makes it tricky to implement for an auto-regressive modeling that benefits from a single, uniform representation. to tackle the issues above, we reformulate text editing as an imitation game controlled by a markov decision process (mdp). to begin with, we define the input sequence as the initial state, the required operations as action sequences, and the output target sequence as the goal state. a learning agent needs to imitate an expert policy, respond to seen states with actions, and interact with the environment until the success of the eventual editing. to convert existing input-output data into stateaction pairs, we utilize trajectory generation (tg), a skill to leverage dynamic programming (dp) for an efficient search of the minimum operations given a predefined edit metric. we backtrace explored editing paths and automatically express operations as action sequences. regarding the length misalignment, we first take advantage of the flexibility at the sequence-level to fix actions to be of the same length. secondly, we employ a linear layer after the encoder to transform the length dimension of the context matrix into the action length. by that, we introduce a dual decoders (d2) structure that not only parallels the decoding but also retains capturing interdependencies among action tokens. taking a further step, we propose trajectory augmentation (ta) as a solution to the distribution shift problem most il suffers @xcite . through a suite of three arithmetic equation (ae) benchmarks @xcite , namely arithmetic operators restoration (aor), arithmetic equation simplification (aes), and arithmetic equation correction (aec), we confirm the superiority of our learning paradigm. in particular, d2 consistently exceeds standard autoregressive models from performance, efficiency, and robustness perspectives. in theory, our methods also apply to other imitation learning scenarios where a reward function exists to further promote the agent. in this work, we primarily focus on a proof-of-concept of our learning paradigm landing at supervised behavior cloning (bc) in the context of text editing. to this end, our contributions foot_0 are as follows: 1. we frame text editing into an imitation game formally defined as an mdp, allowing the highest degrees of flexibility to design actions at the sequence-level. 2. we involve tg to translate input-output data to state-action demonstrations for il. 3. we introduce d2, a novel non-autoregressive decoder, boosting the learning in terms of accuracy, efficiency, and robustness. 4. we propose a corresponding ta technique to mitigate distribution shift il often suffers. ## imitation game we aim to cast text editing into an imitation game by defining the task as a recurrent sequence generation, as presented in figure 2 (a). in this section, we describe the major components of our proposal, including (1) the problem definition, (2) the data translation, (3) the model structure, and (4) a solution to the distribution shift. using the same example, (b) explains how to achieve shifted state s 2 by skipping action a * 1 and doing a 2 . here we update a * 2 to a 2 accordingly due to the previous skipping. the new state s 2 was not in the expert demonstrations. ## experiments we adapt recurrent inference to our paradigm and evaluate them across ae benchmarks. ## analysis we conduct extensive sensitivity analyses to better illustrate and understand our methods. ## conclusion we reformulate text editing as an imitation game defined by an mdp to allow action design at the sequence-level. we propose d2, a nonautoregressive decoder for state-action learning, coupled with tg for data translation and ta for distribution shift alleviation. achievements on ae benchmarks evidence the advantages of our methods in performance, efficiency, and robustness. sequence-level actions are arguably more controllable, interpretable, and similar to human behavior. turning tasks into games that agents feel more comfortable with sheds light on future studies in the direction of reinforcement learning in the application of text editing. the involvement of a reward function, the optimization of the trajectories, the design of sequence-level actions, and their applications in more practical tasks, to name a few, are interesting for future work. suggesting text editing as a new testbed, we hope our findings will shed light on future studies in reinforcement learning applying to natural language processing.