Data-to-text Generation by Splicing Together Nearest Neighbors (2021)

Abstract: We propose to tackle data-to-text generation tasks by directly splicing together retrieved segments of text from “neighbor” source-target pairs. Unlike recent work that conditions on retrieved neighbors but generates text token-by-token, left-to-right, we learn a policy that directly manipulates segments of neighbor text, by inserting or replacing them in partially constructed generations. Standard techniques for training such a policy require an oracle derivation for each generation, and we prove that finding the shortest such derivation can be reduced to parsing under a particular weighted context-free grammar. We find that policies learned in this way perform on par with strong baselines in terms of automatic and human evaluation, but allow for more interpretable and controllable generation.

## introduction there has been recent interest in text generation systems that make use of retrieved "neighbors" -examples of good text retrieved from a database, perhaps paired with the source information on which these example texts condition -on the hope that these neighbors might make a generation task easier, or the system more interpretable or controllable @xcite @xcite @xcite . whereas most work along these lines has adopted a conventional encoder-decoder approach, conditioning on the retrieved neighbors and then autoregressively generating text token-by-token from left to right, we instead propose to generate text by directly splicing together segments of text from retrieved neighbors. generating in this way aligns with the intuition that in settings such as data-totext generation it ought to be sufficient to retrieve sentences similar to the one that must be generated, and then merely change some details, such as names or dates. there are notable advantages to a generation-bysplicing approach. first, generation becomes more interpretable: it is always clear from which neighbor a particular piece of generated text derives, and it is also clear how these pieces have come together to form the generated text. generation-by-splicing may also increase our control over the generated text, and we suspect that approaches that make clear the provenance of each piece of generated text (as ours does) will be useful in preventing text generation systems from emitting harmful or biased text @xcite @xcite . that is, we might imagine preventing systems from emitting harmful or biased text by only allowing generation from approved neighbor examples. methodologically, we implement this generationby-splicing approach by training a policy to directly insert or replace spans of neighbor text at arbitrary positions within a partially constructed generation, and we define a generalized insert function capable of such manipulations in section 3. we train this policy with "teacher forcing" @xcite , which requires, for each training example, an oracle sequence of insert actions that derive it. accordingly, we define a shortest sequence of actions deriving a training generation from its neighbors to be an oracle one, and we prove that, given some neighbors, an oracle sequence of actions can be obtained by parsing under a particular weighted context-free grammar, introduced in section 3.1.1. empirically, we find our proposed approach yields text of comparable quality to strong baselines under automatic metrics and human evaluation on the e2e dataset @xcite and wikibio datasets @xcite , but with added interpretability and controllability (see section 5.2). our reduction of minimum-insertion generation to wcfg parsing may also be of independent interest. our code is available at https://github.com/swiseman/ neighbor-splicing . ## background and notation conditional text generation tasks involve generating a sequence of tokens ŷ1 , . . . , ŷt = ŷ1:t conditioned on some @xmath0, where each generated token ŷt is from a vocabulary v. we will consider in particular the task of table-to-text generation, where x is some tabular data and ŷ1:t is a natural language description of it. for supervision, we will assume we have access to a dataset, which pairs an input x with a true corresponding reference text y 1:@xmath1. since we are interested in nearest neighbor-based generation, we will also assume that along with each input x we have a set @xmath2:tn } n @xmath3, with each ν (n) @xmath4. we will be interested in learning to form y 1:tx from its corresponding x and neighbor set n in a way that will be made more precise below. we note that finding an appropriate set of neighbor sequences to allow for successful generation with respect to an input x is an interesting and challenging problem (see, e.g., hashimoto et al. 2018 ), but for the purposes of our exposition we will assume these neighbor sequences are easy to obtain given only x (and without knowledge of y). we give the details of our simple retrieval approach in section 5. ## splicing nearest neighbors given a canvas ŷ1:@xmath5:tn } n @xmath6, we define a generalized insertion function, which forms a new canvas from ŷ1:m . this generalized insertion function implements the following mapping name: ayelet nahmias-verbin | dob: 19 june 1970 | birthplace: ... <@xmath7. </@xmath8, who served ... </@xmath9. </@xmath10. </@xmath11. </@xmath12. </@xmath13, ν (1) , ν (2) ; ν (0) is from the corresponding table. bottom right: a sequence of insert operations (see equation ( 1 )) deriving the sentence from the neighbors above. left: the parse of the target sentence under the grammar in section 3.1.1 corresponding to the derivation on the bottom right. intuitively, this generalized insertion function attempts to capture a generation scheme where text is generated by making only minor insertions or replacements in some existing text. for example, we might imagine generating a new sentence by copying a neighbor sentence to our canvas, and then simply replacing the names or dates in this neighbor sentence to form a new sentence; see figure 1 for an example. having defined this insertion function, we can generate text with a policy that consumes an input x, a set of neighbors, and a canvas, 2 and outputs the arguments of the insert function, or else the stop action. thus, for a given canvas and neighbor set, we take our policy to produce actions in or else the stop action. we show examples of generating with such a policy in figure 1 and figure 3 . ## models, training, and generation to avoid directly parameterizing a distribution over the impractically large number of combinations of arguments to the insert function, we factorize the distribution over its arguments as more concretely, we obtain token-level representations x 1 , . . . , x s and ŷ0 , . . . , ŷm+1 , all in r d , of source sequence @xmath14:s and of canvas sequence ŷ1:m , padded on each side with a special token, by feeding them to an encoder-decoder style transformer @xcite with no causal masking. we obtain neighbor token representations ν (n) k by feeding neighbor sequences through the same encoder transformer that consumes x. we provide additional architectural details in appendix d. viewing the source sequence x as the 0th neighbor, we then define while the above holds for full policies, lrt policies always insert after the most recently inserted token, and so they require only a π θ (n, k | i) policy, and only w 0 and w 2 transformations. ## experiments our experiments are designed to test the quality of the text produced under full policies and lrt policies, as well as whether such policies allow for more controllable or interpretable generation. ## related work nlp systems have incorporated neighbors for decades. early work focused on machine translation @xcite , syntactic disambiguation @xcite , and tagging @xcite . while some more recent work has made use of retrieved neighbors for problems such as sequence labeling @xcite , auditing multi-label text classification predictions @xcite , and reasoning over knowledge bases @xcite , the majority of re-cent nlp work involving neighbor-based methods has focused on conditioning neural text generation systems on retrieved neighbors. this conditioning is variously accomplished using a conventional encoder in an encoder-decoder setup @xcite @xcite @xcite , by allowing the parameters of the decoder to depend on the retrieved neighbor @xcite , or by viewing the unknown neighbor as a latent variable @xcite @xcite . recent work @xcite @xcite has also used retrieved neighbors at decoding time to modify the next-token distribution of the decoder. our work differs from these approaches in that we explicitly parameterize the splicing operations that form a generation from neighbors, rather than conditioning or otherwise modifying a left-to-right token generation model using retrieved neighbors. our parameterization is motivated by trying to increase the interpretability and controllability of the generation process, which also motivates recent work making explicit the template or plan being followed by the generation @xcite @xcite @xcite . this more structural or syntactic flavor of controllability differs slightly from foundational work on controlling content or stylistic attributes of text @xcite @xcite . our approach is also related to work in non-left-to-right text generation, including treebased @xcite , non-autoregressive @xcite , masked language modelbased @xcite , and, most closely, insertion-based @xcite ,a, inter alia) approaches. our work differs from this last category in several important respects: first, we insert and replace (and model) full spans rather than tokens. our policies are trained to minimize the number of insertion operations rather than to insert (centrally positioned) correct tokens in available slots, as is insertion transformer @xcite , or to mimic a levenshtein distance-based oracle, as is levt @xcite . our policies are also fundamentally sequential, unlike these partially autoregressive alternatives, which can generate tokens in parallel. the sequential nature of our approach makes us3: a visualization of how two randomly selected model generations from the wikibio validation setkyle johansen (born july 13, 1967) is an american politician. and l. yves fortier (born september 11, 1935) is a former canadian ambassador the the united nations from 1988 to 1991. -are derived. in both cases only one non-table neighbor is used, and is depicted in the top-most solid box (in blue). the solid box second from the top (in orange) represents the linearized source table, ν (0) , and the third (in green) represents all word types in the vocabulary. derivations should be read top-down; the current canvas is represented by a dotted box, carets indicate insertion, struck through text has been replaced, and straight arrows indicate the provenance of a span. beginningand end-of-sequence tokens are omitted. our work also relates to recent work on sentencelevel transduction tasks, like grammatical error correction (gec), which allows for directly predicting certain span-level edits @xcite . these edits are different from our insertion operations, requiring token-level operations except when copying from the source sentence, and are obtained, following a long line of work in gec @xcite @xcite , by heuristically merging token-level alignments obtained with a damerau-levenshtein-style algorithm @xcite . ## conclusion we have presented an approach to data-to-text generation, which directly splices together retrieved neighbors. we believe this line of work holds promise for improved interpretability and controllability of text generation systems. in future work we hope to tackle more ambitious text generation tasks, which will likely require retrieving many more neighbors, perhaps dynamically, from larger data-stores, and with more sophisticated retrieval techniques, such as those currently being used in retrieval-based pretraining @xcite . we also hope to consider more sophisticated models, which explicitly capture the history of produced canvases, and more sophisticated training approaches, which search for optimal insertions while training, rather than as a preprocessing step @xcite .