Twist Decoding: Diverse Generators Guide Each Other (2022)

Abstract: Many language generation models are now available for a wide range of generation tasks, including machine translation and summarization. Combining such diverse models may lead to further progress, but ensembling generation models is challenging during inference: conventional ensembling methods (e.g., shallow fusion) require that the models share vocabulary/tokenization schemes. We introduce Twist decoding, a simple and general text generation algorithm that benefits from diverse models at inference time. Our method does not assume the vocabulary, tokenization or even generation order is shared. Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available. Twist decoding also consistently outperforms the popular reranking heuristic where output candidates from one model are rescored by another. We hope that our work will encourage researchers and practitioners to examine generation models collectively, not just independently, and to seek out models with complementary strengths to the currently available models.

## introduction natural language generation is an important building block for many applications, such as machine translation, summarization, and question answering @xcite @xcite @xcite . researchers have recently explored and advanced models for generation in various aspects, in-figure 1 : twist decoding of two generation models, f and g, that does not assume a shared vocabulary, tokenization, or generation order. beam search is first applied to f to generate y (0) , followed by output mapping to ˜︁ y (0) (e.g., f 's detokenization and g's tokenization or sequence reversal). g is then decoded with beam search augmented with distances from the set of previouslygenerated outputs (here only one sequence y is shown): one well-established ensembling technique is "shallow fusion" @xcite @xcite , which aggregates models' scores during beam search. this approach requires, however, that the models use the same vocabulary/tokenization scheme and organize the search in the same way (e.g., autoregressive, left-to-right factorization). we introduce a new inference algorithm, twist decoding (fig. 1 ), that enables more diverse generators to guide each other. twist decoding can combine generators with different vocabularies, (de)tokenization, and even generation order without any additional training or finetuning. our method decodes a model by standard beam search, but the scores at every step incorporate a simple function that measures the distance from outputs of the other model. we run this procedure on each generation model in turn, so that both can benefit from each other. we present extensive experiments on machine translation and scientific paper summarization and show that twist decoding can improve performance over each model decoded in isolation across several scenarios: combining 1) generic and domain-specific models, 2) left-to-right and right-to-left generation models, and 3) models that generate using different conditioning inputs. our results show consistent performance gains from combining generic and domain-specific translation models over a wide range of domains, including medical and legal translation. applications in these domains require particularly high accuracy, and twist decoding is a desirable alternative to standard beam search on a single model. interestingly, we find that twist decoding between generic and domain models is effective even when parallel data from the domain are scarce and the domain model yields poor performance by itself, suggesting complementary strengths of diverse generators ( §3.4). twist decoding can be seen as a generalization of reranking heuristics that have proven effective in syntactic parsing @xcite ; char-twist decoding g generates z with guidance from f at iteration t k: beam size. m : maximum length. vg: vocabulary of g. g(•): scoring function. y (t-1) : set of output sequences from f . z (t) : new outputs. bn: beam of continuing sequences. h: expanded hypotheses before beam selection. d(•, •) : distance between partial sequences. λ f : scalar coefficient for the distance. figure 2: twist decoding when g is guided by f . swap f and g and y and z to obtain y (t) . map_output converts outputs from f to g; e.g., f 's detokenization followed by g's tokenization. it would also include sequence reversal if f or g is a right-to-left model. the highlighted line is the only modification that twist decoding introduces to standard beam search. the input sequence to g is omitted. see also @xcite for the stopping criterion and implementation details (the first come, first served heuristic). @xcite , speech recognition @xcite , and machine translation @xcite : one model generates candidate sequences, followed by rescoring from another model. we present extensive comparisons with reranking baselines and demonstrate that twist decoding outperforms reranking consistently. we also observe that since the encoder computations on two models can be parallelized, the inference time required for twist decoding is much shorter than the sum of the two models, resulting only in a 50% increase, relative to decoding of a single model in isolation ( §4). twist decoding is therefore a viable alternative to standard beam search on a single model and the widespread reranking heuristic. algorithm is a simple modification in standard beam search (highlighted in fig. 2 ); we incorporate into a scoring function the distance from outputs that are previously generated by another model. ## experiments we present experiments across three scenarios: combining domain and generic models for machine translation ( §3.1), left-to-right and right-to-left machine translation models ( §3.2), and scientific paper summarization models that take as input different parts of the paper ( §3.3). we empirically compare twist decoding with decoding in isolation and the widely-adopted reranking baselines, illustrating that twist decoding offers performance improvements in various situations without any change to the trained models. ## analysis iterations. so far, we have only applied one iteration of twist decoding, but fig. 4 plots performance over multiple iterations. iteration 0 signifies f 's initial decoding (y (0) in fig. 1 ), and every iteration involves g's decoding with f 's guidance (z inference time. figure 4 : effects of iterations on dev. performance. iteration 0 refers to the initial decoding from f . every iteration consists of g's decoding with f 's guidance followed by f 's decoding with g's guidance. the values of λs are kept the same over all iterations for simplicity. initially, we explored gradually increasing the λs as f and g's outputs become closer, but we found no substantial performance gain. ## conclusion we presented twist decoding, a general inference algorithm that generates text from diverse models without the assumption of a shared vocabulary, tokenization, or generation order. our method enables diverse models to guide each other, thereby outperforming individual models over various scenarios, even when one of the models is much weaker because of limited data. we also demonstrated that twist decoding can be viewed as a generalization and improvement of the commonly-adopted reranking heuristic. as it only requires a small change in code, we hope that researchers and practitioners will explore complementary strengths of diverse generation models through twist decoding.