COD3S : Diverse Generation with Discrete Semantic Signatures (2020)

Abstract: We present COD3S, a novel method for generating semantically diverse sentences using neural sequence-to-sequence (seq2seq) models. Conditioned on an input, seq2seqs typically produce semantically and syntactically homogeneous sets of sentences and thus perform poorly on one-to-many sequence generation tasks. Our two-stage approach improves output diversity by conditioning generation on locality-sensitive hash (LSH)-based semantic sentence codes whose Hamming distances highly correlate with human judgments of semantic textual similarity. Though it is generally applicable, we apply to causal generation, the task of predicting a proposition’s plausible causes or effects. We demonstrate through automatic and human evaluation that responses produced using our method exhibit improved diversity without degrading task performance.

## introduction open-ended sequence generation problems such as dialog, story generation, image captioning, or causal generation pose a practical challenge to neural sequence-to-sequence (seq2seq) models, as they necessitate a diverse set of predicted outputs. the typical sampling method for seq2seq decoding is beam search, which produces a set of candidate sequences that generally have high syntactic, lexical, and semantic overlap. recent methods for improved diversity generation make slight modifications to the neural architecture or beam search algorithm @xcite , or impose lexical constraints during decoding @xcite . @xcite propose the use of sentence codes, a technique in which generation is conditioned on a discrete code that aims to induce diversity in syntax or semantics. while their approach is effective for syntactic codes, it is less so for semantics. in this work, we introduce an improved method for diverse generation conditioned on inferred sentence codes that explicitly capture meaningful semantic differences. we use the contextual sentence embeddings from sentence-bert (sbert; @xcite , the cosine distances between which correlate highly with human scalar judgments of semantic textual similarity (sts). we construct discrete codes from these embeddings using locality-sensitive hashing @xcite , producing short binary signatures whose hamming distances well-preserves the cosine distances between inputs. our method induces a bitwise hierarchy of semantic bins whose similarities in signature imply similarities in semantics. conditioning generation on a signature as a target-side prefix indicates the bin into which the generated sequence falls. we implement a two-stage decoding process that (1) infers the most relevant signatures and (2) decodes sequences via separate prefix-conditioned beams. we term our method cod3s: constrained decoding with semantic sentence signatures. we demonstrate the effectiveness of cod3s in the context of causal sequence generation @xcite through bleu-and cosine-based diversity measures as well as human evaluation. ## related work we draw inspiration from recent work in multilingual machine translation (mt) @xcite and domain adaptation @xcite in which a language code (e.g. en, de) is prepended to the target to guide generation. our method for encoding sentence diversity is closely related to mt work by @xcite , who condition generation on prefixed sentence codes. they improve the syntactic diversity of sampled translations using codes produced from improved semantic hashing @xcite with a treelstmbased autoencoder. their experiments with semantic coding via clustering of bert @xcite and fasttext @xcite embeddings lead to negligible or negative effects. outside of @xcite in a similar vein condition on manually categorized "control codes" that specify style and content, and @xcite condition on annotated syntactic or lexical change markers that can be learnt from data. we refer readers to @xcite for an overview of diverse decoding methods. few to our knowledge explicitly and effectively encode open-domain semantic diversity. text-based causal knowledge acquisition is a well-studied challenge in nlp @xcite . recent efforts have investigated open ended causal generation using neural models @xcite . the latter train a conditional generation model to propose cause or effect statements for a given proposition. the model is trained on the co-released corpus causalbank, which comprises causal statements harvested from english common crawl @xcite . applications of lsh @xcite in nlp began with @xcite who demonstrated its use in fast lexical similarity comparison; later, van durme and lall (2010) showed such hashing could be performed online. more similar to our use case, @xcite binned tweets via lsh to enable fast first story detection. most related to ours is work by @xcite , who describe a generative sentence model that edits a 'prototype' sentence using lexically similar ones retrieved via lsh. ## cod3s approach our signature construction method, depicted in figure 1 (a), produces a sequence of bits that collectively imply a highly specific bin of sentences with similar semantic meaning. this is accomplished by encoding sentences into high-dimensional vectors that encode degrees of semantic difference and then discretizing the vectors in a way that approximately preserves the difference. ## experiments we apply cod3s to the task of open-ended causal generation for free-form textual inputs as considered by @xcite . given an input statement, the model must suggest a diverse set of possible causes or effects. we train models on sentence pairs from li et al.'s released dataset, causalbank, which is scraped from common crawl using templatic causal patterns. following their work, we use 10 million sentence pairs that match the patterns "x, so y" to train cause-to-effect models and "x because y" for effect-to-cause models. we experiment with 16-bit lsh signatures of sbert embeddings. 5 after prepending targetside bit signatures, pairs are encoded with bytepair encoding (bpe; @xcite using a vocabulary size of 10k. we train transformer models @xcite using the fairseq library @xcite . appendix b provides details for reproducibility. 6 evaluation we show that cod3s induces sensible inference of diverse but relevant semantic bins and causal statements. examples of generation are shown in ## conclusion we have outlined cod3s, a method for producing semantically diverse statements in open-ended generation tasks. we design sentence lsh signatures that encode bitwise the semantic similarity of underlying statements; conditioning generation on different signatures yields outputs that are semantically heterogeneous. cod3s leads to more diverse outputs in a multi-target generation task in a controllable and interpretable manner, suggesting the potential of semantically guided diverse decoding for a variety of text generation tasks in the future.