Lexicon Learning for Few Shot Sequence Modeling (2021)

Abstract: Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following. The neural network models that provide the dominant solution to these problems are brittle, especially in low-resource settings: they fail to generalize correctly or systematically from small datasets. Past work has shown that many failures of systematic generalization arise from neural models’ inability to disentangle lexical phenomena from syntactic ones. To address this, we augment neural decoders with a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. We describe how to initialize this mechanism using a variety of lexicon learning algorithms, and show that it improves systematic generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation.

## introduction humans exhibit a set of structured and remarkably consistent inductive biases when learning from language data. for example, in both natural language acquisition and toy language-learning problems like the one depicted in fig. 1 , human learners exhibit a preference for systematic and compositional interpretation rules (guasti 2017, chapter 4; @xcite . these inductive biases in turn support behaviors like one-shot learning of new concepts @xcite . but in natural language processing, recent work has found that state-of-the-art neural models, while highly effective at in-domain prediction, fail to generalize in human-like ways when faced with rare phenomena pause for a moment to fill in the missing labels in fig. 1 . while doing so, which training examples did you pay the most attention to? how many times did you find yourself saying means or maps to? explicit representations of lexical items and their meanings play a key role diverse models of syntax and semantics @xcite @xcite . but one of the main findings in existing work on generalization in neural models is that they fail to cleanly separate lexical phenomena from syntactic ones @xcite . given a dataset like the one depicted in fig. 1 , models conflate (lexical) information about the correspondence between zup and y with the (syntactic) fact that y appears only in a sequence of length 1 at training time. longer input sequences containing the word zup in new syntactic contexts cause models to output tokens only seen in longer sequences (section 5). in this paper, we describe a parameterization for sequence decoders that facilitates (but does not enforce) the learning of context-independent word meanings. specifically, we augment decoder output layers with a lexical translation mechanism which generalizes neural copy mechanisms (e.g. @xcite and enables models to generate token-level translations purely attentionally. while the lexical translation mechanism is quite general, we focus here on its ability to improve few-shot learning in sequence-to-sequence models. on a suite of challenging tests of few-shot semantic parsing and instruction following, our model exhibits strong generalization, achieving the highest reported results for neural sequence models on datasets as diverse as cogs @xcite training examples) and colors @xcite . our approach also generalizes to real-world tests of few-shot learning, improving bleu scores @xcite by 1.2 on a low-resource english-chinese machine translation task (2.2 on test sentences requiring one-shot word learning). in an additional set of experiments, we explore effective procedures for initializing the lexical translation mechanism using lexicon learning algorithms derived from information theory, statistical machine translation, and bayesian cognitive modeling. we find that both mutual-informationand alignment-based lexicon initializers perform well across tasks. surprisingly, however, we show that both approaches can be matched or outperformed by a rule-based initializer that identifies high-precision word-level token translation pairs. we then explore joint learning of the lexicon and decoder, but find (again surprisingly) that this gives only marginal improvements over a fixed initialization of the lexicon. in summary, this work: a great deal of past work has suggested that neural models come equipped with an inductive bias that makes them fundamentally ill-suited to human-like generalization about language data, especially in the low-data regime (e.g. @xcite . our results suggest that the situation is more complicated: by offloading the easier lexicon learning problem to simpler models, neural sequence models are actually quite effective at modeling (and generalizing about) about syntax in synthetic tests of generalization and real translation tasks. recent years have seen tremendous amount of modeling work aimed at encouraging these generalizations in neural models, primarily by equipping them with symbolic scaffolding in the form of program synthesis engines @xcite , stack machines @xcite , or symbolic data transformation rules @xcite . a parallel line of work has investigated the role of continuous representations in systematic generalization, proposing improved methods for pretraining @xcite and procedures for removing irrelevant contextual information from word representations @xcite @xcite . the latter two approaches proceed from similar intuition to ours, aiming to disentangle word meanings from syntax in encoder representations via alternative attention mechanisms and adversarial training. our approach instead focuses on providing an explicit lexicon to the decoder; as discussed below, this appears to be considerably more effective. copying and lexicon learning in neural encoder-decoder models, the clearest example of benefits from special treatment of word-level production rules is the copy mechanism. a great deal of past work has found that neural models inputs outputs lexicon entries a crocodile blessed william . william needed to walk . crocodile(x_1) and bless.agent (x_2, x_1) and bless.theme (x_2, william) need.agent (x_1 , william) and need.xcomp(x_1, x_3) and walk.agent (x_3, william) @xmath0. 許多 衛星 繞著 土星 運行. 地球 是 一個 行星. @xmath1: we present example (input,output) pairs from cogs, english-to-chinese machine translation and scan datasets. we also present some of the lexicon entries which can be learned by proposed lexicon learning methods and that are helpful to make generalizations required in each of the datasets. as will be discussed, the core of our approach is a (non-contextual) lexicon that maps individual input tokens to individual output tokens. learning lexicons like this is of interest in a number of communities in nlp and language science more broadly. a pair of representative approaches @xcite will be discussed in detail below; other work on lexicon learning for semantics and translation includes @xcite ; @xcite ; @xcite among numerous others. finally, and closest to the modeling contribution in this work, several previous papers have proposed alternative generalized copy mechanisms for tasks other than semantic lexicon learning. concurrent work by @xcite introduces a similar approach for grapheme-to-phoneme translation (with a fixed functional lexicon rather than a trainable parameter matrix), and nguyen and chiang (2018) and @xcite describe less expressive mechanisms that cannot smoothly interpolate between lexical translation and ordinary decoding at the token level. @xcite incorporate lexicon entries by rewriting input sequences prior to ordinary sequence-to-sequence translation. @xcite describe a model in which a copy mechanism is combined with a retrieval-based generative model; like the present work, that model effectively disentangles syntactic and lexical information by using training examples as implicit representations of lexical correspondences. we generalize and extend this previous work in a number of ways, providing a new parameterization of attentive token-level translation and a detailed study of initialization and learning. but perhaps the most important contribution of this work is the observation that many of the hard problems studied as "compositional generalization" have direct analogues in more conventional nlp problems, especially machine translation. research on systematicity and generalization would benefit from closer attention to the ingredients of effective translation at scale. ## sequence-to-sequence models with lexical translation mechanisms this paper focuses on sequence-to-sequence language understanding problems like the ones depicted in neural encoder-decoders our approach builds on the standard neural encoder-decoder model with attention @xcite . in this model, an encoder represents the input sequence next, a decoder generates a distribution over output sequences y according to the sequentially: here we specifically consider decoders with attention. 2 when predicting each output token y i , we assign each input token an attention weight α j i as in eq. ( 3 ). then, we construct a context representation c i as the weighted sum of encoder representations e i : the output distribution over v y , which we denote p write,i , is calculated by a final projection layer: copying a popular extension of the model described above is the copy mechanism, in which output tokens can be copied from the input sequence in addition to being generated directly by the decoder @xcite . using the decoder hidden state h i from above, the model first computes a gate probability: and then uses this probability to interpolate between the distribution in eq. ( 5 ) and a copy distribution that assigns to each word in the output vocabulary a probability proportional to that word's weight in the attention vector over the input: content-independent copying is particularly useful in tasks like summarization and machine translation where rare words (like names) are often reused between the input and output. our model: lexical translation when the input and output vocabularies are significantly different, copy mechanisms cannot provide further improvements on a sequence-to-sequence model. however, even for disjoint vocabularies as in fig. 1 , there may be strict correspondences between individual words on input and output vocabularies, e.g. @xmath4. 1 . following this intuition, the lexical translation mechanism we introduce in this work extends the copy mechanism by introducing an additional layer of indirection between the input sequence x and the output prediction y i as shown in fig. 2 . specifically, after selecting an input token x @xmath5, the decoder can "translate" it to a context-independent output @xmath6. we equip the model with an additional lexicon parameter l, a |v x | × |v y | matrix in which w l @xmath7, and finally define the model is visualized in fig. 2 . note that when v @xmath8, this is identical to the original copy mechanism. however, this approach can in general be used to produce a larger set of tokens. as shown in ## initializing the lexicon the lexicon parameter l in the preceding section can be viewed as an ordinary fully-connected layer inside the copy mechanism, and trained end-to-end with the rest of the network. as with other neural network parameters, however, our experiments will show that the initialization of the parameter l significantly impacts downstream model performance, and specifically benefits from initialization with a set of input-output mappings learned with an offline lexicon learning step. indeed, while not widely used in neural sequence models (though c.f. section 2), lexicon-based initialization was a standard feature of many complex non-neural sequence transduction models, including semantic parsers @xcite and phrase-based machine translation systems @xcite . but an important distinction between our approach and these others is the fact that we can handle outputs that are not (transparently) compositional. not every fragment of an input will correspond to a fragment of an output: for example, thrice in scan has no corresponding output token and instead describes a structural transformation. moreover, the lexicon is not the only way to generate: complex mappings can also be learned by p write without going through the lexicon at all. thus, while most existing work on lexicon learning aims for complete coverage of all word meanings, the model described in section 3 benefits from a lexicon with high-precision coverage of rare phenomena that will be hard to learn in a normal neural model. lexicon learning is widely studied in language processing and cognitive modeling, and several approaches with very different inductive biases exist. to determine how to best initialize l, we begin by reviewing three algorithms in section 4.1, and identify ways in which each of them fail to satisfy the high precision criterion above. in section 4.2, we introduce a simple new lexicon learning rule that addresses this shortcoming. ## experiments we investigate the effectiveness of the lexical translation mechanism on sequence-to-sequence models for four tasks, three focused on compositional generalization and one on low-resource machine translation. in all experiments, we use an lstm encoder-decoder with attention as the base predictor. we compare our approach (and variants) with two other baselines: geca (andreas 2020; a data augmentation scheme) and synatt @xcite ; an alternative seq2seq model parameterization). hyper-parameter selection details are given in the appendix c. unless otherwise stated, we use τ = 0 and do not fine-tune l after initialization. ## conclusion we have described a lexical translation mechanism for representing token-level translation rules in neural sequence models. we have additionally described a simple initialization scheme for this lexicon that outperforms a variety of existing algorithms. together, lexical translation and proper initialization enable neural sequence models to solve a diverse set of tasks-including semantic parsing and machine translation-that require 1-shot word learning and 0-shot compositional generalization. future work might focus on generalization to longer sequences, learning of atomic but nonconcatenative translation rules, and online lexicon learning in situated contexts.