Original: O NE - SHOT AND FEW - SHOT LEARNING OF WORD EMBEDDINGS
Revised: O NE - SHOT AND FEW - SHOT LEARNING OF WORD EMBEDDINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.
Revised: Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.

Original: By contrast, humans have an incredible ability to do one-shot or few-shot learning.
Revised: By contrast, humans have an incredible ability to do one-shot or few-shot learning.

Original: For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.
Revised: For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.

Original: Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.
Revised: Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.

Original: This could make natural language processing systems much more ﬂexible, by allowing them to learn continually from the new words they encounter.
Revised: This could make natural language processing systems much more ﬂexible, by allowing them to learn continually from the new words they encounter.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans are often able to infer approximate meanings of new words from context.
Revised: Humans are often able to infer approximate meanings of new words from context.

Original: For example, consider the following stanza from the poem “Jabberwocky” by Lewis Carroll:
Revised: For example, consider the following stanza from the poem “Jabberwocky” by Lewis Carroll:

Original: He took his vorpal sword in hand:
Revised: He took his vorpal sword in hand:

Original: Long time the manxome foe he sought
Revised: Long time the manxome foe he sought

Original: So rested he by the Tumtum tree,
Revised: So rested he by the Tumtum tree,

Original: And stood awhile in thought.
Revised: And stood awhile in thought.

Original: Despite the fact that there are several nonsense words, we can follow the narrative of the poem and understand approximately what many of the words mean by how they relate other words.
Revised: Despite the fact that there are several nonsense words, we can follow the narrative of the poem and understand approximately what many of the words mean by how they relate other words.

Original: This a vital skill for interacting with the world – we constantly need to learn new words and ideas from context.
Revised: This a vital skill for interacting with the world – we constantly need to learn new words and ideas from context.

Original: Even beyond language, humans are often able adapt quickly to gracefully accomodate situations that differ radically from what they have seen before.
Revised: Even beyond language, humans are often able adapt quickly to gracefully accomodate situations that differ radically from what they have seen before.

Original: Complementary learning systems theory (Kumaran et al., 2016) suggests that it is the interaction between a slow-learning system that learns structural features of the world (i.e. a deep-learning like system) and a fast-learning system (i.e. a memory-like system) that allows humans to adapt rapidly from few experiences.
Revised: Complementary learning systems theory (Kumaran et al., 2016) suggests that it is the interaction between a slow-learning system that learns structural features of the world (i.e. a deep-learning like system) and a fast-learning system (i.e. a memory-like system) that allows humans to adapt rapidly from few experiences.

Original: By comparison, standard deep learning systems usually require much more data to learn a concept or task, and sometimes generalize poorly Lake et al.
Revised: By comparison, standard deep learning systems usually require much more data to learn a concept or task, and sometimes generalize poorly Lake et al.

Original: They can be trained to learn a concept in one-shot if this is their sole task (Vinyals et al., 2016, e.g.), but this limits the types of tasks that can be performed.
Revised: They can be trained to learn a concept in one-shot if this is their sole task (Vinyals et al., 2016, e.g.), but this limits the types of tasks that can be performed.

Original: Furthermore, these models typically discard this information after a single use.
Revised: Furthermore, these models typically discard this information after a single use.

Original: In order for deep learning systems to be adaptible, they will need to build on their prior knowledge to learn effectively from a few pieces of information.
Revised: In order for deep learning systems to be adaptible, they will need to build on their prior knowledge to learn effectively from a few pieces of information.

Original: In other words, they will need to integrate learning experiences across different timescales, as complementary learning systems theory suggests that humans and other animals do.
Revised: In other words, they will need to integrate learning experiences across different timescales, as complementary learning systems theory suggests that humans and other animals do.

Original: In this paper, we explore this broad issue in the speciﬁc context of creating a useful representation for a new word based on its context.
Revised: In this paper, we explore this broad issue in the speciﬁc context of creating a useful representation for a new word based on its context.

Original: 1. B ACKGROUND
Revised: 1. B ACKGROUND

Original: Continuous representations of words have proven to be very effective (Mikolov et al., 2013; Pennington et al., 2014, e.g.).
Revised: Continuous representations of words have proven to be very effective (Mikolov et al., 2013; Pennington et al., 2014, e.g.).

Original: These approaches represent words as vectors in a space, which are learned from large corpuses of text data.
Revised: These approaches represent words as vectors in a space, which are learned from large corpuses of text data.

Original: Using these vectors, deep learning systems have achieved success on tasks ranging from natural language translation (Wu et al., 2016, e.g.) to question answering (Santoro et al., 2017, e.g.).
Revised: Using these vectors, deep learning systems have achieved success on tasks ranging from natural language translation (Wu et al., 2016, e.g.) to question answering (Santoro et al., 2017, e.g.).

Original: However, these word vectors are typically trained on very large datasets, and there has been surprisingly little prior work on how to learn embeddings for new words once the system has been trained.
Revised: However, these word vectors are typically trained on very large datasets, and there has been surprisingly little prior work on how to learn embeddings for new words once the system has been trained.

Original: Cotterell et al.
Revised: Cotterell et al.

Original: (2016) proposed a method for incorporating morphological information into word embeddings that allows for limited generalization to new words (e.g. generalizing to unseen conjugations of a known verb).
Revised: (2016) proposed a method for incorporating morphological information into word embeddings that allows for limited generalization to new words (e.g. generalizing to unseen conjugations of a known verb).

Original: However, this is not a general system for learning representations for new words, and requires building in rather strong structural assumptions about the language and having appropriately labelled data to exploit them.
Revised: However, this is not a general system for learning representations for new words, and requires building in rather strong structural assumptions about the language and having appropriately labelled data to exploit them.

Original: More recently Lazaridou et al.
Revised: More recently Lazaridou et al.

Original: (2017) explored multi-modal word learning (similar to what children do when an adult points out a new object), and in the process suggested a simple heuristic for inferring a word vector from context: simply average together all the surrounding word vectors.
Revised: (2017) explored multi-modal word learning (similar to what children do when an adult points out a new object), and in the process suggested a simple heuristic for inferring a word vector from context: simply average together all the surrounding word vectors.

Original: This is sensible, since the surrounding words will likely occur in similar contexts.
Revised: This is sensible, since the surrounding words will likely occur in similar contexts.

Original: However, it ignores all syntactic information, by treating all the surrounding words identically, and it relies on the semantic information being linearly combinable between different word embeddings.
Revised: However, it ignores all syntactic information, by treating all the surrounding words identically, and it relies on the semantic information being linearly combinable between different word embeddings.

Original: Both of these factors will likely limit its performance.
Revised: Both of these factors will likely limit its performance.

Original: Can we do better?
Revised: Can we do better?

Original: A deep learning system which has been trained to perform a language task must have learned a great deal of semantic and syntactic structure which would be useful for inferring and representing the meaning of a new word.
Revised: A deep learning system which has been trained to perform a language task must have learned a great deal of semantic and syntactic structure which would be useful for inferring and representing the meaning of a new word.

Original: However, this knowledge is opaquely encoded in its weights.
Revised: However, this knowledge is opaquely encoded in its weights.

Original: Is there a way we can exploit this knowledge when learning about a new word?
Revised: Is there a way we can exploit this knowledge when learning about a new word?

Original:  A PPROACH
Revised:  A PPROACH

Original: We suggest that we already have a way to update the representations of a network while accounting for its current knowledge and inferences – this is precisely what backpropagation was invented for!
Revised: We suggest that we already have a way to update the representations of a network while accounting for its current knowledge and inferences – this is precisely what backpropagation was invented for!

Original: Of course, we cannot simply train the whole network to accomodate this new word, this would lead to catastrophic interference.
Revised: Of course, we cannot simply train the whole network to accomodate this new word, this would lead to catastrophic interference.

Original: However, Rumelhart & Todd (1993) showed that a simple network could be taught about a new input by freezing all its weights except for those connecting the new input to the ﬁrst hidden layer, and optimizing these by gradient descent as usual.
Revised: However, Rumelhart & Todd (1993) showed that a simple network could be taught about a new input by freezing all its weights except for those connecting the new input to the ﬁrst hidden layer, and optimizing these by gradient descent as usual.

Original: They showed that this resulted in the network making appropriate generalizations about the new input, and by design the training procedure does not interfere with the network’s prior knowledge.
Revised: They showed that this resulted in the network making appropriate generalizations about the new input, and by design the training procedure does not interfere with the network’s prior knowledge.

Original: They used this as a model for human concept learning (as have other authors, for example Rogers & McClelland (2004)).
Revised: They used this as a model for human concept learning (as have other authors, for example Rogers & McClelland (2004)).

Original: We take inspiration from this work to guide our approach.
Revised: We take inspiration from this work to guide our approach.

Original: To learn from one sentence (or a few) containing a new word, we freeze all the weights in the network except those representing the new word (in a complex NLP system, there may be more than one such set of weights, for example the model we evaluate has distinct input and output embeddings for each word).
Revised: To learn from one sentence (or a few) containing a new word, we freeze all the weights in the network except those representing the new word (in a complex NLP system, there may be more than one such set of weights, for example the model we evaluate has distinct input and output embeddings for each word).

Original: We then use stochastic gradient descent ( η = 0 . 01 ) to update the weights for the new word using 100 epochs of training over the sentence(s) containing it.
Revised: We then use stochastic gradient descent ( η = 0 . 01 ) to update the weights for the new word using 100 epochs of training over the sentence(s) containing it.

Original: Of course, there are a variety of possible initializations for the embeddings before optimizing.
Revised: Of course, there are a variety of possible initializations for the embeddings before optimizing.

Original: In this paper, we consider three possibilities:
Revised: In this paper, we consider three possibilities:

Original: Beginning with an embedding for a token that was placed in the softmax but never used during training (for the purpose of having a useful initialization for new words).
Revised: Beginning with an embedding for a token that was placed in the softmax but never used during training (for the purpose of having a useful initialization for new words).

Original: This might help separate the new embedding from other embeddings.
Revised: This might help separate the new embedding from other embeddings.

Original: Beginning with a vector of zeros.
Revised: Beginning with a vector of zeros.

Original: Beginning with the centroid of the other words in the sentence, which Lazaridou et al.
Revised: Beginning with the centroid of the other words in the sentence, which Lazaridou et al.

Original: (2017) suggested was a useful estimate of an appropriate embedding.
Revised: (2017) suggested was a useful estimate of an appropriate embedding.

Original: We compare these to two baselines:
Revised: We compare these to two baselines:

Original: The centroid of the embeddings of the other words in the sentence Lazaridou et al.
Revised: The centroid of the embeddings of the other words in the sentence Lazaridou et al.

Original: Training the model with the 10 training sentences included in the corpus from the beginning (i.e. the “standard” deep-learning approach).
Revised: Training the model with the 10 training sentences included in the corpus from the beginning (i.e. the “standard” deep-learning approach).

Original: (A reader of an earlier draft of this paper noted that Herbelot & Baroni (2017) independently tried some similar strategies, but our results go farther in a variety of ways, see Appendix E.)
Revised: (A reader of an earlier draft of this paper noted that Herbelot & Baroni (2017) independently tried some similar strategies, but our results go farther in a variety of ways, see Appendix E.)

Original: 2. T ASK , M ODEL , AND A PPROACH
Revised: 2. T ASK , M ODEL , AND A PPROACH

Original: The framework we have described for updating embeddings could be applied very generally, but for the sake of this paper we ground it in a simple task: predicting the next word of a sentence based on the previous words, on the Penn Treebank dataset (Marcus et al., 1993).
Revised: The framework we have described for updating embeddings could be applied very generally, but for the sake of this paper we ground it in a simple task: predicting the next word of a sentence based on the previous words, on the Penn Treebank dataset (Marcus et al., 1993).

Original: Speciﬁcally, we will use the (large) model architecture and approach of Zaremba et al.
Revised: Speciﬁcally, we will use the (large) model architecture and approach of Zaremba et al.

Original: , see Appendix B.1 for details.
Revised: , see Appendix B.1 for details.

Original: Of course, human language understanding is much more complex than just prediction, and grounding language in situations and goals is likely important for achieving deeper understanding of language (Gauthier & Mordatch, 2016).
Revised: Of course, human language understanding is much more complex than just prediction, and grounding language in situations and goals is likely important for achieving deeper understanding of language (Gauthier & Mordatch, 2016).

Original: More recent work has begun to do this (Hermann et al., 2017, e.g), and it’s likely that our approach would be more effective in settings like these.
Revised: More recent work has begun to do this (Hermann et al., 2017, e.g), and it’s likely that our approach would be more effective in settings like these.

Original: The ability of humans to make rich inferences about text stems from the richness of our knowledge.
Revised: The ability of humans to make rich inferences about text stems from the richness of our knowledge.

Original: However, for simplicity, we have chosen to ﬁrst demonstrate it on the prediction task.
Revised: However, for simplicity, we have chosen to ﬁrst demonstrate it on the prediction task.

Original: In order to test our one-shot word-learning algorithm on the Penn Treebank, we chose a word which appeared only 20 times in the training set.
Revised: In order to test our one-shot word-learning algorithm on the Penn Treebank, we chose a word which appeared only 20 times in the training set.

Original: We removed the sentences containing this word and then trained the model with the remaining sentences for 55 epochs using the learning rate decay strategy of Zaremba et al.
Revised: We removed the sentences containing this word and then trained the model with the remaining sentences for 55 epochs using the learning rate decay strategy of Zaremba et al.

Original: Because the PTB dataset contains over 40,000 sentences, themissing ones had essentially no effect on the networks overall performance.
Revised: Because the PTB dataset contains over 40,000 sentences, themissing ones had essentially no effect on the networks overall performance.

Original: We then split thesentences containing the new word into 10 train and 10 test, and trained on 1 - 10 of these indifferent permutations (via a balanced Latin square (Campbell & Geller, 1980), which ensures that each sentence was used for one-shot learning once, and enforces diversity in the multi-shot examples).
Revised: We then split thesentences containing the new word into 10 train and 10 test, and trained on 1 - 10 of these indifferent permutations (via a balanced Latin square (Campbell & Geller, 1980), which ensures that each sentence was used for one-shot learning once, and enforces diversity in the multi-shot examples).

Original: In other words, we performed 100 training runs for each word, 10 training runs each with a distinct single sentence, 10 with a distinct pair of sentences, etc.
Revised: In other words, we performed 100 training runs for each word, 10 training runs each with a distinct single sentence, 10 with a distinct pair of sentences, etc.

Original:  R ESULTS
Revised:  R ESULTS

Original: We ﬁrst evaluated our approach on the words “bonuses,” “explained,” “marketers,” and “strategist,” either initializing from the never-seen embedding that the network is optimized to never produce, the zero vector, or the centroid of the surrounding words, and compared to the baselines of just taking the centroid of the surrounding words and full training with the words, see Fig. 1.
Revised: We ﬁrst evaluated our approach on the words “bonuses,” “explained,” “marketers,” and “strategist,” either initializing from the never-seen embedding that the network is optimized to never produce, the zero vector, or the centroid of the surrounding words, and compared to the baselines of just taking the centroid of the surrounding words and full training with the words, see Fig. 1.

Original: Optimizing from the centroid outperforms all other approaches for learning a new word for all datasets, including the centroid approach of Lazaridou et al.
Revised: Optimizing from the centroid outperforms all other approaches for learning a new word for all datasets, including the centroid approach of Lazaridou et al.

Original: (2017), and even outperforms full training with the word 1 in three of the four cases (however, this likely comes with a tradeoff, see below).
Revised: (2017), and even outperforms full training with the word 1 in three of the four cases (however, this likely comes with a tradeoff, see below).

Original: The optimizing approaches are strongly affected by embedding initialization with few training sentences (e.g. one-shot learning), but by 10 sentences they all perform quite similarly.
Revised: The optimizing approaches are strongly affected by embedding initialization with few training sentences (e.g. one-shot learning), but by 10 sentences they all perform quite similarly.

Original: Of course, this learning might still cause interference with the networks prior knowledge.
Revised: Of course, this learning might still cause interference with the networks prior knowledge.

Original: In order to evaluate this, we replicated these ﬁndings with four new words (“borrow,” “cowboys”, “immune,” and “rice”), but also evaluated the change in perplexity on the PTB test corpus (see Appendix A, Fig. 5).
Revised: In order to evaluate this, we replicated these ﬁndings with four new words (“borrow,” “cowboys”, “immune,” and “rice”), but also evaluated the change in perplexity on the PTB test corpus (see Appendix A, Fig. 5).

Original: Indeed, we found that while the centroid method does not substantially change the test perplexity on the corpus, the optimizing method causes increasingly more interference as more training sentences are provided, up to a 1% decline in the case of training on all 10 sentences.
Revised: Indeed, we found that while the centroid method does not substantially change the test perplexity on the corpus, the optimizing method causes increasingly more interference as more training sentences are provided, up to a 1% decline in the case of training on all 10 sentences.

Original: This is not necessarily surprising, the base rate of occurrence of the new word in the training data is artiﬁcially inﬂated relative to its true base rate probability.
Revised: This is not necessarily surprising, the base rate of occurrence of the new word in the training data is artiﬁcially inﬂated relative to its true base rate probability.

Original: This problem of learning from data which is locally highly correlated and biased has been solved in many recent domains by the use of replay buffers to interleave learning, especially in reinforcement learning (Mnih et al., 2015, e.g).
Revised: This problem of learning from data which is locally highly correlated and biased has been solved in many recent domains by the use of replay buffers to interleave learning, especially in reinforcement learning (Mnih et al., 2015, e.g).

Original: The importance of replay is also highlighted in the complementary learning systems theory (Kumaran et al., 2016) that helped inspire this work.
Revised: The importance of replay is also highlighted in the complementary learning systems theory (Kumaran et al., 2016) that helped inspire this work.

Original: We therefore tested whether using a replay buffer while learning the new word would ameliorate this interference.
Revised: We therefore tested whether using a replay buffer while learning the new word would ameliorate this interference.

Original: Speciﬁcally, we sampled 100 negative sentences from the without-word corpus the network was pre-trained on, and interleaved these at random with the new word training sentences (the same negative samples were used every epoch).
Revised: Speciﬁcally, we sampled 100 negative sentences from the without-word corpus the network was pre-trained on, and interleaved these at random with the new word training sentences (the same negative samples were used every epoch).

Original: Indeed, interleaving sentences without the word substantially reduced the interference caused by the new word, see Fig.
Revised: Indeed, interleaving sentences without the word substantially reduced the interference caused by the new word, see Fig.

Original: The maximum increase in perplexity was 0 .
Revised: The maximum increase in perplexity was 0 .

Original: This interleaving did result in somewhat less improvement on the new-word test sentences, but this is probably simply because the test sentences over-represent the new word and the network was overﬁtting to this and predicting the new word much more than is warranted.
Revised: This interleaving did result in somewhat less improvement on the new-word test sentences, but this is probably simply because the test sentences over-represent the new word and the network was overﬁtting to this and predicting the new word much more than is warranted.

Original: The optimizing approach still reduces perplexity onthe dataset containing the new word  by up to 33% (about 10 percentage points better than the centroid approach).
Revised: The optimizing approach still reduces perplexity on new word dataset by up to 33% (about 10 percentage points better than the centroid approach).

Original: 3. W HERE IS THE MAGIC HAPPENING ?
Revised: 3. W HERE IS THE MAGIC HAPPENING ?

Original: Because the model we are using has distinct input and output embeddings, we are able to evaluate their distinct contributions to learning about the new word.
Revised: Because the model we are using has distinct input and output embeddings, we are able to evaluate their distinct contributions to learning about the new word.

Original: Speciﬁcally, we compared learning only the softmax weights and bias (output embedding), to learning only the input embedding, as well as to learning both, for one- and ten-shot learning 2 .
Revised: Speciﬁcally, we compared learning only the softmax weights and bias (output embedding), to learning only the input embedding, as well as to learning both, for one- and ten-shot learning 2 .

Original: See Fig.
Revised: See Fig.

Original: 3 for our results.
Revised: 3 for our results.

Original: We found that it was mostly the output embeddings that are improving.
Revised: We found that the changes in the output embeddings were almost entirely responsible for the overall improvement.

Original: In one-shot learning, changing the input embedding alone causes almost no improvement, and changing both embeddings does not seem substantially different from changing just the output embedding.
Revised: In one-shot learning, changing the input embedding alone causes almost no improvement, and changing both embeddings does not seem substantially different from changing just the output embedding.

Original: However, with ten training sentences the updated input embedding is producing some improvement, both alone and when trained together with the output embedding.
Revised: However, with ten training sentences the updated input embedding is producing some improvement, both alone and when trained together with the output embedding.

Original: Even in this case, however, the effect of the input embedding is still much smaller than the effect of the output embedding.
Revised: Even in this case, however, the effect of the input embedding is still much smaller than the effect of the output embedding.

Original: That is, the model is mostly improving at predicting the new word in context, rather than predicting context based on the new word.
Revised: From this evidence, it seems likely that the model is mostly improving in predicting the new word in context, rather than predicting context based on the new word.

Original: This is sensible for several reasons.
Revised: This is sensible for several reasons.

Original: First, whatever the new word conveys about the context will already be partly conveyed by the other words in the context.
Revised: First, whatever the new word conveys about the context will already be partly conveyed by the other words in the context.

Original: Second, our training approach wasdoes full training with the word result in lowered perplexity on the test data in three out of four cases? This may just be chance variation, of course but in general we would expect learning about a word to be not just for the sake of learning about that word but because each word is a small piece of the signal by which the network learns about language more generally
Revised: Second, our training approach was

Original: [Figure]
Revised: [Figure]

Original: more unnatural than the situations in which a human might experience a new word, or even than the pretraining for the network, in that the sentences were presented without the context of surrounding sentences.
Revised: more unnatural than the situations in which a human might experience a new word, or even than the pretraining for the network, in that the sentences were presented without the context of surrounding sentences.

Original: This means that the model has less data to learn about how the new word predicts surrounding context, and less information about the context which predicts this word.
Revised: This means that the model has less data to learn about how the new word predicts surrounding context, and less information about the context which predicts this word.

Original: This may also explain why full training with the word still produced better results in some cases than updating for it.
Revised: This may also explain why full training with the word still produced better results in some cases than updating for it.

Original: Finally, efﬁciently passing information about the new word through the model from the input might require adjustments to the intermediate weights, which were frozen.
Revised: Finally, efﬁciently passing information about the new word through the model from the input might require adjustments to the intermediate weights, which were frozen.

Original: 3. D IGGING DEEPER INTO MODEL PERFORMANCE
Revised: 3. D IGGING DEEPER INTO MODEL PERFORMANCE

Original: New word iscorrect
Revised: New word is

Original:  Wrong but relevant Wrong and irrelevant
Revised: [Table] Wrong [Table]

Original: Full training with the word
Revised: Full training with the word

Original: [Table]
Revised: [Table]

Original: Centroid

Original: [Table]

Original: Optimizing from centroid
Revised: Optimizing from centroid

Original: [Table]
Revised: [Table]

Original: Of course, as the above results illustrate, evaluating solely on change in perplexity limits insight. Thus we conducted more detailed analyses of the model’s predictions.
Revised: In order to dig more deeply into the effects of learning the new words embedding, we conducted more detailed analyses of the models predictions of the probability of the new word in different contexts.

Original: Since we know that most of the beneﬁt of the one-shot training in our model comes from predicting the new word in context, we evaluated how well the word was predicted in three cases: when it was the actual target word, when it was not the current target but did appear in the sentence, and  in “irrelevant” sentences.
Revised: Speciﬁcally, we evaluated how well the word was predicted in three cases: when it was the actual target word, when it was not the current target but did appear in the sentence (“wrong but relevant”) and when it was did not even appear in the sentence at all (“wrong and irrelevant”).

Original: This allowed us to investigate whether the model was learning something useful, or simply overﬁtting to the new data.
Revised: This allowed us to investigate whether the model was learning something useful, or simply overﬁtting to the new data.

Original: We compared the average log-probability the model assigned to the new word in each of these cases for the full training with the word baseline, the centroid approach to learning from 10 words, and our approach (with a replay buffer).
Revised: We compared the average log-probability the model assigned to the new word in each of these cases for the full training with the word baseline, the centroid approach to learning from 10 words, and our approach (with a replay buffer).

Original: The relevant cases were evaluated on our held-out test data for that word; the irrelevant case was evaluated on the ﬁrst 10 sentences of the PTB test corpus (an article that does not contain any of the words we used).
Revised: The relevant cases were evaluated on our held-out test data for that word; the irrelevant case was evaluated on the ﬁrst 10 sentences of the PTB test corpus (an article that does not contain any of the words we used).

Original: See Table 1 for our results.
Revised: See Table 1 for our results.

Original: The model fully trained with the word shows clear distinctions between the three conditions – the word is estimated to be about 10 times more likely in contexts where it appears than in irrelevant contexts, and is estimated to be about 25 times more likely again when it actually appears.
Revised: The model fully trained with the word shows clear distinctions between the three conditions – the word is estimated to be about 10 times more likely in contexts where it appears than in irrelevant contexts, and is estimated to be about 25 times more likely again when it actually appears.

Original: However, the model severely underestimates the probability of the word when it does appear; the word would have a similar log probability under a uniform distribution over the whole vocabulary.
Revised: However, the model severely underestimates the probability of the word when it does appear; the word would have a similar log probability under a uniform distribution over the whole vocabulary.

Original: The centroid method also has this issue, but in addition it does not even distinguish particularly well between contexts.
Revised: The centroid method also has this issue, but in addition it does not even distinguish particularly well between contexts.

Original: The word is only estimated to be about 4 times more likely when it is the target than in completely irrelevant contexts.
Revised: The word is only estimated to be about 4 times more likely when it is the target than in completely irrelevant contexts.

Original: By contrast, our approach results in a good distinction between contexts – the word is predicted to be about 5 times as likely in contexts where it appears compared to the irrelevant context, and abouttimes as likely again when the word actually appears.
Revised: By contrast, our approach results in a good distinction between contexts – the word is predicted to be about 5 times as likely in contexts where it appears compared to the irrelevant context, and abouttimes as likely again when the word actually appears.

Original: These relative probabilities are quite similar to those exhibited by the model fully trained with the word.
Revised: These relative probabilities are quite similar to those exhibited by the model fully trained with the word.

Original: In both respects, it appears superior to the centroid approach.
Revised: In both respects, it appears superior to the centroid approach.

Original: When compared to the full training with the word, however, it appears that the base rate estimates of the prevalence of the word are inﬂated (which is sensible, even withnegative samples per positive sample in our training data the prevalence of the word is much higher than in the actual corpus).
Revised: When compared to the full training with the word, however, it appears that the base rate estimates of the prevalence of the word are inﬂated (which is sensible, even withnegative samples per positive sample in our training data the prevalence of the word is much higher than in the actual corpus).

Original: This explains the residual small increase in test perplexity on the dataset not containing the new word.
Revised: This explains the residual small increase in test perplexity on the dataset not containing the new word.

Original: It is possible that this could be ameliorated either by setting a prior on the bias for the new word (perhaps penalize the (cid:96) 2 norm of the distance of the bias from the values for other rare words), or with a validation set, or just by using more negative samples during training.
Revised: It is possible that this could be ameliorated either by setting a prior on the bias for the new word (perhaps penalize the (cid:96) 2 norm of the distance of the bias from the values for other rare words), or with a validation set, or just by using more negative samples during training.

Original: In any case, the optimized embeddings are capturing some of the important features of when the word does and does not appear, and are doing so more effectively than the centroid approach.
Revised: In any case, the optimized embeddings are capturing some of the important features of when the word does and does not appear, and are doing so more effectively than the centroid approach.

Original: 3. M ORE WORDS
Revised: 3. M ORE WORDS

Original: Up to this point, we have presented all the results in this paper broken down by word rather than as averages, because there were large word-by-word differences on almost every analysis, and we evaluated on relatively few new words.
Revised: Up to this point, we have presented all the results in this paper broken down by word rather than as averages, because there were large word-by-word differences on almost every analysis, and we evaluated on relatively few new words.

Original: In order to establish the generality of our ﬁndings with a large sample of words, we ran an additional experiment that spans the space of words more broadly.
Revised: In order to establish the generality of our ﬁndings with a large sample of words, we ran an additional experiment that spans the space of words more broadly.

Original: In this experiment, we selected 100 of the 150 words that appear exactly 20 times in the PTB train corpus (omitting the words we used in prior experiments, see Appendix B.1.3 for a complete list).
Revised: In this experiment, we selected 100 of the 150 words that appear exactly 20 times in the PTB train corpus (omitting the words we used in prior experiments, see Appendix B.1.3 for a complete list).

Original: Instead of training separate models without each word as we had previously, we trained a single model with none of these words included in the train set.
Revised: Instead of training separate models without each word as we had previously, we trained a single model with none of these words included in the train set.

Original: We then tested our few-shot learning technique with a replay buffer (optimizing from centroid) and the centroid technique on these sentences, and compared to results obtained from full training with all words – a model trained with the entire train corpus, including the train sentences for each of the hundred words.
Revised: We then tested our few-shot learning technique with a replay buffer (optimizing from centroid) and the centroid technique on these sentences, and compared to results obtained from full training with all words – a model trained with the entire train corpus, including the train sentences for each of the hundred words.

Original: (In all cases, the same 10 of the 20 sentences containing the new word were used in training, and the other 10 were used for testing.)
Revised: (In all cases, the same 10 of the 20 sentences containing the new word were used in training, and the other 10 were used for testing.)

Original: Notice that the comparison to “full training with all words” is not as precise as our previous experiments – the model receives about 2.5 % more training data overall than any of the few-shot learning models, which means it will have more linguistic structure to learn the new words from, as well as the advantage of interleaving them.
Revised: Notice that the comparison to “full training with all words” is not as precise as our previous experiments – the model receives about 2.5 % more training data overall than any of the few-shot learning models, which means it will have more linguistic structure to learn the new words from, as well as the advantage of interleaving them.

Original: However, the comparisons between our technique and the centroid technique are still valid, and the comparison to the full training with all words gives a worst-case bound on how poorly one-shot methods will do compared to full training.
Revised: However, the comparisons between our technique and the centroid technique are still valid, and the comparison to the full training with all words gives a worst-case bound on how poorly one-shot methods will do compared to full training.

Original: With this in mind, see Fig.
Revised: With this in mind, see Fig.

Original: 4 (and Appendix A Fig. 6) for our results.
Revised: 4 (and Appendix A Fig. 6) for our results.

Original: As before, optimizing from the centroid performed much better than simply using the centroid – on average it produced a 64% (11 percentage point) improvement over the centroid result, and on none of the 100 words did it perform worse than the centroid method.
Revised: As before, optimizing from the centroid performed much better than simply using the centroid – on average it produced a 64% (11 percentage point) improvement over the centroid result, and on none of the 100 words did it perform worse than the centroid method.

Original: More quantitatively, the optimizing method performed signiﬁcantly better (paired t -test, t (99) = − 20 .
Revised: More quantitatively, the optimizing method performed signiﬁcantly better (paired t -test, t (99) = − 20 .

Original: 5 , p < 1 · 10 − 16 ).
Revised: 5 , p < 1 · 10 − 16 ).

Original: Furthermore, despite the disadvantage of being exposed to less total data, the optimizing approach seems to do approximately as well as the full training approach on average – although the full training approach sometimes results in much larger improvements, the results did not signiﬁcantly differ (paired t -test, t (99) = 0 .
Revised: Furthermore, despite the disadvantage of being exposed to less total data, the optimizing approach seems to do approximately as well as the full training approach on average – although the full training approach sometimes results in much larger improvements, the results did not signiﬁcantly differ (paired t -test, t (99) = 0 .

Original: 9 , p = 0 . 39 ).
Revised: 9 , p = 0 . 39 ).

Original: Across a wide variety of words, optimizing improves on the centroid approach, and performs comparably to full training.
Revised: Across a wide variety of words, optimizing improves on the centroid approach, and performs comparably to full training.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Overall, using our technique of updating only the embedding vectors of a word while training on sentences containing it and negative sampled sentences from the networks past experience seems quite effective.
Revised: Overall, using our technique of updating only the embedding vectors of a word while training on sentences containing it and negative sampled sentences from the networks past experience seems quite effective.

Original: It allows for substantial reductions in perplexity on text containing the new word, without greatly interfering with knowledge about other words.
Revised: It allows for substantial reductions in perplexity on text containing the new word, without greatly interfering with knowledge about other words.

Original: Furthermore, it seems to be capturing more useful structure about how the word is used in context than previous approaches, and performs close to as well as full training with the word.
Revised: Furthermore, it seems to be capturing more useful structure about how the word is used in context than previous approaches, and performs close to as well as full training with the word.

Original: These results are exciting beyond their potential applications to natural language processing – this technique could easily be extended to adapting systems to other types of new experiences, for example a vision network for an RL agent could have a few new ﬁlters per layer added and trained to accomodate a new type of object.
Revised: These results are exciting beyond their potential applications to natural language processing – this technique could easily be extended to adapting systems to other types of new experiences, for example a vision network for an RL agent could have a few new ﬁlters per layer added and trained to accomodate a new type of object.

Original: Under what circumstances will this strategy fail?
Revised: Under what circumstances will this strategy fail?

Original: Complementary learning systems theory (Kumaran et al., 2016), from which we drew inspiration, suggests that information which is schema-consistent (i.e. ﬁts in with the network’s previous knowledge) can be integrated easily, whereas schemainconsistent knowledge (i.e. knowledge that differs from the network’s previous experience) will cause interference.
Revised: Complementary learning systems theory (Kumaran et al., 2016), from which we drew inspiration, suggests that information which is schema-consistent (i.e. ﬁts in with the network’s previous knowledge) can be integrated easily, whereas schemainconsistent knowledge (i.e. knowledge that differs from the network’s previous experience) will cause interference.

Original: Similar principles should apply here.
Revised: Similar principles should apply here.

Original: Our approach should work for learning a new word on a topic which is already somewhat familiar, but would likely fail to learn from a new word in a context that is not well understood.
Revised: Our approach should work for learning a new word on a topic which is already somewhat familiar, but would likely fail to learn from a new word in a context that is not well understood.

Original: For example, it would be difﬁcult to learn a new
Revised: For example, it would be difﬁcult to learn a new

Original: German word from context if the model has only experienced English.
Revised: German word from context if the model has only experienced English.

Original: On the other hand, this perspective also provides promises.
Revised: On the other hand, this perspective also provides promises.

Original: We expect that our technique would perform even better in a system that had a more sophisticated understanding of language, because it would have more prior knowledge from which to bootstrap understanding of new words.
Revised: We expect that our technique would perform even better in a system that had a more sophisticated understanding of language, because it would have more prior knowledge from which to bootstrap understanding of new words.

Original: Thus it would be very interesting to apply our technique on more complicated tasks like question answering, such as Santoro et al.
Revised: Thus it would be very interesting to apply our technique on more complicated tasks like question answering, such as Santoro et al.

Original: (2017), or in a grounded context,
Revised: (2017), or in a grounded context,

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have presented a technique for doing one- or few-shot learning of word embeddings from text data: freeze all the weights in the network except the embeddings for the new word, and then optimize these embeddings for the sentence, interleaving with negative examples from network’s prior experience and stopping early.
Revised: We have presented a technique for doing one- or few-shot learning of word embeddings from text data: freeze all the weights in the network except the embeddings for the new word, and then optimize these embeddings for the sentence, interleaving with negative examples from network’s prior experience and stopping early.

Original: This results in substantial improvement of the ability to predict the word in context, with minimal impairment of prediction of other words.
Revised: This results in substantial improvement of the ability to predict the word in context, with minimal impairment of prediction of other words.

Original: This technique could allow natural language processing systems to adapt more ﬂexibly to a changing world, like humans do.
Revised: This technique could allow natural language processing systems to adapt more ﬂexibly to a changing world, like humans do.

Original: More generally, it could serve as a model for how to integrate rapid adaptation into deep learning systems
Revised: More generally, it could serve as a model for how to integrate rapid adaptation into deep learning systems

Original: A S UPPLEMENTARY FIGURES
Revised: A S UPPLEMENTARY FIGURES

Original: [Figure]
Revised: [Figure]

Original: B I MPLEMENTATION DETAILS B. M ODEL
Revised: B I MPLEMENTATION DETAILS B. M ODEL

Original: We used the “large” model described by Zaremba et al.
Revised: We used the “large” model described by Zaremba et al.

Original: (2014), and use all their hyper-parameters for the pre-training.
Revised: (2014), and use all their hyper-parameters for the pre-training.

Original: Speciﬁcally, the model consists of 2 layers of stacked LSTMs with a hidden size of 1500 units, 35 recurrent steps, and dropout ( p keep = 0 . 35 ) applied to the non-recurrent connections.
Revised: Speciﬁcally, the model consists of 2 layers of stacked LSTMs with a hidden size of 1500 units, 35 recurrent steps, and dropout ( p keep = 0 . 35 ) applied to the non-recurrent connections.

Original: The gradients were clipped to a max global norm of 10.
Revised: The gradients were clipped to a max global norm of 10.

Original: Weights were initialized uniformly from [ − 0 . 04 , 0 . 04] .
Revised: Weights were initialized uniformly from [ − 0 . 04 , 0 . 04] .

Original: B.1. T RAINING ON FULL CORPUS
Revised: B.1. T RAINING ON FULL CORPUS

Original: The loss function was the cross-entropy loss of the model predictions.
Revised: The loss function was the cross-entropy loss of the model predictions.

Original: The model was trained for 55 epochs by stochastic gradient descent with a batch size of 20.
Revised: The model was trained for 55 epochs by stochastic gradient descent with a batch size of 20.

Original: The learning rate was set to 1 for the ﬁrst 14 epochs, and then was decayed by a multiplier of 1 / 1 . 15 per epoch for the remainder of training.
Revised: The learning rate was set to 1 for the ﬁrst 14 epochs, and then was decayed by a multiplier of 1 / 1 . 15 per epoch for the remainder of training.

Original: B.1. T RAINING ON A NEW WORD
Revised: B.1. T RAINING ON A NEW WORD

Original: Centroid: We extracted the input embeddings, softmax weights, and softmax biases of every word in the sentence except for the new word.
Revised: Centroid: We extracted the input embeddings, softmax weights, and softmax biases of every word in the sentence except for the new word.

Original: We then averaged together these input embeddings to produce the new input embedding for the new word, etc.
Revised: We then averaged together these input embeddings to produce the new input embedding for the new word, etc.

Original: Optimizing:
Revised: Optimizing:

Original: We initialized the new input embedding in one of three ways: with the centroid from above, with a vector of zeros, or with the current embedding vector for the word (note that this last would not be possible in a general context where a new word is not “expected”).
Revised: We initialized the new input embedding in one of three ways: with the centroid from above, with a vector of zeros, or with the current embedding vector for the word (note that this last would not be possible in a general context where a new word is not “expected”).

Original: Starting from this point, we ran 100 epochs of batch gradient descent (batch size was equal to the number of training sentences), with a learning rate of 0.01.
Revised: Starting from this point, we ran 100 epochs of batch gradient descent (batch size was equal to the number of training sentences), with a learning rate of 0.01.

Original: As a loss, we used the cross entropy loss plus 0.01 times the (cid:96) 2 norm of the new embedding.
Revised: As a loss, we used the cross entropy loss plus 0.01 times the (cid:96) 2 norm of the new embedding.

Original: B.1. L IST OF HUNDRED WORDS
Revised: B.1. L IST OF HUNDRED WORDS

Original: [ ’ ab ’ , ’ a b s o l u t e l y ’ , ’ a g r i c u l t u r a l ’ , ’ aim ’ , ’ animals ’ , ’ announcing (cid:44) → ’ , ’ arguments ’ , ’ a s s i s t ’ , ’ averaged ’ , ’ bass ’ , ’ b u l l i s h ’ , ’ (cid:44) → c a l c u l a t i o n s ’ , ’ c a r e f u l l y ’ , ’ claiming ’ , ’ compare ’ , ’ conceded (cid:44) → ’ , ’ congressman ’ , ’ consortium ’ , ’ c o n t e s t ’ , ’ c
Revised: [ ’ ab ’ , ’ a b s o l u t e l y ’ , ’ a g r i c u l t u r a l ’ , ’ aim ’ , ’ animals ’ , ’ announcing (cid:44) → ’ , ’ arguments ’ , ’ a s s i s t ’ , ’ averaged ’ , ’ bass ’ , ’ b u l l i s h ’ , ’ (cid:44) → c a l c u l a t i o n s ’ , ’ c a r e f u l l y ’ , ’ claiming ’ , ’ compare ’ , ’ conceded (cid:44) → ’ , ’ congressman ’ , ’ consortium ’ , ’ c o n t e s t ’ , ’ c

Original: r e a t i o n ’ , ’ (cid:44) → cumulative ’ , ’ danger ’ , ’ darman ’ , ’ die ’ , ’ d i s c
Revised: r e a t i o n ’ , ’ (cid:44) → cumulative ’ , ’ danger ’ , ’ darman ’ , ’ die ’ , ’ d i s c

Original: r i m i n a t i o n ’ , ’ (cid:44) → disney ’ , ’ dominant ’ , ’ dorrance ’ , ’ edwards ’ , ’ e f f i
Revised: r i m i n a t i o n ’ , ’ (cid:44) → disney ’ , ’ dominant ’ , ’ dorrance ’ , ’ edwards ’ , ’ e f f i

Original: c i e n c y ’ , ’ (cid:44) → e l d e r l y ’ , ’ enable ’ , ’ encouraging ’ , ’ e n t r y ’ , ’ (cid:44) → e n v i r o n m e n t a l i s t s ’ , ’ e x e c u t i o n ’ , ’ e x p e n d i t u r e s ’ , ’ f a c t s ’ , ’ (cid:44) → formula ’ , ’ gaf ’ , ’ geneva ’ , ’ globe ’ , ’ g o l f ’ , ’ h e a l t h c a r e ’ , ’ (cid:44) → homeless ’ , ’ honor ’ , ’ horse ’ , ’ i n c e s t ’ , ’ informed ’ , ’ (cid:44) → i n v e s t i g a t
Revised: c i e n c y ’ , ’ (cid:44) → e l d e r l y ’ , ’ enable ’ , ’ encouraging ’ , ’ e n t r y ’ , ’ (cid:44) → e n v i r o n m e n t a l i s t s ’ , ’ e x e c u t i o n ’ , ’ e x p e n d i t u r e s ’ , ’ f a c t s ’ , ’ (cid:44) → formula ’ , ’ gaf ’ , ’ geneva ’ , ’ globe ’ , ’ g o l f ’ , ’ h e a l t h c a r e ’ , ’ (cid:44) → homeless ’ , ’ honor ’ , ’ horse ’ , ’ i n c e s t ’ , ’ informed ’ , ’ (cid:44) → i n v e s t i g a t

Original: o r s ’ , ’ i r o n ’ , ’ jackson ’ , ’ judgment ’ , ’ knight ’ , ’ (cid:44) → lake ’ , ’ lend ’ , ’ l o u
Revised: o r s ’ , ’ i r o n ’ , ’ jackson ’ , ’ judgment ’ , ’ knight ’ , ’ (cid:44) → lake ’ , ’ lend ’ , ’ l o u

Original: i s v i l l e ’ , ’ lowest ’ , ’ l u c r a t i v e ’ , ’ (cid:44) → maturing ’ , ’ minute ’ , ’ m
Revised: i s v i l l e ’ , ’ lowest ’ , ’ l u c r a t i v e ’ , ’ (cid:44) → maturing ’ , ’ minute ’ , ’ m

Original: i s s
Revised: i s s

Original: i s s i p p
Revised: i s s i p p

Original: i ’ , ’ motorola ’ , ’museum ’ , ’ (cid:44) → nabisco ’ , ’ n e t h e r l a n d s ’ , ’ n i g e l ’ , ’ nine − month ’ , ’ owning ’ , ’ (cid:44) → p e t r o c h
Revised: i ’ , ’ motorola ’ , ’museum ’ , ’ (cid:44) → nabisco ’ , ’ n e t h e r l a n d s ’ , ’ n i g e l ’ , ’ nine − month ’ , ’ owning ’ , ’ (cid:44) → p e t r o c h

Original: e
Revised: e

Original: m i c a l ’ , ’ pionee r ’ , ’ pre par e ’ , ’ p r i n t ’ , ’ pro − choice ’ , (cid:44) → ’ recognized ’ , ’ r e f e r r e d ’ , ’ regarded ’ , ’ r e j e c t i o n ’ , ’ (cid:44) → r e q u e s t s ’ , ’ r e s o r t s ’ , ’ r e s p o n s i
Revised: m i c a l ’ , ’ pionee r ’ , ’ pre par e ’ , ’ p r i n t ’ , ’ pro − choice ’ , (cid:44) → ’ recognized ’ , ’ r e f e r r e d ’ , ’ regarded ’ , ’ r e j e c t i o n ’ , ’ (cid:44) → r e q u e s t s ’ , ’ r e s o r t s ’ , ’ r e s p o n s i

Original: b i l
Revised: b i l

Original: i t i e s ’ , ’ r o l l e d ’ , ’ s a n s u i ’ (cid:44) → , ’ s e r v i n g ’ , ’ setba ck ’ , ’ s i m i l a r l y ’ , ’ somewhere ’ , ’ sounds ’ , (cid:44) → ’ s t a f f e r s ’ , ’ s t o l e n ’ , ’ t r e a s u r y s ’ , ’ t r e a t ’ , ’ t r u t h ’ , ’ utah ’ , (cid:44) → ’ v u l n e r a b l e ’ , ’ ward ’ , ’ warsaw ’ , ’ wedtech ’ , ’ wheat ’ , ’ (cid:44) → wisconsin ’ ]
Revised: i t i e s ’ , ’ r o l l e d ’ , ’ s a n s u i ’ (cid:44) → , ’ s e r v i n g ’ , ’ setba ck ’ , ’ s i m i l a r l y ’ , ’ somewhere ’ , ’ sounds ’ , (cid:44) → ’ s t a f f e r s ’ , ’ s t o l e n ’ , ’ t r e a s u r y s ’ , ’ t r e a t ’ , ’ t r u t h ’ , ’ utah ’ , (cid:44) → ’ v u l n e r a b l e ’ , ’ ward ’ , ’ warsaw ’ , ’ wedtech ’ , ’ wheat ’ , ’ (cid:44) → wisconsin ’ ]

Original: C L OG - PROBABILITY ANALYSIS BROKEN OUT
Revised: C L OG - PROBABILITY ANALYSIS BROKEN OUT

Original: new word num train
Revised: new word num train

Original: Approach
Revised: Approach

Original: New word is correct
Revised: New word is correct

Original: Wrong and irrelevant
Revised: Wrong and irrelevant

Original: Wrong but relevant borrow centroid -9. borrow opt centroid -6.
Revised: Wrong but relevant borrow centroid -9. borrow opt centroid -6.

Original: borrow centroid -9.
Revised: borrow centroid -9.

Original: borrow opt centroid -5.
Revised: borrow opt centroid -5.

Original: borrow centroid -9.
Revised: borrow centroid -9.

Original: borrow opt centroid -6.
Revised: borrow opt centroid -6.

Original: borrow with word -8.
Revised: borrow with word -8.

Original: cowboys centroid -9.
Revised: cowboys centroid -9.

Original: cowboys opt centroid -7.
Revised: cowboys opt centroid -7.

Original: cowboys centroid -9.
Revised: cowboys centroid -9.

Original: cowboys opt centroid -6.
Revised: cowboys opt centroid -6.

Original: cowboys centroid -9.
Revised: cowboys centroid -9.

Original: cowboys opt centroid -6.
Revised: cowboys opt centroid -6.

Original: cowboys with word -8.
Revised: cowboys with word -8.

Original: immune centroid -9.
Revised: immune centroid -9.

Original: immune opt centroid -8.
Revised: immune opt centroid -8.

Original: immune centroid -9.
Revised: immune centroid -9.

Original: immune opt centroid -7.
Revised: immune opt centroid -7.

Original: immune centroid -9.
Revised: immune centroid -9.

Original: immune opt centroid -6.
Revised: immune opt centroid -6.

Original: immune with word -9.
Revised: immune with word -9.

Original: rice centroid -8.
Revised: rice centroid -8.

Original: rice opt centroid -7.
Revised: rice opt centroid -7.

Original: rice centroid -8.
Revised: rice centroid -8.

Original: rice opt centroid -6.
Revised: rice opt centroid -6.

Original: rice centroid -8.
Revised: rice centroid -8.

Original: rice opt centroid -5.
Revised: rice opt centroid -5.

Original: rice with word -9.
Revised: rice with word -9.

Original: D E MBEDDING SIMILARITY ANALYSES
Revised: D E MBEDDING SIMILARITY ANALYSES

Original: It has often been noted that relationships between word embeddings capture semantic features of the words, such as analogies between words (Mikolov et al., 2013).
Revised: It has often been noted that relationships between word embeddings capture semantic features of the words, such as analogies between words (Mikolov et al., 2013).

Original: Thus it is natural to ask to what extent the word vectors produced by one-shot learning produce similarity structures close to those produced by full training with the word.
Revised: Thus it is natural to ask to what extent the word vectors produced by one-shot learning produce similarity structures close to those produced by full training with the word.

Original: To address this question, we computed the dot product 3 of the new word output embedding with all other word output embeddings.
Revised: To address this question, we computed the dot product 3 of the new word output embedding with all other word output embeddings.

Original: This vector of dot-product results can be thought of as a similarity map for the new word.
Revised: This vector of dot-product results can be thought of as a similarity map for the new word.

Original: We then compared these similarity maps to the similarity maps produced by full training with the word by computing correlations between the similarity maps.
Revised: We then compared these similarity maps to the similarity maps produced by full training with the word by computing correlations between the similarity maps.

Original: (See Fig.
Revised: (See Fig.

Original: 7 for our results.)
Revised: 7 for our results.)

Original: Different runs of full training with the word produced very correlated similarity structures (all correlations greater than 0.9).
Revised: Different runs of full training with the word produced very correlated similarity structures (all correlations greater than 0.9).

Original: Thus there does appear to be some consistent semantic structure that the embeddings are capturing.
Revised: Thus there does appear to be some consistent semantic structure that the embeddings are capturing.

Original: However, neither the centroid nor the optimizing one-shot learning methods seemed to be capturing this structure.
Revised: However, neither the centroid nor the optimizing one-shot learning methods seemed to be capturing this structure.

Original: The centroid method actually produced similarity structures that were negatively correlated with the full training similarity structures in three out of four cases!
Revised: The centroid method actually produced similarity structures that were negatively correlated with the full training similarity structures in three out of four cases!

Original: The optimizing method produced slightly more similar structures, but only in so far as most of its similarity structures were effectively uncorrelated with the full training similarity structure.
Revised: The optimizing method produced slightly more similar structures, but only in so far as most of its similarity structures were effectively uncorrelated with the full training similarity structure.

Original: Only for one word each did the methods produce a similarity structure that was strongly positively correlated with the structure produced by full training with the word.
Revised: Only for one word each did the methods produce a similarity structure that was strongly positively correlated with the structure produced by full training with the word.

Original: These results are difﬁcult to interpret.
Revised: These results are difﬁcult to interpret.

Original: On the one hand, the consistency of the semantic knowledge produced by the network when fully trained with the word suggests that it is discovering important structure.
Revised: On the one hand, the consistency of the semantic knowledge produced by the network when fully trained with the word suggests that it is discovering important structure.

Original: On the other hand, the later learning approaches actually produce even more internally consistency, at least when training with several sentences.
Revised: On the other hand, the later learning approaches actually produce even more internally consistency, at least when training with several sentences.

Original: Full training with the word produces an average correlation of 0.93 between different training runs, whereas optimizing with all ten sentences produces an average correlation of 0.97 (and optimizing with 1 sentence produces an average correlation of 0.50).
Revised: Full training with the word produces an average correlation of 0.93 between different training runs, whereas optimizing with all ten sentences produces an average correlation of 0.97 (and optimizing with 1 sentence produces an average correlation of 0.50).

Original: Furthermore, the results of section 3.2 above suggest that our approach is doing about as well at distinguishing contexts where the new word does and does not appear.
Revised: Furthermore, the results of section 3.2 above suggest that our approach is doing about as well at distinguishing contexts where the new word does and does not appear.

Original: That is, the optimizing runs are extracting consistent structure as well, they are just creating different representational structures than the full training runs.
Revised: That is, the optimizing runs are extracting consistent structure as well, they are just creating different representational structures than the full training runs.

Original: These different representations may be explained by the removal of some contextual cues when the sentences are presented in isolation (noted above) – the networks doing full training with the word are able to pay attention to relationships across sentence boundaries, which may give them exposure to word co-occurrences that the one-shot learning approach is missing.
Revised: These different representations may be explained by the removal of some contextual cues when the sentences are presented in isolation (noted above) – the networks doing full training with the word are able to pay attention to relationships across sentence boundaries, which may give them exposure to word co-occurrences that the one-shot learning approach is missing.

Original: Including the surrounding context sentences in the training might result in more similar representational structures.
Revised: Including the surrounding context sentences in the training might result in more similar representational structures.

Original: E O THER RELATED WORK
Revised: E O THER RELATED WORK

Original: A commenter on a draft of this paper also noted that Herbelot & Baroni (2017) pursued related questions and independently came to some of the same conclusions we reached.
Revised: A commenter on a draft of this paper also noted that Herbelot & Baroni (2017) pursued related questions and independently came to some of the same conclusions we reached.

Original: However, we believe our results improve upon their work in several important ways.
Revised: However, we believe our results improve upon their work in several important ways.

Original: First, while they were only able to show beneﬁts on training from deﬁnitions of a word, we show beneﬁts of learning from a word in context.
Revised: First, while they were only able to show beneﬁts on training from deﬁnitions of a word, we show beneﬁts of learning from a word in context.

Original: Furthermore, they only evaluated on embedding similarity, while we show behaviorally relevant improvements.
Revised: Furthermore, they only evaluated on embedding similarity, while we show behaviorally relevant improvements.

Original: This is important, because one conclusion from our results of our embedding similarity analyses in Appendix D show that dissimilar embeddings may nevertheless offer comparable performance.
Revised: This is important, because one conclusion from our results of our embedding similarity analyses in Appendix D show that dissimilar embeddings may nevertheless offer comparable performance.

Original: Finally, we explore in depth the effects of varying data parameters like the number of training sentences, and analyze where the improvements are occurring, both from the perspectives of parameters and outputs.
Revised: Finally, we explore in depth the effects of varying data parameters like the number of training sentences, and analyze where the improvements are occurring, both from the perspectives of parameters and outputs.