Original: L EARNING TO O RGANIZE K NOWLEDGE WITH N-G RAM M ACHINES
Revised: L EARNING TO O RGANIZE K NOWLEDGE WITH N-G RAM M ACHINES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) had great success on NLP tasks such as language modeling, machine translation and certain question answering (QA) tasks.
Revised: Deep neural networks (DNNs) had great success on NLP tasks such as language modeling, machine translation and certain question answering (QA) tasks.

Original: However, the success is limited at more knowledge intensive tasks such as QA from a big corpus.
Revised: However, the success is limited at more knowledge intensive tasks such as QA from a big corpus.

Original: Existing end-to-end deep QA models (Miller et al., 2016; Weston et al., 2014) need to read the entire text after observing the question and therefore their complexity is linear in the document size.
Revised: Existing end-to-end deep QA models (Miller et al., 2016; Weston et al., 2014) need to read the entire text after observing the question and therefore their complexity is linear in the document size.

Original: This is prohibitive for practical tasks such as QA from Wikipedia, a novel, or the Web.
Revised: This is prohibitive for practical tasks such as QA from Wikipedia, a novel, or the Web.

Original: We propose to solve this scalability issue by using symbolic meaning representations, which can be indexed and retrieved efﬁciently with complexity that is independent of the text size.
Revised: We propose to solve this scalability issue by using symbolic meaning representations, which can be indexed and retrieved efﬁciently with complexity that is independent of the text size.

Original: More speciﬁcally, we use sequence-to-sequence models to encode knowledge symbolically and generate programs to answer questions from the encoded knowledge.
Revised: More speciﬁcally, we use sequence-to-sequence models to encode knowledge symbolically and generate programs to answer questions from the encoded knowledge.

Original: We apply our approach, called the N-Gram Machine (NGM), to the bAbI tasks (Weston et al., 2015) and a special version of them (“life-long bAbI”) which has stories of up to 10 million sentences.
Revised: We apply our approach, called the N-Gram Machine (NGM), to the bAbI tasks (Weston et al., 2015) and a special version of them (“life-long bAbI”) which has stories of up to 10 million sentences.

Original: Our experiments show that NGM can successfully solve both of these tasks accurately and efﬁciently.
Revised: Our experiments show that NGM can successfully solve both of these tasks accurately and efﬁciently.

Original: Unlike fully differentiable memory models, NGM’s time complexity and answering quality are not affected by the story length.
Revised: Unlike fully differentiable memory models, NGM’s time complexity and answering quality are not affected by the story length.

Original: The whole system of NGM is trained end-to-end with REINFORCE (Williams, 1992).
Revised: The whole system of NGM is trained end-to-end with REINFORCE (Williams, 1992).

Original: To avoid high variance in gradient estimation, which is typical in discrete latent variable models, we use beam search instead of sampling.
Revised: To avoid high variance in gradient estimation, which is typical in discrete latent variable models, we use beam search instead of sampling.

Original: To tackle the exponentially large search space, we use an auto-encoding text summarization objective and a structure tweak procedure to iteratively reduce and reﬁne the search space.
Revised: To tackle the exponentially large search space, we use an auto-encoding text summarization objective and a structure tweak procedure to iteratively reduce and reﬁne the search space.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Although there is a great deal of recent research on extracting structured knowledge from text (Dong et al., 2014; Mitchell et al., 2015) and answering questions from structured knowledge stores (Dong & Lapata, 2016; Jia & Liang, 2016; Liang et al., 2017), much less progress has been made on either the problem of unifying these approaches in an end-to-end model or the problem of removing the bottleneck of relying on human experts to design the schema and annotate examples for information extraction.
Revised: Although there is a great deal of recent research on extracting structured knowledge from text (Dong et al., 2014; Mitchell et al., 2015) and answering questions from structured knowledge stores (Dong & Lapata, 2016; Jia & Liang, 2016; Liang et al., 2017), much less progress has been made on either the problem of unifying these approaches in an end-to-end model or the problem of removing the bottleneck of relying on human experts to design the schema and annotate examples for information extraction.

Original: In particular, traditional natural language processing and information extraction approaches are too labor-intensive and brittle for answering open domain questions from large corpus, and existing end-to-end deep QA models (e.g., (Miller et al., 2016; Weston et al., 2014)) lack scalability and the ability to integrate domain knowledge.
Revised: In particular, traditional natural language processing and information extraction approaches are too labor-intensive and brittle for answering open domain questions from large corpus, and existing end-to-end deep QA models (e.g., (Miller et al., 2016; Weston et al., 2014)) lack scalability and the ability to integrate domain knowledge.

Original: This paper presents a new QA system that treats both the schema and the content of a structured storage as discrete hidden variables, and infers these structures automatically only under weak supervisions (such as QA pair examples).
Revised: This paper presents a new QA system that treats both the schema and the content of a structured storage as discrete hidden variables, and infers these structures automatically only under weak supervisions (such as QA pair examples).

Original: The structured storage we consider is simply a set of “n-grams”, which we show can represent a wide range of semantics, and can be indexed for efﬁcient computations at scale.
Revised: The structured storage we consider is simply a set of “n-grams”, which we show can represent a wide range of semantics, and can be indexed for efﬁcient computations at scale.

Original: We present an end-to-end trainable system which combines an text auto-encoding component for encoding knowledge, and a memory enhanced sequence to sequence component for answering questions from the encoded knowledge.
Revised: We present an end-to-end trainable system which combines an text auto-encoding component for encoding knowledge, and a memory enhanced sequence to sequence component for answering questions from the encoded knowledge.

Original: We show that the method scales well on artiﬁcially generated stories of up to 10 million lines long (Figure 4).
Revised: We show that the method scales well on artiﬁcially generated stories of up to 10 million lines long (Figure 4).

Original: The system we present here illustrates how end-to-end learning and scalability can be made possible through a symbolic knowledge storage.
Revised: The system we present here illustrates how end-to-end learning and scalability can be made possible through a symbolic knowledge storage.

Original: 1. Q UESTION A NSWERING A S A T ESTBED FOR T EXT U NDERSTANDING
Revised: 1. Q UESTION A NSWERING A S A T ESTBED FOR T EXT U NDERSTANDING

Original: We ﬁrst deﬁne question answering as producing the answer a given sentences s =
Revised: We ﬁrst deﬁne question answering as producing the answer a given sentences s =

Original: ( s 1 , . . .
Revised: ( s 1 , . . .

Original: , s | s | ) and question q .
Revised: , s | s | ) and question q .

Original: Each sentence s i is represented as a sequence of words, i.e. s i =
Revised: Each sentence s i is represented as a sequence of words, i.e. s i =

Original: ( w i 1 , . . .
Revised: ( w i 1 , . . .

Original: , w in ) .
Revised: , w in ) .

Original: And the question q is also represented as a sequence of words, i.e. q = ( w q 1 , . . .
Revised: And the question q is also represented as a sequence of words, i.e. q = ( w q 1 , . . .

Original: , w qm ) .
Revised: , w qm ) .

Original: We focus on extractive question answering, where the answer a is always a word in one of the sentences.
Revised: We focus on extractive question answering, where the answer a is always a word in one of the sentences.

Original: Despite its simple form, question answering can be incredibly challenging.
Revised: Despite its simple form, question answering can be incredibly challenging.

Original: Consider answering the question “Who was Adam Smith’s wife?’
Revised: Consider answering the question “Who was Adam Smith’s wife?’

Original: from the Web.
Revised: from the Web.

Original: There exists the following snippet from a reputable website “Smith was born in Kirkcaldy, ... (skipped 35 words) ...
Revised: There exists the following snippet from a reputable website “Smith was born in Kirkcaldy, ... (skipped 35 words) ...

Original: In 1720, he married
Revised: In 1720, he married

Original: Margaret Douglas” .
Revised: Margaret Douglas” .

Original: An ideal system needs to identify that “Smith” in this text is equivalent to “Adam
Revised: An ideal system needs to identify that “Smith” in this text is equivalent to “Adam

Original: Smith” in the question; “he” is referencing “Smith”; and text expressions of the form “X married Y” answer questions of the form “Who was X’s wife?”.
Revised: Smith” in the question; “he” is referencing “Smith”; and text expressions of the form “X married Y” answer questions of the form “Who was X’s wife?”.

Original: There are three main challenges in this process:
Revised: There are three main challenges in this process:

Original: Scalability A typical QA system, such as Watson (Ferrucci et al., 2010) or any of the commercial search engines, processes millions or even billions of documents for answering a question.
Revised: Scalability A typical QA system, such as Watson (Ferrucci et al., 2010) or any of the commercial search engines, processes millions or even billions of documents for answering a question.

Original: Yet the response time is restricted to a few seconds or even fraction of seconds.
Revised: Yet the response time is restricted to a few seconds or even fraction of seconds.

Original: Answering any possible question that is answerable from a large corpus with limited time means that the information need to be organized and indexed for fast access and reasoning.
Revised: Answering any possible question that is answerable from a large corpus with limited time means that the information need to be organized and indexed for fast access and reasoning.

Original: Representation
Revised: Representation

Original: A fundamental building block for text understanding is paraphrasing, e.g., knowing that “X married Y” leads to “X’s wife is Y”.
Revised: A fundamental building block for text understanding is paraphrasing, e.g., knowing that “X married Y” leads to “X’s wife is Y”.

Original: By observing users’ interactions with a search engine a system may capture certain equivalence relationships among expressions in questions (Baker, 2010).
Revised: By observing users’ interactions with a search engine a system may capture certain equivalence relationships among expressions in questions (Baker, 2010).

Original: However, given these observations, there is still a wide range of choices for how the meaning of expressions can be represented.
Revised: However, given these observations, there is still a wide range of choices for how the meaning of expressions can be represented.

Original: Open information extraction approaches (Angeli et al., 2015) represent expressions by themselves, and relies on corpus statistics to calculate their similarities.
Revised: Open information extraction approaches (Angeli et al., 2015) represent expressions by themselves, and relies on corpus statistics to calculate their similarities.

Original: This approach leads to data sparsity, and brittleness on out-of-domain text.
Revised: This approach leads to data sparsity, and brittleness on out-of-domain text.

Original: Vector space approaches (Mikolov et al., 2013; Weston et al., 2014; Neelakantan et al., 2015; Miller et al., 2016) embeds text expressions into latent continuous spaces.
Revised: Vector space approaches (Mikolov et al., 2013; Weston et al., 2014; Neelakantan et al., 2015; Miller et al., 2016) embeds text expressions into latent continuous spaces.

Original: They allow soft matching of semantics for arbitrary expressions, but are hard to scale to knowledge intensive tasks, which require inference with large amount of data.
Revised: They allow soft matching of semantics for arbitrary expressions, but are hard to scale to knowledge intensive tasks, which require inference with large amount of data.

Original: Inference The most basic form of reasoning is to combine pieces of information and form a new piece, for example, from co-reference (“He”, “Adam Smith”) and has_spouse (“He”, “Margaret
Revised: Inference The most basic form of reasoning is to combine pieces of information and form a new piece, for example, from co-reference (“He”, “Adam Smith”) and has_spouse (“He”, “Margaret

Original: Douglas”) to has_spouse (“Adam Smith”, “Margaret Douglas”).
Revised: Douglas”) to has_spouse (“Adam Smith”, “Margaret Douglas”).

Original: As the number of pieces that need to be put together grows the search space grows exponentially – making it a hard search problem (Lao et al., 2011).
Revised: As the number of pieces that need to be put together grows the search space grows exponentially – making it a hard search problem (Lao et al., 2011).

Original: Good representations dramatically reduce the search space.
Revised: Good representations dramatically reduce the search space.

Original: Among these challenges, scalability is the main blocker to a practical end-to-end solution.
Revised: Among these challenges, scalability is the main blocker to a practical end-to-end solution.

Original: Now we present our new framework, which meets all three challenges described above.
Revised: Now we present our new framework, which meets all three challenges described above.

Original: [Figure]
Revised: [Figure]

Original: We propose to solve the scalability issue of DNN text understanding models by learning to represent the meaning of text as a symbolic knowledge storage.
Revised: We propose to solve the scalability issue of DNN text understanding models by learning to represent the meaning of text as a symbolic knowledge storage.

Original: Because the storage can be indexed before being used for question answering, the inference step can be done very efﬁciently with complexity that is independent of the original text size.
Revised: Because the storage can be indexed before being used for question answering, the inference step can be done very efﬁciently with complexity that is independent of the original text size.

Original: More speciﬁcally the structured storage we consider is simply a set of “n-grams”, which we show can represent complex semantics presented in bAbI tasks (Weston et al., 2015) and can be indexed for efﬁcient computations at scale.
Revised: More speciﬁcally the structured storage we consider is simply a set of “n-grams”, which we show can represent complex semantics presented in bAbI tasks (Weston et al., 2015) and can be indexed for efﬁcient computations at scale.

Original: Each n-gram consists of a sequence of tokens, and each token can be a word, or any predeﬁned special symbol.
Revised: Each n-gram consists of a sequence of tokens, and each token can be a word, or any predeﬁned special symbol.

Original: Different from conventional n-grams, which are contiguous chunks of text, the “n-grams” considered here can be any combination of arbitrary words and symbols.
Revised: Different from conventional n-grams, which are contiguous chunks of text, the “n-grams” considered here can be any combination of arbitrary words and symbols.

Original: The whole system (Figure 1) consists of learnable components which convert text into symbolic knowledge storage and questions into programs (details in Section 2).
Revised: The whole system (Figure 1) consists of learnable components which convert text into symbolic knowledge storage and questions into programs (details in Section 2).

Original: A deterministic executor executes the programs against the knowledge storage and produces answers.
Revised: A deterministic executor executes the programs against the knowledge storage and produces answers.

Original: The whole system is trained end-to-end with no human annotation other than the expected answers to a set of question-text pairs.
Revised: The whole system is trained end-to-end with no human annotation other than the expected answers to a set of question-text pairs.

Original: Training highly expressive discrete latent variable models on large datasets is a challenging problem due to the difﬁculties posed by inference (Hinton et al., 2006; Mnih & Gregor, 2014) – speciﬁcally the huge variance in the gradient estimation.
Revised: Training highly expressive discrete latent variable models on large datasets is a challenging problem due to the difﬁculties posed by inference (Hinton et al., 2006; Mnih & Gregor, 2014) – speciﬁcally the huge variance in the gradient estimation.

Original: Mnih & Gregor (2014) applies REINFORCE (Williams, 1992) to optimize a variational lower-bound of the data log-likelihood, but relies on complex schemes to reduce variance in the gradient estimation.
Revised: Mnih & Gregor (2014) applies REINFORCE (Williams, 1992) to optimize a variational lower-bound of the data log-likelihood, but relies on complex schemes to reduce variance in the gradient estimation.

Original: We use a different set of techniques to learn N-Gram
Revised: We use a different set of techniques to learn N-Gram

Original: Machines, which are simpler and with less model assumptions.
Revised: Machines, which are simpler and with less model assumptions.

Original: Instead of Monte Carlo integration, which is known for high variance and low data efﬁciency, we apply beam search .
Revised: Instead of Monte Carlo integration, which is known for high variance and low data efﬁciency, we apply beam search .

Original: Beam search is very effective for deterministic environments with sparse reward (Liang et al., 2017; Guu et al., 2017), but it leads to a search problem.
Revised: Beam search is very effective for deterministic environments with sparse reward (Liang et al., 2017; Guu et al., 2017), but it leads to a search problem.

Original: At inference time, since only a few top hypotheses are kept in the beam, search could get stuck and not receive any reward, preventing learning.
Revised: At inference time, since only a few top hypotheses are kept in the beam, search could get stuck and not receive any reward, preventing learning.

Original: We solve this hard search problem by 1) having an auto-encoding objective to bias the knowledge encoder to more interesting hypotheses; and 2) having a structural tweak procedure which retrospectively corrects the inconsistency among multiple hypotheses so that reward can be achieved (See Section 2.2).
Revised: We solve this hard search problem by 1) having an auto-encoding objective to bias the knowledge encoder to more interesting hypotheses; and 2) having a structural tweak procedure which retrospectively corrects the inconsistency among multiple hypotheses so that reward can be achieved (See Section 2.2).

Original:  F RAMEWORK
Revised:  F RAMEWORK

Original: In this section we ﬁrst describe the N-Gram Machine (NGM) model structure, which contains three sequence to sequence modules, and an executor that executes programs against knowledge storage.
Revised: In this section we ﬁrst describe the N-Gram Machine (NGM) model structure, which contains three sequence to sequence modules, and an executor that executes programs against knowledge storage.

Original: Then we describe how this model can be trained end-to-end with reinforcement learning.
Revised: Then we describe how this model can be trained end-to-end with reinforcement learning.

Original: We use the bAbI dataset (Weston et al., 2015) as running examples.
Revised: We use the bAbI dataset (Weston et al., 2015) as running examples.

Original: 2. NGM M ODEL S TRUCTURE
Revised: 2. NGM M ODEL S TRUCTURE

Original: Knowledge storage Given a sequence of sentences s = { s 1 , . . .
Revised: Knowledge storage Given a sequence of sentences s = { s 1 , . . .

Original: , s T } , our knowledge storage is a collection of knowledge tuples Γ =
Revised: , s T } , our knowledge storage is a collection of knowledge tuples Γ =

Original: { Γ 1 , . . .
Revised: { Γ 1 , . . .

Original: , Γ T } .
Revised: , Γ T } .

Original: The tuple Γ i has two parts: a time stamp i and a sequence of symbols ( γ 1 , . .
Revised: The tuple Γ i has two parts: a time stamp i and a sequence of symbols ( γ 1 , . .

Original: . , γ N ) , where each symbol γ j is either a word from the sentence
Revised: . , γ N ) , where each symbol γ j is either a word from the sentence

Original: s i or a special symbol from a pre-deﬁned set.
Revised: s i or a special symbol from a pre-deﬁned set.

Original: The time stamps in the tuple are useful for reasoning about time and are just sentence indices for the bAbI task.
Revised: The time stamps in the tuple are useful for reasoning about time and are just sentence indices for the bAbI task.

Original: The knowledge storage is probabilistic – each tuple Γ i also has a probability, and the probability of the knowledge storage is the product of the probabilities of all its tuples (Equation 1).
Revised: The knowledge storage is probabilistic – each tuple Γ i also has a probability, and the probability of the knowledge storage is the product of the probabilities of all its tuples (Equation 1).

Original: An example of a knowledge storage is shown in Table 1.
Revised: An example of a knowledge storage is shown in Table 1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Programs Programs in the N-Gram Machine are similar to those introduced in Neural Symbolic
Revised: Programs Programs in the N-Gram Machine are similar to those introduced in Neural Symbolic

Original: Machine (Liang et al., 2017), except that our functions operate on n-grams (i.e. knowledge tuples)instead of Freebase triples.
Revised: Machine (Liang et al., 2017), except that our functions operate on n-grams (i.e. knowledge tuples)instead of Freebase triples.

Original: In general, functions specify how symbols can be retrieved from a knowledge storage.
Revised: In general, functions specify how symbols can be retrieved from a knowledge storage.

Original: Speciﬁcally, a function in NGM use a preﬁx (or sufﬁx) to retrieve symbols from tuples – i.e. if a preﬁx “matches” a tuple, the immediate next symbol in the tuple is returned.
Revised: Speciﬁcally, a function in NGM use a preﬁx (or sufﬁx) to retrieve symbols from tuples – i.e. if a preﬁx “matches” a tuple, the immediate next symbol in the tuple is returned.

Original: For the bAbI tasks, we deﬁne four functions, which are illustrated in Table 2: Function Hop and HopFR return symbols from all the matched tuples while function Argmax and ArgmaxFR return symbols from the latest matches (i.e. the tuples with the latest the time stamp in all the matches)
Revised: For the bAbI tasks, we deﬁne four functions, which are illustrated in Table 2: Function Hop and HopFR return symbols from all the matched tuples while function Argmax and ArgmaxFR return symbols from the latest matches (i.e. the tuples with the latest the time stamp in all the matches)

Original: [Table]
Revised: [Table]

Original: More formally a program C is a list of expressions c 1 ...
Revised: More formally a program C is a list of expressions c 1 ...

Original: c N , where each c i is either a special expression
Revised: c N , where each c i is either a special expression

Original: Return indicating the end of the program, or is of the form of ( F, A 1 ...
Revised: Return indicating the end of the program, or is of the form of ( F, A 1 ...

Original: A L ) where F is a function in
Revised: A L ) where F is a function in

Original: Table 2 and A 1 ...
Revised: Table 2 and A 1 ...

Original: A L are L input arguments of F .
Revised: A L are L input arguments of F .

Original: When an expression is executed, it returns a set of symbols by matching its arguments in Γ , and stores the result in a new variable symbol (e.g., ( V1 ) ) to reference the result.
Revised: When an expression is executed, it returns a set of symbols by matching its arguments in Γ , and stores the result in a new variable symbol (e.g., ( V1 ) ) to reference the result.

Original: Consider executing the expression ( Hop V1 to ) against the knowledge storage in Table 1, assuming that V1 is a variable that stores { mary } from previous executions.
Revised: Consider executing the expression ( Hop V1 to ) against the knowledge storage in Table 1, assuming that V1 is a variable that stores { mary } from previous executions.

Original: The execution returns a set of two symbols { kitchen , garden }.
Revised: The execution returns a set of two symbols { kitchen , garden }.

Original: Similarly, executing ( Argmax V1 to ) would instead produces { garden }.
Revised: Similarly, executing ( Argmax V1 to ) would instead produces { garden }.

Original: Though executing a program on a knowledge storage as described above is deterministic, probabilities are assigned to the execution results, which are the products of probabilities of the corresponding program and knowledge storage.
Revised: Though executing a program on a knowledge storage as described above is deterministic, probabilities are assigned to the execution results, which are the products of probabilities of the corresponding program and knowledge storage.

Original: Since the knowledge storage can be indexed using a data structure such as hash tables, the program execution time is independent of the size of the knowledge storage.
Revised: Since the knowledge storage can be indexed using a data structure such as hash tables, the program execution time is independent of the size of the knowledge storage.

Original: [Figure]
Revised: [Figure]

Original: Seq2Seq components Our N-Gram Machine uses three sequence-to-sequence (Sutskever et al., 2014) neural network models to deﬁne probability distributions over knowledge tuples and programs.
Revised: Seq2Seq components Our N-Gram Machine uses three sequence-to-sequence (Sutskever et al., 2014) neural network models to deﬁne probability distributions over knowledge tuples and programs.

Original: As illustrated in Figure 2, these models are:
Revised: As illustrated in Figure 2, these models are:

Original: • A knowledge encoder that converts sentences to knowledge tuples and deﬁnes a distribution P (Γ i | s i , s i − 1 ; θ enc ) .
Revised: • A knowledge encoder that converts sentences to knowledge tuples and deﬁnes a distribution P (Γ i | s i , s i − 1 ; θ enc ) .

Original: It is conditioned on the previous sentence s i − 1 to handle cross sentence linguistic phenomenons such as co-references 2 .
Revised: It is conditioned on the previous sentence s i − 1 to handle cross sentence linguistic phenomenons such as co-references 2 .

Original: The probability of a knowledge storage Γ = { Γ 1 . . .
Revised: The probability of a knowledge storage Γ = { Γ 1 . . .

Original: Γ n } deﬁned as the product of its knowledge tuples’ probabilities:
Revised: Γ n } deﬁned as the product of its knowledge tuples’ probabilities:

Original: [Equation]
Revised: [Equation]

Original: • A knowledge decoder that converts tuples back to sentences and deﬁnes a distribution P ( s i | Γ i , s i − 1 ; θ dec ) .
Revised: • A knowledge decoder that converts tuples back to sentences and deﬁnes a distribution P ( s i | Γ i , s i − 1 ; θ dec ) .

Original: It enables auto-encoding training, which is crucial for ﬁnding good knowledge representations (See Section 2.3).
Revised: It enables auto-encoding training, which is crucial for ﬁnding good knowledge representations (See Section 2.3).

Original: • A programmer that converts questions to programs and deﬁnes a distribution P ( C | q, Γ ; θ prog ) .
Revised: • A programmer that converts questions to programs and deﬁnes a distribution P ( C | q, Γ ; θ prog ) .

Original: It is conditioned on the knowledge storage Γ for code assistance (Liang et al., 2017) – before generating each token the programmer can query Γ about the valid next tokens given a tuple preﬁx, and therefore avoid writing invalid programs.
Revised: It is conditioned on the knowledge storage Γ for code assistance (Liang et al., 2017) – before generating each token the programmer can query Γ about the valid next tokens given a tuple preﬁx, and therefore avoid writing invalid programs.

Original: For all of these neural networks we use the CopyNet (Gu et al., 2016) architecture, which has copy (Vinyals et al., 2015) and attention (Bahdanau et al., 2014) mechanisms.
Revised: For all of these neural networks we use the CopyNet (Gu et al., 2016) architecture, which has copy (Vinyals et al., 2015) and attention (Bahdanau et al., 2014) mechanisms.

Original: 2. B EAM S EARCH
Revised: 2. B EAM S EARCH

Original: Given an example ( s , q, a ) from our training set, we would like to maximize the expected reward
Revised: Given an example ( s , q, a ) from our training set, we would like to maximize the expected reward

Original: [Equation]
Revised: [Equation]

Original: where the reward function R ( · ) returns 1 if and only if executing C on Γ produces a .
Revised: where the reward function R ( · ) returns 1 if and only if executing C on Γ produces a .

Original: We approximate the expectation with beam searches.
Revised: We approximate the expectation with beam searches.

Original: More speciﬁcally, the summation over all programs is replaced by summing over programs found by beam search based on the programmer model P ( C | q, Γ ; θ prog ) .
Revised: More speciﬁcally, the summation over all programs is replaced by summing over programs found by beam search based on the programmer model P ( C | q, Γ ; θ prog ) .

Original: For the summation over knowledge storages Γ , we ﬁrst run beam search for each sentence based on the knowledge encoder model P
Revised: For the summation over knowledge storages Γ , we ﬁrst run beam search for each sentence based on the knowledge encoder model P

Original: (Γ i | s i , s i − 1 ; θ enc ) , and then sample a set of knowledge storages by independently sampling from the knowledge tuples of each sentence.
Revised: (Γ i | s i , s i − 1 ; θ enc ) , and then sample a set of knowledge storages by independently sampling from the knowledge tuples of each sentence.

Original: However, since the beam searches explore exponentially large spaces, it is very challenging to optimize O QA .
Revised: However, since the beam searches explore exponentially large spaces, it is very challenging to optimize O QA .

Original: We introduce two special techniques to iteratively reduce and improve the search space:
Revised: We introduce two special techniques to iteratively reduce and improve the search space:

Original: Auto-Encoding (AE)
Revised: Auto-Encoding (AE)

Original: We add an auto-encoding objective to our framework, similar to the text summarization model proposed by Miao & Blunsom (2016).
Revised: We add an auto-encoding objective to our framework, similar to the text summarization model proposed by Miao & Blunsom (2016).

Original: The training of this objective can be done by variational inference (Kingma & Welling, 2014; Mnih & Gregor, 2014):
Revised: The training of this objective can be done by variational inference (Kingma & Welling, 2014; Mnih & Gregor, 2014):

Original: [Equation]
Revised: [Equation]

Original: where x is text, and z is the hidden discrete structure.
Revised: where x is text, and z is the hidden discrete structure.

Original: However, it suffers from instability due to the strong coupling between encoder and decoder – the training of the decoder θ dec relies solely on a distribution parameterized by the encoder θ enc , which changes throughout the course of training.
Revised: However, it suffers from instability due to the strong coupling between encoder and decoder – the training of the decoder θ dec relies solely on a distribution parameterized by the encoder θ enc , which changes throughout the course of training.

Original: Instead we propose to augment the decoder training with a more stable objective – predict the data x back from noisy partial observations of x , which are independent of θ enc .
Revised: Instead we propose to augment the decoder training with a more stable objective – predict the data x back from noisy partial observations of x , which are independent of θ enc .

Original: More speciﬁcally, for NGM we force the knowledge decoder to decode from a ﬁxed set of hidden sequences z ∈ Z N ( x ) , which includes all knowledge tuples of length N and consist of only words from the text x :
Revised: More speciﬁcally, for NGM we force the knowledge decoder to decode from a ﬁxed set of hidden sequences z ∈ Z N ( x ) , which includes all knowledge tuples of length N and consist of only words from the text x :

Original: [Equation]
Revised: [Equation]

Original: The knowledge decoder θ dec converts knowledge tuples back to sentences and the reconstruction log-likelihoods approximate how informative the tuples are, which can be used as reward for the knowledge encoder.
Revised: The knowledge decoder θ dec converts knowledge tuples back to sentences and the reconstruction log-likelihoods approximate how informative the tuples are, which can be used as reward for the knowledge encoder.

Original: We also drop the KL divergence (last two terms in Equation 3) between language model p ( z ) and the encoder, since the z ’s are produced for NGM computations instead of human reading, and do not need to be ﬂuent in natural language.
Revised: We also drop the KL divergence (last two terms in Equation 3) between language model p ( z ) and the encoder, since the z ’s are produced for NGM computations instead of human reading, and do not need to be ﬂuent in natural language.

Original: Structure Tweaking (ST) Even with AE training, the knowledge encoder is encoding tuples without the understanding of how they are going to be used, and may encode them inconsistently across sentences.
Revised: Structure Tweaking (ST) Even with AE training, the knowledge encoder is encoding tuples without the understanding of how they are going to be used, and may encode them inconsistently across sentences.

Original: At the later QA stage, such inconsistency can lead to no reward when the programmer tries to reason with multiple knowledge tuples.
Revised: At the later QA stage, such inconsistency can lead to no reward when the programmer tries to reason with multiple knowledge tuples.

Original: To retrospectively correct the inconsistency in tuples, we apply structure tweak , a procedure which is similar to code assist (Chen et al., 2017), but works in an opposite direction.
Revised: To retrospectively correct the inconsistency in tuples, we apply structure tweak , a procedure which is similar to code assist (Chen et al., 2017), but works in an opposite direction.

Original: While code assist uses the knowledge storage to inform the programmer, structure tweak adjusts the knowledge encoder to cooperate with an uninformed programmer.
Revised: While code assist uses the knowledge storage to inform the programmer, structure tweak adjusts the knowledge encoder to cooperate with an uninformed programmer.

Original: More speciﬁcally, at training the programmer always performs an extra beam search with code assist turned off.
Revised: More speciﬁcally, at training the programmer always performs an extra beam search with code assist turned off.

Original: If such programs lead to execution failure, they can be used to propose tweaked knowledge tuples (Algorithm 1).
Revised: If such programs lead to execution failure, they can be used to propose tweaked knowledge tuples (Algorithm 1).

Original: For example, when executing an expression ( Hop mary journeyed ) – generated without code assist – on a knowledge storage with tuples ( john the milk ), ( mary went bathroom ) and ( mary picked milk ), matching the preﬁx ( mary journeyed ) fails at token journeyed and returns empty result.
Revised: For example, when executing an expression ( Hop mary journeyed ) – generated without code assist – on a knowledge storage with tuples ( john the milk ), ( mary went bathroom ) and ( mary picked milk ), matching the preﬁx ( mary journeyed ) fails at token journeyed and returns empty result.

Original: At this point, the programmer uses journeyed to replace inconsistent symbols in partially matched tuples (i.e. tuples that start with mary ), and produces tuples ( mary journeyed bathroom ) and ( mary journeyed milk ) .
Revised: At this point, the programmer uses journeyed to replace inconsistent symbols in partially matched tuples (i.e. tuples that start with mary ), and produces tuples ( mary journeyed bathroom ) and ( mary journeyed milk ) .

Original: These tweaked tuples are then added into the beams for the knowledge encoder.
Revised: These tweaked tuples are then added into the beams for the knowledge encoder.

Original: In this way, the search space of the knowledge storage is reﬁned, and the knowledge encoder can learn to generate tuples in a consistent fashion in the future.
Revised: In this way, the search space of the knowledge storage is reﬁned, and the knowledge encoder can learn to generate tuples in a consistent fashion in the future.

Original: Algorithm f one of Hop or
Revised: Algorithm f one of Hop or

Original: Argmax for HopFR and ArgmaxFR can be deﬁned similarly.
Revised: Argmax for HopFR and ArgmaxFR can be deﬁned similarly.

Original: if ( a 1 . . .
Revised: if ( a 1 . . .

Original: a L ) can be matched in Γ , or ( a 1 ) can not be matched in Γ then return
Revised: a L ) can be matched in Γ , or ( a 1 ) can not be matched in Γ then return

Original: Let p =
Revised: Let p =

Original: ( a 1 , . .
Revised: ( a 1 , . .

Original: . , a m ) be the longest preﬁx matched in Γ , and Γ p be the set of matched tuples.
Revised: . , a m ) be the longest preﬁx matched in Γ , and Γ p be the set of matched tuples.

Original: for Γ =
Revised: for Γ =

Original: ( γ 1 , . . . , γ N ) ∈
Revised: ( γ 1 , . . . , γ N ) ∈

Original: Γ p do
Revised: Γ p do

Original: Add ( a 1 , . . . , a m , γ m +1 ,  . . . , γ N ) to T
Revised: Add ( a 1 , . . . , a m , a m +1 , γ m +2 . . . , γ N ) to T

Original: 2. P ARAMETERS E STIMATION
Revised: 2. P ARAMETERS E STIMATION

Original: Now the whole model has parameters θ =
Revised: Now the whole model has parameters θ =

Original: [ θ enc , θ dec , θ prog ] , and the training objective function is
Revised: [ θ enc , θ dec , θ prog ] , and the training objective function is

Original: [Equation]
Revised: [Equation]

Original: where β (Γ i ) is 1 if Γ i only contains tokens from s i and 0 otherwise.
Revised: where β (Γ i ) is 1 if Γ i only contains tokens from s i and 0 otherwise.

Original: For training stability and to overcome search failures, we augment this objective with experience replay (Schaul et al., 2016), and the gradients with respect to each set of parameters are:
Revised: For training stability and to overcome search failures, we augment this objective with experience replay (Schaul et al., 2016), and the gradients with respect to each set of parameters are:

Original: [Equation]
Revised: [Equation]

Original: where R ( G ) =
Revised: where R ( G ) =

Original: (cid:80) Γ ∈G [Equation] C P ( Γ | s ; θ enc ) P ( C | q, Γ ; θ prog )
Revised: (cid:80) Γ ∈G (cid:80) C P ( Γ | s ; θ enc ) P ( C | q, Γ ; θ prog )

Original: R ( Γ , C, a ) is the total expected reward for a set of valid knowledge stores G , G
Revised: R ( Γ , C, a ) is the total expected reward for a set of valid knowledge stores G , G

Original: (Γ i ) is the set of knowledge stores which contains the tuple Γ i , and G (cid:48)
Revised: (Γ i ) is the set of knowledge stores which contains the tuple Γ i , and G (cid:48)

Original: (Γ i ) is the set of knowledge stores which contains the tuple Γ i through tweaking.
Revised: (Γ i ) is the set of knowledge stores which contains the tuple Γ i through tweaking.

Original: R ( G (cid:48)
Revised: R ( G (cid:48)

Original: (Γ i ))
Revised: (Γ i ))

Original: implements “tweaking” as a form of experience replay.
Revised: implements “tweaking” as a form of experience replay.

Original: [Equation]
Revised: [Equation]

Original: where C ∗ ( s , q ) is the experience replay buffer for ( s , q ) .
Revised: where C ∗ ( s , q ) is the experience replay buffer for ( s , q ) .

Original: α = 0 .
Revised: α = 0 .

Original: 1 is a constant.
Revised: 1 is a constant.

Original: During training, the program with the highest weighted reward (i.e. P ( Γ | s ; θ enc ) R ( Γ , C, a ) ) is added to the replay buffer.
Revised: During training, the program with the highest weighted reward (i.e. P ( Γ | s ; θ enc ) R ( Γ , C, a ) ) is added to the replay buffer.

Original: Because the knowledge storage and the program are non-differentiable discrete structures, we optimize our objective by a coordinate ascent approach – optimizing the three components in alternation with
Revised: Because the knowledge storage and the program are non-differentiable discrete structures, we optimize our objective by a coordinate ascent approach – optimizing the three components in alternation with

Original: REINFORCE (Williams, 1992).
Revised: REINFORCE (Williams, 1992).

Original:  R ESULTS
Revised:  R ESULTS

Original: We apply the N-Gram Machine (NGM) to solve a set of text reasoning tasks in the bAbI dataset (Weston et al., 2015).
Revised: We apply the N-Gram Machine (NGM) to solve a set of text reasoning tasks in the bAbI dataset (Weston et al., 2015).

Original: In Section 3.1, we demonstrate that the model can learn to build knowledge storage and generate programs that accurately answer the questions.
Revised: In Section 3.1, we demonstrate that the model can learn to build knowledge storage and generate programs that accurately answer the questions.

Original: In Section 3.2, we show the scalability advantage of NGMs by applying it to longer stories up to 10 million sentences.
Revised: In Section 3.2, we show the scalability advantage of NGMs by applying it to longer stories up to 10 million sentences.

Original: The Seq2Seq components are implemented as one-layer recurrent neural networks with Gated
Revised: The Seq2Seq components are implemented as one-layer recurrent neural networks with Gated

Original: Recurrent Unit (Chung et al., 2014).
Revised: Recurrent Unit (Chung et al., 2014).

Original: The hidden dimension and the vocabulary embedding dimension are both 8.
Revised: The hidden dimension and the vocabulary embedding dimension are both 8.

Original: During decoding, the per sentence knowledge tuple beam size is 2, the knowledge store sample size is 5, and the program beam size is 30.
Revised: During decoding, the per sentence knowledge tuple beam size is 2, the knowledge store sample size is 5, and the program beam size is 30.

Original: The neural networks are implemented in
Revised: The neural networks are implemented in

Original: TensorFlow (Abadi et al., 2016) and optimized using Adam (Kingma & Ba, 2014).
Revised: TensorFlow (Abadi et al., 2016) and optimized using Adam (Kingma & Ba, 2014).

Original: 3. B A B I
Revised: 3. B A B I

Original: The bAbI dataset contains twenty tasks in total.
Revised: The bAbI dataset contains twenty tasks in total.

Original: We consider the subset of them that are extractive question answering tasks (as deﬁned in Section1.1).
Revised: We consider the subset of them that are extractive question answering tasks (as deﬁned in Section1.1).

Original: Each task is learned separately.
Revised: Each task is learned separately.

Original: For all tasks, we set the knowledge tuple length to three.
Revised: For all tasks, we set the knowledge tuple length to three.

Original: In Table 3, we report results on the test sets.
Revised: In Table 3, we report results on the test sets.

Original: NGM outperforms MemN2N (Sukhbaatar et al., 2015) on all tasks listed.
Revised: NGM outperforms MemN2N (Sukhbaatar et al., 2015) on all tasks listed.

Original: The results show that autoencoding is essential to bootstrapping the learning.
Revised: The results show that autoencoding is essential to bootstrapping the learning.

Original: Without auto-encoding, the expected rewards are near zero; but auto-encoding alone is not sufﬁcient to achieve high rewards (See Section 2.2).
Revised: Without auto-encoding, the expected rewards are near zero; but auto-encoding alone is not sufﬁcient to achieve high rewards (See Section 2.2).

Original: Since multiple discrete latent structures (i.e. knowledge tuples and programs) need to agree with each other over the choice of their representations for QA to succeed, the search becomes combinatorially hard.
Revised: Since multiple discrete latent structures (i.e. knowledge tuples and programs) need to agree with each other over the choice of their representations for QA to succeed, the search becomes combinatorially hard.

Original: Structure tweaking is an effective way to reﬁne the search space – improving the performance of more than half of the tasks.
Revised: Structure tweaking is an effective way to reﬁne the search space – improving the performance of more than half of the tasks.

Original: [Table]
Revised: [Table]

Original: To illustrate the effect of auto-encoding, we show in Figure 3 how informative the knowledge tuples are by computing the reconstruction log-likelihood using the knowledge decoder for the sentence "john went back to the garden".
Revised: To illustrate the effect of auto-encoding, we show in Figure 3 how informative the knowledge tuples are by computing the reconstruction log-likelihood using the knowledge decoder for the sentence "john went back to the garden".

Original: As expected, the tuple ( john went garden ) is the most informative.
Revised: As expected, the tuple ( john went garden ) is the most informative.

Original: Other informative tuples include ( john the garden ) and ( john to garden ) .
Revised: Other informative tuples include ( john the garden ) and ( john to garden ) .

Original: Therefore, with auto-encoding training, useful hypotheses have large chance to be found by a small knowledge encoder beam size (2 in our case).
Revised: Therefore, with auto-encoding training, useful hypotheses have large chance to be found by a small knowledge encoder beam size (2 in our case).

Original: Table 4 lists sampled knowledge storages learned with different objectives and procedures.
Revised: Table 4 lists sampled knowledge storages learned with different objectives and procedures.

Original: Knowledge storages learned with auto-encoding are much more informative compared to the ones without.
Revised: Knowledge storages learned with auto-encoding are much more informative compared to the ones without.

Original: After structure tweaking, the knowledge tuples converge to use more consistent symbols – e.g., using went instead of back or travelled .
Revised: After structure tweaking, the knowledge tuples converge to use more consistent symbols – e.g., using went instead of back or travelled .

Original: Our experiment results show the tweaking procedure can help NGM to deal with various linguistic phenomenons such as singular/plural (“cats” vs “cat”) and synonyms (“grabbed” vs “got”).
Revised: Our experiment results show the tweaking procedure can help NGM to deal with various linguistic phenomenons such as singular/plural (“cats” vs “cat”) and synonyms (“grabbed” vs “got”).

Original: More examples are included in the supplementary material A.1.
Revised: More examples are included in the supplementary material A.1.

Original: [Table]
Revised: [Table]

Original: there there there
Revised: there there there

Original: [Table]
Revised: [Table]

Original: is is cat gertrude is cat gertrude is cat
Revised: is is cat gertrude is cat gertrude is cat

Original: [Equation]
Revised: [Equation]

Original: To demonstrate the scalability advantage of the N-Gram Machine, we conduct experiments on question answering data where the number of sentences may increase up to 10 million.
Revised: To demonstrate the scalability advantage of the N-Gram Machine, we conduct experiments on question answering data where the number of sentences may increase up to 10 million.

Original: More speciﬁcally we generated longer bAbI stories using the open-source script from Facebook 3 .
Revised: More speciﬁcally we generated longer bAbI stories using the open-source script from Facebook 3 .

Original: We measure the answering time and answer quality of MemN2N (Sukhbaatar et al., 2015) 4 and NGM at different scales.
Revised: We measure the answering time and answer quality of MemN2N (Sukhbaatar et al., 2015) 4 and NGM at different scales.

Original: The answering time is measured by the amount of time used to produce an answer when a question is given.
Revised: The answering time is measured by the amount of time used to produce an answer when a question is given.

Original: For MemN2N, this is the neural network inference time.
Revised: For MemN2N, this is the neural network inference time.

Original: For NGM, because the knowledge storage can be built and indexed in advance, this is the programmer decoding time.
Revised: For NGM, because the knowledge storage can be built and indexed in advance, this is the programmer decoding time.

Original: in each QA pair.
Revised: in each QA pair.

Original: Figure 4 compares MemN2N and NGM.
Revised: Figure 4 compares MemN2N and NGM.

Original: In terms of answering time, MemN2N scales poorly – the inference time increases linearly as the story length increases.
Revised: In terms of answering time, MemN2N scales poorly – the inference time increases linearly as the story length increases.

Original: While for NGM, the answering time is not affected by story length.
Revised: While for NGM, the answering time is not affected by story length.

Original: The crossover of the two lines is when the story length is around 1000, which is due to the difference in neural network architectures – NGM uses recurrent networks while MemN2N uses feed-forward networks.
Revised: The crossover of the two lines is when the story length is around 1000, which is due to the difference in neural network architectures – NGM uses recurrent networks while MemN2N uses feed-forward networks.

Original: To compare the answer quality at scale, we apply
Revised: To compare the answer quality at scale, we apply

Original: MemN2N and NGM to solve three life-long bAbI tasks (Task 1, 2, and 11).
Revised: MemN2N and NGM to solve three life-long bAbI tasks (Task 1, 2, and 11).

Original: For each life-long task,
Revised: For each life-long task,

Original: MemN2N is run for 10 trials and the test accuracy of the trial with the best validation accuracy is used.
Revised: MemN2N is run for 10 trials and the test accuracy of the trial with the best validation accuracy is used.

Original: For NGM, we use the same models trained on regular bAbI tasks.
Revised: For NGM, we use the same models trained on regular bAbI tasks.

Original: We compute the average and standard deviation of test accuracy from these three tasks.
Revised: We compute the average and standard deviation of test accuracy from these three tasks.

Original: MemN2N performance is competitive with NGM when story length is no greater than 400, but decreases drastically when story length further increases.
Revised: MemN2N performance is competitive with NGM when story length is no greater than 400, but decreases drastically when story length further increases.

Original: On the other hand, NGM answering quality is the same for all story lengths.
Revised: On the other hand, NGM answering quality is the same for all story lengths.

Original: These scalability advantages of NGM are due to its “machine” nature – the symbolic knowledge storage can be computed and indexed in advance, and the program execution is robust on stories of various lengths.
Revised: These scalability advantages of NGM are due to its “machine” nature – the symbolic knowledge storage can be computed and indexed in advance, and the program execution is robust on stories of various lengths.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The auto-encoding part of our model (Figure 2) is similar to the auto-encoding text summarization model proposed by Miao & Blunsom (2016).
Revised: The auto-encoding part of our model (Figure 2) is similar to the auto-encoding text summarization model proposed by Miao & Blunsom (2016).

Original: Miao & Blunsom (2016) solved the large search space problem (when generating hidden sequences) by 1) restricting hidden sequences to only consist of tokens in the source sequences (through a PointerNet (Vinyals et al., 2015)), and 2) a language model pre-trained on a separate corpus.
Revised: Miao & Blunsom (2016) solved the large search space problem (when generating hidden sequences) by 1) restricting hidden sequences to only consist of tokens in the source sequences (through a PointerNet (Vinyals et al., 2015)), and 2) a language model pre-trained on a separate corpus.

Original: We adopt a different approach, which does not require a separate corpus, or a restricted hidden sequence space: 1) use a less restricted hidden space (through a CopyNet (Gu et al., 2016)) by allowing both copied tokens and generated tokens; 2) stabilize the decoder by forcing it (through experience replay) to train from randomly generated hidden sequences; and 3) use the log-likelihood of the pre-trained decoder to guide the training of the encoder.
Revised: We adopt a different approach, which does not require a separate corpus, or a restricted hidden sequence space: 1) use a less restricted hidden space (through a CopyNet (Gu et al., 2016)) by allowing both copied tokens and generated tokens; 2) stabilize the decoder by forcing it (through experience replay) to train from randomly generated hidden sequences; and 3) use the log-likelihood of the pre-trained decoder to guide the training of the encoder.

Original: The question answering part of our model (Figure 2) is similar to the Neural Symbolic Machine (NSM)
Revised: The question answering part of our model (Figure 2) is similar to the Neural Symbolic Machine (NSM)

Original: (Liang et al., 2017), which is a memory enhanced sequence-to-sequence model that translates questions into programs in λ -calculus
Revised: (Liang et al., 2017), which is a memory enhanced sequence-to-sequence model that translates questions into programs in λ -calculus

Original: (Liang et al., 2011).
Revised: (Liang et al., 2011).

Original: The programs, when executed on a knowledge graph, can produce answers to the questions.
Revised: The programs, when executed on a knowledge graph, can produce answers to the questions.

Original: Our work extends NSM by removing the assumption of a given knowledge bases or schema, and instead learns to generate storage by end-to-end training to answer questions.
Revised: Our work extends NSM by removing the assumption of a given knowledge bases or schema, and instead learns to generate storage by end-to-end training to answer questions.

Original: The reminder of this section analyzes why existing text understanding technologies are fundamentally limited and draws connections of our proposed solution to psychology and neuroscience.
Revised: The reminder of this section analyzes why existing text understanding technologies are fundamentally limited and draws connections of our proposed solution to psychology and neuroscience.

Original: 4. T EXT U NDERSTANDING : C URRENT P RACTICE
Revised: 4. T EXT U NDERSTANDING : C URRENT P RACTICE

Original: In recent years, several large-scale knowledge bases (KBs) have been constructed, such as
Revised: In recent years, several large-scale knowledge bases (KBs) have been constructed, such as

Original: YAGO (Suchanek et al., 2007), Freebase (Bollacker et al., 2008), NELL (Mitchell et al., 2015),
Revised: YAGO (Suchanek et al., 2007), Freebase (Bollacker et al., 2008), NELL (Mitchell et al., 2015),

Original: Google Knowledge Graph (Singhal, 2012), Microsoft Satori (Qian, 2013), and others.
Revised: Google Knowledge Graph (Singhal, 2012), Microsoft Satori (Qian, 2013), and others.

Original: However, all of them suffer from a completeness problem (Dong et al., 2014) relative to large text corpora, which which stems from an inability to convert arbitrary text to graph structures, and accurately answering questions from these structures.
Revised: However, all of them suffer from a completeness problem (Dong et al., 2014) relative to large text corpora, which which stems from an inability to convert arbitrary text to graph structures, and accurately answering questions from these structures.

Original: There are two core issues in this difﬁculty:
Revised: There are two core issues in this difﬁculty:

Original: The schema (or representation) problem Traditional text extraction approaches need some ﬁxed and ﬁnite target schema (Riedel et al., 2013).
Revised: The schema (or representation) problem Traditional text extraction approaches need some ﬁxed and ﬁnite target schema (Riedel et al., 2013).

Original: For example, the deﬁnition of marriage 5 is very complex and includes a whole spectrum of concepts, such as common-law marriage, civil union, putative marriage, handfasting, nikah mut’ah, and so on.
Revised: For example, the deﬁnition of marriage 5 is very complex and includes a whole spectrum of concepts, such as common-law marriage, civil union, putative marriage, handfasting, nikah mut’ah, and so on.

Original: It is prohibitively expensive to have experts to clearly deﬁne all these differences in advance, and annotate enough utterances from which they are expressed.
Revised: It is prohibitively expensive to have experts to clearly deﬁne all these differences in advance, and annotate enough utterances from which they are expressed.

Original: However, given a particular context, many of this distinctions might be irrelevant.
Revised: However, given a particular context, many of this distinctions might be irrelevant.

Original: For example when a person say “I just married Lisa.”
Revised: For example when a person say “I just married Lisa.”

Original: and then ask “Who is my wife?”, a system should be able to respond correctly, without ﬁrst learning about all types of marriage.
Revised: and then ask “Who is my wife?”, a system should be able to respond correctly, without ﬁrst learning about all types of marriage.

Original: A desirable solution should induce schema automatically from the corpus such that knowledge encoded with this schema can help downstream tasks such as QA.
Revised: A desirable solution should induce schema automatically from the corpus such that knowledge encoded with this schema can help downstream tasks such as QA.

Original: The end-to-end training (or inference) problem The state-of-the-art approaches break down
Revised: The end-to-end training (or inference) problem The state-of-the-art approaches break down

Original: QA solutions into independent components, such as schema deﬁnition and manually annotating examples (Mitchell et al., 2015; Bollacker et al., 2008), relation extraction (Mintz et al., 2009), entity resolution (Singla & Domingos, 2006), and semantic parsing of questions (Berant et al., 2013; Liang et al., 2017).
Revised: QA solutions into independent components, such as schema deﬁnition and manually annotating examples (Mitchell et al., 2015; Bollacker et al., 2008), relation extraction (Mintz et al., 2009), entity resolution (Singla & Domingos, 2006), and semantic parsing of questions (Berant et al., 2013; Liang et al., 2017).

Original: However, these components need to work together and depend on each other.
Revised: However, these components need to work together and depend on each other.

Original: For example, the meaning of text expressions (e.g., “father of IBM”, “father of Adam Smith”, “father of electricity”) depend on entity types.
Revised: For example, the meaning of text expressions (e.g., “father of IBM”, “father of Adam Smith”, “father of electricity”) depend on entity types.

Original: The type information, however, depends on the entity resolution decisions, and the content of the knowledge base (KB).
Revised: The type information, however, depends on the entity resolution decisions, and the content of the knowledge base (KB).

Original: The content of the KBs, if populated from text, will in return depend on the meaning of text expressions.
Revised: The content of the KBs, if populated from text, will in return depend on the meaning of text expressions.

Original: Existing systems (Dong et al., 2014;
Revised: Existing systems (Dong et al., 2014;

Original: Berant et al., 2013) leverage human annotation and supervised training for each component separately, which create consistency issues, and preclude directly optimizing the performance of the QA task.
Revised: Berant et al., 2013) leverage human annotation and supervised training for each component separately, which create consistency issues, and preclude directly optimizing the performance of the QA task.

Original: A desirable solution should use few human annotations of intermediate steps, and rely on end-to-end training to directly optimize the QA quality.
Revised: A desirable solution should use few human annotations of intermediate steps, and rely on end-to-end training to directly optimize the QA quality.

Original: 4. T EXT U NDERSTANDING : D EEP N EURAL N ETS
Revised: 4. T EXT U NDERSTANDING : D EEP N EURAL N ETS

Original: More recently there has been a lot of progress in applying deep neural networks (DNNs) to text understanding (Mikolov et al., 2013; Weston et al., 2014; Neelakantan et al., 2015; Graves et al., 2016; Miller et al., 2016).
Revised: More recently there has been a lot of progress in applying deep neural networks (DNNs) to text understanding (Mikolov et al., 2013; Weston et al., 2014; Neelakantan et al., 2015; Graves et al., 2016; Miller et al., 2016).

Original: The key ingredient to these solutions is embedding text expressions into a latent continuous space.
Revised: The key ingredient to these solutions is embedding text expressions into a latent continuous space.

Original: This removes the need to manually deﬁne a schema, while the vectors still allow complex inference and reasoning.
Revised: This removes the need to manually deﬁne a schema, while the vectors still allow complex inference and reasoning.

Original: Continuous representations greatly simpliﬁed the system design of QA systems, and enabled end-to-end training, which directly optimizes the QA quality.
Revised: Continuous representations greatly simpliﬁed the system design of QA systems, and enabled end-to-end training, which directly optimizes the QA quality.

Original: However, a key issue that prevents these models from being applied to many applications is scalability.
Revised: However, a key issue that prevents these models from being applied to many applications is scalability.

Original: After receiving a question, all text in a corpus need to be analyzed by the model.
Revised: After receiving a question, all text in a corpus need to be analyzed by the model.

Original: Therefore it leads to at least O ( n ) complexity, where n is the text size.
Revised: Therefore it leads to at least O ( n ) complexity, where n is the text size.

Original: Approaches which rely on a search subroutine (e.g., DrQA (Chen et al., 2017)) lose the beneﬁt of end-to-end training, and are limited by the quality of the retrieval stage, which itself is difﬁcult.
Revised: Approaches which rely on a search subroutine (e.g., DrQA (Chen et al., 2017)) lose the beneﬁt of end-to-end training, and are limited by the quality of the retrieval stage, which itself is difﬁcult.

Original: 4. R ELATIONSHIP TO P SYCHOLOGY AND N EUROSCIENCE
Revised: 4. R ELATIONSHIP TO P SYCHOLOGY AND N EUROSCIENCE

Original: Previous studies (Wickman, 2012; Bartol et al., 2015) of hippocampus suggests that human memory capacity may be somewhere between 10 terabytes and 100 terabytes.
Revised: Previous studies (Wickman, 2012; Bartol et al., 2015) of hippocampus suggests that human memory capacity may be somewhere between 10 terabytes and 100 terabytes.

Original: This huge capacity is very advantageous for humans’ survival, yet puts human brain in the same situation of a commercial search engine – facing the task of organizing vast amounts of information in a form which support fast retrieval and inference.
Revised: This huge capacity is very advantageous for humans’ survival, yet puts human brain in the same situation of a commercial search engine – facing the task of organizing vast amounts of information in a form which support fast retrieval and inference.

Original: Our choice of using n-grams as the unit of knowledge storage is partially motivated by the Adaptive
Revised: Our choice of using n-grams as the unit of knowledge storage is partially motivated by the Adaptive

Original: Resonance Theory (ART) (Carpenter & Grossberg, 2003; Tan et al., 2007; Wang et al., 2012), which models each piece of human episodic memory as a set of symbols, each representing one aspect of the experience such as location, color, object etc.
Revised: Resonance Theory (ART) (Carpenter & Grossberg, 2003; Tan et al., 2007; Wang et al., 2012), which models each piece of human episodic memory as a set of symbols, each representing one aspect of the experience such as location, color, object etc.

Original: The structure of our model (Figure 2) resembles the complementary learning theory (McClelland et al., 1995; O’Reilly et al., 2014; Kumaran et al., 2016), which hypothesizes that intelligent agents possess two learning systems, instantiated in mammals in the neocortex and hippocampus.
Revised: The structure of our model (Figure 2) resembles the complementary learning theory (McClelland et al., 1995; O’Reilly et al., 2014; Kumaran et al., 2016), which hypothesizes that intelligent agents possess two learning systems, instantiated in mammals in the neocortex and hippocampus.

Original: The ﬁrst gradually acquires structured knowledge representations, while the second quickly learns the speciﬁcs of individual experiences.
Revised: The ﬁrst gradually acquires structured knowledge representations, while the second quickly learns the speciﬁcs of individual experiences.

Original: In our model the stories in training examples (episodic memories) and knowledge store (semantic memory) represents the fast learning neurons (declarative memory), while the sequence to sequence DNN models represent the slow learning neurons (implicit memory).
Revised: In our model the stories in training examples (episodic memories) and knowledge store (semantic memory) represents the fast learning neurons (declarative memory), while the sequence to sequence DNN models represent the slow learning neurons (implicit memory).

Original: The DNN training procedure, which goes over the past experience (stories) over and over again, resembles the “replay” of hippocampal memories that allows goal-dependent weighting of experience statistics (Kumaran et al., 2016).
Revised: The DNN training procedure, which goes over the past experience (stories) over and over again, resembles the “replay” of hippocampal memories that allows goal-dependent weighting of experience statistics (Kumaran et al., 2016).

Original: The structure tweak procedure (Section 2.2) in our model is critical for its success, and bears resemblance to the reconstructive (or generative) memory theory (Piaget, 1977; Block, 1982), which hypothesizes that by employing reconstructive processes, individuals supplement other aspects of available personal knowledge into the gaps found in episodic memory in order to provide a fuller and more coherent version.
Revised: The structure tweak procedure (Section 2.2) in our model is critical for its success, and bears resemblance to the reconstructive (or generative) memory theory (Piaget, 1977; Block, 1982), which hypothesizes that by employing reconstructive processes, individuals supplement other aspects of available personal knowledge into the gaps found in episodic memory in order to provide a fuller and more coherent version.

Original: Our analysis of the QA task showed that at knowledge encoding stage information is often encoded without understanding how it is going to be used.
Revised: Our analysis of the QA task showed that at knowledge encoding stage information is often encoded without understanding how it is going to be used.

Original: So it might be encoded in an inconsistent way.
Revised: So it might be encoded in an inconsistent way.

Original: At the later QA stage an inference procedure tries to derive the expected answer by putting together several pieces of information, and fails due to inconsistency.
Revised: At the later QA stage an inference procedure tries to derive the expected answer by putting together several pieces of information, and fails due to inconsistency.

Original: Only at that time can a hypothesis be formed to retrospectively correct the inconsistency in memory.
Revised: Only at that time can a hypothesis be formed to retrospectively correct the inconsistency in memory.

Original: These “tweaks” in memory later participate in training the knowledge encoder in the form of experience replay.
Revised: These “tweaks” in memory later participate in training the knowledge encoder in the form of experience replay.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We present an end-to-end trainable system which combines an text auto-encoding component for encoding the meaning of text in symbolic representations, and a memory enhanced sequence-tosequence component for answering questions from the storage.
Revised: We present an end-to-end trainable system which combines an text auto-encoding component for encoding the meaning of text in symbolic representations, and a memory enhanced sequence-tosequence component for answering questions from the storage.

Original: We show that the method achieves good scaling properties and robust inference on artiﬁcially generated stories of up to 10 million sentences long.
Revised: We show that the method achieves good scaling properties and robust inference on artiﬁcially generated stories of up to 10 million sentences long.

Original: The system we present here illustrates how end-to-end learning and scalability can be made possible through a symbolic knowledge storage.
Revised: The system we present here illustrates how end-to-end learning and scalability can be made possible through a symbolic knowledge storage.

Original: To further improve the system we are interested in investigating whether the proposed n-gram representation is sufﬁcient for natural languages.
Revised: To further improve the system we are interested in investigating whether the proposed n-gram representation is sufﬁcient for natural languages.

Original: More complex representations, such as Abstract
Revised: More complex representations, such as Abstract

Original: Meaning Representations Banarescu et al.
Revised: Meaning Representations Banarescu et al.

Original: (2013), are possible alternatives
Revised: (2013), are possible alternatives

Original: but it remains unclear how to design effective weakly supervised learning techniques to induce such representations
Revised: but it remains unclear how to design effective weakly supervised learning techniques to induce such representations

Original: A S UPPLEMENTARY M ATERIAL A. M ODEL GENERATED KNOWLEDGE STORAGES AND PROGRAMS
Revised: A S UPPLEMENTARY M ATERIAL A. M ODEL GENERATED KNOWLEDGE STORAGES AND PROGRAMS

Original: A B I TASKS
Revised: A B I TASKS

Original: [Table]
Revised: [Table]

Original: the bedroom
Revised: the bedroom

Original: [Table]
Revised: [Table]

Original: journeyed to the hallway
Revised: journeyed to the hallway

Original: [Table]
Revised: [Table]

Original: travelled to the garden
Revised: travelled to the garden

Original: [Table]
Revised: [Table]

Original: John went back to the kitchen.
Revised: John went back to the kitchen.

Original: [Table]
Revised: [Table]

Original: back to the hallway
Revised: back to the hallway

Original: [Table]
Revised: [Table]

Original: Where is Daniel?
Revised: Where is Daniel?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Sandra journeyed to the hallway.
Revised: Sandra journeyed to the hallway.

Original: [Table]
Revised: [Table]

Original: John journeyed to the bathroom.
Revised: John journeyed to the bathroom.

Original: [Table]
Revised: [Table]

Original: Sandra grabbed the football.
Revised: Sandra grabbed the football.

Original: [Table]
Revised: [Table]

Original: Daniel travelled to the bedroom.
Revised: Daniel travelled to the bedroom.

Original: [Table]
Revised: [Table]

Original: John got the milk.
Revised: John got the milk.

Original: [Table]
Revised: [Table]

Original: got milk
Revised: got milk

Original: John dropped the milk.
Revised: John dropped the milk.

Original: [Table]
Revised: [Table]

Original: Question
Revised: Question

Original: Program
Revised: Program

Original: Where is the milk?
Revised: Where is the milk?

Original: ArgmaxFR milk got
Revised: ArgmaxFR milk got

Original: Argmax V1 journeyed
Revised: Argmax V1 journeyed

Original: Story
Revised: Story

Original: [Table]
Revised: [Table]

Original: John went to the bathroom.
Revised: John went to the bathroom.

Original: John went bathroom
Revised: John went bathroom

Original: After that he went back to the hallway.
Revised: After that he went back to the hallway.

Original: John he hallway
Revised: John he hallway

Original: Sandra journeyed to the bedroom
Revised: Sandra journeyed to the bedroom

Original: Sandra Sandra bedroom
Revised: Sandra Sandra bedroom

Original: After that she moved to the garden
Revised: After that she moved to the garden

Original: Sandra she garden
Revised: Sandra she garden

Original: Question
Revised: Question

Original: Program
Revised: Program

Original: Where is Sandra?
Revised: Where is Sandra?

Original: Argmax Sandra
Revised: Argmax Sandra

Original: she
Revised: she

Original: Story
Revised: Story

Original: [Table]
Revised: [Table]

Original: Sheep are afraid of cats.
Revised: Sheep are afraid of cats.

Original: Sheep afraid cats
Revised: Sheep afraid cats

Original: Cats are afraid of wolves.
Revised: Cats are afraid of wolves.

Original: Cat afraid wolves
Revised: Cat afraid wolves

Original: Jessica is a sheep.
Revised: Jessica is a sheep.

Original: Jessica is sheep
Revised: Jessica is sheep

Original: Mice are afraid of sheep.
Revised: Mice are afraid of sheep.

Original: Mouse afraid sheep
Revised: Mouse afraid sheep

Original: Wolves are afraid of mice.
Revised: Wolves are afraid of mice.

Original: Wolf afraid mice
Revised: Wolf afraid mice

Original: Emily is a sheep.
Revised: Emily is a sheep.

Original: Emily is sheep
Revised: Emily is sheep

Original: Winona is a wolf.
Revised: Winona is a wolf.

Original: Winona is wolf
Revised: Winona is wolf

Original: Gertrude is a mouse.
Revised: Gertrude is a mouse.

Original: Gertrude is mouse
Revised: Gertrude is mouse

Original: Question
Revised: Question

Original: Program
Revised: Program

Original: What is Emily afraid of?
Revised: What is Emily afraid of?

Original: Hop Emily is
Revised: Hop Emily is

Original: Hop V1 afraid
Revised: Hop V1 afraid

Original: Story
Revised: Story

Original: Knowledge Storage
Revised: Knowledge Storage

Original: Berhard is a rhino.
Revised: Berhard is a rhino.

Original: Bernhard a rhino
Revised: Bernhard a rhino

Original: Lily is a swan.
Revised: Lily is a swan.

Original: Lily a swan
Revised: Lily a swan

Original: Julius is a swan.
Revised: Julius is a swan.

Original: Julius a swan
Revised: Julius a swan

Original: Lily is white.
Revised: Lily is white.

Original: Lily is white
Revised: Lily is white

Original: Greg is a rhino.
Revised: Greg is a rhino.

Original: Greg a rhino
Revised: Greg a rhino

Original: Julius is white.
Revised: Julius is white.

Original: Julius is white
Revised: Julius is white

Original: Brian is a lion.
Revised: Brian is a lion.

Original: Brian a lion
Revised: Brian a lion

Original: Bernhard is gray.
Revised: Bernhard is gray.

Original: Bernhard is gray
Revised: Bernhard is gray

Original: Brian is yellow.
Revised: Brian is yellow.

Original: Brian is yellow
Revised: Brian is yellow

Original: Question
Revised: Question

Original: Program
Revised: Program

Original: What color is Greg?
Revised: What color is Greg?

Original: Hop Greg a
Revised: Hop Greg a

Original: HopFR V

Revised: HopFR V

Original: a
Revised: a

Original: Hop V2 is
Revised: Hop V2 is