Paper:
A NEURAL KNOWLEDGE LANGUAGE MODEL
Current language models have significant limitations in their ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge based on statistical co-occurrences, even if most of the knowledge words are rarely observed named entities. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model. The model predicts whether the word to generate has an underlying fact or not. Then, a word is either generated from the vocabulary or copied from the description of the predicted fact. We train and test the model on a new dataset, WikiFacts. In experiments, we show that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words. In addition, we demonstrate that the sampled descriptions include named entities which used to be the unknown words in RNN language models.
1 INTRODUCTION
Kanye West, a famous <unknown> and the husband of <unknown>, released his latest album <unknown> in <unknown>.
A core purpose of language is to communicate knowledge. Thus, for human-level language understanding, it is important for a language model to take advantage of knowledge. Although traditional language models are good at capturing statistical co-occurrences of entities as long as they are observed frequently in a corpus (e.g., words like verbs, pronouns, and prepositions), they are in general limited in their ability to encode or decode knowledge, which is often represented by named entities such as person names, place names, years, etc. (as shown in the above example sentence of Kanye West.) When trained with a very large corpus, traditional language models have demonstrated to some extent the ability to encode/decode knowledge (Vinyals & Le, 2015; Serban et al., 2015). However, we claim that simply feeding a larger corpus into a bigger model hardly results in a good knowledge language model.
The primary reason for this is the difficulty in learning good representations for rare or unknown words because these are a majority of the knowledge-related words. In particular, for applications such as question answering (Iyyer et al., 2014; Weston et al., 2016; Bordes et al., 2015) and dialogue modeling (Vinyals & Le, 2015; Serban et al., 2015), these words are of our main interest. Specifically, in the recurrent neural network language model (RNNLM) (Mikolov et al., 2010) the computational complexity is linearly dependent on the number of vocabulary words. Thus, including all words of a language is computationally prohibitive. Instead, we typically fill our vocabulary with a limited number of frequent words and regard all the other words as the unknown (UNK) word. Even if we can include a large number of words in the vocabulary, according to Zipf’s law, a large portion of the words will be rarely observed in the corpus and thus learning good representations for these words remains a problem.
The fact that languages and knowledge can change over time also makes it difficult to simply rely on a large corpus. Media produce an endless stream of new knowledge every day (e.g., the results of baseball games played yesterday) that is even changing over time (e.g., “the current president of the
∗This work was done while HC was in Samsung Advanced Institute of Technology
United States is ”). Furthermore, a good language model should exercise some level of reasoning. For example, it may be possible to observe several occurrences of Barack Obama’s year of birth in a large corpus and thus the model may be able to predict it. However, after seeing mentions of his year of birth, presented with a simple reformulation of that piece of knowledge into a sentence such as “Barack Obama’s age is ”, one would not expect current language models to handle the required amount of reasoning in order to predict the next word (i.e. the age) easily. However, a good model should be able to reason the answer from this context1.
In this paper, we propose a Neural Knowledge Language Model (NKLM) as a step towards addressing the limitations of traditional language modeling when it comes to exploiting factual knowledge. In particular, we incorporate symbolic knowledge provided by a knowledge graph (Nickel et al., 2015) into the RNNLM. A knowledge graph (KG) is a collection of facts which have a form of (subject, relationship, object). We observe particularly the following properties of KGs that make the connection to the language model sensible. First, facts in KGs are mostly about rare words in text corpora. KGs are managed and updated in a similar way that Wikipedia pages are managed to date. The KG embedding methods (Bordes et al., 2011; 2013) provide distributed representations for the entities in the KG. The graph can be traversed for reasoning (Gu et al., 2015). Finally, facts come along with textual representations which we call the fact description and take advantage of here.
There are a few differences between the NKLM and the traditional RNNLM. First, we assume that a word generation is either based on a fact or not. Thus, at each time step, before predicting a word, we predict whether the word to generate has an underlying fact or not. As a result, our model provides the predictions over facts in a topic in addition to the word predictions. Similarly to how context information of previous words flows through the hidden states in the RNNLM, in the NKLM the previous information on both facts and words flow through an RNN and provide richer context. Second, the model has two ways to generate the next word. One option is to generate a “vocabulary word” from the vocabulary softmax as is in the RNNLM. The other option is to generate a “knowledge word” by copying a word contained in the description of the predicted fact. Considering that the fact description is often short and consists of out-of-vocabulary words, we predict the position of the word to copy within the fact description. This knowledge-copy mechanism makes it possible to generate words which are not in the predefined vocabulary. Thus, it does not require to learn explicit embeddings of the words to generate, and consequently resolves the rare/unknown word problem. Lastly, the NKLM can immediately adapt to adding or modifying knowledge because the model learns to predict facts, which can easily be modified without having to retrain the model.
Training the above model in a supervised way requires to align words with facts. To this end, we introduce a new dataset, called WikiFacts. For each topic in the dataset, a set of facts from the Freebase KG (Bollacker et al., 2008) and a Wikipedia description of the same topic is provided along with the alignment information. This alignment is done automatically by performing string matching between the fact description and the Wikipedia description.
2 RELATED WORK
There have been remarkable advances in language modeling research based on neural networks (Bengio et al., 2003; Mikolov et al., 2010). In particular, the RNNLMs are interesting for their ability to take advantage of longer-term temporal dependencies without a strong conditional independence assumption. It is especially noteworthy that the RNNLM using the Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) has recently advanced to the level of outperforming carefully-tuned traditional n-gram based language models (Jozefowicz et al., 2016).
There have been many efforts to speed up the language models so that they can cover a larger vocabulary. These methods approximate the softmax output using hierarchical softmax (Morin & Bengio, 2005; Mnih & Hinton, 2009), importance sampling (Jean et al., 2015), noise contrastive estimation (Mnih & Teh, 2012), etc. Although helpful to mitigate the computational problem, these approaches still suffer from the statistical problem due to rare or unknown words. Having the UNK word as the output of a generative language model is also inconvenient (e.g, dialogue system).
1We do not investigate the reasoning ability in this paper but highlight this example because the explicit representation of facts would help to handle such examples.
To help deal with the rare/unknown word problem, the pointer networks (Vinyals et al., 2015) have been adopted to implement the copy mechanism (Gulcehre et al., 2016; Gu et al., 2016) and applied to machine translation and text summarization. With this approach, the (unknown) word to copy from the context sentence is inferred from neighboring words. However, because in our case the context can be very short and often contains no known relevant words (e.g., person names), we cannot use the existing approach directly.
Our knowledge memory is also related to the recent literature on neural networks with external memory (Bahdanau et al., 2014; Weston et al., 2015; Graves et al., 2014). In Weston et al. (2015), given simple sentences as facts which are stored in the external memory, the question answering task is studied. In fact, the tasks that the knowledge-based language model aims to solve (i.e. predict the next word) can be considered as a fill-in-the-blank type of question answering. The idea of jointly using Wikipedia and knowledge graphs has also been used in the context of enriching word embedding (Celikyilmaz et al., 2015; Long et al., 2016).
3 MODEL
3.1 PRELIMINARY
A topic2 k in a set of entities E is associated with topic knowledge Fk (e.g., from Freebase) and topic description Wk (e.g., from Wikipedia). Topic knowledge Fk is a set of facts {ak,1, ak,2, . . . , ak,|Fk|} where each fact a is a triple of subject ∈ E , relationship, and object ∈ E , e.g., (Barack Obama, Married-To, Michelle Obama). Topic description Wk is a sequence of words (wk1 , w k 2 , . . . , w k |Wk|) describing the topic (e.g., a description of a topic in Wikipedia). Because the subject entities in Fk are all equal to the topic entity k3 and the words describing relationships can easily be found in the vocabulary, we use the description of the object entity (e.g., Michelle Obama) as our fact description.
Given Fk and Wk, we perform simple string matching between words in Wk and words in the fact descriptions in Fk and thereby build a sequence of augmented observations Yk = {ykt = (wt, at, zt)}t=1:|Wk|. Here, wt ∈ Wk is an observed word, at ∈ Fk a fact on which the generated word wt is based, and zt a binary variable indicating whether wt is in the vocabulary V (including UNK) or not. Because not all words are based on a fact (e.g., words like, is, a, the, have), we introduce a special type of fact, called Not-a-Fact (NaF), and assign NaF to such words.
For example, a description “Rogers was born in Latrobe, Pennsylvania in 1928” from a topic Fred Rogers in Wikipedia, is augmented to, Y = {(w=“Rogers”, a=0, z=0), (“was”, NaF, 1), (“born”, NaF, 1), (“in”, NaF, 1), (“Latrobe”, 42, 0), (“Pennsylvania”, 42, 1), (“in”, NaF, 1), (“1928”, 83, 0)}. Here, we use facts on Fred Rogers, a42 = (Fred Rogers, Place of Birth, Latrobe Pennsylvania), a83 = (Fred Rogers, Year of Birth, 1928), and a special fact a0 = (Fred Rogers, Topic Itself, Fred Rogers) which we define in order to refer to the topic string itself. We also assume here that the words Rogers, Latrobe and 1928 are not in the vocabulary.
During the inference and training of topic k, we assume that the topic knowledge Fk is loaded in the knowledge memory in a form of a matrix Fk ∈ RDa×|Fk| where the i-th column is a fact embedding ak,i ∈ RDa . The fact embedding is the concatenation of subject, relationship, and object embeddings. We obtain these entity embeddings from a preliminary run of a knowledge graph embedding method such as TransE (Bordes et al., 2013). Note that we fix the fact embedding during the training of our model to help the model predict new facts at test time. But, we learn the embedding of the Topic Itself. For notation, to denote the vector representation of any object of our interest, we use bold lowercase characters. For example, the embedding of a word wt is represented by wt = W[wt] where WDw×|V| is the word embedding matrix, and W[wt] denotes the wt-th column of W.
2In this work, a topic is one of the entities which exist in both Wikipedia and Freebase. This is different to the concept in topic modeling where a topic is represented by a distribution over words.
3Although in Freebase the topic entity can be either the subject or the object, for convenience we process them such that the subject is always equal to the topic entity k.
a1 a2 a3 a4 … aN NaF
𝑤𝑡𝑜
O1 O2 O3 O4 … ON
Topic Knowledge
𝑤𝑡- 𝑧/
ℎ/ ℎ/𝑎/
𝑘/
𝑎/34 𝑤𝑡34- 𝑤𝑡345
ℎ/34 ℎ/ 𝑒 𝑥/
Figure 1: The NKLM model. The input consisting of a word (either wot−1 or w v t−1) and a fact (at−1) goes into LSTM. The LSTM’s output ht together with the knowledge context e generates the fact key kt. Using the fact key, the fact embedding at is retrieved from the topic knowledge memory. Using at and ht, knowledge-copy switch zt is determined, which in turn determines the next word generation source wvt or w o t . The copied word w o t is a symbol taken from the fact description Oat .
3.2 INFERENCE
At each time step, the NKLM follows four sub-steps. First, using both the word and fact outputs from the previous time step as the input of the current time step, we update the LSTM controller. Second, given the output of the LSTM, the NKLM predicts a fact (including NaF) and extracts corresponding fact embedding from the knowledge memory. Thirdly, with the extracted fact and the state of the LSTM controller, the NKLM makes a binary decision to choose the source of word generation. Finally, a word is generated according to the chosen source. A model diagram is depicted in Fig. 1. In the following, we describe these four steps in more detail.
1) Input Representation and LSTM Controller. As shown in Fig. 1, the input at time step t is the concatenation of three embedding vectors corresponding to a fact at−1, a vocabulary word wvt−1, and a copied word wot−1, all predicted in the previous time step. However, because at a time step, the predicted word comes only either from the vocabulary or by copying from the fact description, we set either wvt−1 or w o t−1 to a zero vector when it is not selected in the previous step. As we shall see, we use position embeddings to represent the copied words by its position within the fact description. And, because the dimensions of the vocabulary word embedding and the position embedding for copied words are different, we use such concatenation of wvt−1 and w o t−1 to represent the word input. The resulting input representation xt = fconcat(at−1,wvt−1,w o t−1) is then fed into the LSTM controller, and obtain the output states (ht, ct) = fLSTM(xt,ht−1). Note that at−1 and wot−1 (e.g., corresponding to n-th position) together can deliver information that a symbol in n-th position in the description of fact at−1 was used in the previous time step.
2) Fact Extraction. Then, we predict a relevant fact at on which the word wt will be based. If the word wt is supposed to be irrelevant to any fact, the NaF type is predicted. Unlike the fact embeddings, we learn the NaF embedding during training.
Predicting a fact is done in two steps. First, a fact-key kfact ∈ RDa is generated by kfact = ffactkey(ht, ek). Here, ek ∈ RDa is the topic context embedding (or a subgraph embedding of the topic) which encodes information about what facts are available in the knowledge memory so that the key generator adapts to changes in the knowledge memory. For example, if we remove a fact from the memory, without retraining, the fact-key generator should be aware of the absence of that information and thus should not generate a key vector for the removed fact. Although, in the experiments, we use mean-pooling (average of the all fact embeddings in the knowledge memory) to obtain ek, one can also consider using the soft-attention mechanism (Bahdanau et al., 2014). For the fact-key generator ffactkey, we use an MLP with one hidden layer of ReLU nonlinearity.
Then, using the generated fact-key kfact, we perform key-value lookup over the knowledge memory Fk to predict a fact and retrieve its embedding at,
P (at|ht) = exp(k>factFk[at])∑ a′ exp(k > factFk[a ′]) , (1)
at = argmax at∈Fk
P (at|ht), (2)
at = Fk[at]. (3)
Note that in order to perform the copy mechanism, we need to pick a single fact from the knowledge memory instead of using the weighted average of the fact embeddings as in the soft-attention.
3) Knowledge-Copy Switch. Given the encoding of the context ht and the embedding of the extracted fact at, the model decides the source for the next word generation: either from the vocabulary or from the fact description by copy. As zt = 1 if the word wt is in the vocabulary, we define the probability of selecting copy as:
ẑt = p(1− zt|ht) = sigmoid(fcopy(ht,at)). (4) Here, fcopy is an MLP with one ReLU hidden layer and a single linear output unit. For facts about attributes such as nationality or profession, the words in the fact description (e.g., “American” or “actor”) are likely to be in the vocabulary, but for facts like the year of birth or father name, the model is likely to choose to copy.
4) Word Generation. Word wt is generated from the source indicated by the copy-switch ẑt as follows:
wt = { wvt ∈ V, if ẑt < 0.5, wot ∈ Oat , otherwise.
For vocabulary word wvt ∈ V , we use the softmax function where each output dimension corresponds to a word in the vocabulary including UNK,
P (wvt = w|ht) = exp(k>vocaW[w])∑
w′∈V exp(k > vocaW[w
′]) . (5)
where kvoca ∈ RDw is obtained by fvoca(ht,at) which is an MLP with a ReLU hidden layer and linear output units of dimension Dw.
For knowledge word wot ∈ Oat , we predict the position of the word in the fact description and then copy the word on the predicted position to output. This is because, unlike with the traditional copy mechanism, our context words (i.e., the fact description) often consist of all unknown words and/or are short in length. Copying allows us not to rely on the word embeddings for the knowledge words. Instead, we learn the position embeddings shared among all knowledge words. This makes sense because words in the fact description usually appear one by one in increasing order. Thus, given that the first symbol o1 = “Michelle” was used in the previous time step and prior to that other words such as “President” and “US” were also observed, the model can easily predict that it is time to select the second symbol, i.e., o2 = “Obama”.
For this copy-by-position, we first generate the position key kpos ∈ RDo by a function fposkey(ht,at) which is again an MLP with one hidden layer and linear outputs whose dimension is equal to the maximum length of the fact descriptions Nomax = maxa∈F |Oa| where F = ∪kFk. Then, the n-th symbol on ∈ Oat is chosen by
P (wot = on|ht, at) = exp(k>posP[n])∑ n′ exp(k > posP[n ′]) , (6)
with n′ running from 0 to |Oat | − 1. Here, PDo×N o max is the position embedding matrix. Note that Nomax is typically a much smaller number (e.g., 20 in our experiments) than the size of vocabulary. The position embedding matrix P is learned during training.
Although in this paper we find that the simple position prediction performs well, we note that one could also consider a more advanced encoding such as one based on a convolutional network (Kim, 2014) to model the fact description. At test time, to compute p(wkt |wk<t), we can obtain {zk<t, ak<t} from {wk<t} and Fk using the automatic labeling script, and perform the above inference process with hard decisions taken about zt and at based on the model’s predictions.
3.3 LEARNING
Given word observations {Wk}Kk=1 and knowledge {Fk}Kk=1, our objective is to maximize the log-likelihood of the observed words w.r.t the model parameter θ,
θ∗ = argmax θ ∑ k logPθ(Wk|Fk). (7)
Because, given Wk and Fk, a sequence of Yk = {yt = (wt, zt, at)}t=1:|Wk| is deterministically induced for each word wt, the following equality is satisfied
Pθ(Wk|Fk) = Pθ(Yk|Fk). (8)
By the chain rule, we can decompose the probability of the observation Yk as
logPθ(Yk|Fk) = |Yk|∑ t=1 logPθ(y k t |yk1:t−1,Fk). (9)
Then, after omitting Fk and k for simplicity, we can rewrite the single step conditional probability as
Pθ(yt|y1:t−1) = Pθ(wt, at, zt|ht) = Pθ(wt|at, zt, ht)Pθ(at|ht)Pθ(zt|ht). (10)
We maximize the above objective using stochastic gradient optimization.
4 EVALUATION
4.1 WIKIFACTS DATASET
An obstacle in developing the above model is the lack of the dataset where the text corpus is aligned with facts at the word level. To this end, we produced the WikiFacts dataset by aligning Wikipedia descriptions with corresponding Freebase facts. Because many Freebase topics provide a link to its corresponding topic in Wikipedia, we choose a set of topics for which both a Freebase entity and a Wikipedia description exist. In the experiments, we used a version called WikiFacts-FilmActor-v0.1 where the domain is restricted to the /Film/Actor in Freebase.
For all object entity descriptions {Oak} associated with Fk, we performed string matching to the Wikipedia description Wk. We used the summary part (first few paragraphs) of the Wikipedia page as text to be modeled but discarded topics for which the number of facts is greater than 1000 or the Wikipedia description is too short (< 3 sentences). For the string matching, we also used the synonyms and alias provided by WordNet (Miller, 1995) and Freebase.
We augmented the fact set Fk with the anchor facts Ak whose relationship is all set to UnknownRelation. That is, observing that an anchor (words under hyperlink) in Wikipedia descriptions has a corresponding Freebase entity as well as being semantically closely related to the topic in which the anchor is found, we make a synthetic fact of the form (Topic, UnknownRelation, Anchor). This potentially compensates for some missing facts in Freebase. Because we extract the anchor facts from the full Wikipedia page and they all share the same relation, it is more challenging for the model to use these anchor facts than using the Freebase facts. As a result, for each word w in the dataset, we have a tuple (w, zw, aw, kw). Here, kw is the topic where w appears. We provide a summary of the dataset statistics in Table 1. The dataset will be available on a public webpage4.
4.2 EXPERIMENTS
Setup. We split the dataset into 80/10/10 for train, validation, and test. As a baseline model, we use the RNNLM. For both the NKLM and the RNNLM, two-layer LSTMs with dropout regularization (Zaremba et al., 2014) are used. We tested models with different numbers of LSTM hidden units [200, 500, 1000], and report results from the 1000 hidden-unit model. For the NKLM, we set the symbol embedding dimension to 40 and word embedding dimension to 400. Under this setting, the number of parameters in the NKLM is slightly smaller than that of the RNNLM. We used
4https://bitbucket.org/skaasj/wikifact_filmactor
100-dimension TransE embeddings for Freebase entities and relations, and concatenate the relation and object embeddings to obtain fact embeddings. We averaged all fact embeddings in Fk to obtain the topic context embedding ek. We unrolled the LSTMs for 30 steps and used minibatch size 20. We trained the models using stochastic gradient ascent with gradient clipping range [-5,5]. The initial learning rate was set to 0.5 for the NKLM and 1.5 for the RNNLM, and decayed after every epoch by a factor of 0.98. We trained for 50 epochs and report the results chosen by the best validation set results.
Evaluation metric. The perplexity exp(− 1N ∑N i=1 log pwi) is the standard performance metric for language modeling. This, however, has a problem in evaluating language models for a corpus containing many named entities: a model can get good perplexity by accurately predicting UNK words. As an extreme example, when all words in a sentence are unknown words, a model predicting everything as UNK will get a good perplexity. Considering that unknown words provide virtually no useful information, this is clearly a problem in tasks such as question answering, dialogue modeling, and knowledge language modeling.
To this end, we introduce a new evaluation metric, called the Unknown-Penalized Perplexity (UPP), and evaluate the models on this metric as well as the standard perplexity (PPL). Because the actual word underlying the UNK should be one of the out-of-vocabulary (OOV) words, in UPP, we penalize the likelihood of unknown words as follows:
PUPP(wunk) = P (wunk)/|Vtotal \ Vvoca|. Here, Vtotal is a set of all unique words in the corpus, and Vvoca is the vocabulary used in the softmax. In other words, in UPP we assume that the OOV set is equal to |Vtotal\Vvoca| and thus assign a uniform probability to OOV words. In another version, UPP-fact, we consider the fact that the RNNLM can also use the knowledge given to the NKLM to some extent, but with limited capability (because the model is not designed for it). For this, we assume that the OOV set is equal to the total knowledge vocabulary of a topic k, i.e., PUPP-fact(wunk) = P (wunk)/|Ok|, where Ok = ∪iOak,i . In other words, by using UPP-fact, we assume that, for an unknown word, the RNNLM can pick one of the knowledge words with uniform probability. We describe the detail results and discussion on the experiments in the captions of Table 2, 3, and 4.
Observations from the experiment results. Our observations from the experiment results are as follows. (a) The NKLM outperforms the RNNLM in all three perplexity measures. (b) The copy mechanism is the key of the significant performance improvement. Without the copy mechanism, the NKLM still performs better than the RNNLM due to its usage of the fact information, but the improvement is not so significant. (c) The NKLM results in a much smaller number of UNKs (roughly, a half of the RNNLM). (d) When no knowledge is available, the NKLM performs as well as the
RNNLM. (e) KG embedding using TransE is an efficient way to initialize the fact embeddings. (f) The NKLM generates named entities in the provided facts whereas the RNNLM generates many more UNKs. (g) The NKLM shows its ability to adapt immediately to the change of the knowledge. (h) The standard perplexity is significantly affected by the prediction accuracy on the unknown words. Thus, one need carefully consider it as a metric for knowledge-related language models.
5 CONCLUSION
In this paper, we presented a novel Neural Knowledge Language Model (NKLM) that brings the symbolic knowledge from a knowledge graph into the expressive power of RNN language models. The
NKLM significantly outperforms the RNNLM in terms of perplexity and generates named entities which are not observed during training, as well as immediately adapting to changes in knowledge. We believe that the WikiFact dataset introduced in this paper, can be useful in other knowledge-related language tasks as well. In addition, the Unknown-Penalized Perplexity introduced in this paper in order to resolve the limitation of the standard perplexity, can be useful in evaluating other language tasks. The task that we investigated in this paper is limited in the sense that we assume that the true topic of a given description is known. Relaxing this assumption by making the model search for proper topics on-the-fly will make the model more practical. We believe that there are many more open research challenges related to the knowledge language models.
ACKNOWLEDGMENTS
The authors would like to thank Alberto Garcı́a-Durán, Caglar Gulcehre, Chinnadhurai Sankar, Iulian Serban and Sarath Chandar for feedback and discussions as well as the developers of Theano (Bastien et al., 2012), NSERC, CIFAR, Samsung and Canada Research Chairs for funding, and Compute Canada for computing resources.
APPENDIX: HEATMAPS

Review:
This paper addresses the practical problem of generating rare or unseen words in the context of language modeling. Since language follows a Zipf’s law, most approaches limit the vocabulary (because of computation reasons) and hence rare words are often mapped to a UNK token. Rare words are especially important in context of applications such as question answering. MT etc. This paper proposes a language modeling technique which incorporates facts from knowledge bases (KBs) and thus has the ability to generate (potentially unseen) words from KBs. This paper also releases a dataset by aligning words with Freebase facts and corresponding Wikipedia descriptions.
The model first selects a KB fact based on the previously generated words and facts. Based on the selected fact, it then predicts whether to generate a word based on the vocabulary or to output a symbolic word from the KB. For the latter, the model is trained to predict the position of the word from the fact description.
Overall the paper could use some rewriting especially the notations in section 3. The experiments are well executed and they definitely get good results. The heat maps at the end are very insightful. 
Comments
This contributions of this paper would be much stronger if it showed improvements in a practical applications such as Question Answering (although the paper clearly mentions that this technique could be applied to improve QA)
In section 3, it is unclear why the authors refer the entity as a ‘topic'. This makes the text a little confusing since a topic can also be associated with something abstract, but in this case the topic is always a freebase entity. 
Is it really necessary to predict a fact at every step before generating a word. In other words, how many distinct facts on average does the model choose to generate a sentence. Intuitively a natural language sentence would be describe few facts about an entity. If the fact generation step could be avoided (by adding a latent variable which decides if the fact should be generated or not), the model will also be faster.
In equation 2, the model has to make a hard decision to choose the fact. For this to be end to end trained, every word needs to be annotated with a corresponding fact which might not be always a realistic scenario. For e.g., in domains such as social media text.
Learning position embeddings for copying knowledge words seems a little counter-intuitive. Does the sequence of knowledge words follow any particular structure like word O_2 is always the last name (e.g. Obama).
It would also be nice to compare to char-level LM's which inherently solves the unknown token problem.