Paper:
REFERENCE-AWARE LANGUAGE MODELS
We propose a general class of language models that treat reference as an explicit stochastic latent variable. This architecture allows models to create mentions of entities and their attributes by accessing external databases (required by, e.g., dialogue generation and recipe generation) and internal state (required by, e.g. language models which are aware of coreference). This facilitates the incorporation of information that can be accessed in predictable locations in databases or discourse context, even when the targets of the reference may be rare words. Experiments on three tasks show our model variants outperform models based on deterministic attention.
1 INTRODUCTION
Referring expressions (REs) in natural language are noun phrases (proper nouns, common nouns, and pronouns) that identify objects, entities, and events in an environment. REs occur frequently and they play a key role in communicating information efficiently. While REs are common, previous works neglect to model REs explicitly, either treating REs as ordinary words in the model or replacing them with special tokens. Here we propose a language modeling framework that explicitly incorporates reference decisions.
In Figure 1 we list examples of REs in the context of the three tasks that we consider in this work. Firstly, reference to a database is crucial in many applications. One example is in task oriented dialogue where access to a database is necessary to answer a user’s query (Young et al., 2013; Li et al., 2016; Vinyals & Le, 2015; Wen et al., 2015; Sordoni et al., 2015; Serban et al., 2016; Bordes & Weston, 2016; Williams & Zweig, 2016; Shang et al., 2015; Wen et al., 2016). Here we consider the domain of restaurant recommendation where a system refers to restaurants (name) and their attributes (address, phone number etc) in its responses. When the system says “the nirala is a nice restaurant”, it refers to the restaurant name the nirala from the database. Secondly, many models need to refer to a list of items (Kiddon et al., 2016; Wen et al., 2015). In the task of recipe generation from a list of ingredients (Kiddon et al., 2016), the generation of the recipe will frequently reference these items. As shown in Figure 1, in the recipe “Blend soy milk and . . . ”, soy milk refers to the ingredient summaries. Finally, we address references within a document (Mikolov et al., 2010; Ji et al., 2015; Wang & Cho, 2015), as the generation of words will ofter refer to previously generated words. For instance the same entity will often be referred to throughout a document. In Figure 1, the entity you refers to I in a previous utterance.
In this work we develop a language model that has a specific module for generating REs. A series of latent decisions (should I generate a RE? If yes, which entity in the context should I refer to? How should the RE be rendered?) augment a traditional recurrent neural network language model and the two components are combined as a mixture model. Selecting an entity in context is similar to familiar models of attention (Bahdanau et al., 2014), but rather than being a deterministic function that reweights representations of elements in the context, it is treated as a distribution over contextual elements which are stochastically selected and then copied or, if the task warrants it, transformed (e.g., a pronoun rather than a proper name is produced as output). Two variants are possible for updating the RNN state: one that only looks at the generated output form; and a second that looks at values of the latent variables. The former admits trivial unsupervised learning, latent decisions are conditionally independent of each other given observed context, whereas the latter enables more
∗Work completed at DeepMind.
expressive models that can extract information from the entity that is being referred to. In each of the three tasks, we demonstrate our reference aware model’s efficacy in evaluations against models that do not explicitly include a reference operation.
Our contributions are as follows:
• We propose a general framework to model reference in language and instantiate it in the context of dialogue modeling, recipe generation and coreference based language models.
• We build three data sets to test our models. There lack existing data sets that satisfy our need, so we build these data sets ourselves. These data sets are either built on top existing data set (we constructed the table for DSTC2 data set for dialogue evaluation), crawled from websites (we crawled all recipes in www.allrecipes.com) or annotated with NLP tools (we annotate the coreference with Gigaword corpus for our evaluation).
• We perform comprehensive evaluation of our models on the three data sets and verify our models perform better than strong baselines.
2 REFERENCE-AWARE LANGUAGE MODELS
Here we propose a general framework for reference-aware language models.
We denote each document as a series of tokens x1, . . . , xL, where L is the number of tokens in the document. Our goal is to maximize the probabilities p(xi | ci), for each word in the document based on its previous context ci = x1, . . . , xi−1. In contrast to traditional neural language models, we introduce a variable at each position zi, which controls the decision on which source xi is generated from. The token conditional probably is then obtained by:
p(xi | ci) = p(xi | zi, ci)p(zi | ci). (1)
In dialogue modeling and recipe generation, zi will simply taken on values in {0, 1}. Where zi = 1 denotes that xi is generated as a reference, either to a database entry or an item in a list. However, zi can also be defined as a distribution over previous entities, allowing the model to predict xi conditioned on its a previous mention word. This will be the focus of the coreference language model. When zi is not observed (which it generally will not be), we will train our model to maximize the marginal probability in Eq. 1 directly.
2.1 DIALOGUE MODEL WITH DATABASE SUPPORT
We first apply our model on task-oriented dialogue systems in the domain of restaurant recommendations, and work on the data set from the second Dialogue State Tracking Challenge (DSTC2) (Henderson et al., 2014). Table. 1 is one example dialogue from this dataset.
We can observe from this example, users get recommendations of restaurants based on queries that specify the area, price and food type of the restaurant. We can support the system’s decisions by incorporating a mechanism that allows the model to query the database allowing the model to find restaurants that satisfy the users queries. Thus, we crawled TripAdvisor for restaurants in the
Cambridge area, where the dialog dataset was collected. Then, we remove restaurants that do not appear in the data set and create a database with 109 entries with restaurants and their attributes (e.g. food type). A sample of our database is shown in Table. 2. We can observe that each restaurant contains 6 attributes that are generally referred in the dialogue dataset. As such, if the user requests a restaurant that serves “indian” food, we wish to train a model that can search for entries whose “food” column contains “indian”. Now, we describe how we deploy a model that fulfills these requirements.
2.1.1 DIALOGUE MODEL
We build a model based on the hierarchical RNN model described in (Serban et al., 2016), as in dialogues, the generation of the response is not only dependent on the previous sentence, but on all sentences leading to the response. We assume that a dialogue is alternated between a machine and a user. An illustration of the model is shown in Figure 2.
Consider a dialogue with T turns, and the utterance from a user is denoted as X = {xi}Ti=1, where i is the i-th utterance, whereas the utterance from a machine is denoted as Y = {yi}Ti=1, where i is the i-th utterance. We define xi = {xij}|xi|j=1, yi = {yiv} |yi| v=1, where xij denotes the j-th token in the i-th utterance from the user, whereas yiv denotes the v-th token in the i-th utterance from the machine. Finally, |xi| and |yi| denote the number of tokens in the user and machine utterances, respectively. The dialogue sequence starts with machine utterance {y1, x1, y2, x2, . . . , yT , xT }. We would like to model the utterances from the machine
p(y1, y2, . . . , yT |x1, x2, . . . , xT ) = ∏ i p(yi|y<i, x<i) = ∏ i,v p(yi,v|yi,<v, y<i, x<i),
where y<i denotes all the utterances before i and yi,<v denotes the first v − 1 tokens in the i-th utterance of the machine. A neural model is employed to predict p(yi,v|yi,<v, y<i, x<i), which operates as follows:
Sentence Encoder: We first encode previous utterances y<i and x<i into continuous space by generating employing a LSTM encoder. Thus, for a given utterance xi, and start with the initial LSTM state hxi,0 and apply the recursion h x i,j = LSTME(WExi,j , h x i,j−1), where WExi,j denotes a word
embedding lookup for the token xi,j , and LSTME denotes the LSTM transition function described in Hochreiter & Schmidhuber (1997). The representation of the user utterance is represented by the final LSTM state hxi = h x i,|xi|. The same process is applied to obtain the machine utterance representation hyi = h y i,|yi|.
Turn Encoder: Then, combine all the representations of all the utterances with a second LSTM, which encodes the sequence {hy1, hx1 , ..., h y i , h x i } into a continuous vector. Once again, we start with an initial state u0 and feed each of the utterance representation to obtain the following LSTM state, until the final state is obtained. For simplicity, we shall refer to this as ui, which can be seen as the hierarchical encoding of the previous i utterances.
Seq2Seq Decoder: As for decoding, in order to generate each utterance yi, we can feed ui−1 into the decoder LSTM as the initial state si,0 = ui−1 and decode each token in yi. Thus, we can express the decoder as:
syi,v = LSTMD(WEyi,v−1, si,v−1), pyi,v = softmax(Ws y i,v),
where the desired probability p(yi,v|yi,<v, y<i, x<i) is expressed by pyi,v .
Attention based decoder: We can also incorporate the attention mechanism in our hierarchical model. An attention model builds a representation d by averaging over a set of vectors p. We define the attention function as a = ATTN(p, q), where a is a probability distribution over the set of vectors p, conditioned on any input representation q. A full description of this operation is described in (Bahdanau et al., 2014). Thus, for each generated token yi,v, we compute the attentions ai,v , conditioned on the current decoder state syi,v, obtaining the attentions over input tokens from previous turn (i−1). We denote the vector of all tokens in previous turn as hx,yi−1 = [{hxi−1,j} |xi−1| j=1 , {h y i−1,v} |yi−1| v=1 ]. Let K = |hx,yi−1| be the number of tokens in previous turn. Thus, we obtain the attention probabilities over all previous tokens ai,v as ATTN(s y i,v, h x,y i−1). Then, the weighted sum is computed over these
probabilities di,v = ∑ k∈K ai,v,kh x,y i−1,k, where ai,v,k is the probability of aligning to the k-th token from previous turn. The resulting vector di,v is used to obtain the probability of the following word pyi,v . Thus, we express the decoder as:
syi,v = LSTMD([WEyi,v−1, di,v−1], si,v−1), ai,v = ATTN(h x,y i−1, s y i,v),
di,v = ∑ k∈K ai,v,kh x,y i−1,k, pyi,v = softmax(W [s y i,v, di,v]).
2.1.2 INCORPORATING TABLE ATTENTION
We now extend the attention model in order to allow the attention to be computed over a table, allowing the model to condition the generation on a database.
We denote a table with R rows and C columns as {fr,c}, r ∈ [1, R], c ∈ [1, C], where fr,c is the cell in row r and column c. The attribute of each column is denoted as sc, where c is the c-th attribute. fr,c and sc are one-hot vector.
Table Encoding: To encode the table, we build an attribute vector gc for each column. For each cell fr,c of the table, we concatenate it with the corresponding attribute gc and then feed it through a one-layer MLP as follows: gc = WEsc and then er,c = tanh(W [WEfr,c, gc]).
Table Attention: The diagram for table attention is shown in Figure 3a. The attention over cells in the table is conditioned on a given vector q, similarly to the attention model for sequences ATTN(p, q). However, rather than a sequence p, we now operate over a table f . Our attention model computes a attribute attention followed by row attention of the table. We first use the attention mechanism on the attributes to find out which attribute the user asks about. Suppose a user says cheap, then we should focus on the price attribute. After we get the attention probability pa = ATTN({gc}, q), over the attribute, we calculate the weighted representation for each row er = ∑ c p a cerc conditioned on p
a. Then er has the price information of each row. We further use attention mechanism on er and get the probability pr = ATTN({er}, q) over the rows. Then restaurants with cheap price will be picked. Then, using the probabilities pr, we compute the weighted average over the all rows ec = ∑ r p
r rer,c, which is used in the decoder. The detailed process is: pa = ATTN({gc}, q), (2)
er = ∑ c pacerc ∀r, (3)
pr = ATTN({er}, q), (4) ec = ∑ r prrer,c ∀c. (5)
This is embedded in the decoder by replacing the conditioned state q as the current decoder state syi,0 and then at each step, conditioning the prediction of yi,v on {ec} by using attention mechanism at each step. The detailed diagram of table attention is shown in Figure 3a.
2.1.3 INCORPORATING TABLE POINTER NETWORKS
We now describe the mechanism used to refer to specific database entries during decoding. At each timestep, the model needs to decide whether to generate the next token from an entry of the database or from the word softmax. This is performed as follows.
Pointer Switch: We use zi,v ∈ [0, 1] to denote the decision of whether to copy one cell from the table. We compute this probability as follows:
p(zi,v|si,v) = sigmoid(W [si,v, di,v]). Thus, if zi,v = 1, the next token yi,v will be generated from the database, whereas if zi,v = 0, then the following token is generated from a softmax. We shall now describe how we generate tokens from the database.
Table Pointer: If zi,v = 1, the token is generated from the table. The detailed process of calculating the probability distribution over the table is shown in Figure 3b. This is similar to the attention mechanism, except that we perform a column attention to compute the probabilities of copying from each column after Equation. 5. More formally:
pc = ATTN({ec}, q), (6) pcopy = pr ⊗ pc, (7)
where pc is a probability distribution over columns, whereas pr is a probability distribution over rows. In order to compute a matrix with the probability of copying each cell, we simply compute the outer product pcopy = pr ⊗ pc. Objective: As we treat zi as a latent variable, we wish to maximize the marginal probability of the sequence yi over all possible values of zi. Thus, our objective function is defined as:
p(yi,v|si,v) = pvocabp(0|si,v) + pcopyp(1|si,v) = pvocab(1− p(1|si,v)) + pcopyp(1|si,v). (8)
The model can also be trained in a fully supervised fashion, if zi,v is observed. In such cases, we simply maximize the likelihood of p(zi,v|si,v), based on the observations, rather than using the marginal probability over zi,v .
2.2 RECIPE GENERATION
Next, we consider the task of recipe generation conditioning on the ingredient lists. In this task, we must generate the recipe from a list of ingredients. Table. 3 illustrates the ingredient list and recipe for Spinach and Banana Power Smoothie. We can see that the ingredients soy milk, spinach leaves, and banana occur in the recipe.
Let the ingredients of a recipe be X = {xi}Ti=1 and each ingredient contains L tokens xi = {xij}Lj=1. The corresponding recipe is y = {yv}Kv=1. We first use a LSTM to encode each ingredient:
hi,j = LSTME(WExij , hi,j−1) ∀i. Then, we sum the resulting state of each ingredient to obtain the starting LSTM state of the decoder. Once again we use an attention based decoder:
sv = LSTMD(sv−1, dv−1,WEyv−1),
pcopyv = ATTN({{hi,j}Ti=1}Lj=1, sv), dv = ∑ ij pv,i,jhi,j ,
p(zv|sv) = sigmoid(W [sv, dv]), pvocabv = softmax(W [sv, dv]).
Similar to the previous task, the decision to copy from the ingredient list or generate a new word from the softmax is performed using a switch, denoted as p(zv|sv). We can obtain a probability distribution of copying each of the words in the ingredients by computing pcopyv = ATTN({{hi,j}Ti=1}Lj=1, sv) in the attention mechanism. For training, we optimize the marginal likelihood function employed in the previous task.
2.3 COREFERENCE BASED LANGUAGE MODEL
Finally, we build a language model that uses coreference links to point to previous words. Before generating a word, we first make the decision on whether it is an entity mention. If so, we decide
which entity this mention belongs to, then we generate the word based on that entity. Denote the document as X = {xi}Li=1, and the entities are E = {ei}Ni=1, each entity has Mi mentions, ei = {mij}Mij=1, such that {xmij} Mi j=1 refer to the same entity. We use a LSTM to model the document, the hidden state of each token is hi = LSTM(WExi, hi−1). We use a set he = {he0, he1, ..., heM} to keep track of the entity states, where hej is the state of entity j.
um and [I]1 think that is whats - Go ahead [Linda]2. Well and thanks goes to [you]1 and to [the media]3 to help [us]4...So [our]4 hat is off to all of [you]5...
Word generation: At each time step before generating the next word, we predict whether the word is an entity mention:
pcoref(vi|hi−1, he) = ATTN(he, hi−1), di = ∑ vi p(vi)h e vi
p(zi|hi−1) = sigmoid(W [di, hi−1]),
where zi denotes whether the next word is an entity and if yes vi denotes which entity the next word corefers to. If the next word is an entity mention, then p(xi|vi, hi−1, he) = softmax(W1 tanh(W2[hevi , hi−1])) else p(xi|hi−1) = softmax(W1hi−1),
p(xi|x<i) = { p(xi|hi−1)p(zi|hi−1, he) if zi = 0. p(xi|vi, hi−1, he)pcoref(vi|hi−1, he)p(zi|hi−1, he) if zi = 1.
(9)
Entity state update: We update the entity state he at each time step. In the beginning, he = {he0}, he0 denotes the state of an virtual empty entity and is a learnable variable. If zi = 1 and vi = 0, then it indicates the next word is a new entity mention, then in the next step, we append hi to he, i.e., he = {he, hi}, if ei > 0, then we update the corresponding entity state with the new hidden state, he[vi] = hi. Another way to update the entity state is to use one LSTM to encode the mention states and get the new entity state. Here we use the latest entity mention state as the new entity state for simplicity. The detailed update process is shown in Figure 5.
3 EXPERIMENTS
4 DATA SETS AND PREPROCESSING
Dialogue: We use the DSTC2 data set. We only extracted the dialogue transcript from data set. There are about 3,200 dialogues in total. Since this is a small data set, we use 5-fold cross validation and report the average result over the 5 partitions. There may be multiple tokens in each table cell, for example in Table.2, the name, address, post code and phone number have multiple tokens, we replace them with one special token. For the name, address, post code and phone number of the j-th row, we replace the tokens in each cell with NAME j, ADDR j, POSTCODE j, PHONE j. If a table cell is empty, we replace it with an empty token EMPTY. We do a string match in the transcript and replace the corresponding tokens in transcripts from the table with the special tokens.
Each dialogue on average has 8 turns (16 sentences). We use a vocabulary size of 900, including about 400 table tokens and 500 words.
Recipes: We crawl all recipes from www.allrecipes.com. There are about 31, 000 recipes in total, and every recipe has a ingredient list and a corresponding recipe. We exclude the recipes that have less than 10 tokens or more than 500 tokens, those recipes take about 0.1% of all data set. On average each recipe has 118 tokens and 9 ingredients. We random shuffle the whole data set and take 80% as training and 10% for validation and test. We use a vocabulary size of 10,000 in the model.
Coref LM: We use the Xinhua News data set from Gigaword Fifth Edition and sample 100,000 documents from it that has length in range from 100 to 500. Each document has on average 234 tokens, so there are 23 million tokens in total. We use a tool to annotate all the entity mentions and use the annotation in the training. We take 80% as training and 10% as validation and test respectively. We ignore the entities that have only one mention and for the mentions that have multiple tokens, we take the token that is most frequent in the all the mentions for this entity. After the preprocessing, tokens that are entity mentions take about 10% of all tokens. We use a vocabulary size of 50,000 in the model.
4.1 MODEL TRAINING AND EVALUATION
We train all models with simple stochastic gradient descent with clipping. We use a one-layer LSTM for all RNN components. Hyper-parameters are selected using grid search based on the validation set. We use dropout after the input embedding and LSTM output. The learning rate is selected from [0.1, 0.2, 0.5, 1], maximum gradient norm is selected from [1, 2, 5, 10] and drop ratio is selected from [0.2, 0.3, 0.5]. The batch size and LSTM dimension size is slightly different for different tasks so as to make the model fit into memory. The number of epochs to train are different for each task and we drop the learning rate after reaching a given number of epochs. We report the per-word perplexity for all tasks, specifically, we report the perplexity of all words, words that can be generated from reference and non-reference words. For recipe generation, we also generate the recipe using beam size of 10 and evaluate the generated recipe with BLEU.
4.2 RESULTS AND ANALYSIS
The results for dialogue, recipe generation and coref language model are shown in Table 4, 5 and 6 respectively. We can see from Table 4 that models that condition on table performs better in predicting table tokens in general. Table pointer has the lowest perplexity for token in the table. Since the table token appears rarely in the dialogue, the overall perplexity does not differ much and the non-table tokens perplexity are similar. With attention mechanism over the table, the perplexity of table token improves over basic seq2seq model, but not as good as directly pointing to cells in the table. As expected, using sentence attention improves significantly over models without sentence attention. Surprisingly, table latent performs much worse than table pointer. We also measure the perplexity of table tokens that appear only in test set. For models other than table pointer, because the tokens never appear in training set, the perplexity is quite high, while table pointer can predict these tokens much more accurately. The recipe results in Table 5 in general follows that findings from the dialogue. But the latent model performs better than pointer model since that tokens in ingredients that match with recipe does not necessarily come from the ingredients. Imposing a supervised signal will give wrong information to the model and hence make the result worse. Hence with latent decision, the model learns to when to copy and when to generate it from the vocabulary. The coref LM results are shown in Table 6. We find that coref based LM performs much better on the entities perplexities, but however is a little bit worse than for non-entity words. We found it is an optimization problem and perhaps the model is stuck in local optimum. So we initialize the pointer model with the weights learned from LM, the pointer model performs better than LM both for entity perplexity and non-entity words perplexity.
5 RELATED WORK
Recently, there has been great progresses in modeling languages based on neural network, including language modeling (Mikolov et al., 2010; Jozefowicz et al., 2016), machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), question answering (Hermann et al., 2015) etc. Based on the success of seq2seq models, neural networks are applied in modeling chit-chat dialogue (Li et al., 2016; Vinyals & Le, 2015; Sordoni et al., 2015; Serban et al., 2016; Shang et al., 2015) and task oriented dialogue (Wen et al., 2015; Bordes & Weston, 2016; Williams & Zweig, 2016; Wen et al., 2016). Most of the chit-chat neural dialogue models are simply applying the seq2seq models. For the task oriented dialogues, most of them embed the seq2seq model in traditional dialogue systems, in which the table query part is not differentiable. while our model queries the database directly. Recipe generation was proposed in (Kiddon et al., 2016). Their model extents previous work on attention models (Allamanis et al., 2016) to checklists, whereas our work models explicit references to those checklists. Context dependent language models (Mikolov et al., 2010; Ji et al., 2015; Wang & Cho, 2015) are proposed to capture long term dependency of text. There are also lots of works on coreference resolution (Haghighi & Klein, 2010; Wiseman et al., 2016). We are the first to combine coreference with language modeling, to the best of our knowledge. Much effort has been invested in embedding a copying mechanism for neural models (Gülçehre et al., 2016; Gu et al., 2016; Ling et al., 2016). In general, a gating mechanism is employed to combine the softmax over observed words and a pointer network (Vinyals et al., 2015). These gates can be trained either by marginalizing over both outcomes, or using heuristics (e.g. copy low frequency words). Our models are similar to models proposed in (Ahn et al., 2016; Merity et al., 2016), where the generation of each word can be conditioned on a particular entry in knowledge lists and previous words. In our work, we describe a model with broader applications, allowing us to condition, on databases, lists and dynamic lists.
6 CONCLUSION
We introduce reference-aware language models which explicitly model the decision of from where to generate the token at each step. Our model can also learns the decision by treating it as a latent variable. We demonstrate on three tasks, table based dialogue modeling, recipe generation and coref based LM, that our model performs better than attention based model, which does not incorporate this decision explicitly. There are several directions to explore further based on our framework. The current evaluation method is based on perplexity and BLEU. In task oriented dialogues, we can also try human evaluation to see if the model can reply users’ query accurately. It is also interesting to use reinforcement learning to learn the actions in each step.

Review:
This paper introduces pointer-network neural networks, which are applied to referring expressions in three small-scale language modeling tasks: dialogue modeling, recipe modeling and news article modeling. When conditioned on the co-reference chain, the proposed models outperform standard sequence-to-sequence models with attention.
The proposed models are essentially variants of pointer networks with copy mechanisms (Gulcehre et al., 2016; Gu et al., 2016; Ling et al., 2016), which have been modified to take into account reference chains. As such, the main architectural novelty lies in 1) restricting the pointer mechanism to focus on co-referenced entities, 2) applying pointer mechanism to 2D arrays (tables), and 3) training with supervised alignments. Although useful in practice, these are minor contributions from an architectural perspective.
The empirical contributions are centred around measuring perplexity on the three language modeling tasks. Measuring perplexity is typical for standard language modeling tasks, but is really an unreliable proxy for dialogue modeling and recipe generation performance. In addition to this, both the dialogue and recipe tasks are tiny compared to standard language modeling tasks. This makes it difficult to evaluate the impact of the dialogue and recipe modeling results. For example, if one was to bootstrap from a larger corpus, it seems likely that a standard sequence-to-sequence model with attention would yield performance comparable to the proposed models (with enough data, the attention mechanism could learn to align referring entities by itself). The language modeling task on news article (Gigaword) seems to yield the most conclusive results. However, the dataset for this task is non-standard and results are provided for only a single baseline. Overall, this limits the conclusions we can draw from the empirical experiments.
Finally, the paper itself contains many errors, including mathematical errors, grammatical errors and typos:
- Eq. (1) is missing a sum over $z_i$.
- "into the a decoder LSTM" -> "into the decoder LSTM"
- "denoted as his" -> "denoted as"
- "Surprising," -> "Surprisingly,"
- "torkens" -> "tokens"
- "if follows that the next token" -> "the next token"
- In the "COREFERENCE BASED LANGUAGE MODEL" sub-section, what does $M$ denote?
- In the sentence: "The attribute of each column is denoted as $s_c, where $c$ is the c-th attribute". For these definitions to be make sense, $s_c$ has to be a one-hot vector. If yes, please clarify this in the text.
- "the weighted sum is performed" -> "the weighted sum is computed"
- "a attribute" -> "an attribute"
- In the paragraph on Pointer Switch, change $p(z_{i,v} |s_{i,v}) = 1$ -> $p(z_{i,v} |s_{i,v}) = 0$.
- In the "Table Pointer" paragraph, I assume you mean outer-product instead of cross-product? Otherwise, I don't see how the equations add up.
Other comments:
- For the "Attention based decoder", is the attention computed using the word embeddings themselves or the hidden states of the sentence encoder? Also, it applied only to the previous turn of the dialogue or to the entire dialogue history? Please clarify this.
- What's the advantage of using an "Entity state update" rule, compared to a pointer network or copy network, which you used in the dialogue and recipe tasks? Please elaborate on this.
- In the Related Work section, the following sentence is not quite accurate: "For the task oriented dialogues, most of them embed the seq2seq model in traditional dialogue systems while our model queries the database directly.". There are task-oriented dialogue models which do query databases during natural language generation. See, for example, "A Network-based End-to-End Trainable Task-oriented Dialogue System" by Wen et al.