A N EURAL M ETHOD FOR G OAL -O RIENTED D IALOG S YSTEMS TO INTERACT WITH N AMED E NTITIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Many goal-oriented dialog tasks, especially ones in which the dialog system has to interact with external knowledge sources such as databases, have to handle a large number of Named Entities (NEs).
There are at least two challenges in handling NEs using neural methods in such settings: individual NEs may occur only rarely making it hard to learn good representations of them, and many of the
Out Of Vocabulary words that occur during test time may be NEs.
Thus, the need to interact well with these NEs has emerged as a serious challenge to building neural methods for goal-oriented dialog tasks.
In this paper, we propose a new neural method for this problem, and present empirical evaluations on a structured
Question answering task and three related goal-oriented dialog tasks that show that our proposed method can be effective in interacting with NEs in these settings.
I NTRODUCTION AND P ROBLEM D ESCRIPTION
Goal-oriented dialog systems (Wen et al., 2016; Bordes & Weston, 2016a;
Dhingra et al., 2017) are those in which the system tries to achieve an explicit goal by the end of the dialog.
Examples include restaurant reservation, ﬂight ticket booking, and course selection advising.
One common aspect of many of these goal-oriented dialog systems is their need to interact with a large number of Named Entities (NEs).
This is especially true in cases where the system has to interact with external knowledge sources such as DataBases (DB), as is the case in all of the examples mentioned above.
NEs in these systems include restaurant names, place names, phone numbers, addresses, time, faculty names, course numbers, etc.
Recently,there has been a lot of interest in building neural methods for goal-oriented dialog systems.
The presence of and the need to interact with large numbers of Named Entities in these tasks has emerged as a serious challenge to achieving good performance.
There are different ways in which past work has tried to handle NEs in these neural systems.
One straightforward way is to add each and every NE (including those in the DB) to the vocabulary, which was been evaluated for only synthetic or small tasks (Neelakantan et al., 2015).
For real world tasks with large DBs, this causes an explosion in the vocabulary size and hence the number of parameters to learn.
There is also the problem of not being able to learn good representations for individual NEs, as individual NEs (e.g. a particular phone number) generally occur only a few times in a dataset.
Another possibility is to encode all the NEs with random representations and keep them ﬁxed throughout (Yin et al., 2015), but here we lose the meaning associated with the neural embeddings and risk their representations interfering and correlating with those of others in unexpected ways.
There is another simple way in which NEs are handled in many real world systems, which is to ﬁrst recognize the NEs with either NE taggers (Finkel et al., 2005) or entity linkers (Cucerzan, 2007; Guo et al., 2013; Yang &
Chang, 2015), and then replace them with NE-type tags.
For example, all location names could be replaced with a tag NE location and all faculty names could be replaced with a tag NE faculty .
This prevents the explosion in vocabulary size, but the system loses the ability to distinguish and reference different NEs of the same type.
If there are several location names mentioned in a dialog, the system loses its ability to distinguish and use them separately as needed in future dialog.
In addition to this, there is also the possibility of new NEs arising during the test time.
In fact many of the Out Of
Vocabulary (OOV) words that arise during test time in many Natural Language Processing (NLP) tasks are NEs.
In this paper, we propose a simple idea for neural methods to interact with NEs that handles all the aforementioned issues, including robustness to OOV NEs during test time.
The core idea is to not include any of the NEs in the vocabulary, but rather to create a distributed representation for them on the ﬂy when the agent actually encounters them and then use these representations to retrieve and use the actual NE value whenever required.
We demonstrate our idea ﬁrst on a simple structured
Question Answering (QA) task and then on three related goal-oriented dialog tasks that are extended versions of some of the dialog bAbI tasks proposed in Bordes & Weston (2016b).
We use a multipleattention based neural retrieval mechanism to retrieve items from a DB table.
The results suggest that our proposed way of handling NEs can be very useful in these tasks compared to not handling them separately.
D ETAILS OF P ROPOSED S OLUTION
Consider a neural dialog system participating in a dialog with a user.
It has a predeﬁned vocabulary set obtained from the training data by excluding all NEs.
When a user makes an utterance, the dialog system processes that utterance word by word.
A separate Named Entity Recognizer (NER) is used to classify if a given word in the user utterance is a NE or not (possibly along with its NE type).
In most of the goal-oriented dialog systems that interact with DB, these NEs come from the DB.
So, there might not be a need for a separate NER, as the NEs, along with its type can be obtained easily by referring to the DB.
If it is a NE, then it will not be a part of the vocabulary and hence will not have a neural embedding.
When the Dialog system encounters a NE while encoding a user utterance, it uses its knowledge of the previous dialog history and the knowledge of the current user utterance to generate a distributed representation (key) for the newly encountered NE (value) and stores it in a separate table called the NE-Table .
The NE-Table has key-value pairs, where the key is the representation generated by the dialog system when it encountered the NE and the value is the actual NE corresponding to the representation generated.
When the dialog system wants to refer/get back to this NE value in the future, it can do so by generating a key to match the keys in the NE-Table and then retrieve the corresponding value and use it.
For example, it can refer to a NE that it came across earlier in the dialog from the NE-Table , and use that to match over an attribute’s values in an external DB to retrieve necessary information from the DB.
This could for example, be another NE.
This NE retrieved from the DB can also be a part of the system response.
The system utterance, just like the user utterance, is again processed word by word.
In case of the presence of a
NE, a new representation is generated on the ﬂy as a function of the dialog so far and stored in the
NE-Table along with its value.
Thus, all and only the NEs that have appeared in the dialog so far will be present in the NE-Table with their generated neural keys and actual values.
Also, the generated key representations are used, in place of the NE-word representation, by the sentence encoder while encoding the sentence.
In short, the Dialog system learns to generate representations for the NEs as they come in, such that the representations have relevant and enough information which would allow it to match and retrieve them when required later in the dialog.
Figure 1 shows an instantiation of this idea in a Hierarchical Recurrent Encoder Decoder (HRED) model from Serban et al.
(2016), along with a multiple-attention based mechanism for retrieving information from a DB.
HRED model comprises of 3 different Recurrent Neural Networks (RNN) in its architecture.
The ﬁrs RNN acts as an encoder and works at the word level.
It encodes the given input sentence to a ﬁxed length vector.
The second RNN acts as the Dialog Manager and works at the level of sentences.
It takes as input the ﬁxed length sentence representation generated by the encoder RNN at each time step and generates a representation which captures the context, i.e state of dialog so far.
This representation is then passed on to the third RNN, the decoder, which also works at the word level and produces the output sentence word by word.
The multiple-attention based system shown in Figure 1 and later used in the experiments can attend to multiple columns and rows of a table.
Consider the scenario depicted in Figure 1.
At time step t , the user makes a sentence utterance who teaches EECS545?
The encoder RNN goes through the input utterance word by word.
When it encounters a NE, in this case, a course number EECS545 , it generates a representation for the NE
EECS545 as a function of the hidden state representation of the encoder RNN and the hidden state representation of the Dialog manager (at the previous time step) and stores it in the NE table, shown in the bottom right of the ﬁgure.
This generated representation is also used by the encoder RNN as the neural representation for EECS545 during its sentence encoding.
The encoder RNN then moves to the next and last word ‘?’
and ﬁnishes its sentence encoding.
The Dialog manager then uses the encoded representation from the encoder RNN, and its hidden representation at the previous time step to generate its new hidden representation for time step t .
This is passed on to the decoder
RNN which starts producing the system output utterance word by word, with the ﬁrst word being
Professor .
For the second word, in this case, the dialog system decides to query the DB to get the information, which is the name of the Instructor for that course.
In this paper, we perform tasks on datasets where the information is present in a single database table, where each row corresponds to a new entity of interest and the columns of the table correspond to the different attributes associated with it.
For example, for the restaurant reservation domain, each row of the table corresponds to a new restaurant and the columns correspond to restaurant attributes such as cuisine, location, phone, address etc
For the course selection advising domain, each to columns to course attributes, such as number
The information in the DB is represented in 2 ways.
If it is a NE, then it is represented by its exact value, that is, it is not a part of the vocabulary and has no neural embedding associated with it.
The non-NEs become a part of the vocabulary are represented by their neural embeddings.
In order to retrieve a particular cell from the table, the system needs to ﬁnd the correct column and row corresponding to it.
The Dialog system does so by generating 3 different attention key embeddings:
Attention-Column-Answer , Attention-Column-Query and Attention-Rows-Query .
The columns that the ﬁnal retrieved answers belong to, are selected by matching Attention-ColumnAnswer key embeddings with the neural embeddings of the column headings (Course Number, Instructor, Credits etc).
A separate Attention-Column-Answer key embedding is generated for every column heading.
The Attention-Column-Answer key embeddings generated by system are matched with embeddings learned for column headings, to provide attention scores for all the columns.
For the example, Who teaches EECS545? , the system would want to retrieve the name of the Instructor who teaches the course with Course Number EECS545 .
Therefore, the Instructor column heading alone will have high attention score and be selected.
Now that the column(s) are chosen, the system has to select row(s), so that it can get the cell(s) it is looking for.
Each row in the table contains the values (EECS545, Machine Learning, Scott
Mathew etc) of several attributes (Course Number, Course Name, Instructor etc).
But we want to assign attention scores to the rows based on particular attributes that are of interest to us ( Course Number in this example).
The column/attribute headings that the system has to attend to for selecting these relevant attributes are obtained by matching Attention-Column-Query key embeddings with the neural embeddings of the different column headings.
Now, a new representation for each row in the DB is generated based on the attributes selected by
Attention-Column-Query .
This new representation is used by the row retrieval process as explained below.
For the example that we are working with, each row in the DB table is represented by its course number values (EECS392, EECS545 and so on; which are exact values, since they are NEs).
The last step in the database retrieval process is to select the relevant rows using the AttentionRows-Query key embedding.
Attention-Rows-Query is split into two parts- Attention-Rows-QueryNE-Table and Attention-Rows-Query-DB .
In a general setting, Attention-Column-Query can select multiple columns to represent the rows.
For each selected column that is a NE column, a separate
NE value is retrieved from the NE-Table using separate Attention-Rows-Query-NE-Table embedding for each of them.
These NE values are used to do exact match search (along the corresponding columns) over the DB table to select the matching rows.
For the non-NE columns, their values are combined together to get a ﬁxed vector representation for each row in the DB (weighted sum of their embeddings, weighted by the corresponding column attention scores).
Attention-Rows-Query-DB is then used to match these representations for selecting rows.
The intersection of the rows selected by the exact match through the NE values obtained using Attention-Rows-Query-NE-Table s and rows selected using Attention-Rows-Query-DB , is the ﬁnal set of selected rows.
In short, the dialog system can use distributed representational matching for non-NEs, exact matching for NEs or a combination of both to decide which rows to attend to.
Depending on the number of columns and rows we match with, we can select one or more output cells, to be used in system output utterance.
For our running example, Attention-Rows-Query-NE-Table is used to match with the keys in the
NE-Table to select the row corresponding to EECS545 and the value EECS545 is returned to do an exact match over the row representations (represented by the course number values) in the DB.
This gives us the row corresponding to EECS545 and hence the cell Scott Mathew .
We could use our
NE-Table idea with potentially many types of neural retrieval mechanisms to retrieve information from the DB.
The multiple-attention based retrieval mechanism, described above, is only one such possible mechanism.
E XPERIMENTS AND R ESULTS
We evaluate our idea on two types of tasks - a simple structured Question-Answering (QA) task and goal-oriented dialog tasks, which are extended versions of some of the dialog bAbI tasks from (Bordes & Weston, 2016b).
3. S TRUCTURED Q UESTION A NSWERING FROM DB
The task here is to retrieve answers (single cell in a table) from a DB in response to structured one line questions.
We used the details of course offerings at a University to create these questionanswer pairs.
Each row in the DB table corresponds to a unique course, and the different columns correspond to the different attributes of the course.
Example structured question-answer pair:
[Equation]
This is a simple task where 400 question-answer pairs were used for training and 100 questionanswer pairs were used for testing.
The DB is a single table of 100 rows and 4 columns (Course
Number, Course Name, Department, Credits).
The experiments were performed with two models.
The ﬁrst model uses the proposed NE idea to handle the NEs ( With-NE-Table ), which are course number, course name and department in this task.
The other ( W/O-NE-Table ) model does not distinguish NEs from normal words.
In the W/O-NE-Table model, all words including the NEs that occur in the questions and those that occur in the DB are part of the vocabulary and have individual word representations.
Both models use a simple RNN to encode the question and the multiple-attention based retrieval mechanism discussed above to retrieve answers.
For the example question above, both the models would have to attend to the column Credits using Attention-Column-Answer key.
For selecting the attributes essential for row matching, both the models would have to attend to the column Course
Number using Attention-Column-Query key.
In the Model W/O-NE-Table , each row will be represented by the neural representation associated with the different course numbers, as all the different course numbers are part of the vocabulary.
The Attention-Rows-Query-DB key will then be used for matching and return the selected row.
For With-NE-Table model, the Course Number column will be identiﬁed as a NE column and hence the Attention-Rows-Query-NE-Table key matching will happen in the NE-Table .
The returned NE value, EECS545 , will be used to do an exact match with the row representations.
Note that in the model With-NE-Table , the rows of the DB table are going to be represented with exact course number values (no neural representations).
The test accuracy for the model W/O-NE-Table is 81% and 100% for the model With-NE-Table .
The train accuracy for both the models was 100%.
We found that the main difference between the train and test set for this task, is the presence of new NEs (also called OOV terms) in the test set.
This suggests that the 19% drop in performance mainly comes from the NEs encountered in the questions during test time which were not seen during training time.
For the model W/O-NE-Table , these are the NEs that are in the DB, and hence are part of the vocabulary, but have random representations which did not change during the training time.
The task was speciﬁcally constructed to be simple and with a small table to show that, even in this very simple task involving OOV-NEs, where the W/ONE-Table model achieves 100% accuracy at training time, its test accuracy is affected signiﬁcantly due to OOV-NEs.
However, this does not pose a problem for our model With-NE-Table .
The WithNE-Table model can also easily scale to large datasets with thousands of NEs without any drop in performance.
3. G OAL -O RIENTED D IALOG T ASKS
We test our idea in extended versions of three dialog bAbI tasks from Bordes & Weston (2016b) to evaluate our idea’s usefulness in different aspects of goal-oriented dialog.
Dialog bAbI dataset has 5 goal-oriented dialog tasks - Task 1: Issuing API calls, Task 2: Updating API calls, Task3:
Displaying Options, Task 4: Providing extra information and Task5: Conducting full dialogs (which is a combination of tasks 1-4).
Each of the four tasks (1-4) test different capabilities required in a general (commonly required) goal-oriented dialog system.
The system is evaluated in a retrieval setting.
At each turn of the dialog, the system has to select a candidate response from a list of possible candidates.
Issuing an api call corresponds to selecting a response which starts with api call followed by the argument values.
We evaluate our idea on extended versions of task 1, 2 and 4 (with extensions described below).
Task 3 requires the agent to learn to sort the retrieved restaurants based on their ratings and then display the options.
As shown in Bordes & Weston (2016b), it is tough for the agent to learn this sorting with the limited amount of data. Their model achieves close to 0% accuracy on full dialog for this task.
So we decided to skip task 3 and hence task 5 (which includes task 3 dialogs), as we are more interested in testing our idea to handle NEs, rather than trying to solve the sorting problem.
3.2. E XTENDED DIALOG B A B I TASK 1: I SSUING API CALLS AND RETRIEVING INFORMATION
In the original bAbI task 1, the the conversation between system and the user involves getting information necessary to make an api call and once user has provided their preferences, issue an api call with the argument values.
We extend this task as follows.
In the extended task, the system has to perform an additional task.
Once the system determines that the next utterance is an api call , the system also performs the actual query on the database to retrieve the restaurant details which match user preferences.
In the original task, the system simply responds with an utterance api call along with user preferences and the database retrieval is not considered.
The system is evaluated on all the above three aspects of the task (having the conversation with the user, issuing api call , retrieving the correct information).
The DB is represented as a single table, with each row corresponding to a unique restaurant.
The different columns of the table correspond to different information about the restaurant, such as its location, address etc.
Example:
[Table]
should
[Table]
be ? U:
[Table]
S:
Ok, let me look into some options for you.
S: api call British, London, Six,
Expensive
[Table]
let me look into some options for you.
S: api call
S (Retrieved Information): { The Place, british, london, ..., } { The Fancy Pub, british, london, ..., }
3.2. E XTENDED DIALOG B A B I TASK 2: U PDATING API CALLS AND RETRIEVING INFORMATION
In the original bAbI task 2, just like bAbI task 1, the conversation between system and the user involves getting the information necessary to make an api call and then issue an api call .
After this, the user might ask the system to update his/her request for information by changing some of the argument values.
The system has to take this into account and make an updated api call at the end of the conversation.
The extension task 2 is very similar to task 1: whenever the system decides to make an api call , it now has to actually do the work of the api call and retrieve the information.
Appendix A gives an example of the original and the extended dialog bAbI task
. 3.2. E XTENDED DIALOG B A B I TASK 4: P ROVIDING INFORMATION
The dialog bAbI task 4 starts as if the user has agreed to book a table in a particular restaurant.
The system is given all the information about only that particular restaurant.
The user can ask for the phone number, address or both.
The system must learn to use the given information about the restaurant of interest to answer these questions.
We make several changes in the extended version to make the task harder.
First we do not consider information just about the particular restaurant of interest to the user, instead the system needs to search and provide an answer from the full DB of all the restaurants.
The candidate responses are also changed.
All the NEs in the candidate responses are now replaced with their NE type tags.
For example, The Fancy Pub phone and Suvai phone are now both replaced with NE phone .
The system has to select these candidates with NE-type tags and later replace the tags with the actual NE values by retrieving them from the DB. Example [Table] [Table]
[Table] [Table] [Table] [Table]
The
[Table]
address,
[Table]
:six
[Table]
S
[Table]
Hello, what can I help you with today ?
U:
I’d like to book a table at The Place S:
Great, let me do the reservation.
U:
May I have the address of the restaurant.
S:
Here it is, The Place address.
[Table]
DB
[Table]
: Entire DB U: Hello S:
Hello, what can I help you with today ?
U:
I’d like to book a table at The Place S:
Great, let me do the reservation.
U:
May I have the address of the restaurant.
S:
Here it is, NE address.
S:
The Place address ( retrieved information)
This experiment setting is closer to how a human would do this task.
When someone asks for phone number/address of a restaurant, we don’t try to memorize it or ﬁgure out how one phone number is related to another phone number, etc., rather, we search for the phone number/ address in the DB and return the information to the user as part of the response.
3.2. M ODEL DESCRIPTION AND RESULTS FOR THE GOAL - ORIENTED DIALOG TASKS
The model that we use for the goal-oriented dialog task experiments is an instantiation of our idea (of using NE-Table ) in an end-to-end memory network (Sukhbaatar et al., 2015).
It is similar to the model used in Bordes & Weston (2016b) paper, except that we encode the sentences using an RNN, while, they use a bag of words representation.
All the previous dialog history embeddings are stored in the memory.
The new user utterance embedding is used to attend over the memory to get relevant information from the memory.
This is done multiple times and the ﬁnal embedding obtained is transformed and used to select both the candidate response, and to generate the keys for performing the retrieval operation from the DB.
For the With-NE-Table model, the NE key is generated (when a NE is encountered) and stored in the NE-Table during the process of encoding the dialog sentences using an RNN.
In all the experiments involving retrieval from DB, the ground truth attention labels were used for training.
Evaluation measures’ description
• A-C-Q : ( Attention-Column-Query ) Accuracy of attention over the columns relevant in deciding the attributes to use for representing the rows.
• A-C-A : ( Attention-Column-Answer )
Accuracy of attention over the columns from which the ﬁnal answers are to be retrieved.
• A-R-Q-DB : ( Attention-Rows-Query-DB ) Accuracy of attention over the rows in the DB, returned by matching the key representation with the representation of the rows generated by the relevant non-NE column values.
• A-R-Q-NE ( Attention-Rows-Query-NE-table ): Accuracy of attention over the NE-Table to select the correct NE values for different relevant NE columns of the DB . • Retrieval : Retrieval percentage accuracy for rows (task 1,2) and a particular cell (task 4).
This would implicitly depend on the above mentioned attention accuracy measures.
• Per-response : Percentage of dialog responses that are correct.
• Per-Dialog : Percentage of dialogs where every dialog response is correct.
• Per-Dialog + Retrieval : Percentage of dialogs where every dialog response and all the information retrievals involved are correct.
Results on extended dialog bAbI tasks 1 and 2:
The results for task 1 and task 2 are shown in Table 1.
With-NE-Table : For issuing an api call in tasks 1 and 2, four argument values are required - cuisine, location, price range and number of people.
We consider cuisine and location to be NEs.
So whenever cuisine and location names occur in the dialog, a NE key is generated on the ﬂy and is stored in the NE-Table along with the NE values.
Here we are interested in retrieving rows from the table.
So, we need to ﬁrst ﬁnd the column attributes with which we want to represent the rows ( Attention-Column-Query ) and then use that to represent the rows.
Out of the four column attributes that are of interest, two are NEs and two are not.
This splits the the attention over rows into two parts.
Attention-Rows-Query-DB attends to rows based on the non-NE attributes, which are price range and number of people.
So each row in the DB is represented by its price range and number of people values (weighted vector sum) and key matching is done to get the attention scores for all the rows in the DB.
This retrieves all the rows in the DB that match the price range and number of people values that we are looking for.
The second part of the row attention is Attention-RowsQuery-NE-Table where the system attends over the NE-Table by matching its generated key with the key present in the NE-Table to retrieve NE values.
The selected NE values are then matched (exact-match) with the cuisine and location values in the DB to retrieve the relevant rows from the DB (the rows that have that particular cuisine and location values).
The ﬁnal retrieved rows are the intersection of the rows selected by both these parts.
[Table]
W/O-NE-Table : Here, ll input words are part of the vocabulary and are not categorized into NEs and non-NEs.
Similar to the case above, Attention-Column-Query is used to attend to the four relevant columns.
However, unlike the case above, each row is represented by the combined representation of the all the four attribute values, cuisine, location, price range and number of people.
AttentionColumn-Query-DB is used to retrieve the relevant rows.
From Table
[Table] we can see that both the models perform well in selecting the relevant columns [Table] but the model W/O-NE-Table performs poorly in retrieving the rows
[Table]
while With-NE-Table performs very well.
This is because, in the With-NE-Table model, the retrieving rows task is split into two simpler tasks.
The NEs are chosen from the NE-Table , and then exact matching is used (which helps in handling OOV-NEs).
The non-NEs, price range and number of people, have limited set of possible values, low, moderate or expensive for price range and 2,4,6 or 8 for number of people respectively.
This allows the system to learn good representation for them and hence have high accuracy in Attention-Rows-Query-DB .
Whereas in W/O-NE-Table model, Attention-Rows-QueryDB involves the neural representations of cuisine and location values as well, where a particular location and cuisine value will occur only a few number of times in the dataset.
In addition to that, new cuisine and location values can occur during the test time (Test OOV dataset, performance shown in parenthesis).
For the dialog part of extended tasks 1 and 2, the system utterances do not have any NEs in them.
However, the user utterances contain NEs and so the system has to understand them in order to select the right system utterance.
The accuracy in performing the dialog (by selecting responses from candidate set) is similar for both the models on the test set.
However, in the OOV-test set, for task 1, where the system has to maintain the dialog state to track which attribute values have not been provided by the user yet, W/O-NE-Table model seems to get affected (Per-Dialog accuracy), while the With-NE-Table model is robust to that.
Results on extended dialog bAbI task 4:
The results for task 4 are shown in table 2.
As mentioned in the task description, all the NE values are replaced with their respective NE-type tags in the candidate responses.
This greatly reduces the number of possible candidate responses in the dataset (’Here it is The place phone’ and ’Here it is Suvai phone’ will now be both ’ Here it is NE phone’).
So, in order to give some information such as the phone number, the system has to select a general candidate response that gives phone number (eg.
Here it is NE phone) and then select the exact NE value to put in separately from the DB (e.g., The place phone).
This setting is similar to the system action templates proposed in Hybrid Code
Networks from (Williams et al., 2017).
In this task, the restaurant name, phone number and address are NEs.
In a general generation setting, the non-NE part of the response is generated word by word by the decoder and the NE can be retrieved and inserted into the output sentence
With-NE-Table : In task 4, the user tells the system the restaurant in which he/she wants to book a table.
The restaurant name, which is a NE, is stored in the NE-Table along with it’s generated key.
When the user asks for information about the restaurant such as, phone number, the NE restaurant name stored in the NE table is selected and used for retrieving its corresponding phone number from the DB.
For this particular case, Attention-Column-Query attends over Restaurant Name and
Attention-Column-Answer attends over the column Phone .
The NE value (here the actual restaurant name given by the user) is retrieved using Attention-Rows-Query-NE-Table over the NE-Table . The
[Table]
retrieved NE value is used to do an exact match over the DB column selected by Attention-ColumnQuery to select the rows.
The cell that intersects the selected row and the column selected by Attention-Column-Answer is returned as the retrieved information and used to replace the NE type tag in the output response.
W/O-NE-Table :
The candidate response retrieval (dialog) is same as the above model and the column attentions are also similar.
However, the models differ with respect to attention over rows.
Since NEs are not treated as special here, attention over rows happens through Attention-Rows-QueryDB .
For this task, when Attention-Column-Query is selected correctly (restaurant name), each row will be represented by the neural representation of its restaurant names.
Attention-Rows-QueryDB generates a key to match these neural representation to attend to the row corresponding to the restaurant name mentioned by the user.
From table 2, we observe that the dialog performance (retrieving candidate responses) for both models is very good and similar, but the W/O-NE-Table model fails in row retrieval while the WithNE-Table performs well.
The difﬁculty for the W/O-NE-Table model to retrieve rows comes from the need to learn neural representations for all restaurant names, that can be used later to match and retrieve the same.
Performance comparison with the original dialog bAbI tasks:
The extended dialog bAbI tasks that we present results for above require the dialog system to do strictly more work compared to the original dialog bAbI tasks.
For tasks 1 and 2, while in the original version of the task, the dialog system has to have a conversation with the user and issue api call s along with the arguments, in the extended version, the dialog system also has to retrieve the relevant information from the DB (in addition to having the conversation and issuing api call s).
For task 4, in the original task the dialog system is given just the details of the particular restaurant that the user will ask for.
The dialog system has to ﬁnd the correct attribute from it, depending on what the user asks for.
In the extended version of task 4, the dialog system is not given information just about the restaurants the user is interested in, instead it has to use the restaurant name that the user provides during the conversation to retrieve the relevant information that the user asks for, from the full DB and use it in dialog.
Table 3 compares the performance of the With-NE-Table method with that of a baseline method on the original bAbI tasks.
The baseline method here is an end-to-end memory network with RNN encoding for its sentences (similar to the architecture used for With-NE-Table model), without any
NE-Table or retrieval mechanism.
The baseline method is also given match type features which gives extra information about the NE type when a NE occurs (as done in Bordes & Weston (2016b)).
The performance that we report here for the baseline model is higher than that reported in Bordes &
Weston (2016b).
This is probably because we use RNN encoding for sentences, while the above paper uses bag of words representation for sentences.
We compare our method’s accuracy in getting every dialog response and every information retrieval (per-Dialog + Retrieval) in a dialog correct with the baseline model’s accuracy in getting every dialog response (Per-Dialog) in a dialog correct.
Though not a strictly fair comparison for our model, from
Table 3, we observe that the performance our With-NE-Table model in the extended bAbI tasks are as good as the performance of the baseline model in the original bAbI tasks.
In the case of bAbI task 2 OOV test set, the performance of the With-NE-Table model is actually much higher compared to the baseline model.
[Table]
R ELATED W ORKS
There are several papers in the Question Answering (QA) domain that focus on DB retrieval with neural methods and suffer from the Named Entity (NE) issues described in the Introduction.
Some end-to-end systems (Neelakantan et al., 2015; Yin et al., 2015) were proposed to transform a natural language question/query to a program that could run on DBs, but those approaches are only veriﬁed on small or synthetic databases.
Other papers dealing with large knowledge bases (KB) usually rely on entity linking techniques (Cucerzan, 2007;
Guo et al., 2013), which links entity mentions in texts to knowledge base queries.
For example, in knowledge base QA papers (Yih et al., 2015;
Yin et al., 2016; Yu et al., 2017), the text spans in questions are compared with KB entity names at the character-level for entity linking; then after the linked entities have their properties extracted, the corresponding text spans are replaced with special NE tags for further text processing like KB relation extraction.
Recently, Liang et al.
(2016) extended end-to-end neural methods to question answering over knowledge base, which could handle large KB and large number of entities.
However, their method still relies on entity linking (Yang & Chang, 2015) to generate a short list of entities linked from text spans in questions in advance.
Yin et al.
(2015) propose ’Neural Enquirer’, a neural network architecture similar to our neural retrieval mechanism, to execute natural language queries on DB.
However, they keep the randomly initialized embeddings of the NEs ﬁxed as a way to handle NEs and OOV words.
We could potentially use our idea of NE-Table with a Neural Enquirer to retrieve information from tables.
There has been a lot of recent interest in end-to-end training of dialog systems (Vinyals & Le (2015);
Serban et al.
; Lowe et al.
(2015); Kadlec et al.
; Shang et al.
Research on this topic tends to focus on large-scale training corpora such as movie subtitles, social media chats, or technical support logs.
For large corpora it is natural to use supervised training techniques where the Recurrent Neural Networks (RNNs) attempt to replicate the recorded human utterances.
However, there are also approaches that envision training via reinforcement learning techniques, given a suitably deﬁned reward function in the dialog (Wen et al.
; Su et al.
(2015b;a)).
In more recent work on end-to-end learning of task-oriented dialog such as Bordes & Weston (2016a); Dodge et al.
this paradigm is extended to decompose the main task into smaller tasks each of which must be learned by the agent and composed to accomplish the main task.
Williams & Zweig (2016) use an LSTM model that learns to interact with APIs on behalf of the user.
Weston (2016) (bAbIdialog) combines dialog and reasoning to explore how an agent can learn dialog when interacting with a teacher.
Guo et al.
(2017) apply reinforcement learning and supervised learning in interactive reasoning tasks.
Dhingra et al.
(2017) use reinforcement learning to build the KB look-up in task-oriented dialog systems.
But the look-up actions are deﬁned over each entity in the KB and is therefore hard to scale up.
Most of these papers actually do not discuss the issue of interacting with NEs though they are present.
In our work, we showed using simple tasks the usefulness in identifying and handling them (NEs) in the way proposed.
Recently, a new hybrid approach was also proposed by (Williams et al., 2017).
They proposed Hybrid Code Networks which combine an RNN with domain-speciﬁc knowledge encoded as software and system action templates.
They achieved state-of-the-art performance on the Facebook bAbI dataset, but their approach involves a developer writing domain-speciﬁc software components e.g. the dialog state is maintained in the code.
For certain tasks such as Machine Translation and summarization, neural copying mechanisms (Gulcehre et al.
(2016), Gu et al.
(2016)) have been proposed as means to handle OOV words and NEs.
Here, the system has the option to copy certain words from the input to the output.
We could use our NE-Table idea along with such copying mechanisms for cases like dialog generation.
C ONCLUSION AND F UTURE WORK
In this paper we proposed a method for interacting with NEs in a goal-oriented dialog setting, which typically also involves interacting with a DB.
Our experiments show that, handling NEs this way can indeed work and be useful for certain aspects of goal-oriented dialog.
In future, we are interested in using the NE-Table idea with other retrieval mechanisms such the ’Neural Enquirer’ proposed in (Yin et al., 2015) which can work with multiple tables, instead of the multiple-attention mechanism used in our work.
We are also interested in extending the multiple-attention mechanism for retrieval to a multiple table setting, and work on datasets, which have data distributed across multiple tables.
Our proposed idea is generic enough to be thought of as follows: in many NLP tasks, goal-oriented dialogs in particular, there are places where we are interested in encoding the meaning and relation of words/sentences by learning neural representations for them (for understanding a user question for example), there are also places where we are not interested in the meaning, but rather just in the exact values (phone numbers, address etc).
One reason could be that working with these exact values can make certain tasks easier and accurate (compared to having a neural embedding for them) and another practical reason could be that learning good neural representation for these particular values that occur rarely (sometimes only during test time) is difﬁcult.
The NE-Table idea provides a way to work with exact values of entities but still remain within the neural learning scheme, which allows the overall task to be end-to-end trainable using back propagation.
In future work, we are interested in exploring the use of this idea in a more general setting of working with both symbols and distributed representations, exploiting the beneﬁts of both of them wherever appropriate and doing all this within a end-to-end trainable neural setting.