A SSOCIATIVE C ONVERSATION M ODEL : G ENERATING V ISUAL I NFORMATION FROM T EXTUAL I NFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose an Associative Conversation Model that generates visual information from textual information and uses it for generating sentences in order to utilize visual information in a dialogue system without image input.
In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information is useful for generating translated sentences.
However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input.
Our approach generates (associates) visual information from input text and generates response text using context vector fusing associative visual information and sentence textual information.
A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences.
Furthermore, analysis experiment of association showed that our proposed model generates (associates) visual information effective for sentence generation.
I NTRODUCTION
In human conversation, communication is carried out on the premise that each other has common knowledge.
This is also true when humans use the dialogue system, and we conduct dialogue on the premise that the system has common sense.
Therefore, it is essential that the dialogue system has common sense.
However, it is nearly impossible for humans to organize and describe a huge amount of common sense, when the necessary time and cost are considered.
In the case of a human being, it is possible to acquire the knowledge through conversation with a person.
Hence, it is desirable for dialogue systems to acquire knowledge through conversation as well.
This has the advantage that human beings need not explicitly give knowledge to the system.
As a model that can extract knowledge from conversations, the encoder-decoder model has been proposed (Sutskever et al., 2014) (Vinyals & Le, 2015).
It consists of an encoder that encodes the input information into a context vector and a decoder that generates sentences using the context.
Vinyals & Le (2015) showed that it is possible to extract commonsense knowledge and to conduct conversation by learning pairs of dialogues with the model.
For example, Vinyals & Le (2015) reported that when asked who is skywalker, NCM responded “he is a hero. ” .
Their conversation model (NCM) has a problem that it is not possible to respond properly to the input texts that require visual information.
For example, Vinyals & Le (2015) reported that when asked how many legs a spider have, NCM responded “three, i think. ” .
Further, it may contain more detailed information statement in the image.
For example, in the news, statements and skaters that “won the ﬁgure skating game” is when it is presented the video to have a gold medal, but there is no information of the gold medal in the statement are present in the video.
I thought that it would be able generation of beneﬁcial sentence including “gold medal” if extracting such detailed information from the image.
In recent years, studies have been reported in which translated sentences are generated by adding image features to the context vector encoded by the encoder-decoder model (Calixto et al., 2017)
(Elliott & K´ad´ar, 2017)
(Nakayama & Nishida, 2017)
[Figure]
(Saha et al., 2016)
(Toyama et al., 2016).
These studies showed that visual information works effectively for generating translation.
Meanwhile, visual information is not considered in many text-based dialogue systems, because what is given to the input is only the utterance text.
How can the visual information be used without accepting visual information as the input to the dialogue system?
Based on the discussion above, we propose an Associative Conversation Model that associates the input text with the visual information and generates the response using both the text and the associated visual information.
In our proposed method, we attempted to generate response texts using visual information without inputting images.
The contribution of this research is as follows:
(cid:15)
We made it possible to generate visual information related to sentence textual information through end-to-end dialogue learning.
(cid:15)
We made it possible to generate sentences using visual information without directly inputting visual information by association.
(cid:15)
Our proposed model can generate response texts including useful information compared with a model without association by associating visual information related to input text.
This is a useful method for constructing the dialogue systems that automatically extract information from the text and the video data (e.g. TV news) to generate sentences.
A SSOCIATIVE C ONVERSATION M ODEL 2. O VERVIEW OF ARCHITECTURE
Fig.
1 shows our overall model.
The objective of this model is to enable for the decoder to generate a more appropriate response text which cannot be obtained only by textual information, by adaptively referring to the context vector based on visual information.
However, in this task, only the textual description is given as input.
Therefore, we adopted the following approach so that visual information can be utilized from the textual information.
(cid:15) First, visual association is performed from the input text, and the visual information corresponding to the text is generated.
(For example, Fig. 1 shows that the input text “Today is Skating Championships” is used to generate the visual information corresponding to a scene where a player in Skating Championships won the gold medal.)
(cid:15)
Next, by fusing the textual information and the associated visual information is obtained the information reﬂecting either or both as necessary.
[Figure] [Figure]
(cid:15) Finally, a response text is generated based on the fused information.
(Fig.
1 shows that the fused information was used to decode the ﬁnal response “Japanese female skater won the gold medal”.)
Our approach is simple.
When generating sentences from the input texts and videos, an encoder for text and another for video can be used to acquire context vectors for text and for video, respectively.
In our task, however, the videos cannot be directly obtained from the input because only the textual information is given as input.
Therefore, our idea is to replace the encoder for video with a mechanism for generating the context vectors of the video from texts.
The conﬁguration of this associative conversation model is simple, and is composed of the encoder for text and the decoder equipped with attention mechanism, as well as the visual association encoder from texts.
Associative Encoder is an
RNN which generates visual context vectors from textual context vectors (see Sec 2.2.2).
We used
LSTMs for encoder, decoder, and the associative encoder, respectively (Hochreiter & Schmidhuber, 1997).
The input to the associative conversation model is textual descriptions only, and the output is a response text to the input.
This model generates the response text by inputting to the decoder the synthesized context vector obtained by fusing the textual context vector and the visual context vector generated from the input text.
The associative conversation model is able to learn the pairs of dialogue texts by end-to-end, like a normal encoder-decoder model.
Actually, however, two prior learnings are performed before training the associative conversation model.
One is the training of multimodal encoder-decoder model which generates a response text from the video and text.
The other is the training of the associative encoder which predicts the visual context vector from the textual context vector obtained by the multimodal encoder-decoder model.
2. L EARNING M ETHOD
Learning consists of the following three steps.
Step1 and Step2 are the prior learning models for learning an Associative Conversation Models in Step3.
Step 1 : Extraction of context vectors between textual and visual information
Fig.
2a is the network of models used in this step.
In this step, use a multimodal encoderdecoder model consisting of a textual encoder, a visual encoder, and a sentence decoder including Attention (Bahdanau et al., 2014).
The model of Step 1 encodes a video and a sentence with each encoders, and fuse the context vector of the sentence and the video to predict response texts.
Then, using this learned model, correspondence between textual information and visual information is extracted (Fig. 2a).
Step 2 : Learning association
In this step, use an RNN.
The model of Step 2 associates visual information from textual information.
In other words, the model learns predicting video context vector (associative representation) from the sentence context vector.
Step 3 : Generating responses using association
Fig.
2b is the network of models used in this step.
In this step, we use a multimodal encoder-decoder model using associative encoder instead of visual encoder.
In other words, the network architecture in Step3 is a decoder of a sentence including a textual encoder, an associative encoder, and Attention.
This is Associative Conversation Model.
Association is performed using an associative encoder from textual information of input text to generate associative representation.
Then, it combines textual information and associative representation, and generates response text (Fig. 2b).
2.2. S TEP 1 : E XTRACTION OF CONTEXT VECTORS THE CORRESPONDENCE BETWEEN THE TEXTUAL AND VISUAL INFORMATION
The training data used in this step are the utterance text, the video corresponding to the utterance text, and the response text for the utterance.
We used videos rather than images because the texts often include the expression of actions.
We created the dataset based on TV programs as they contain the utterance texts, their corresponding videos, and their response texts.
First, the closed caption texts were extracted from TV news programs as the utterance texts X txt .
Also, the scenes corresponding the temporal intervals where the utterances occurred were extracted as the sequence of images X vis from the TV news programs.
For more information on the dataset, see Sec ?? .
Here, the utterance
X txt is represented by the sequence of the corresponding word x txt .
Similarly, the video X vis corresponding to the utterance text is represented by the sequence of image features x vis .
In this work, the image features acquired by the already learned CNN were used instead of learning the image features directly from the input video.
Therefore, the input to the prior learning models are the sequence of word vectors X txt =
( x txt 1 ; :::::; x txtL ) and the sequence of image feature vectors
X vis = ( x vis 1 ; :::::; x visF ) .
Also, the output is the sequence of words vectors Y =
( y 1 ; :::::; y T ) .
Here,
L , F , and T represent the length of each input text, the number of the input images, and the length of the output text, respectively.
The model used in step 1 is the multimodal encoder-decoder model consisting of a textual encoder, a visual encoder, and a decoder with attention (Bahdanau et al., 2014) .
In step 1, the textual and visual encoders encode the input text and video, respectively, to obtain the context vectors C txtt and C vist .
Then, C t which is the fused context vector is input to the decoder to generate a response text.
For extracting context vector C t , the attention mechanism was used.
[Equation]
Attention mechanism can extract a context vector that strongly reﬂects some parts of the sequence corresponding to the words of the response text.
For example, when the word “spider” of the text and the image of the spider were paid attention to, the textual and visual context vectors of the spider are generated, respectively.
In this case, there is a correspondence relations between the textual and visual context vectors.
So, in step 2, the associative encoder which predicts the visual context vectors from the textual context vectors is trained.
The purpose of step 1 is to extract the textual and visual context vectors having correspondence relations to be used for learning in step 2.
The usual attention-based encoder-decoder model calculates the context vectors by Eq. , , .
h i is the intermediate vector of the bidirectional LSTM computed from the input sequence h
i
=
[ (cid:0)!
h i ; (cid:0) h i ] .
[Equation]
Meanwhile, the multimodal encoder-decoder model acquires two context vectors C txtt and C vist .
Thus, the multimodal encoder-decoder model uses the following equations instead of Eq.(3) (4) and (5).
[Equation] [Equation]
Also, there are some responses that do not always require visual information in the dialogue.
Therefore, the decoder needs to be able to “measure” which of the textual and visual information should be paid attention to and how strongly it should be focused.
The degree to which the textual and visual context vectors C txtt and C vist should be referred to can be learnt by introducing the weight matrices and by multiplying them to C txtt and C vist , respectively.
Therefore, the context vector eventually passed to the decoder is the fused context vector having how strongly the textual and visual information should be referred to.
[Equation]
This fused context vector C t is used to predict the next word in the decoder.
We call this layer the fusion layer of the context vectors.
After training the multi-modal encoder-decoder model, the training data is again input to the model to save the textual and visual context vectors.
2.2. S TEP 2 : L EARNING FOR VISUAL ASSOCIATION
In step 2, the visual associative encoder is trained.
The associative encoder is trained so that it can predict the visual context vector from the textual context vector.
In addition, the associative encoder decides the next visual context vector by referring to the past textual context vectors.
Let us consider the following example.
The visual information (visual context vector) associated with the word “squeeze” is diverse.
For example, the phrases “squeeze a lemon”, “squeeze a person’s hand”, and “squeeze toothpaste out” would make us imagine the different visual situations representing “squeeze”.
In this example, the phrases “lemon”, “person’s hand”, and “toothpaste” make the situation of “squeeze” more concrete.
In this work, the association is to generate the corresponding visual information from the textual information.
Namely, there is a demand that the model can imagine what to “squeeze”.
Here, suppose that a context vector “squeeze” is obtained by the attention mechanism.
If the context vector “lemon” was obtained before this point, it will be possible to generate the visual information of “squeeze a lemon” from the context vectors of “lemon” and “squeeze”.
This indicates the utilization of temporal features.
Therefore, RNN is used for the associative encoder.
Accordingly, the associative encoder can be seen as a regression model with the input sequence of context vectors of words C txt =
( C txt 1 ; :::; C txtt ; :::; C txtT ) and the output sequence of context vectors of images C vis = ( C vis 1 ; :::::; C vist ; :::::; C visT ) (Eq. 13).
Here, T is the length of the output text.
We used LSTM which is capable of learning long-term dependence.
[Equation] [Equation]
2.2. S TEP 3 : G ENERATING RESPONSES WITH VISUAL ASSOCIATION
In step 3, the associative conversation model learns response texts using the associative visual information instead of visual information directly obtained from video(Fig. 2b).
The input to the associative conversation model is the sequence of words X txt =
( x txt 1 ; :::; x txtL ) .
The output is also the sequence of words Y =
( y 1 ; :::::; y T ) .
In the associative conversation model, the visual encoder obtained in the prior learning of step1 is replaced with the associative encoder.
Therefore, the architecture of the associative conversation model can be obtained by replacing Eq.(7) in step1 with Eq.(13).
In other words, the mechanism of blending the textual and visual information trained in step1 is left unchanged, and the visual context vector is predicted from the textual context vector by the associative encoder.
The visual context vectors as well as the textual context vectors are given to the decoder, and the decoder, the attention, and the fusion layer (Eq. 12) are trained.
The weights of the decoder, the attention and the fusion layer are initialized, and the weights obtained by prior learning of step1 is not used.
The weights in the textual and associative encoders are not updated, and the same parameters used in the prior learning of step 1 are given.
The reason for re-training the decoder, the attention and the fusion layer is because the associative visual representation has different property from the visual context vector generated in step1.
The associative visual representation is different from the visual context vector obtained in step1.
We thought that re-training parts other than encoding of semantic representation enables to generate response texts using the associative visual elements in common knowledge.
D ATASET
The model was learned using the following data collected independently: subtitles in TV news and the corresponding video where the subtitles were displayed.
The visual encoder of the model of step1 encodes a sequence of images.
The sequence of images are a set of images cut out from a moving picture at ﬁxed intervals.
Note that the subtitles were delimited by “ ! ” , “ ? ”, or Japanese period to acquire sentences, and a pair of one sentence and the following sentence was regarded as one dialogue.
Therefore, the response sentence of a certain utterance becomes the next input utterance.
Also, the video was cut out as a frame sequence with a frame rate of 5 fps.
Each frame was input to the prior learnined convolutional neural network, and the output of the last pooling layer was used as the image features.
VGG16 was used for the convolutional neural network (Simonyan & Zisserman, 2014).
The recorded programs were 163 Japanese TV news broadcast in from December, 2016 to
March, 2017, having 38K dialogues and 19K vocabulary words.
Data was divided into 34K dialogue for training data and 4K dialogue for test data.
E XPERIMENTS
In this section, we describe the result of comparison between an Associative Conversation Model and a model without association.
We used a encoder-decoder model with attention that caused the pair of dialogue sentences for the baseline model.
The baseline model generate a sentence Y as input sentence X txt .
Both the proposed model and the baseline model was trained with the same dialogue sentences, and we investigated the effect of association.
In addition, we visualized the associated objects, and investigated whether associative conversation model acquired effective visual information for response generation.
As a result of the experiments, we found that the association works effectively to generate sentences with useful information.
We also found that our proposed model associates visual information related to input texts.
4. M ODEL C ONFIGURATION
In the step1, we used single layer LSTMs as both the encoders and the decoder.
The hidden layer dimension of the textual encoder was set to 512, the hidden layer dimension of the visual encoder to 512, the dimension of the fused context vector to 1024, and the hidden layer dimension of the decoder to 1024, the dimensionality word embeddings was set to 512, the dimensionality image feature was set to 512, the batch size was set to 64.
Adagrad was used for optimization.
In the step2, the associative encoder is composed of 4-layer LSTM with the hidden layers dimension being 1024, the batch size was 64.
Adam was used for optimization.
In the step3, the model is composed of a textual encoder, decoder, and a associative encoder instead of visual encoder.
The hidden layer dimension of the textual encoder was set to 512, the dimension of the fused context vector to 1024, and the hidden layer dimension of the decoder to 1024, the dimensionality word embeddings was set to 512, the batch size was set to 64.
Adagrad was used for optimization
Also, in the step 3, weights and biases of the decoder and attention were used the weights in step and not updated.
The weights of associative encoder were not updated either.
The weights of the decoder are initialized in the step3 (See Eq.(1), (2), (10), (11), and (12)) .
The baseline model is composed of single layer LSTMs as the encoder and the decoder.
The hidden layer dimension of the textual encoder was set to 512, the dimension of the fused context vector to
Associated image
Input
On the 14th and 15th is the University
Entrance examination exam.
Well, today is All Japan Figure Skating
Championships.
Output (ACM)
Snow and wind are expected to spread.
A player who has won the gold medal three women's singles.
Output (Baseline)
There will be a large-scale ﬁre that is also in western Japan and eastern Japan.
Aiming for four consecutive championships in the women's singles, athletes of the
Japanese championships participated in the tournament.
Generated
[Figure]
1024, and the hidden layer dimension of the decoder to 1024, the dimensionality word embeddings was set to 512, the batch size was set to 64.
Adagrad was used for optimization.
4. R ESULTS
Fig.
3 shows images similar to the visual information associated with our model and the sentences generated by our model and the baseline with respect to the test data.
For example, for the input text “The vicinity of Honshu is expected to be covered widely by mobile high pressure.”
, our proposed model generated speciﬁc weather words (“sunny”) .
On the other hand, the sentence generated by baselines do not contain information on speciﬁc weather forecasts.
In addition to this example, we also found examples where our proposed model with association produces useful sentences with more speciﬁc information than the baseline model.
From these results, we speculated that association works effectively to generate sentences with concrete information.
4. A NALYSIS OF ASSOCIATION
Fig.
3 is the analysis result of association.
The above image is an actual image with the highest degree of similarity to the associative representation generated by association of the proposed method from the input text.
We input the video of the learning data (sequence of images) to the model of step1 and we measured the meaning expression and the associative representation of the animation generated by the visual encoder with cosine similarity.
Although it is indirect, the context vector of the movie with the highest similarity among them and the corresponding input image were made to be associative visual information.
In other words, the image in Fig.
3 is visualized visual information associated with the sentence using the image in the learning data.
From the result of Fig.
3, you can see that the sentence and associated visual information are associated.
For example, in the example on the left of Fig. 3, the topic of “Japan’s wide range is covered with high pressure” is reminiscent of “Japan covered with high pressure” It generates the word “sunny”.
Actually, if the high pressure is covered widely, the weather will be ﬁne, so the output sentence of the proposed method contains useful information.
Also, although an example on the right of Fig.
3 misunderstood ﬁgure skating as speed skating, the skating player has won the gold “skating championship” We associate the scene and generate the word “gold medal” from the associative expression.
There is also a problem.
In this research, there was a case that the input text was misunderstood and associated with visual information.
For example, in the example to the right of Fig. 3, it is reminiscent of a scene where skate player is acquiring gold against topics about ﬁgure skating.
This will be improved by increasing the learning data.
Also, because I did learning in the news sentence,
I could not respond well to spoken conversation.
As for the solution to this, it is conceivable to learn with a general dialogue sentence instead of a news sentence when relearning the model using association in step3.
That is, using the knowledge extracted by encoder in step1 and 2, in step3, relearning is performed using a general dialogue sentence.
If knowledge can be extracted, learning can be performed more efﬁciently by performing Step3 only in accordance with the task.
R ELATED W ORK
Mostafazadeh et al.
(2017) assumed a conversation on social media about a shared image and proposed a model that performs natural conversation from images and sentences.
We assume a case where we do not use images when we talk.
In the sentence generation algorithm, there is an attempt to obtain a useful output sentence by simultaneously giving visual information and textual information (Calixto et al., 2017)
(Elliott & K´ad´ar, 2017)
(Nakayama & Nishida, 2017)
(Saha et al., 2016)
(Toyama et al., 2016).
Approaches using visual information only during training are similar to our approach, with the approach of Toyama et al.
and Elliott & K´ad´ar (2017).
Toyama et al.
(2016) proposed a model for modeling latent variables from text and images and generating sentences using the latent variables.
In addition to encoderdecoder with normal attention, their model has a mechanism to generate latent variables.
When a sentence is input to encoder and predict translation with decoder, target word is generated by using latent variable generated from source sentence / target sentence / image as hidden layer of decoder.
Next, we incorporate the latent variable into the decoder and ﬁnally output the target.
Elliott & K´ad´ar (2017) shows that learning context vector using a multitasking learning framework with two types of output, sentence generation and image feature generation, is used to generate decoder’s translation sentence, thereby improving accuracy It was.
The structure of their model consists of textual encoder, decoder of sentence, decoder of image, input text and predict translation by decoder of sentence.
At the same time, hidden layers of encoder are averaged and input to the decoder of the image to predict image features.
There are three major differences between our approach and their approach.
First, while their approach gives images as visual information during training, our approach extends it and gives videos.
With this extension we believe that it will be possible for models to learn the correspondence between words representing motion.
Secondly, we deal with dialogue tasks, not translation tasks.
Thirdly, our task is based on the assumption that our model extracts knowledge from data not necessarily corresponding to the visual information at that time such as general conversation.
For this reason, our approach is equipped with a noise removal mechanism that learns the degree of usage of visual information and textual information Eq.
The other difference between (Elliott & K´ad´ar, 2017) is that we do not adopt multitasking learning in our method and therefore input visual information.
Therefore, since it is possible to attention to visual information during training, visual information can be extracted according to the response word.
C ONCLUSIONS
In a study applying a sentence generation algorithm of translation sentence to a conversation model, there was a problem that it was not possible to respond well to an input text which requires visual information.
However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input.
Based on the discussion above, we propose an Associative Conversation Model that associates the input text with the visual information and generates the response using both the text and the associated visual information.
Comparative experiments with models that do not use association show that association of visual information related to input texts produces response texts that contain valuable information compared to models without association.
Analysis of association also showed that our proposed method can generate visual information related to sentence textual information through end-to-end learning of dialogue.