A SSOCIATIVE CONVERSATION MODEL : GENERATING VISUAL INFORMATION FROM LANGUAGE INFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose an Associative Conversation Model that generates visual information from language information and uses it for generating sentences in order to utilize visual information in a dialog system without image input.
In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information is useful for generating translated sentences.
However, it is not possible to use sentence generation algorithms using images for the conversation system since many dialog systems only accept text input.
Our approach generates (associates) visual information from input text and generates response text using context vector fusing associative visual information and sentence language information.
A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences.
Furthermore, analysis experiment of association showed that our proposed model generates (associates) visual information effective for sentence generation.
I NTRODUCTION
In human conversation, communication is carried out on the premise that each other has common knowledge.
This is also true when humans use the dialogue system, and we conduct dialogue on the premise that the system has common sense.
Therefore, it is essential that the dialogue system has common sense.
However, it is nearly impossible for humans to organize and describe a huge amount of common sense, when the necessary time and cost are considered.
In the case of a human being, it is possible to acquire the knowledge through conversation with a person.
Hence, it is desirable for dialogue systems to acquire knowledge through conversation as well.
This has the advantage that human beings need not explicitly give knowledge to the system.
As a model that can extract knowledge from conversations, the encoder-decoder model has been proposed (Sutskever et al., 2014) (Vinyals & Le, 2015).
It consists of an encoder that encodes the input information into a context vector and a decoder that generates sentences using the context.
Vinyals & Le (2015) showed that it is possible to extract commonsense knowledge and to conduct conversation by learning pairs of dialogues with the model.
Their conversation model (NCM) has a problem that it is not possible to respond properly to the input texts that require visual information.
For example, Vinyals & Le (2015) reported that when asked how many legs a spider have, NCM responded “three, i think. ” .
In recent years, studies have been reported in which translated sentences are generated by adding image features to the context vector encoded by the encoder-decoder model (Toyama et al., 2016)
(Calixto et al., 2017) (Elliott & K´ad´ar, 2017)
(Nakayama & Nishida, 2017)
(Saha et al., 2016).
These studies showed that visual information works effectively for generating translation.
Meanwhile, visual information is not considered in many text-based dialogue systems, because what is given to the input is only the utterance text.
How can the visual information be used without accepting visual information as the input to the dialogue system?
Today, there was the Skating Championships.
[Figure]
Fusion
Japanese female skater won the gold medal.
[Figure]
Based on the discussion above, we propose an Associative Conversation Model that associates the input text with the visual information and generates the response using both the text and the associated visual information.
In our proposed method, we attempted to generate response texts using visual information without inputting images.
The contribution of this research is as follows:
(cid:15)
We made it possible to generate visual information related to sentence language information through end-to-end learning.
(cid:15)
We made it possible to generate sentences using visual information without directly inputting visual information by association.
(cid:15)
Our proposed model can generate response texts including useful information compared with a model without association by associating visual information related to input text.
A SSOCIATIVE CONVERSATION MODEL 2. O VERVIEW OF A RCHITECTURE
Figure 1 shows our overall model.
The objective of this model is to enable for the decoder to generate a more appropriate response text which cannot be obtained only by textual information, by adaptively referring to the semantic representation based on visual information.
However, in this task, only the comment text is given as input.
Therefore, we adopted the following approach so that visual information can be utilized from the textual information.
(cid:15) First, visual association is performed from the input text, and the visual information corresponding to the text is generated.
(cid:15)
Next, by fusing the textual information and the associated visual information is obtained the information reﬂecting either or both as necessary.
(cid:15)
Finally, a response text is generated based on the fused information.
Therefore, we introduced the encoder that uses language information to generate a visual context vector .
The structure of Associative Conversation Model consists of Associative encoder in addition to encoder and decoder with Attention Mechanism.
We used LSTM for encoder, decoder, and Associative encoder (Hochreiter & Schmidhuber, 1997).
Input to the Associative Conversation
Model is only sentences, and output is response texts to input texts.
Fusing the context vector of the visual information associated with the input text and the context vector of the input text, and inputting it into the decoder, the response text is generated.
We call the process of generating the
[Figure]
corresponding visual information from the meaning of the sentence associative, and call the generated visual information as associative representation.
The reason to call associative representation rather than context vector is to distinguish it from visual information obtained from actual images by preliminary learning (see Sec. 2.2).
It is possible to respond using visual information without associating an image etc. by association.
2. L EARNING METHOD
Learning consists of the following three steps.
Step1 and Step2 are the pre-train models for learning an Associative Conversation Models in Step3.
Step 1 : Extraction of context vectors between language information and visual information
Fig. 2a is the network of models used in this step.
In this step, use a multimodal encoderdecoder model consisting of a sentence encoder, a video encoder, and a sentence decoder including Attention (Bahdanau et al., 2014).
The model of Step 1 encodes a video and a sentence with each encoders, and fuse the context vector of the sentence and the video to predict response texts.
Then, using this learned model, correspondence between language information and visual information is extracted (Fig. 2a).
Step 2 : Learning association
In this step, use an RNN.
The model of Step 2 associates visual information from language information.
In other words, the model learns predicting video context vector (associative representation) from the sentence context vector.
Step 3 :
Generating responses using association
Fig. 2b is the network of models used in this step.
In this step, we use a multimodal encoder-decoder model using associative encoder instead of video encoder.
In other words, the network architecture in Step3 is a decoder of a sentence including a sentence encoder, an Associative encoder, and Attention.
This is Associative Conversation Model.
Association is performed using an Associative
encoder from language information of input text to generate associative representation Then, it combines language information
and associa
tive representation, and generates response text (Fig. 2b).
There are two pre-train steps before the learning Associative Conversation Model.
One is the Learning Associative encoder.
The Learning Associative encoder predicts the context vector of an image from the sentence context vector by using the context vector of the sentence extracted in advance and the video context vector.
The other is the extraction of context vectors of sentence and video.
We used a multimodal encoder-decoder model which performs end-to-end learning with two encoders and one decoder.
The input data is a sentence and a video (a sequence of images).
Enter a sentence in the sentence encoder and enter the image sequence in the video encoder.
Then encode the sentence by Attention Mechanism and obtain the context vector of the sentence (Bahdanau et al., 2014).
In the same way, we encode the sequence of images and obtain the video context vector.
Then, we fuse these context vectors, input them into the decoder, and generate sentences.
After learning this, the model extracts the context vector of the sentence and the video context vector and uses it for the learning Associative encoder.
2.2. S TEP 1 : E XTRACTION OF CONTEXT VECTORS BETWEEN LANGUAGE INFORMATION AND VISUAL INFORMATION
We are challenging the model to extract knowledge from human utterances and videos at that time.
The important point here is that human utterances and videos at that time may not correspond to each other.
For example, when we are talking about a spider, we talk in situations where a spider are not in front of us.
Therefore, we did not use data with correspondence between visual information and language information in advance (e.g., MS COCO (Lin et al., 2014)).
Instead, we used the following data which may NOT correspond to language information and visual information.
(cid:15) Subtitle and the video of the scene where the subtitle was displayed Language information has expressions representing actions.
Therefore we used video rather than image.
For details on data sets, see Sec. 3 .
The inputs to the model in Step1 are a video (sequence of images) and a sentence.
The output is a response text to the input text.
In addition, we input image features, instead of using images directly.
We use features obtained by pre-trained CNN (without fully connected layers).
Therefore, the input to this preliminary learning model is a series of words X lang =
( x lang; :::; x langT l ) and the image feature sequence X vis = ( x vis 1 ; :::::; x visT v ) .
In addition, the output is the sequence of words
Y =
( y 1 ; :::::; y T ) .
Here, T l , T v and T represent the length of the input text, the number of input images and the length of the output sentence, respectively.
The network architecture of the model of Step1 is a multimodal encoder-decoder model consisting of a sentence encoder, a video encoder, and a decoder with Attention.
In Step 1, the sentence encoder encodes the input text, the video encoder encodes the input video, and each encoders obtain context vectors C langt and C vist .
Then, C t which fused context vector is input to the decoder to generate a response word.
[Equation]
For extracting context, we use Attention mechanism (Fig. 2a)
(Bahdanau et al., 2014).
In normal seq2seq (Sutskever et al., 2014), the context vector is compressed to a ﬁxed length vector, but the attention mechanism solves it.
Furthermore, in the Attention mechanism, it is possible to extract context vectors strongly reﬂecting the information of a part of the sequence corresponding to the word of the response text.
An Attention Mechanism is a point where you can associate a certain scene in a video with words of an input text.
It is suitable for learning the Associative Conversation
Model in two points that can record a pair of sentence context vector and video context vector as the training data for Associative encoder.
The Attention mechanism receives the input series
X = ( x 1 ; :::::; x T ) and calculates the context vector C attt .
h i is the intermediate vector of bidirectional
LSTM computed by receiving the input sequence h i =
[ (cid:0)!
h i ; (cid:0) h i ] .
[Equation]
On the other hand, the multimodal encoder-decoder model acquires two context vectors C langt and
C vis t .
Therefore, the multimodal encoder-decoder model uses the following equations instead of
Eq.(3) (4) and (5).
[Equation]
There are also responses that do not require visual information in the conversation.
Therefore, the decoder side needs to have a “scale” on how strongly to refer to sentence information (sentence context) or visual information (video context vector).
By multiplying by the weight matrix it is possible to learn the reference condition of the context vector C vist and the context vector C langt (Eq. 12).
Therefore, the context vector eventually passed to decoder is a vector having a reference condition of sentence context vector (language information) and video context vector (visual information).
[Equation]
2.2. S TEP 2 : L EARNING ASSOCIATION
In Step2, the Associative encoder learns correspondence between visual information and language information.
We thought that learning can extract the most probable visual information for the context vector of sentences.
In other words, Associative encoder expects to acquire common sense association of language and vision (e.g., “apple” word and apple visual information).
In addition, the Associative encoder decides the next video context vector by referring to past context vectors in the sentence.
We present and explain examples.
The visual information (video context vector) that corresponds to the context vector of the word “squeeze” is diverse.
For example, the phrases “squeeze a lemon”, “squeeze a person’s hand”, “squeeze toothpaste out” would be considered.
In this example, there are phrases that explain concretely “squeeze” in the sentence.
For example, “lemon”, “person’s hand”, and “toothpaste”.
In this paper, the Association is to generate corresponding visual information from language information.
Therefore, there is a demand to associate as to what is.
Here, it is assumed that a context vector “squeeze” is obtained by the Attention Mechanism.
If the context vector “lemon” was obtained before this, it would be possible to generate visual information of “squeeze a lemon” from the context vector “lemon” and “squeeze”.
This is a temporal feature.
Therefore, RNN is used for the Associative encoder.
Therefore, the Associative encoder is a regression model with the input sequence of context vectors of word C lang =
( C lang; :::; C langt ; :::; C langT ) and output sequence of context vectors of image
C vis =
( C vis 1 ; :::::; C vist ; :::::; C visT ) (Eq. 13).
Here, T is the length of the output sentence.
We used
LSTM which is capable of learning long-term dependence.
[Equation]
2.2. S TEP 3 : G ENERATING RESPONSES USING ASSOCIATION
In Step 3, the Associative Conversation Model learns response text using association instead of video(Fig. 2b).
The input to the Associative Conversation Model is the sequence of words (sentence)
X lang =
( x lang; :::; x langT l ) .
The output is the sequence of words (sentence)
Y =
( y 1 ; :::::; y T ) .
In the Associative Conversation Model, the video encoder of the preliminary learning model of Step is changed to an Associative encoder.
Therefore, the architecture of the Associative Conversation
Model is Eq.(7) in Step1 replaced by Eq.(13).
In other words, we do not change the mechanism of blending the language and visual information that was done in Step1, predicting the video context vector from the context vector of the input text by the Associative encoder.
We input associative representation generated by the Associative encoder and sentence context vector into decoder, learn the decoder, the Attention, and a fully connected layer (Eq. 12).
The weights of decoder, Attention and fully connected layer are initialized, and the weight of the preliminary learning model of Step is not used.
In the sentence encoder and Associative encoder, weight update is not performed and parameters of the preliminary learning model of Step1 are used.
The reason for re-training the decoder, the Attention and a fully connected layer is because the associative representation has a different property from the context vector of the animation generated in Step1.
The associative representation is different from the video context vector generated in Step1.
The Associative encoder associates with reference to context vectors of past sentences.
Therefore, decoder use representations not obtained in Step1.
Furthermore, in order to associate with reference to the context vector of past sentences, we obtained the representations not obtained in Step1.
We thought that by re-training parts other than encoding of context, it is possible to generate response texts using associative visual common sense.
D ATASET
The model was learned using the following data collected independently: subtitles in TV news and the corresponding video where the subtitles were displayed.
The video encoder of the model of
Step1 encodes a sequence of images.
The sequence of images are a set of images cut out from a moving picture at ﬁxed intervals.
Note that the subtitles were delimited by Japanese period , “ ! ” or “ ? ” to acquire sentences, and a pair of one sentence and the following sentence was regarded as one dialogue.
Therefore, the response sentence of a certain utterance becomes the next input utterance.
Also, the video was cut out as a frame sequence with a frame rate of 5 fps.
Each frame was input to the pre-trained convolutional neural network, and the output of the last convolutional layer was used as the image features.
VGG16 was used for the convolutional neural network (Simonyan & Zisserman, 2014).
The recorded programs were 163 Japanese TV news broadcast in from December, 2016 to March, 2017, having 38K dialogues and 19K vocabulary words.
Data was divided into 34K dialogue for training data and 4K dialogue for test data.
E XPERIMENTS
In this section, we describe the result of comparison between associative conversation model and model without association.
We use a encoder-decoder model with attention that caused the pair of conversation sentences to be learned in the baseline.
The input and output of the baseline model are only sentences.
We learned both the proposed model and the baseline model with the same data and investigated the effect of association.
In addition, we visualized associative visual information and investigated whether associative conversation model acquired effective visual information for sentence generation.
As a result of the experiment, we found that the association works effectively to generate sentences with concrete information.
We also found that our proposed model associates visual information related to input texts.
4. M ODEL C ONFIGURATION
In the conﬁguration of the multimodal encoder-decoder model in Step 1, single layer of LSTM was used for two encoders and decoders.
We set the hidden layer dimension number of encoder todimensions, the dimension of fused context vector to 1024 dimensions, and the number of dimension of decoder hidden layer to 1024 dimensions.
The associative encoder in Step2 used 4 layers LSTM.
The hidden layer dimension number was 1024 in all 4 layers, the optimization method was adam, the batch size was 64.
In the associative conversation model in Step3, we set the hidden layer dimension number of Encode of the sentence to 512 dimension, the dimension of synthetic meaning expression to 1024 dimension, and the number of dimension of hidden layer of decoder to 1024 dimension.
Also, in Step 3, weights of the encoder of the sentence and biases are not updated by using the weights in Step1.
Likewise, associative encoder does not update weights.
In addition, weights and biases of Decoding, Attention
Associated image
Input
The vicinity of Honshu is expected to be covered widely by mobile high pressure.
Well, today is the ﬁgure skating All Japan
Championship.
Output (ACM)
Tomorrow morning, it will be sunny in many places from western Japan to eastern Japan, and western Japan side of the Sea side.
A player who has won the gold medal three women's singles.
Output (Baseline)
Then, the weather in various places of tomorrow.
Aiming for four consecutive championships in the women's singles, athletes of the
Japanese championships participated in the tournament.
Generated word
Sunny
Gold medal addition of and Combination Semantic Expression are not weighted in Step 1 and weights are initialized and learning is performed (See Eq.(1), (2), (10), (11), and (12)) .
The baseline model consists of one encoder and one decoder with Attention.
Baseline model does not include associative encoder nor video encoder.
Both encoder and decoder used one layer of
LSTM.
We set encoder’s hidden layer dimension number to 512 dimension and decoder hidden layer dimension number to 1024 dimension.
In Step1 of the baseline and Step 1 and Step 3 of the proposed method, adagrad was used as the optimization method.
The batch size was set to 64.
In addition, I made a 512-dimensional embedding expression of a word and entered it into sentence encoder.
In Step1 of the proposed method, an image is input to the VGG 16, and the output of the last convolved layer is made a one-dimensional vector.
This vector is 25088 dimensions.
This dimension is compressed to 512 dimensions and entered into the video encoder.
4. R ESULTS
Fig. (3) shows the proposed method and the sentence generated by the baseline. This result shows that our proposed model generates more useful sentences with more speciﬁc information than the baseline model.
For example, for the input text “The vicinity of Honshu is expected to be covered widely by mobile high pressure.”
, our proposed model generated speciﬁc weather words (“sunny”) .
On the other hand, the sentence generated by baselines do not contain information on speciﬁc weather forecasts.
In addition to this example, We also found examples where our proposed model with association produces useful sentences with more speciﬁc information than the baseline model.
From these results, we speculated that association works effectively to generate sentences with concrete information.
4. A NALYSIS OF ASSOCIATION
Fig.(3) is the analysis result of association.
The above image is an actual image with the highest degree of similarity to the associative representation generated by association of the proposed method from the input text.
We input the video of the learning data (sequence of images) to the model of Step1 and we measured the meaning expression and the associative representation of the animation generated by the video encoder with cosine similarity.
Although it is indirect, the semantic representation of the movie with the highest similarity among them and the corresponding input image were made to be associative visual information.
In other words, the image in Fig.(3) is visualized visual information associated with the sentence using the image in the learning data.
From the result of Fig.(3) , you can see that the sentence and associated visual information are associated.
For example, in the example on the left of Fig.(3) , the topic of “Japan’s wide range is covered with high pressure” is reminiscent of “Japan covered with high pressure” It generates the word “sunny”.
Actually, if the high pressure is covered widely, the weather will be ﬁne, so the output sentence of the proposed method contains useful information.
Also, although the example on the right of Fig.(3) misunderstood ﬁgure skating as speed skating, the skating player has won the gold medal for the topic of ”skate championship” We associate the scene and generate the word “gold medal” from the associative expression.
There is also a problem.
In this research, there was a case that the input text was misunderstood and associated with visual information.
For example, in the example to the right of Fig.(3) , it is reminiscent of a scene where skate player is acquiring gold against topics about ﬁgure skating.
This will be improved by increasing the learning data.
Also, because I did learning in the news sentence,
I could not respond well to spoken conversation.
As for the solution to this, it is conceivable to learn with a general dialogue sentence instead of a news sentence when relearning the model using association in Step3.
That is, using the knowledge extracted by encoder in Step1 and 2, in Step3, relearning is performed using a general dialogue sentence.
If knowledge can be extracted, learning can be performed more efﬁciently by performing Step3 only in accordance with the task.
R ELATED W ORK
Mostafazadeh et al.
(2017) assumed a conversation on social media about a shared image and proposed a model that performs natural conversation from images and sentences.
We assume a case where we do not use images when we talk.
In the sentence generation algorithm, there is an attempt to obtain a useful output sentence by simultaneously giving visual information and language information (Toyama et al., 2016) (Calixto et al., 2017) (Elliott & K´ad´ar, 2017)
(Nakayama & Nishida, 2017)
(Saha et al., 2016).
Approaches using visual information only during training are similar to our approach, with the approach of Toyama et al.
and Elliott & K´ad´ar (2017).
Toyama et al.
(2016) proposed a model for modeling latent variables from text and images and generating sentences using the latent variables.
In addition to encoderdecoder with normal attention, their model has a mechanism to generate latent variables.
When a sentence is input to encoder and predict translation with decoder, target word is generated by using latent variable generated from source sentence / target sentence / image as hidden layer of decoder.
Next, we incorporate the latent variable into the decoder and ﬁnally output the target.
Elliott & K´ad´ar (2017) shows that learning semantic representation using a multitasking learning framework with two types of output, sentence generation and image feature generation, is used to generate decoder’s translation sentence, thereby improving accuracy It was.
The structure of their model consists of encoder of sentence, decoder of sentence, decoder of image, input text and predict translation by decoder of sentence.
At the same time, hidden layers of encoder are averaged and input to the decoder of the image to predict image features.
There are three major differences between our approach and their approach.
First, while their approach gives images as visual information during training, our approach extends it and gives animation.
With this extension we believe that it will be possible for models to learn the correspondence between words representing motion and visual information.
Secondly, we deal with dialog tasks, not translation tasks.
Thirdly, we are assuming that we learn such as knowledge extraction from data which does not necessarily correspond to the sentence and the visual information at that time, such as general conversation.
For this reason, our approach is equipped with a noise removal mechanism that learns the degree of usage of visual information and language information Eq.
The other difference between (Elliott & K´ad´ar, 2017) is that we do not adopt multitasking learning in our method and therefore input visual information.
Therefore, since it is possible to attention to visual information during training, visual information can be extracted according to the response word.
C ONCLUSIONS
In a study applying a sentence generation algorithm of a translation sentence to a conversation model there was a problem that it was not possible to respond well to an input text which requires visual information.
However, in many practical dialogue systems, there are many cases where there is no input of visual information.
Therefore, our proposed method generates (associates) visual information from input texts and generates response texts using fused context vector with associative visual information and sentence language information.
Comparative experiments with models that do not use association show that association of visual information related to input texts produces response texts that contain valuable information compared to models without association.
Analysis of association also showed that our proposed method can generate visual information related to sentence language information through end-to-end learning of dialogue.
A PPENDIX
Associated image
Input
The Grand Sumo Tournament is the second day.
In a pitcher, three people including Otani has been selected from the Nippon Ham.
Output (ACM)
Today, Yokozuna Hakuho will aim for the ﬁrst victory.
This is an athlete .
Output (Baseline)
No. 1 is No. 1 in 1 meter.
Baseball is out in the professional this season.
Generated word
Yokozuna (The highest rank in sumo)
Athlete (a) Topic : Sports
Associated image
Input
On the 14th and 15th is the University
Entrance examination exam.
On the 14th and 15th is the University
Entrance examination exam.
Output (ACM)
The expectation of snowy wind is spreading. The expectation of snowy wind is spreading .
Output (Baseline)
There will be a large-scale ﬁre that is also in western Japan and eastern Japan. There will be a large-scale ﬁre that is also in western Japan and eastern Japan.
Generated word
Snowy
Spreading (b) Topic : Weather