C onversational W ord E mbedding for R etrieval- B ased D ialog S ystem (2020)

Abstract: Human conversations contain many types of information, e.g., knowledge, common sense, and language habits. In this paper, we propose a conversational word embedding method named PR-Embedding, which utilizes the conversation pairs <post, reply> to learn word embedding. Different from previous works, PR-Embedding uses the vectors from two different semantic spaces to represent the words in post and reply. To catch the information among the pair, we first introduce the word alignment model from statistical machine translation to generate the cross-sentence window, then train the embedding on word-level and sentence-level. We evaluate the method on single-turn and multi-turn response selection tasks for retrieval-based dialog systems. The experiment results show that PR-Embedding can improve the quality of the selected response.

## introduction word embedding is one of the most fundamental work in the nlp tasks, where low-dimensional word representations are learned from unlabeled corpora. the pre-trained embeddings can reflect the semantic and syntactic information of words and help various downstream tasks get better performance @xcite . the traditional word embedding methods train the models based on the co-occurrence statistics, such as word2vec @xcite , glove @xcite . those methods are widely used in dialog systems, not only in retrievalbased methods @xcite but also the generation-based models (serban et al., 1 in this paper, we name the first utterance in the conversation pair as 'post,' and the latter is 'reply' 2 pr-embedding source code is available at https:// github.com/wtma/pr-embedding . 2016; @xcite . the retrieval-based methods predict the answer based on the similarity of context and candidate responses, which can be divided into single-turn models @xcite and multi-turn models @xcite @xcite based on the number of turns in context. those methods construct the representations of the context and response with a single vector space. consequently, the models tend to select the response with the same words . on the other hand, as those static embeddings can not cope with the phenomenon of polysemy, researchers pay more attention to contextual representations recently. elmo @xcite , bert @xcite , and xlnet @xcite have achieved great success in many nlp tasks. however, it is difficult to apply them in the industrial dialog system due to their low computational efficiency. in this paper, we focus on the static embedding, for it is flexible and efficient. the previous works learn the embedding from intra-sentence within a single space, which is not enough for dialog systems. specifically, the semantic correlation beyond a single sentence in the conversation pair is missing. for example, the words 'why' and 'because' usually come from different speakers, and we can not catch their relationship by context window within the sentence. furthermore, when the words in post and reply are mapped into the same vector space, the model tends to select boring replies with repeated content because repeated words can easily get a high similarity. to tackle this problem, we propose pr-embedding (post-reply embedding) to learn representations from the conversation pairs in different spaces. firstly, we represent the post and the reply in two different spaces similar to the source and target languages in the machine translation. then, the word alignment model is introduced to gener- p_from post: reply: p_hi p_, p_where p_are p_you r_i r_am r_from r_alabama r_how r_about r_you r_, p_you r_i r_alabama figure 1: an example of conversational word alignment from the personachat dataset (section 3.1). 'p ' and 'r ' identify the vocabulary the words come from. for the word 'where,' we find the most related word 'alabama' based on the alignment model and generate the cross-sentence window with the size of 3 centered on the word. ate the cross-sentence window. lastly, we train the embeddings based on the word-level co-occurrence and a sentence-level classification task. the main contributions of our work are: (1) we propose a new method to learn the conversational word embedding from human dialogue in two different vector spaces; (2) the experimental results show that pr-embedding can help the model select better responses and catch the semantic correlation among the conversation pair. ## conclusions in this paper, we have proposed a conversational word embedding method named pr-embedding, which is learned from conversation pairs for retrieval-based dialog system. we use the word alignment model from machine translation to calculate the cross-sentence co-occurrence and train the embedding on word and sentence level. we find that pr-embedding can help the models select the better response both in single-turn and multi-turn conversation by catching the information among the pairs. in the future, we will adapt the method to more neural models especially the generationbased methods for the dialog system.